multisensory processing

Multisensory processing in primates, like information processing on the Internet, is extensively decentralized. A recent neuroscience review article declared:

The pervasiveness of multisensory influences on all levels of cortical processing compels us to reconsider thinking about neural processing in unisensory terms. Indeed, the multisensory nature of most, possibly all, of the neocortex forces us to abandon the notion that senses ever operate independently during real-world cognition.[1]

Within the ascending auditory pathway prior to the neocortex, neurons in the inferior colliculus respond to auditory signals. Recent research found that about two-thirds of neurons in the inferior colliculus also respond to visual signals. In addition, about 20% of the neurons fire with eye movement motor activity.[2] These empirical results are consistent with the general model of perception-action cycles embedded throughout all levels of sensory processing.

Important communications services filter multisensory information at significant cost. Text and text messaging constrain communication to visual symbols. Telephone calls limit communication to auditory signals. Human beings successfully communicate despite these sensory constraints. Real-world communication behavior, however, indicates that these constraints impose significant bodily costs. Communications services that provide multisensory communication, i.e. audiovisual streams like television, vastly predominate in the allocation of persons’ time using communication services.

Sensory-constrained communication services have some offsetting advantages. Creating a PicturePhone was not enough to create a valuable communication service. Different sensory forms have different implications for multi-person communication, emotional distance, time patterns in dialogue, and many other pragmatic aspects of communication. For example, text messaging can be done surreptitiously in circumstances, such as a class or meeting, where other forms of communication would be more readily noticed. Sensory-limited communication is likely to persist even when multisensory communication is cheap, highly capable, and widely available.

Human beings make multisensory sense of unimodal communications relatively well compared to current non-biological information processors. An efficient distribution of computation between humans and non-biological devices apparently favors non-biological devices creating television programs rather than similar radio programs. But given a stream of silent images, humans can generate a descriptive stream of audio for those images much more quickly, cheaply, and accurately than can current non-biological technology. Understanding better how humans make multisensory sense from unimodal communication might provide insights into improving the weak performance of non-biological devices in such tasks.


[1] Ghazanfar, Asif A. and Charles E. Schroeder, “Is neocortex essentially multisensory?” Trends in Cognitive Sciences, v. 10, n. 6 (June 2006) pp. 278-86.

[2] Porter, Kristen Kelly, Ryan R. Metzger, and Jennifer M. Groh, “Visual- and saccade-related signals in the primate inferior colliculus,” PNAS v. 104, n. 45 (Nov. 6, 2007) pp. 17855-17860. Cited (approximate) figures are based on the figures reported in Table 1 (63.9% and 19.4% are figures to the full precision given).

solving the crisis in boys' education

“I feel squirmy,” said my student I was tutoring on the subject of the Italian Renaissance to the tune of $50/hour. “I feel like wiggling around, you know?” he complained. “Why IS that?”–which is what makes him interesting.

“Because you’re a twelve-year-old boy, that’s why. Can you stand on your head?”

For more, see the excellent and sadly defunct blog WriteOutLoud

This is an important issue and an innovative approach clearly worthy of a National Academies’ report.


diversity of flowers

Pollination is the act of transferring pollen grains from the male anther of a flower to the female stigma. Over 80 percent of the world’s flowering plants require a pollinator to reproduce. Animals that assist plants in their reproduction as pollinators include species of bat, butterflies, moths, flies, birds, beetles, ants, and bees. Without pollinators, the human race and all of Earth’s terrestrial ecosystems would not survive.”

The above text comes (with some re-arrangement) from the USDA publication, The Simple Truth: We Can’t Live Without Pollinators. Think about pollination the next time you squash a fly or crush a beetle. Given the importance of pollination to human survival, I favor mandatory pollination education classes in schools.