danorman@UCSD.EDU (Donald A Norman-UCSD Cog Sci Dept) (12/04/90)
Summary: I think it senseless to argue about which modality is most important. There are real data that can decide the issue. The conclusion is clear: every modality is essential if you want a complete experience. Every modality contributes an essential aspect of our experiential phenomena. BUT, the human is clearly a visual animal, with about half the cortex devoted to visual processing. Even so, this does not diminish the importance of the other sensory modalities. ---- Some facts and some speculation about the various sensory modalities. Although I am a fan of auditory phenomena, and although sound, touch, feel, kinesthesis, and smell are essential ingredients to human life and each contributes essential information and essential phenomena, the facts are that the human is a visual animal. Not only is the retina the most complex sensory receptor we have (huge in terms of numbers of receptors and sophistication of its pre-processing) and not only is the optic nerve the largest sensory fiber of information to the brain, but a huge percentage of the brain is devoted to vision, much more volume than to any of the other senses (hell, then to all the other senses combined). The data-flow rate in the optic nerves is measured in tens or hundreds of megabits/second, and this is *after* considerable preprocessing. As one cognitive neuroscientist put it "over 50% of the cortex in primates, probably including humans, consists of areas devoted to specifically visual processing." (Sereno, 1990) But any argument about the essential aspects of vision should not be used to deny the importance of other modalities. Sound gives much better temporal information than does vision. Vision is slow with resolution times measured in hundreds of milliseconds: we can resolve arrival times of sounds with a 10 microsecond precision, yielding good localization. We are sensitive to tiny phase and amplitude deviances caused by sound hitting the pinna (the funny ridges and creases people's outer ears) so that we can detect distance and elevation of sounds with remarkable accuracy. And sounds tell us about the nature of substances, so that we can detect roughly how big, how dense, and even the composition of an object just by hearing the sound made by something else hitting it (tap a plate, a glass, the table, a door, a piece of metal -- and we can distinguish each by the sound alone). My favorite paper about the importance of sound is Bill Gaver's study of "naturalistic sound." (Gaver, 1986, 1988). And sound conveys low-level emotion better than other media. We seem to have special mechanism for rhythm. Sound timbre is where the information is carried, and this is the least-well understood aspect of sound. Smell and taste are less studied, but years ago von Bekesy showed that we can actually localize smells -- tell from what direction a smell was coming by tiny differences in arrival times at the two nostrils. We may have lost the art of using this information, but clearly other animals make heavy use of smell localization. And what about the other senses: touch and feel, temperature, body position (limbs and head). Mess up on these and the person can feel ill: simulator sickness (virtual reality sickness?) -- a variant of sea-sickness when vision gives one message and proprioceptive detectors give conflicting messages. If you are not careful, your viewers will throw up all over your nice new display screen. Note too that basically each sensory receptor is about as sensitive as is possible. In hearing, we can detect Brownian motion -- the amplitude displacement of the eardrum that is audible is less than the diameter of a hydrogen molecule. In vision, we can detect one or two photons. In smell and taste, single molecules. And the range is enormous -- a 120 dB range in both audition and vision, which means the light and sound that is painful or just starting to do damage is 10 ^12 times more intense than the amount just dectectable. No human-made instrument has that range. This also means that tiny artifacts in your displays and sound-producing apparatus will be detected and give continual low-level reminders that this is a simulation, not real. In terms of science, we know the most about vision, followed closely by audition. We know little of the other senses. And we know almost nothing of perception -- how the visual and auditory information is translated into percepts, which includes pattern recognition. The relevant sciences are in psychology and, more precisely, psychophysics. If you really want to simulate virtual realities, you need to know sensory psychology, not just just think about it. There are a lot of surprises about the way the sensory systems really work. its not quite like you might have thought. When you start talking about enhancing or changing human perceptions, then technology alone won't do it -- you had better learn about the person -- which means you've got to learn psychology and/or cognitive science. Aside: I neglected to discuss motor output: speech, sound, movement of the limbs. Another much-neglected topic. Note that we should not talk about the sensory systems, we should only talk about the sensori-motor systems. We are a feedback, closed loop, servo system, and the relationship between motor action and perception is essential for our phenomenal experience with the world. But I leave that for another message, although it probably requires a book. don norman Gaver, W. (1986). Auditory icons: Using sound in computer interfaces. Human Computer Interaction, 2, 167-177. Gaver, W. (1988). Everyday listening and auditory icons. Unpublished PhD dissertation, University of California, San Diego Sereno, M. (1990, August). Language and the primate brain. Newsletter for the Center for Research on Language, 4 (4), pp. 3-12. Don Norman INTERNET: dnorman@ucsd.edu Department of Cognitive Science 0515 BITNET: dnorman@ucsd University of California, San Diego AppleLink: dnorman La Jolla, California 92093 USA FAX: (619) 534-1128