McGurk Effect - Background

Background

The McGurk effect is sometimes called the McGurk-MacDonald effect. It was first described in 1976 in a paper by Harry McGurk and John MacDonald entitled "Hearing Lips and Seeing Voices". This effect was discovered by accident when McGurk and his research assistant, MacDonald, asked a technician to dub a video with a different phoneme from the one spoken while conducting a study on how infants perceive language at different developmental stages. When the video was played back, both researchers heard a third phoneme rather than the one spoken or mouthed in the video.

This effect may be experienced when a video of one phoneme's production is dubbed with a sound-recording of a different phoneme being spoken. Often, the perceived phoneme is a third, intermediate phoneme. As an example, the syllable /ba-ba/ is spoken over the lip movements of /ga-ga/, and the perception is of /da-da/. McGurk and McDonald originally believed that this resulted from the common phonetic and visual properties of /b/ and /g/. Two types of illusion in response to incongruent audiovisual stimuli have been observed: fusions ('ba' auditory and 'ga' visual produce 'da') and combinations ('ga' auditory and 'ba' visual produce 'bga'). This is the brain's effort to provide the consciousness with its best guess about the incoming information. The information coming from the eyes and ears is contradictory, and in this instance, the eyes (visual information) have had a greater effect on the brain and thus the fusion and combination responses have been created.

Vision is the primary sense for humans, but speech perception is multimodal, which means that it involves information from more than one sensory modality, in particular, audition and vision. The McGurk effect arises during phonetic processing because the integration of audio and visual information happens early in speech perception. The McGurk effect is very robust; that is, knowledge about it seems to have little effect on one's perception of it. This is different from certain optical illusions, which break down once one 'sees through' them. Some people, including those that have been researching the phenomenon for more than twenty years, experience the effect even when they are aware that it is taking place. With the exception of people who can identify most of what is being said from speech-reading alone, most people are quite limited in their ability to identify speech from visual-only signals. A more extensive phenomenon is the ability of visual speech to increase the intelligibility of heard speech in a noisy environment. Visible speech can also alter the perception of perfectly audible speech sounds when the visual speech stimuli are mismatched with the auditory speech. Normally, speech perception is thought to be an auditory process, however, our use of information is immediate, automatic, and, to a large degree, unconscious and therefore, despite what is widely accepted as true, speech is not only something we hear. Speech is perceived by all of the senses working together (seeing, touching, and listening to a face move). The brain is often unaware of the separate sensory contributions of what it perceives. Therefore, when it comes to recognizing speech the brain cannot differentiate whether it is seeing or hearing the incoming information.

The McGurk effect is being used to produce more accurate speech recognition programs by making use of a video camera and lip reading software. It has also been examined in relation to witness testimony. Wareham and Wright's 2005 study showed that inconsistent visual information can change the perception of spoken utterances, suggesting that the McGurk effect may have many influences in everyday perception. Not limited to syllables, the effect can occur in whole words and have an effect on daily interactions that people are unaware of. Research into this area can provide information on not only theoretical questions, but also it can provide therapeutic and diagnostic relevance for those with disorders relating to audio and visual integration of speech cues.