Re: Auditory-visual comodulation (grant )

Subject: Re: Auditory-visual comodulation
From:    grant  <grant(at)NICOM.COM>
Date:    Fri, 3 Apr 1998 12:56:15 -0500

This is a multi-part message in MIME format. --------------C385372B13CCDA95623DE1D9 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit BRUNO H. Repp wrote: > These are very interesting results! However, I disagree that there were "problems" > with our study that led us not to find the "correct" results. The aims of our > study were different. We did not investigate the effect of comodulation of visual > and auditory input on detectability. Rather, we were interested in effects of the > lexical status of the words to be detected. Effects of lexicality are definitely > top-down, whereas comodulation effects of the sort that Ken has demonstrated are > arguably bottom-up, even though they require cross-modal integration of some kind. > In our study, as Ken has pointed out, there was always a considerable degree of > comodulation between auditory and visual inputs. Also, our use of > signal-correlated noise was quite deliberate and not a "problem". Only the > presence of some inaccuracies in temporal alignment may be considered a > shortcoming of our study. However, the degree of synchrony present was sufficient > to lead to very clear lexical bias effects, whereas there was no effect on > sensitivity to the presence of speech in (speech-like) noise. It seems unlikely to > me that there would have been an effect on sensitivity if the synchrony had been > more accurate. Does the enhancement effect that Ken has demonstrated go away when > auditory and visual inputs are misaligned by +/-50 ms? I would be surprised if > that were the case. His "mismatched" lipread information, I suppose, is completely > out of synch with the auditory (masked) input. In the original article by Repp, Frost, and Zsiga (1992) the stated goal (as reported in the abstract) was to "...investigate whether simultaneous speechreading can influence the detection of speech in envelope-matched noise.". The results of the first experiment showed that "Speech detection was not facilitated by an audio-visual match, which suggests that listeners relied on low-level auditory cues whose perception was immune to cross-modal top-down influences.". I have argued that the use of envelope-matched noise and the inability to ensure audio-visual stimulus alignments prevented Repp et al from observing the influence of visual cues on auditory detection. The current experiment underway in our laboratory shows that visible movements of the lips that match an acoustic stimulus do indeed influence auditory detection. The second question of lexical mediation addressed in the Repp et al study is, in my opinion, still left unanswered. In fact, I would suggest that the visual matching signal provides useful temporal and possibly spectral information (in the case of a lipread signal) that can inform the auditory system when in time and where in frequency to listen. This may be a general property of any audio-visual matching stimulus and need not be speech. For example, the visual signal may be a bouncing ball, and the acoustic signal may be any dynamically changing stimulus. At this point we are not sure what stimulus conditions are required to observe the effect. My guess is that any combination of A and V comodulation will do the trick and that by manipulating the degree of audio-visual comodulation we could modulate the strength of the release from masking. Ken --------------C385372B13CCDA95623DE1D9 Content-Type: text/x-vcard; charset=us-ascii; name="vcard.vcf" Content-Transfer-Encoding: 7bit Content-Description: Card for Ken W. Grant Content-Disposition: attachment; filename="vcard.vcf" begin: vcard fn: Ken W. Grant n: Grant;Ken W. org: Army Audiology & Speech Center adr;dom: Research Section;;Walter Reed Army Medical Center;;Washington, DC;20307-5001; email;internet: grant(at) title: Research Audiologist tel;work: (202) 782-8596 tel;fax: (202) 782-9228 x-mozilla-cpt: ;0 x-mozilla-html: FALSE version: 2.1 end: vcard --------------C385372B13CCDA95623DE1D9--

This message came from the mail archive
maintained by:
DAn Ellis <>
Electrical Engineering Dept., Columbia University