[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Auditory-visual comodulation

This is a multi-part message in MIME format.
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

BRUNO H. Repp wrote:

> These are very interesting results! However, I disagree that there were
> with our study that led us not to find the "correct" results. The aims of our
> study were different. We did not investigate the effect of comodulation of
> and auditory input on detectability. Rather, we were interested in effects of
> lexical status of the words to be detected. Effects of lexicality are
> top-down, whereas comodulation effects of the sort that Ken has demonstrated
> arguably bottom-up, even though they require cross-modal integration of some
> In our study, as Ken has pointed out, there was always a considerable degree
> comodulation between auditory and visual inputs. Also, our use of
> signal-correlated noise was quite deliberate and not a "problem". Only the
> presence of some inaccuracies in temporal alignment may be considered a
> shortcoming of our study. However, the degree of synchrony present was
> to lead to very clear lexical bias effects, whereas there was no effect on
> sensitivity to the presence of speech in (speech-like) noise. It seems
 unlikely to
> me that there would have been an effect on sensitivity if the synchrony had
> more accurate. Does the enhancement effect that Ken has demonstrated go away
> auditory and visual inputs are misaligned by +/-50 ms? I would be surprised if
> that were the case. His "mismatched" lipread information, I suppose, is
> out of synch with the auditory (masked) input.

In the original article by Repp, Frost, and Zsiga (1992) the stated goal (as
reported in the abstract) was to "...investigate whether simultaneous
can influence the detection of speech in envelope-matched noise.". The results
the first experiment showed that "Speech detection was not facilitated by an
audio-visual match, which suggests that listeners relied on low-level auditory
whose perception was immune to cross-modal top-down influences.".

I have argued that the use of envelope-matched noise and the inability to ensure
audio-visual stimulus alignments prevented Repp et al from observing the
of visual cues on auditory detection. The current experiment underway in our
laboratory shows that visible movements of the lips that match an acoustic
do indeed influence auditory detection.

The second question of lexical mediation addressed in the Repp et al study is,
 in my
opinion, still left unanswered. In fact, I would suggest that the visual
signal provides useful temporal and possibly spectral information (in the case
 of a
lipread signal) that can inform the auditory system when in time and where in
frequency to listen. This may be a general property of any audio-visual matching
stimulus and need not be speech. For example, the visual signal may be a
ball, and the acoustic signal may be any dynamically changing stimulus. At this
point we are not sure what stimulus conditions are required to observe the
My guess is that any combination of A and V comodulation will do the trick and
by manipulating the degree of audio-visual comodulation we could modulate the
strength of the release from masking.


Content-Type: text/x-vcard; charset=us-ascii; name="vcard.vcf"
Content-Transfer-Encoding: 7bit
Content-Description: Card for Ken W. Grant
Content-Disposition: attachment; filename="vcard.vcf"

begin:          vcard
fn:             Ken W. Grant
n:              Grant;Ken W.
org:            Army Audiology & Speech Center
adr;dom:        Research Section;;Walter Reed Army Medical Center;;Washington,
email;internet: grant@nicom.com
title:          Research Audiologist
tel;work:       (202) 782-8596
tel;fax:        (202) 782-9228
x-mozilla-cpt:  ;0
x-mozilla-html: FALSE
version:        2.1
end:            vcard