Re: Question about latency in CI comprehension

[Willem Christiaan Heerens <heerens1@xxxxxxxxx>]

Dear Tamás, Nathan and List,

Tamás you reported:
?while working with cochlear implants (CI) I often notice that even CI 
listeners with very good speech perception need some extra time (in 
comparison to normal hearing listeners) to comprehend a spoken 
sentence?.

I have no data about relevant literature of this subject.
But maybe my study since January this year of my experiences as an 
?expert in the field? can be of value and interest for you. And maybe 
for others too.

Since May 2013 I have the Advanced Bionics Harmony CI in my to 120 dB 
deaf left ear [In January 2014 my Harmony equipment is replaced by the 
New concept AB Naida. In my right ear, with a 70 dB overall hearing 
loss, I have the Phonak Naida hearing aid which can support to some 
extend the functioning of the AB CI. 

In my rehabilitation period it took me less than 2 weeks to have a 
speech perception score that almost reaches that of a normal hearing 
person even without seeing the speaking person.
My phoneme score was up to 90 % for a normal stimulation of my CI. 
Remarkable enough my phoneme score reduces a few percent in case both 
apparatuses ?cooperate?. 
But that is only under better than normal quit environmental conditions 
and with listening to a single speaker.
As soon as the environment becomes more ?noisy? my hearing abilities 
reduce rapidly. 
When three or more people are discussing more or less chaotically I only 
hear a tremendous loud noise in which I can hardly distinguish a single 
word. My speech perception then is dropped to zero and the latency for 
comprehending spoken sentences can be named infinite.

Only when someone in such an auditory environment is loudly speaking 
[almost screaming] near the microphone  of my CI processor I can 
comprehend just less than approximately 50 % of the sentences.
Far too low to have a pleasant discussion.

Listening to music ? especially classical music ? is for me far from 
joyful. Actually the only aspect in music I experience almost normally 
is rhythm. Pitch perception, timbre, dynamic range and melody 
recognition are all really bad.  Naming a single instrument out of what 
I hear with my CI is for me a hell of a job. 
What I experience in the comparison of my two hearing apparatuses is 
that with my CI I hear all background noises like traffic and cocktail 
party rumble as lower frequencies compared to the frequencies I hear 
with my normal hearing aid. 
In literature such experiences are reported as well. But more as an 
unclear and remarkable phenomenon. 

So being a physicist and with my research in cochlear functioning in 
mind ? what brought me earlier to the statement that the normal 
functioning human hearing sense makes use of the sound energy stimulus 
in the cochlea and not the sound pressure stimulus, what everybody now 
still assumes ? I started with the survey of what actually the CI 
processor software is doing with the incoming sound pressure stimulus.
What I found ? and please correct me if I am wrong ? in a nutshell was 
that for dynamic behavior purposes in the different electrodes this 
stimulus is rectified and there is further no indication that the sound 
pressure stimulus is transferred into the sound energy stimulus, which 
on its turn is used in a frequency selective way as the electrical 
stimulation of the electrode array.

So I hypothesized that if I compose quite simple tone settings for 
listening to beat phenomena I can study with the resulting sound 
fragments how I experience beats with my CI in comparison with my other 
hearing aid. They simply must sound different.
This because a beat phenomenon in the sound pressure domain is clearly 
different from the corresponding beat phenomenon in the sound energy 
domain.

My most illustrative beat experiment is the following:

I combined two tones with equal amplitude  ? 999.99983333 Hz and 
1000.00016667 Hz ? to a sound pressure  stimulus.
This combination results in a beat in a 1000 Hz stimulus with a beat 
period observed as having a duration of 3000 seconds.
Actually the complete beat period T is 6000 seconds. This because the 
shape of the modulation function in the sum of the two sinusoidal 
contributions is a cosine function with frequency equal to half the 
frequency difference of the two combined tones.
Hence equal to cos(2π×0.00016667×t) or equal to cos(2π×t/6000). And the 
modulation envelope is equal to the modulus of this function, so 
|cos(2π×t/6000)|. And that is a function with a period of 3000 seconds.

You must be aware that when you look closely to the shape of this 
stimulus you will find that near halfway the 3000 seconds the signal 
amplitude falls sharply to zero, remains zero during just a split second 
and then rises again sharply to higher values.
However when you calculate the sound energy stimulus connected with this 
sound pressure stimulus you will find that the beat in this signal still 
has a period of 3000 seconds. And when time is approaching the 1500 
seconds halfway this period the sound amplitude is also declining to 
zero. But it does this in an entirely different way.
At first the frequency is not 1000 Hz anymore but an octave higher so 
2000 Hz.
And the shape of the beat envelope of that 2000 Hz stimulus in the 
vicinity of halfway the period is entirely smooth. The sudden transition 
from sharply descending to sharply rising after the 1500 second point is 
completely disappeared. Instead a gradual approach resulting in a smooth 
touching to the zero level followed again by a gradual increase in 
amplitude.

The two striking differences ? 1000 Hz versus 2000 Hz and a sharp 
approach to zero versus a smooth approach ? must give unmistakable 
differences in hearing impression.

And the results of my experiment confirm my hypotheses:

I have cut the 30 seconds period around halfway the period out of the 
calculated soundtrack of the sound pressure stimulus. And with 
sufficient amplification for my observations I have listened separately 
with my CI and my Phonak hearing aid. And even with another amplifier 
connected to a high quality headphone without my Phonak hearing aid.

With my CI I heard without any doubt the sharp continuous decline to 
zero stimulus and after a split second the continuous  increase. I could 
not observe a substantial long period of zero signal.
With my other ear I heard in both cases during the period of 30 seconds 
a smooth decline to zero that was reached approximately 7 ? 8 seconds 
before the halfway moment. This zero signal ended approximately 7 ? 8  
seconds after the halfway moment. So during a period of 14 ? 16 seconds 
the signal remains zero followed by a smooth increase. 
And the tone has without any doubt a doubled frequency ? so 2000 Hz 
instead of 1000 Hz.

I have repeated these experiments with the common series of audiology 
test frequencies except the 125 Hz stimulus ? so starting with 250 Hz up 
to 7000 Hz.
With all frequencies I experienced the same results as for the 1000 Hz 
signal.

My following experiment was modifying the 1000 Hz sound pressure 
stimulus into the sound energy stimulus. And then listening to this 
sound fragment with my CI.
As I expected as result for this experiment I experienced the same sound 
via my CI as I heard from the sound pressure experiment with my Phonak 
hearing aid. 
A 2000 Hz signal and a smooth approach to a zero period of 16 seconds 
followed by a smooth rising of the 2000 Hz signal. 

After that I concluded that also pitch and missing fundamental 
experiments will give different results when a normally functioning 
basilar membrane is apparently stimulated with the sound energy stimulus 
while in the CI processor the sound energy stimulus isn?t generated and 
transferred to the brain but the sound pressure stimulus.

So I composed two tone complexes the first one existing of the 
frequencies:

800 ? 1000 ? 1200 ? 1400 ? 1600 ? 1800 ? 2000 Hz. 

All sine functions.
And the other one with the same frequencies but successively sine and 
cosine functions.
Both functions having a 1/f amplitude frequency relation, which results 
for the sound energy tone complex into equal energy contributions for 
all frequencies. 
>From calculations and experimental results in earlier studies and out of 
literature I know that a normal hearing person experiences with all sine 
functions a pitch of 200 Hz. While with the alternating sine ? cosine ? 
sine composition the listener hears a 400 Hz pitch.

The complete calculation for all sine contributions results in a series 
of missing lower harmonics starting with the  fundamental of 200 Hz 
followed by harmonics 400 ? 600 Hz and then the harmonics 800 ? 1000 ? 
1200 Hz. 
The alternating sine ? cosine composition shows after calculation that 
the series starts with the  missing lower harmonic 400 Hz followed by 
the 800 Hz and 1200 Hz harmonic. All three od harmonics 200 ? 600 ? 1000 
Hz are disappeared in the sound energy frequency spectrum.

The results of these two tone complex experiments are even more 
remarkable.

With my CI apparatus I experience no significant difference between the 
two sound fragments. I hear both sounds as higher tones with identical 
frequency and hardly no difference in intensity.
While with my Phonak hearing aid or amplifier headphone combination I 
hear precisely the missing fundamental as a low 200 Hz tone combined 
with higher tone contributions for the all sine function contributions. 
And a 400 Hz tone with a somewhat altered higher tone contribution ? 
which I can characterize as a change in timbre.  

So now I can draw a number of conclusions out of these results:

When I follow the existing hearing hypotheses or theory I am confronted 
with a serious anomaly: 

It is clear that the implantation of the CI has done nothing at all with 
my auditory brain functions.
However by the stimulation of my CI with the sound pressure signal my 
auditory cortex or other brain areas involved in sound perception don?t 
produce hearable missing fundamentals out of the sound pressure signal.

I can only draw the anomalous conclusion that before any signal is 
transferred to the brain the missing fundamental information must be 
present in this stimulus. 
Hence it must be generated inside the cochlea. And not in the brain.

But when I follow my hearing concept, where the non-stationary Bernoulli 
effect  transfers the incoming sound pressure stimulus into the sound 
energy stimulus in front of the basilar membrane, there doesn?t exist 
any anomaly.    

May I remark that the non-stationary Bernoulli effect is a physically 
correct solution of the Navier-Stokes equation for a non-viscous 
alternating potential flow in a non-compressible fluid? These flow 
conditions exist in the cochlear duct.

Tamás, regarding your remark:
?.In fact, some patients with single-sided deafness and CI in the deaf 
ear report a perceived latency between the normal hearing and the CI 
side, which does not seem to be of technical nature?..

I can give you the following answer:

Related to my experiments in which it is clear to me that the CI program 
does not generate the for normal hearing correct signals I also have my 
strong doubts about your assumption that the latency you mentioned is 
not of technical nature. 

Nathan I agree with you, regarding your remarks to Tamás:

?.In terms of your  specific question with unilateral loss and cochlear 
implants, I would be tempted to look at the engineering side of the 
device, or possibly the settings of the implant programming, but you 
mention you do not think the delay is a technical one?.

As you can conclude with me from the results of the described 
experiments I have done it is not only a technical issue.

It is really fundamental in origin. It is related to the fact that 
already for a long time the scientific hearing community is fully 
convinced that the cochlea transfers the sound pressure stimulus to the 
brain and the brain applies nonlinear functions in its auditory 
perception process.
And for me apparently out of my experimental results I distinguish that 
the cochlea performs the major non-linear process step. It transfers the 
sound pressure stimulus by two successive process steps ? 
differentiation followed by squaring ? into the sound energy stimulus. 
And the latter stimulus is frequency selective transferred to the brain.

In that case the at the best 30 dB dynamic range of the CI processor is 
transferred to 60 dB by the squaring process step as well. Which brings 
the CI dynamic range in balance with the normal hearing apparatus.

For a better perception of sound impressions via the CI it is needed 
that the programming of the CI processor must be changed. And that is a 
technical issue.

Maybe the conclusions out of my experiments that my CI processor isn?t 
well programmed for this transfer of fundamentals ? especially for 
missing fundamentals ? can be of high value for Mandarin speaking 
Chinese users of a CI. This tonal language, spoken by them, in which 
fundamentals play a crucial role, is highly problematic for a good 
speech perception score until now. I know that algorithms are developed 
or in development for extracting the fundamentals together with CIS 
technology.

[See for instance:  
N. Lan*, K. B. Nie, S. K. Gao, and F. G. Zeng:
A Novel Speech-Processing Strategy Incorporating Tonal Information for 
Cochlear Implants
IEEE Transactions on Biomedical Engineering, Vol. 51, No. 5, May 2004]

Nathan relating to your following remark:
?.Another area to consider may be the idea of hemispheric connectivity. 
In your example of a unilateral loss with the CI in the deaf ear, it may 
be that the non-CI (and fully hearing ear) input is processing faster in 
the brain than the CI input is. This is an extension of the concept that 
auditory-deprivation impacts on plasticity ?.

What do you think about the suggestion that a perception latency can be 
observed in the CI activated side related to the more or less normal 
hearing other side because the CI stimulus is fundamentally not correct 
which causes that the brain needs more time to make a correct 
perception. This can be placed perfectly in the category auditory-
deprivation resulting in an impact on the brain plasticity.  

I want to close my remarks with the following:

Of course the scientific auditory community can state that my hearing 
experiences with my CI and Phonak hearing aid in tone experiments are a 
pure personal issue. 
Firstly you can say that I have heard everything erroneously by using 
the wrong arguments and my experiments does not meet the high 
international standards you always use.
And secondly you are right if you say my experiments are purely 
subjective in origin.

My answer to the first comment will be: 

I want to remind you to August Seebeck?s quote [dated 1844] in the 
dispute with Ohm and Helmholtz:

Wodurch kann über die Frage, was zu einem Tone gehöre, entschieden 
werden, als eben durch das Ohr? 
(How else can the question as to what makes out a tone be decided but by 
the ear?)

And to the second comment:

Collect the data of such experiments and show me that I am wrong by 
doing the same experiments I have done. Do that with other subjects who 
are equipped with a hearing aid for moderate hearing loss and a CI in 
the deaf ear. If necessary and applicable  use up-to-date techniques 
like auditory fMRI or high resolution EEG methods to improve the level 
of objectivity.

I have asked a few fellow CI-users for their experiences with these 
phenomena. Their answers made me very confident.


Willem Chr. Heerens