Localization ability with bimodal hearing aids and bilateral cochlear implants

After successful cochlear implantation in one ear, some patients continue to use a hearing aid at the contralateral ear. They report an improved reception of speech, especially in noise, as well as a better perception of music when the hearing aid and cochlear implant are used in this bimodal combination. Some individuals in this bimodal patient group also report the impression of an improved localization ability. Similar experiences are reported by the group of bilateral cochlear implantees. In this study, a survey of 11 bimodally and 4 bilaterally equipped cochlear implant users was carried out to assess localization ability. Individuals in the bimodal implant group were all provided with the same type of hearing aid in the opposite ear, and subjects in the bilateral implant group used cochlear implants of the same manufacturer on each ear. Subjects adjusted the spot of a computer-controlled laser-pointer to the perceived direction of sound incidence in the frontal horizontal plane by rotating a trackball. Two subjects of the bimodal group who had substantial residual hearing showed localization ability in the bimodal configuration, whereas using each single device only the subject with better residual hearing was able to discriminate the side of sound origin. Five other subjects with more pronounced hearing loss displayed an ability for side discrimination through the use of bimodal aids, while four of them were already able to discriminate the side with a single device. Of the bilateral cochlear implant group one subject showed localization accuracy close to that of normal hearing subjects. This subject was also able to discriminate the side of sound origin using the first implanted device alone. The other three bilaterally equipped subjects showed limited localization ability using both devices. Among them one subject demonstrated a side-discrimination ability using only the first implanted device.     

Temporal and spatio-temporal vibrotactile displays for voice fundamental frequency: an initial evaluation of a new vibrotactile speech perception aid with normal-hearing and hearing-impaired individuals.

Four experiments were performed to evaluate a new wearable vibrotactile speech perception aid that extracts fundamental frequency (F0) and displays the extracted F0 as a single-channel temporal or an eight-channel spatio-temporal stimulus. Specifically, we investigated the perception of intonation (i.e., question versus statement) and emphatic stress (i.e., stress on the first, second, or third word) under Visual-Alone (VA), Visual-Tactile (VT), and Tactile-Alone (TA) conditions and compared performance using the temporal and spatio-temporal vibrotactile display. Subjects were adults with normal hearing in experiments I-III and adults with severe to profound hearing impairments in experiment IV. Both versions of the vibrotactile speech perception aid successfully conveyed intonation. Vibrotactile stress information was successfully conveyed, but vibrotactile stress information did not enhance performance in VT conditions beyond performance in VA conditions. In experiment III, which involved only intonation identification, a reliable advantage for the spatio-temporal display was obtained. Differences between subject groups were obtained for intonation identification, with more accurate VT performance by those with normal hearing. Possible effects of long-term hearing status are discussed.     

The relative phonetic contributions of a cochlear implant and residual acoustic hearing to bimodal speech perception

The addition of low-passed (LP) speech or even a tone following the fundamental frequency (F0) of speech has been shown to benefit speech recognition for cochlear implant (CI) users with residual acoustic hearing. The mechanisms underlying this benefit are still unclear. In this study, eight bimodal subjects (CI users with acoustic hearing in the non-implanted ear) and eight simulated bimodal subjects (using vocoded and LP speech) were tested on vowel and consonant recognition to determine the relative contributions of acoustic and phonetic cues, including F0, to the bimodal benefit. Several listening conditions were tested (CI/Vocoder, LP, T(F0-env), CI/Vocoder + LP, CI/Vocoder + T(F0-env)). Compared with CI/Vocoder performance, LP significantly enhanced both consonant and vowel perception, whereas a tone following the F0 contour of target speech and modulated with an amplitude envelope of the maximum frequency of the F0 contour (T(F0-env)) enhanced only consonant perception. Information transfer analysis revealed a dual mechanism in the bimodal benefit: The tone representing F0 provided voicing and manner information, whereas LP provided additional manner, place, and vowel formant information. The data in actual bimodal subjects also showed that the degree of the bimodal benefit depended on the cutoff and slope of residual acoustic hearing.     

Beneficial acoustic speech cues for cochlear implant users with residual acoustic hearing

This study investigated which acoustic cues within the speech signal are responsible for bimodal speech perception benefit. Seven cochlear implant (CI) users with usable residual hearing at low frequencies in the non-implanted ear participated. Sentence tests were performed in near-quiet (some noise on the CI side to reduce scores from ceiling) and in a modulated noise background, with the implant alone and with the addition, in the hearing ear, of one of four types of acoustic signals derived from the same sentences: (1) a complex tone modulated by the fundamental frequency (F0) and amplitude envelope contours; (2) a pure tone modulated by the F0 and amplitude contours; (3) a noise-vocoded signal; (4) unprocessed speech. The modulated tones provided F0 information without spectral shape information, whilst the vocoded signal presented spectral shape information without F0 information. For the group as a whole, only the unprocessed speech condition provided significant benefit over implant-alone scores, in both near-quiet and noise. This suggests that, on average, F0 or spectral cues in isolation provided limited benefit for these subjects in the tested listening conditions, and that the significant benefit observed in the full-signal condition was derived from implantees' use of a combination of these cues.     

Enhancement of temporal periodicity cues in cochlear implants: effects on prosodic perception and vowel identification

Standard continuous interleaved sampling processing, and a modified processing strategy designed to enhance temporal cues to voice pitch, were compared on tests of intonation perception, and vowel perception, both in implant users and in acoustic simulations. In standard processing, 400 Hz low-pass envelopes modulated either pulse trains (implant users) or noise carriers (simulations). In the modified strategy, slow-rate envelope modulations, which convey dynamic spectral variation crucial for speech understanding, were extracted by low-pass filtering (32 Hz). In addition, during voiced speech, higher-rate temporal modulation in each channel was provided by 100% amplitude-modulation by a sawtooth-like wave form whose periodicity followed the fundamental frequency (F0) of the input. Channel levels were determined by the product of the lower- and higher-rate modulation components. Both in acoustic simulations and in implant users, the ability to use intonation information to identify sentences as question or statement was significantly better with modified processing. However, while there was no difference in vowel recognition in the acoustic simulation, implant users performed worse with modified processing both in vowel recognition and in formant frequency discrimination. It appears that, while enhancing pitch perception, modified processing harmed the transmission of spectral information.     

Speech recognition in noise for cochlear implant listeners: benefits of residual acoustic hearing

The purpose of this study was to explore the potential advantages, both theoretical and applied, of preserving low-frequency acoustic hearing in cochlear implant patients. Several hypotheses are presented that predict that residual low-frequency acoustic hearing along with electric stimulation for high frequencies will provide an advantage over traditional long-electrode cochlear implants for the recognition of speech in competing backgrounds. A simulation experiment in normal-hearing subjects demonstrated a clear advantage for preserving low-frequency residual acoustic hearing for speech recognition in a background of other talkers, but not in steady noise. Three subjects with an implanted "short-electrode" cochlear implant and preserved low-frequency acoustic hearing were also tested on speech recognition in the same competing backgrounds and compared to a larger group of traditional cochlear implant users. Each of the three short-electrode subjects performed better than any of the traditional long-electrode implant subjects for speech recognition in a background of other talkers, but not in steady noise, in general agreement with the simulation studies. When compared to a subgroup of traditional implant users matched according to speech recognition ability in quiet, the short-electrode patients showed a 9-dB advantage in the multitalker background. These experiments provide strong preliminary support for retaining residual low-frequency acoustic hearing in cochlear implant patients. The results are consistent with the idea that better perception of voice pitch, which can aid in separating voices in a background of other talkers, was responsible for this advantage.     

Perception of intonation in Mandarin Chinese

There is a tendency across languages to use a rising pitch contour to convey question intonation and a falling pitch contour to convey a statement. In a lexical tone language such as Mandarin Chinese, rising and falling pitch contours are also used to differentiate lexical meaning. How, then, does the multiplexing of the F(0) channel affect the perception of question and statement intonation in a lexical tone language? This study investigated the effects of lexical tones and focus on the perception of intonation in Mandarin Chinese. The results show that lexical tones and focus impact the perception of sentence intonation. Question intonation was easier for native speakers to identify on a sentence with a final falling tone and more difficult to identify on a sentence with a final rising tone, suggesting that tone identification intervenes in the mapping of F(0) contours to intonational categories and that tone and intonation interact at the phonological level. In contrast, there is no evidence that the interaction between focus and intonation goes beyond the psychoacoustic level. The results provide insights that will be useful for further research on tone and intonation interactions in both acoustic modeling studies and neurobiological studies.     

Encoding voice fundamental frequency into vibrotactile frequency.

Measured in this study was the ability of eight hearing and five deaf subjects to identify the stress pattern in a short sentence from the variation in voice fundamental frequency (F0), when presented aurally (for hearing subjects) and when transformed into vibrotactile pulse frequency. Various transformations from F0 to pulse frequency were tested in an attempt to determine an optimum transformation, the amount of F0 information that could be transmitted, and what the limitations in the tactile channel might be. The results indicated that a one- or two-octave reduction of F0 vibrotactile frequency (transmitting every second or third glottal pulse) might result in a significant ability to discriminate the intonation patterns associated with moderate-to-strong patterns of sentence stress in English. However, accurate reception of the details of the intonation pattern may require a slower than normal pronounciation because of an apparent temporal indeterminacy of about 200 ms in the perception of variations in vibrotactile frequency. A performance deficit noted for the two prelingually, profoundly deaf subjects with marginally discriminable encodings offers some support for our previous hypothesis that there is a natural association between auditory pitch and perceived vibrotactile frequency.     

Talker-identification training using simulations of binaurally combined electric and acoustic hearing: generalization to speech and emotion recognition

Understanding speech in background noise, talker identification, and vocal emotion recognition are challenging for cochlear implant (CI) users due to poor spectral resolution and limited pitch cues with the CI. Recent studies have shown that bimodal CI users, that is, those CI users who wear a hearing aid (HA) in their non-implanted ear, receive benefit for understanding speech both in quiet and in noise. This study compared the efficacy of talker-identification training in two groups of young normal-hearing adults, listening to either acoustic simulations of unilateral CI or bimodal (CI+HA) hearing. Training resulted in improved identification of talkers for both groups with better overall performance for simulated bimodal hearing. Generalization of learning to sentence and emotion recognition also was assessed in both subject groups. Sentence recognition in quiet and in noise improved for both groups, no matter if the talkers had been heard during training or not. Generalization to improvements in emotion recognition for two unfamiliar talkers also was noted for both groups with the simulated bimodal-hearing group showing better overall emotion-recognition performance. Improvements in sentence recognition were retained a month after training in both groups. These results have potential implications for aural rehabilitation of conventional and bimodal CI users.     

Pitch ranking ability of cochlear implant recipients: a comparison of sound-processing strategies

Pitch ranking of sung vowel stimuli, separated in fundamental frequency (F0) by half an octave, was measured with a group of eleven Nucleus 24 cochlear implant recipients using different sound coding strategies. In three consecutive studies, either two or three different sound coding strategies were compared to the Advanced Combinational Encoder (ACE) strategy. These strategies included Continuous Interleaved Sampling (CIS), Peak Derived Timing (PDT), Modulation Depth Enhancement (MDE), F0 Synchronized ACE (FOSync), and Multi-channel Envelope Modulation (MEM), the last four being experimental strategies. While pitch ranking results on average were poor compared to those expected for most normal hearing listeners, significantly higher scores were obtained using the MEM, MDE, and FOSync strategies compared to ACE. These strategies enhanced coding of temporal F0 cues by providing deeper modulation cues to F0 coincidentally in time across all activated electrodes. In the final study, speech recognition tests were also conducted using ACE, CIS, MDE, and MEM. Similar results among all strategies were obtained for word tests in quiet and between ACE and MEM for sentence tests in noise. These findings demonstrate that strategies such as MEM may aid perception of pitch and still adequately code segmental speech features as per existing coding strategies.     

Use of intonation contours for speech recognition in noise by cochlear implant recipients

The corruption of intonation contours has detrimental effects on sentence-based speech recognition in normal-hearing listeners Binns and Culling [(2007). J. Acoust. Soc. Am. 122, 1765-1776]. This paper examines whether this finding also applies to cochlear implant (CI) recipients. The subjects' F0-discrimination and speech perception in the presence of noise were measured, using sentences with regular and inverted F0-contours. The results revealed that speech recognition for regular contours was significantly better than for inverted contours. This difference was related to the subjects' F0-discrimination providing further evidence that the perception of intonation patterns is important for the CI-mediated speech recognition in noise.     

The acoustic analysis of tone differentiation as a means for assessing tone production in speakers of Cantonese

This paper reports on a methodology for acoustically analyzing tone production in Cantonese. F0 offset versus F0 onset are plotted for a series of tokens for each of the six tones in the language. These are grouped according to tone type into a set of six ellipses. Qualitative visual observations regarding the degree of differentiation of the ellipses within the tonal space are summarized numerically using two indices, referred to here as Index 1 and Index 2. Index 1 is a ratio of the area of the speaker's tonal space and the average of the areas of the ellipses of the three target tones making up the tonal space. Index 2 is a ratio of the average distance between all six tonal ellipses and the average of the sum of the two axes for all six tone ellipses. Using this methodology, tonal differentiation is compared for three groups of speakers; normally hearing adults; normally hearing children aged from 4-6 years; and, prelinguistically deafened cochlear implant users aged from 4-11 years. A potential conundrum regarding how tone production abilities can outstrip tone perception abilities is explained using the data from the acoustic analyses. It is suggested that young children of the age range tested are still learning to normalize for pitch level differences in tone production. Acoustic analysis of the data thus supports results from tone perception studies and suggests that the methodology is suitable for use in studies investigating tone production in both clinical and research contexts.     

Fundamental frequency is critical to speech perception in noise in combined acoustic and electric hearing

Cochlear implant (CI) users have been shown to benefit from residual low-frequency hearing, specifically in pitch related tasks. It remains unclear whether this benefit is dependent on fundamental frequency (F0) or other acoustic cues. Three experiments were conducted to determine the role of F0, as well as its frequency modulated (FM) and amplitude modulated (AM) components, in speech recognition with a competing voice. In simulated CI listeners, the signal-to-noise ratio was varied to estimate the 50% correct response. Simulation results showed that the F0 cue contributes to a significant proportion of the benefit seen with combined acoustic and electric hearing, and additionally that this benefit is due to the FM rather than the AM component. In actual CI users, sentence recognition scores were collected with either the full F0 cue containing both the FM and AM components or the 500-Hz low-pass speech cue containing the F0 and additional harmonics. The F0 cue provided a benefit similar to the low-pass cue for speech in noise, but not in quiet. Poorer CI users benefited more from the F0 cue than better users. These findings suggest that F0 is critical to improving speech perception in noise in combined acoustic and electric hearing.     

Coding of the fundamental frequency in continuous interleaved sampling processors for cochlear implants

In this study the perception of the fundamental frequency (F0) of periodic stimuli by cochlear implant users is investigated. A widely used speech processor is the Continuous Interleaved Sampling (CIS) processor, for which the fundamental frequency appears as temporal fluctuations in the envelopes at the output. Three experiments with four users of the LAURA (Registered trade mark of Philips Hearing Implants, now Cochlear Technology Centre Europe) cochlear implant were carried out to examine the influence of the modulation depth of these envelope fluctuations on pitch discrimination. In the first experiment, the subjects were asked to discriminate between two SAM (sinusoidally amplitude modulated) pulse trains on a single electrode channel differing in modulation frequency ( deltaf = 20%). As expected, the results showed a decrease in the performance for smaller modulation depths. Optimal performance was reached for modulation depths between 20% and 99%, depending on subject, electrode channel, and modulation frequency. In the second experiment, the smallest noticeable difference in F0 of synthetic vowels was measured for three algorithms that differed in the obtained modulation depth at the output: the default CIS strategy, the CIS strategy in which the F0 fluctuations in the envelope were removed (FLAT CIS), and a third CIS strategy, which was especially designed to control and increase the depth of these fluctuations (F0 CIS). In general, performance was poorest for the FLAT CIS strategy, where changes in F0 are only apparent as changes of the average amplitude in the channel outputs. This emphasizes the importance of temporal coding of F0 in the speech envelope for pitch perception. No significantly better results were obtained for the F0 CIS strategy compared to the default CIS strategy, although the latter results in envelope modulation depths at which sub-optimal scores were obtained in some cases of the first experiment. This indicates that less modulation is needed if all channels are stimulated with synchronous F0 fluctuations. This hypothesis is confirmed in a third experiment where subjects performed significantly better in a pitch discrimination task with SAM pulse trains, if three channels were stimulated concurrently, as opposed to only one.     

Accents, focus distribution, and the perceived distribution of given and new information: An experiment

This article reports on an experiment examining some perceptual consequences of correspondences between accent patterns, the distribution of plus and minus focus, and the distribution of new and given information in Dutch spoken sentences. "Accent patterns" refer here to the distribution of intonational accents over spoken sentences. Each accent marks a sentence constituent as plus focus, i.e., as highlighted by the speaker. Constituents not so marked are called minus focus. The main questions examined here are to what extent are plus focus constituents generally perceived as conveying new information, and minus focus constituents as conveying earlier introduced or given information. The linguistic material for the experiment was formed by brief radio news items, each two sentences long. Leading sentences determined the distribution of new and given information in target sentences. The accent patterns and, hence, the possible focus distributions in the target utterances were varied systematically by manipulating their synthetic pitch contours according to the rules for Dutch intonation. Subjects were asked to rate on a scale from 1-10 the acceptability of each possible combination of a leading with a target utterance. Results showed that the most preferred or acceptable distributions of new and given information closely match the distributions of plus and minus focus. It was also found that new information can hardly ever acceptably be associated with minus focus, but given information can rather often, although not always, acceptably be associated with plus focus. This appears to be limited to certain conditions, defined by a combination of syntactic and focus structure of the sentence. In these conditions, plus focus cannot be perceived only as signaling new information, but also as highlighting thematic relations with the context. These results are related to work on text-to-speech systems.     

Temporal pitch mechanisms in acoustic and electric hearing

Two experiments investigated pitch perception for stimuli where the place of excitation was held constant. Experiment 1 used pulse trains in which the interpulse interval alternated between 4 and 6 ms. In experiment 1a these "4-6" pulse trains were bandpass filtered between 3900 and 5300 Hz and presented acoustically against a noise background to normal listeners. The rate of an isochronous pulse train (in which all the interpulse intervals were equal) was adjusted so that its pitch matched that of the "4-6" stimulus. The pitch matches were distributed unimodally, had a mean of 5.7 ms, and never corresponded to either 4 or to 10 ms (the period of the stimulus). In experiment 1b the pulse trains were presented both acoustically to normal listeners and electrically to users of the LAURA cochlear implant, via a single channel of their device. A forced-choice procedure was used to measure psychometric functions, in which subjects judged whether the 4-6 stimulus was higher or lower in pitch than isochronous pulse trains having periods of 3, 4, 5, 6, or 7 ms. For both groups of listeners, the point of subjective equality corresponded to a period of 5.6 to 5.7 ms. Experiment 1c confirmed that these psychometric functions were monotonic over the range 4-12 ms. In experiment 2, normal listeners adjusted the rate of an isochronous filtered pulse train to match the pitch of mixtures of pulse trains having rates of F1 and F2 Hz, passed through the same bandpass filter (3900-5400 Hz). The ratio F2/F1 was 1.29 and F1 was either 70, 92, 109, or 124 Hz. Matches were always close to F2 Hz. It is concluded that the results of both experiments are inconsistent with models of pitch perception which rely on higher-order intervals. Together with those of other published data on purely temporal pitch perception, the data are consistent with a model in which only first-order interpulse intervals contribute to pitch, and in which, over the range 0-12 ms, longer intervals receive higher weights than short intervals.     

The perception of intonation questions and statements in Cantonese

In tone languages there are potential conflicts in the perception of lexical tone and intonation, as both depend mainly on the differences in fundamental frequency (F0) patterns. The present study investigated the acoustic cues associated with the perception of sentences as questions or statements in Cantonese, as a function of the lexical tone in sentence final position. Cantonese listeners performed intonation identification tasks involving complete sentences, isolated final syllables, and sentences without the final syllable (carriers). Sensitivity (d' scores) were similar for complete sentences and final syllables but were significantly lower for carriers. Sensitivity was also affected by tone identity. These findings show that the perception of questions and statements relies primarily on the F0 characteristics of the final syllables (local F0 cues). A measure of response bias (c) provided evidence for a general bias toward the perception of statements. Logistic regression analyses showed that utterances were accurately classified as questions or statements by using average F0 and F0 interval. Average F0 of carriers (global F0 cue) was also found to be a reliable secondary cue. These findings suggest that the use of F0 cues for the perception of intonation question in tonal languages is likely to be language-specific.     

Pitch Shift in Exsomatized Cochlea Observed by Laser Interferometry

Pitch is one of the most important auditory perception characteristics of sound; however, the mechanism underlying the pitch perception of sound is unclear. Although theoretical researches have suggested that perception of virtual pitch is connected with physics in cochlea of inner ear, there is no direct experimental observation of virtual pitch processing in the cochlea. By laser interferometry, we observe shift phenomena of virtual pitch in basilar membrane vibration of exsomatized cochlea, which is consistent with perceptual pitch shift observed in psychoacoustic experiments. This means that the complex mechanical vibration of basilar membrane in cochlea plays an important role in pitch information processing during hearing.     

The effect of segmental and suprasegmental corrections on the intelligibility of deaf speech

Three experiments were conducted to study the effect of segmental and suprasegmental corrections on the intelligibility and judged quality of deaf speech. By means of digital signal processing techniques, including LPC analysis, transformations of separate speech sounds, temporal structure, and intonation were carried out on 30 Dutch sentences spoken by ten deaf children. The transformed sentences were tested for intelligibility and acceptability by presenting them to inexperienced listeners. In experiment 1, LPC based reflection coefficients describing segmental characteristics of deaf speakers were replaced by those of hearing speakers. A complete segmental correction caused a dramatic increase in intelligibility from 24% to 72%, which, for a major part, was due to correction of vowels. Experiment 2 revealed that correction of temporal structure and intonation caused only a small improvement from 24% to about 34%. Combination of segmental and suprasegmental corrections yielded almost perfectly understandable sentences, due to a more than additive effect of the two corrections. Quality judgments, collected in experiment 3, were in close agreement with the intelligibility measures. The results show that, in order for these speakers to become more intelligible, improving their articulation is more important than improving their production of temporal structure and intonation.     

Tuning curves and pitch matches in a listener with a unilateral, low-frequency hearing loss

Psychoacoustical tuning curves and interaural pitch matches were measured in a listener with a unilateral, moderately severe hearing loss of primarily cochlear origin below 2 kHz. The psychoacoustical tuning curves, measured in a simultaneous-masking paradigm, were obtained at 1 kHz for probe levels of 4.5-, 7-, and 13-dB SL in the impaired ear, and 7-dB SL in the impaired ear, and 7-dB SL in the normal ear. Results show that as the level of the probe increased from 4.5- to 13-dB SL in the impaired ear, (1) the frequency location of the tip of the tuning curve decreased from approximately 2.85 to 2.20 kHz and (2) the lowest level of the masker required to just mask the probe increased from 49- to 83-dB SPL. The tuning curve in the normal ear was comparable to data from other normal listeners. The interaural pitch matches were measured from 0.5 to 6 kHz at 10-dB SL in the impaired ear and approximately 15- to 20-dB SL in the normal ear. Results show reasonable identity matches (e.g., a 500-Hz tone in the impaired ear was matched close to a 500-Hz tone in the normal ear), although variability was significantly greater for pitch matches below 2 kHz. The results are discussed in terms of their implications for models of pitch perception.