Research on the Tadoma method of speech communication

In Tadoma, speech is received by placing a hand on the talker's face and monitoring actions associated with speech production. Our initial research has documented the speech perception, speech production, and linguistic abilities of deaf-blind individuals highly trained in Tadoma. This research has demonstrated that good speech reception can be achieved through the tactile sense: Performance is roughly equivalent to that of normals listening in noise or babble with a signal-to-noise ratio in the range 0-6 dB. It appears that the principal cues employed are lip movement, jaw movement, oral airflow, and laryngeal vibration, and that the errors which occur are caused primarily by inadequate information on tongue position. Our current research includes (1) learning of Tadoma by normal subjects with simulated deafness and blindness, (2) augmenting Tadoma with a supplemental tactile display of tongue position, and (3) developing a synthetic Tadoma system in which signals recorded from a talker's face are used to drive an artificial face. This research is expected to increase our understanding of Tadoma and its relation to other tactile communication methods, show that performance obtained through Tadoma does not represent the ultimate limits of the tactile sense, and provide a research tool for studying transformations of Tadoma.     

Upper lip, lower lip, and jaw interactions during speech: comments on evidence from repetition-to-repetition variability

Six studies purporting to demonstrate complementary covariation in lip and jaw activity during speech are reviewed. The statistical procedures used to assess interactions among the upper lip, lower lip, and jaw movements are discussed for four different experiments analyzing repetition-to-repetition movement variation. The findings from two studies analyzing repetition-to-repetition variation for interactions in electromyographic activity recorded from either the jaw musculature or the labial musculature also are evaluated. It is concluded that these studies do not provide convincing evidence of complementary covariation among the articulators or the muscles.     

Jaw and lip movements of deaf talkers producing utterances with known stress patterns

This investigation determined whether prelingually deaf talkers could correctly produce stressed and unstressed syllables across known changes in stress patterning and phonetic composition. Three deaf and three hearing adults spoke sets of homogeneous syllable strings with stress patterns that they could tap successfully with a finger. Strain gauge transduction of lower lip and jaw movement indicated that both deaf and hearing subjects produced different displacements and durations for the stressed and unstressed syllables, regardless of the stress pattern. Jaw movement did not become more variable with changes in phonetic composition of the syllables. The results show no evidence that motoric abilities (as assessed in lip and jaw movements) limit deaf talkers in producing desired stress patterns.     

Different phase-stable relationships of the upper lip and jaw for production of vowels and diphthongs.

Relational invariants have been reported in the timing of articulatory gestures across suprasegmental changes, such as rate and stress. In the current study, the relative timing of the upper lip and jaw was investigated across changes in both suprasegmental and segmental characteristics of speech. The onset of upper lip movement relative to the vowel-to-vowel jaw cycle during intervocalic bilabial production was represented as a phase angle, and analyzed across changes in stress, vowel height, and vowel/diphthong identity. Results indicated that the relative timing of the upper lip and jaw varied systematically with changes in stress and vowel/diphthong identity, while remaining constant across changes in vowel height. It appears that modifications in relative timing may be due to adjustments in the jaw cycle as a result of the compound nature of jaw movement for diphthongs as compared to vowels, with further modifications due to the effect of stress on these compound movements.     

Lip kinematics in long and short stop and fricative consonants

This paper examines lip and jaw kinematics in the production of labial stop and fricative consonants where the duration of the oral closure/constriction is varied for linguistic purposes. The subjects were speakers of Japanese and Swedish, two languages that have a contrast between short and long consonants. Lip and jaw movements were recorded using a magnetometer system. Based on earlier work showing that the lips are moving at a high velocity at the oral closure, it was hypothesized that speakers could control closure/constriction duration by varying the position of a virtual target for the lips. According to this hypothesis, the peak vertical position of the lower lip during the oral closure/constriction should be higher for the long than for the short consonants. This would result in the lips staying in contact for a longer period. The results show that this is the case for the Japanese subjects and one Swedish subject who produced non-overlapping distributions of closure/ constriction duration for the two categories. However, the peak velocity of the lower lip raising movement did not differ between the two categories. Thus if the lip movements in speech are controlled by specifying a virtual target, that control must involve variations in both the position and the timing of the target.     

Strain gauge transduction of lip and jaw motion in the midsagittal plane: refinement of a prototype system.

This paper describes the design refinements made in an earlier prototype system for transducing lip and jaw movements in the midsagittal plane. Major improvements over the prototype system included: reduced transducer loading, improved two-dimensional separation and resolution, elimination of possible artifacts due to lateral-medial or rotational movement, electrical isolation of the subject from the transducer, increased capability to transduce movements of children as well as adults, and an accessory instrument for orienting the transducers with reference to defined anatomical planes.     

Influence of speaking rate on the upper lip, lower lip, and jaw peak velocity sequencing during bilabial closing movements

Upper lip, lower lip, and jaw movements were recorded in five adult speakers for repeated sequences of the utterance [bae] at different speech rates. The results failed to confirm several earlier reports of an invariant upper lip, lower lip, jaw peak velocity sequencing pattern for bilabial closures. While the earlier reported sequence was the most frequent, a wide variety of different sequences was also commonly observed. In addition, significant intersubject differences in sequencing were found. The present results thus do not support the earlier hypothesis that oral closure gestures reflect aspects of a centrally generated pattern of motor output during speech.     

Kinematic, acoustic, and perceptual analyses of connected speech produced by parkinsonian and normal geriatric adults

Acoustic and kinematic analyses, as well as perceptual evaluation, were conducted on the speech of Parkinsonian and normal geriatric adults. As a group, the Parkinsonian speakers had very limited jaw movement compared to the normal geriatrics. For opening gestures, jaw displacements and velocities produced by the Parkinsonian subjects were about half those produced by the normal geriatrics. Lower lip movement amplitude and velocity also were reduced for the Parkinsonian speakers relative to the normal geriatrics, but the magnitude of the reduction was not as great as that seen in the jaw. Lower lip closing velocities expressed as a function of movement amplitude were greater for the Parkinsonian speakers than for the normal geriatrics. This increased velocity of lower lip movement may reflect a difference in the control of lip elevation for the Parkinsonian speakers, an effect that increased with the severity of dysarthria. Acoustically, the Parkinsonian subjects had reduced durations of vocalic segments, reduced formant transitions, and increased voice onset time compared to the normal geriatrics. These effects were greater for the more severe, compared to the milder, dysarthrics and were most apparent in the more complex, vocalic gestures.     

Patterns of interarticulator phasing and their relation to linguistic structure

Work by Tuller and Kelso [J. Acoust. Soc. Am. 76, 1030-1036 (1984)] and Kelso et al. [J. Phon. 14, 29-59 (1986)] has demonstrated stable relations between jaw and lip movements in (bV#CVb) utterances across rate and stress conditions. Specifically, the onset of lip movement toward the intervocalic consonant was found to be constant with respect to the vowel-to-vowel jaw cycle in both time and relative phasing. An attempt was made to replicate and extend this work by investigating interarticulator phase relations for utterances having a broader range of linguistic organization: In addition to rate and stress, syllable structure (open versus closed syllables) and identity of the intervocalic consonant (/p/ vs /m/) were manipulated. Results showed that the upper lip's lowering onset varied systematically with respect to the jaw vowel cycle as a function of both rate and stress. In addition, syllable structure and consonant identity influenced the relation of lip and jaw gestures. There was a general tendency for any condition that shortened the first vowel to produce earlier onsets of the upper lip relative to the jaw. However, the within-condition jaw cycle duration variability did not correlate with the within-condition variability in phase. Thus it seems that stable interarticulator phase relations maintain not only the integrity of phonological structure, as suggested by Kelso et al., but structural integrity at other levels of linguistic organization as well.     

Lip-jaw and tongue-jaw coordination during rate-controlled syllable repetitions

The present study investigated the relationship between functionally relevant compound gestures and single-articulator component movements of the jaw and the constrictors lower lip and tongue tip during rate-controlled syllable repetitions. In nine healthy speakers, the effects of speaking rate (3 vs 5 Hz), place of articulation, and vowel type during stop consonant-vowel repetitions (/pa/, /pi/, /ta/, /ti/) on the amplitude and peak velocity of differential jaw and constrictor opening-closing movements were measured by means of electromagnetic articulography. Rather than homogeneously scaled compound gestures, the results suggest distinct control mechanisms for the jaw and the constrictors. In particular, jaw amplitude was closely linked to vowel height during bilabial articulation, whereas the lower lip component amplitude turned out to be predominantly rate sensitive. However, the observed variability across subjects and conditions does not support the assumption that single-articulator gestures directly correspond to basic phonological units. The nonhomogeneous effects of speech rate on articulatory subsystem parameters indicate that single structures are differentially rate sensitive. On average, an increase in speech rate resulted in a more or less proportional increase of the steepness of peak velocity/amplitude scaling for jaw movements, whereas the constrictors were less rate sensitive in this respect. Negative covariation across repetitions between jaw and constrictor amplitudes has been considered an indicator of motor equivalence. Although significant in some cases, such a relationship was not consistently observed across subjects. Considering systematic sources of variability such as vowel height, speech rate, and subjects, jaw-constrictor amplitude correlations showed a nonhomogeneous pattern strongly depending on place of articulation.     

An investigation of motor equivalence in the speech of children and adults

Although the concept of motor equivalence (i.e., articulatory intercoordination) is generally accepted as functioning in normal speech production, few studies have experimentally demonstrated its existence. One purpose of the present study was thus to obtain additional data concerning this phenomenon. Because motor equivalence is often assumed to represent a rather sophisticated ability in speakers, another purpose of the study was to determine whether trends could be observed that might demonstrate a developmental progression toward more frequent occurrence of articulatory intercoordination with increasing age. A strain gauge transduction system was used to monitor inferior-superior upper lip, lower lip, and jaw movements produced by a group of adults and three groups of children ranging from 4-11 years of age, as they spoke in a normal condition and in two "perturbed" conditions (bite block and fast rate). Based on the assumption that the presence of a significant negative correlation between two articulators constitutes evidence of articulatory intercoordination, there was little indication of motor equivalence in the speech of the adults or the children.     

Three-dimensional modeling of hearing in Delphinus delphis.

Physical modeling is a fertile approach to investigating sound emission and reception (hearing) in marine mammals. A method for simulation of hearing was developed combining three-dimensional acoustic propagation and extrapolation techniques with a novel approach to modeling the acoustic parameters of mammalian tissues. Models of the forehead and lower jaw tissues of the common dolphin, Delphinus delphis, were created in order to simulate the biosonar emission and hearing processes. This paper outlines the methods used in the hearing simulations and offers observations concerning the mechanisms of acoustic reception in this dolphin based on model results. These results include: (1) The left and right mandibular fat bodies were found to channel sound incident from forward directions to the left and right tympanic bulla and to create sharp maxima against the lateral surfaces of each respective bulla; (2) The soft tissues of the lower jaw improved the forward directivity of the simulated receptivity patterns; (3) A focal property of the lower-jaw pan bones appeared to contribute to the creation of distinct forward receptivity peaks for each ear; (4) The reception patterns contained features that may correspond to lateral hearing pathways. A "fast" lens mechanism is proposed to explain the focal contribution of the pan bones in this dolphin. Similar techniques may be used to study hearing in other marine mammals.     

Kinematic and electromyographic responses to perturbation of the jaw

The task-dependent organization of sensorimotor mechanisms during the production of speech was investigated using a perturbation paradigm. Six subjects received unanticipated jaw perturbations before and during tongue elevation for [aedae], in which the lips do not participate, and bilabial closure for [aebae], in which the tongue does not participate. A strain gauge system was used to monitor inferior-superior displacements of the upper lip, lower lip, and jaw, while hooked-wire electrodes monitored muscle activity in various muscles of the lips, jaw, and tongue. Results indicated significant compensatory kinematic adjustments to jaw perturbations in the lips and/or jaw during [aebae], but no labial compensations during [aedae] (with the exception of one subject). EMG responses were inconsistent and not necessarily indicative of the kinematic findings. Individual subjects responded to perturbations reliably but differently, using different combinations of involved articulators to achieve bilabial closure and lingua-alveolar contact. The current study supports earlier research which suggests that the components of the motor system are flexibly assembled, based on the requirements of the specific task. That is, compensatory responses to sensory information occur only when such responses are functionally necessary.     

Temporal measures of anticipatory labial coarticulation for the vowel/u/: within- and cross-subject variability.

The timing of upper lip protrusion movements and accompanying acoustic events was examined for multiple repetitions of word pairs such as "lee coot" and "leaked coot" by four speakers of American English. The duration of the intervocalic consonant string was manipulated by using various combinations of /s/, /t/, /k/, /h/, and /#/. Pairwise comparisons were made of consonant string duration (acoustic /i/ offset to acoustic /u/ onset) with durations of: protrusion movement beginning to acoustic /u/ onset, maximum acceleration of the movement to acoustic /u/ onset, and acoustic /u/ onset to movement end. There were some consonant-specific protrusion effects, primarily on the movement beginning event for /s/. Inferences from measures of the maximum acceleration and movement end events for the non-/s/ subset suggested the simultaneous and variable expression of three competing constraints: (1) end the protrusion movement during the voiced part of the /u/; (2) use a preferred movement duration; and (3) begin the /u/-related protrusion movement when permitted by relaxation of the perceptually motivated constraint that the preceding /i/ be unrounded. The subjects differed in the degree of expression of each constraint, but the results generally indicate that anticipatory coarticulation of lip protrusion is influenced both by acoustic-phonetic context dependencies and dynamical properties of movements. Because of the extensive variation in the data and the small number of subjects, these ideas are tentative; additional work is needed to explore them further.     

Stimulus uncertainty and insensitivity to pitch-change direction

In a series of experiments, Semal and Demany [(2006). J. Acoust. Soc. Am. 120, 3907-3915] demonstrated that some normally hearing listeners are unable to determine the direction of small but detectable differences in frequency between pure tones. Unlike studies demonstrating similar effects in patients with brain damage, the authors used stimuli in which the standard frequency of the tones was highly uncertain (roved) over trials. In Experiment 1, listeners were identified as insensitive to the direction of pitch changes using stimuli with frequency roving. When listeners were retested using stimuli without roving in Experiment 2, impairments in pitch-direction identification were generally much less profound. In Experiment 3, frequency-roving range had a systematic effect on listeners' thresholds, and impairments in pitch-direction identification tended to occur only when the roving range was widest. In Experiment 4, the influence of frequency roving was similar for continuous frequency changes as for discrete changes. Possible explanations for the influence of roving on listeners' insensitivity to pitch-change direction are discussed.     

Tactile information transfer: a comparison of two stimulation sites

Two experiments on the discrimination of time-varying tactile stimuli were performed, with comparison of stimulus delivery to the distal pad of the right index finger and to the right wrist (palmar surface). Subjects were required to perceive differences in short sequences of computer-generated stimulus elements (experiment 1) or differences in short tactile stimuli derived from a speech signal (experiment 2). The pulse-train stimuli were distinguished by differences in frequency (i.e., pulse repetition rate) and amplitude, and by the presence/absence of gaps (approximately 100-ms duration). Stimulation levels were 10 dB higher at the wrist than at the fingertip, to compensate for the lower vibration sensitivity at the wrist. Results indicate similar gap detection at wrist and fingertip and similar perception of frequency differences. However, perception of amplitude differences was found to be better at the wrist than at the fingertip. Maximum information transfer rates for the stimuli in experiment 1 were estimated at 7 bits s(-1) at the wrist and 5 bits s(-1) at the fingertip.     

Interarticulator programming in VCV sequences: lip and tongue movements

This study examined the temporal phasing of tongue and lip movements in vowel-consonant-vowel sequences where the consonant is a bilabial stop consonant /p, b/ and the vowels one of /i, a, u/; only asymmetrical vowel contexts were included in the analysis. Four subjects participated. Articulatory movements were recorded using a magnetometer system. The onset of the tongue movement from the first to the second vowel almost always occurred before the oral closure. Most of the tongue movement trajectory from the first to the second vowel took place during the oral closure for the stop. For all subjects, the onset of the tongue movement occurred earlier with respect to the onset of the lip closing movement as the tongue movement trajectory increased. The influence of consonant voicing and vowel context on interarticulator timing and tongue movement kinematics varied across subjects. Overall, the results are compatible with the hypothesis that there is a temporal window before the oral closure for the stop during which the tongue movement can start. A very early onset of the tongue movement relative to the stop closure together with an extensive movement before the closure would most likely produce an extra vowel sound before the closure.     

基于时间序列模型的粒子滤波行人跟踪算法研究

针对行人运动的随机性导致运动状态模型适应性差和人在行走过程中可能发生短时全部或局部遮挡导致行人跟踪算法精度较低的问题,提出基于时间序列模型的粒子滤波行人跟踪算法。建立了行人运动时间序列模型;给出了基于对视频序列初始帧的检测,确定行人的位置、宽高等作为跟踪先验信息的方法;由先验信息计算加权颜色直方图构建初始粒子群分布,并利用时间序列运动模型预测粒子在下一时刻的状态分布,并更新粒子权值;根据有效粒子的个数判断是否进行重采样;最后由所有粒子的加权和估计行人的运动状态。仿真实验表明：文中提出根据行人的运动轨迹时间序列运动模型可使行人的状态估计更准确,预测误差进一步减小,预测精度得到了提高。     

Seeing pitch: visual information for lexical tones of Mandarin-Chinese

Mandarin perceivers were tested in visual lexical-tone identification before and after learning. Baseline performance was only slightly above chance, although there appeared to be some visual information in the speakers' neck and head movements. When participants were taught to use this visible information in two experiments, visual tone identification improved significantly. There appears to be a relationship between the production of lexical tones and the visible movements of the neck, head, and mouth, and this information can be effectively used after a short training session.     

Interarticulator cohesion within coronal consonant production

If more than one articulator is involved in the execution of a phonetic task, then the individual articulators have to be temporally coordinated with each other in a lawful manner. The present study aims at analyzing tongue-jaw cohesion in the temporal domain for the German coronal consonants [s, f, t, d, n, l], i.e., consonants produced with the same set of articulators--the tongue blade and the jaw--but differing in manner of articulation. The stability of obtained interaction patterns is evaluated by varying the degree of vocal effort: comfortable and loud. Tongue and jaw movements of five speakers of German were recorded by means of electromagnetic midsagittal articulography (EMMA) during [aCa] sequences. The results indicate that (1) tongue-jaw coordination varies with manner of articulation, i.e., a later onset and offset of the jaw target for the stops compared to the fricatives, the nasal and the lateral; (2) the obtained patterns are stable across vocal effort conditions; (3) the sibilants are produced with smaller standard deviations for latencies and target positions; and (4) adjustments to the lower jaw positions during the surrounding vowels in loud speech occur during the closing and opening movement intervals and not the consonantal target phases.