The Phonetics of Speech Production and Medical Research

Abstract

The production of speech requires the interplay of a number of cognitive and motoric activities, which make it an interesting object of study from both a linguistic and a medical point of view. In this paper, we discuss, first, the features and domain of application of the most used technologies in linguistic research on speech production, focusing on those that have been applied to medicine. Second, we offer an insight into the main results that have been obtained so far in studying dysarthria in Italian Parkinson’s Disease, as an example of the interdisciplinary, experimental research at the border between linguistics and medicine.

1.1. The analysis of speech production 

Investigations on speech production may rely on acoustic analyses, which offer information on the issue, though do not allow a direct and detailed observation of the production mechanism. Nowadays phoneticians easily perform acoustic analyses, thanks to the diffusion of recording facilities and dedicated software. However, acoustic data only represent the starting point of studies whose aim is to deeply investigate speech production. Therefore, a typical speech production study involves the recording of both acoustic and articulatory material, and the following analysis of both types of data. Crucially, data acquisition has to be synchronized (via hardware or post-processing) in order to shed light on the articulatory mechanism responsible for the production of linguistic sounds. The following part of this section offers and overview of the main software nowadays used for acoustic investigation, as well as the main technologies exploited in most of the current studies on speech production.

Acoustic analysis, as well as acoustic recordings, may be performed by means of PRAAT (Boersma and Weenink 2020). As for speech analysis, the software allows the user to segment the audio signal and to time-align multiple levels of text labels, such as those regarding consonant and vowel boundaries, as well as the presence intonational events, words, phrases and larger constituents—Figure 1. The system also allows to semi-automatically performing a wide range of acoustic measurements with reference to the abovementioned labels, i.e., points in time. Crucial information on speech may be already collected by means of this type of material and analyses.

However, investigations on speech production nowadays often relies on the use of appropriate, in some cases purpose-built, technology.

1.2. Electromagnetic articulography 

An example of purpose-built technology is the ElectroMagnetic Articulograph (EMA). Various systems are described in the literature (Kaburagi et al. 2005; Stella et al. 2012; Stella et al. 2013), which are used to track the movements of a set of sensors glued on the main speech articulators, such as tongue and lips (see Figure 2, right panel), as well as on more stable parts, such as the front teeth or the forehead, to compensate for head movement. For instance, the Carstens Medizinelektronik GmbH’s system AG501 (http://www.articulograph.de) is used to identify the Cartesian coordinates x, y and z, as well as azimuth and elevation of directionally sensitive magnetic field sensors (8 to 24) at a sampling frequency of up to 1250 Hz in real time. The measuring sensors are single axis coils. Nine reference coils, along three arms (blue arms in Figure 2, left panel), arranged to form a three-dimensional frame of reference, emit magnetic fields at different well known frequencies between 7500 and 13,750 Hz. During a recording session, the alternating currents induced in the sensors by the magnetic fields of the reference coils are separated by their frequencies, digitized and sent in real-time to the control unit. Dedicated software stores the current values, making them available for the spatial arrangement determination process.

1.3. Ultrasound tongue imaging 

Articulatory studies of tongue motion are becoming popular in phonetics, thanks to the adoption of ultrasound systems which have been used for clinical purposes and have been eventually adapted to suit the investigation of speech production. Such systems, which are both non-invasive and non-obtrusive, are able to provide the profile of the tongue during speech production, although the image of tongue apex and radix may be often occluded by the presence of the jaw and the hyoid bone respectively. Ultrasound images are obtained thanks to a high frequency (2-14 MHz) sound waves emitted from an array of piezoelectric transducers (crystals), multiplexed in time: only one crystal emits sound waves in a given time interval, while all the remaining crystals are used to convert the received echoes to voltage values. The ultrasound wave goes through tissues and is reflected when it reaches an interface between tissues or materials with different impedance properties. Ultrasound images are reconstruction of such interfaces, thanks to the processing of the voltage values of the received echoes. In the case of tongue imaging, the probe is placed under the chin and the wave goes upward, though the tongue (Figure 3, left panel). When the wave reaches the upper surface of the tongue, where the impedance changes (think of the mucous membrane and the air in the oral cave), it is reflected to the probe and the surface of the tongue may be reconstructed (Figure 3, right panel). The tongue profile during speech production may be available as a sequence of images that are timealigned with the audio recording.

Nowadays, increased sampling rates allow researchers to get a more convenient number of frames per second during speech, as producing the verbal chain requires quite a high-speed sequence of gestures. For instance, an Aplio XV machine, by Toshiba Medical System corp. (http://www.medical.toshiba.com), allowed us to collect the first ultrasound data related to speech in Italy (Grimaldi et al. 2008). At that time, during each recording session, the ultrasound pictures were exported as a continuous video stream (at 25 Hz), by means of a dedicated S-Video output; such stream was acquired together with the audio signal (synchronously), by means of an external a/v analog-to-digital acquisition card, and then recorder in real-time on a dedicated PC. Nowadays, more compact and fast systems may be used, that do not require the realization of a constellation of hardware and software means to ensure the acquisition of all the information needed to analyze speech. Systems such as the “Micro Ultrasound system for speech research”, proposed by the Articulate Instruments Ltd. (http://www.articulateinstruments.com/alan-wrench/), for instance, may include the hardware synchronization for audio capturing and the AAA software for data analysis.

Linguists, phoneticians, and laboratory phonologist in particular use the abovementioned technologies to test their hypothesis on the linguistic organization of speech production. The way speech gestures are realized and phased with respect to each other is investigated with respect to various languages, as well as with respect to the changes in their organization in the case a second, or foreign, language is produced. However, the testing of the very same linguistic hypotheses, or a better understanding of speech production in general, may be of interest of medical research too (for an overview, see Gili Fivela, Zmarich 2013).

2. Articulatory studies and speech pathology

For many decades, the description, explanation and rehabilitation of various speech articulatory disorders have been based on data derived from phonetic transcriptions, based on transcriber’s perception, and acoustic analysis of pathological speech. However, these methods reveal limitations concerning the description, the explanation and the rehabilitation of speech pathologies. As for the former, limits are due to the subjectivity of the auditory perception and of the subsequent evaluation (Shriberg and Lof 1991), and to the lack of a two-way correspondence between acoustic and articulatory data (Sondhi 1979). As for the possible explanation, limits are identified, firstly, in the “opacity” introduced by the distance between the cause of the pathology and the measured acoustic or perceptual events, which originate at the periphery of the speech production system; secondly, limits concern the inadequacy of phonetic-phonological theories based on the study of perceptual or acoustic targets, and therefore not suitable to explain motor events of an intrinsically dynamic nature (Weismer, Tjaden, Kent 1995). Finally, concerning rehabilitation, a limit consists in the inability to provide adequate articulatory feedback, which can be only partially provided by auditory and, even less, by acoustic information.

A useful integration to these traditional methods is then represented by the information on the dynamics and kinematics of speech, that is the description of the movement (e.g., duration, extension) and the description of the physical conditions responsible for a given movement (which, in addition to the already mentioned descriptors, include, e.g., mass coefficients, rigidity, damping; Bingham 1988). Kinematic and dynamic data offer reliable qualitative and quantitative information about the movements of the articulatory organs (Gracco 1992); from an explanatory point of view, they can provide a precious alternative to traditional explanations, when framed within a suitable theoretical framework, such as a non-linear dynamic theory (see Port, van Gelder 1995). For instance, according to Articulatory Phonology, the main unit of the motoric control for linguistic purposes is the so-called articulatory gesture (see the review in Goldstein and Fowler 2003). The articulatory gesture is dimensioned with respect to the spatial coordinates that represent the vocal tract and with individual quantities that are proportional according to a gestural score that indicates the organization of gestures and the intervals of activation of the tract variables which are relevant to the production (e.g., Lip Aperture or Lip Protrusion, which according to Gestural Phonology—Browman and Goldstein 1986—are associated with a series of articulators). The impact of such theory on the investigation of speech pathologies is acknowledged (van Lieshout, Goldstein 2008), and is tightly linked with experimental investigations performed with the technologies described in the first section of this paper.

In fact, the usefulness of technologies in studying speech articulation has been clear at least since the review by Thompson-Ward and Murdoch (1998), on the methods to verify the articulatory capacity in dysarthria, and the review by Barlow et al. (2009), on cinematic measures related to speech. However, as recalled in the latter, Sonoda (1974) already observed the usefulness of the real-time collection of kinematic data by means of orofacial magnetometry, because such data can be used in the rehabilitation of dysarthria, thanks to the visualization of the movement of the articulators. Not surprisingly, the first studies employing ultrasound in the investigation of pathological speech date back to the early 1980s. As Thompson-Ward and Murdoch (1998) recall, Shawker et al. (1984) already carried out a study on non-pathological and dysarthric speakers, and noted that ultrasounds allowed to observe significant differences in the articulation of vowels (/a/, /i/) and consonants (/k/), and could have therefore represented a promising technique. The usefulness of Electromagnetic articulography is also acknowledged and quite widespread (Wong et al. 2010), as for the investigation of various pathologies, such as dysarthria (Rong et al. 2012; Jaeger et al. 2000; McAuliffe et al. 2005; Wong et al. 2010a; Wong et al. 2011), apraxia of speech (Katz et al. 2003; Katz MacNeil 2010) and stuttering (van Lieshout et al. 1993, 2004; McClean et al. 2004; McClean, Runyan 2000; Max 2004). Further, several corpora of kinematic data have been collected for the investigation of apraxia, stuttering and dysarthria. For instance, van den Berg et al. (2006) for apraxia, van Lieshout et al. (1993), and Ward (1997) for stuttering, and the TORGO database, which includes video, audio and 3D electromagnetic articulation data recordings (AG500) concerning dysarthria (Rudzicz et al. 2008).

Interestingly, a number of studies investigate on the use of technologies, in particular the electromagnetic articulation, in training during therapies related to pathologies that cause articulatory problems in speech. For example, Bose et al. (2001) reported on a one case study on an adult subject suffering from Broca’s aphasia and apraxia. They demonstrated the usefulness of the PROMPT system (Hayden 1984), originally developed for oral language “teaching” and involving the use of auditory, visual and tactile stimuli.

Further, a dynamic field of investigation and application concerns the use of articulatory information for the realization of rehabilitation and training protocols based on biofeedback. The visualization of ultrasound images, for instance, has been successfully used to provide articulatory feedback in the therapy of articulatory problems (Bacsfalvi et al. 2007; Bernhardt et al. 2005). Systems developed from articulographic data are also of considerable interest. In this context, the BALDI systems are worthy of note (Massaro 2004; for Italian, cf. BALDINI, Cosi et al. 2002, LUCIA, Cosi et al. 2008) and ARTUR (Eriksson 2005; Engwall 2008). They are computer training systems used for teaching pronunciation (not only in the case of speech/hearing problems, but also for foreign language learning): thanks to the use of “talking heads”, i.e., heads and faces animated by computers, these systems show the user how to produce speech sounds, and help them to selfcorrect their speech gestures. In particular, ARTUR has been developed on the basis of acoustic, video and EMA data (or MOVETRACK; Branderud 1985) through the optimization of acoustic-articulatory inversion (Kjellström, Engwall 2009).

With respect to the use of biofeedback, the contribution of Katz and McNeil (2010) who studied the feedback effect provided in real time to verify its usefulness in apraxic patients is also particularly important. The study, carried out by means of EMA and sensors positioned on the subjects’ tongue, describes how it is possible to provide information on the use of biofeedback in real time, regarding the position of the tongue (see also Schulz et al. 2006). The system shows subjects how to reach a target indicated on the computer monitor, and is proved to be a useful aid in the improvement of articulatory problems due to apraxia.

As far as Italian is concerned, the usefulness of articulatory investigations has long been recognized and applied to the analysis of stuttering (Zmarich 1999a, 1999b, and following works), and, more recently, to the investigation of dysarthria in Parkinson’s Disease.

3. Dysarthric speech in Parkinson’s Disease: state of the art of investigations on Italian

Along with the development of Parkinson’s Disease, patients often suffer from hypokinetic dysarthria. They show a reduction in the amplitude and speed of movements (Ackermann, Ziegler 1991; Duffy 2005; Darley et al. 1975), which has an impact on speech production too. Besides these very general characteristics, a considerable intra-speaker variability concerning speech abnormalities is observed, which may also depend on various factors, such as the task subjects are asked to perform.

Nevertheless, some established characteristics have been identified. In early stages, there may be mild phonetic impairment, while at later stages articulation becomes less precise, and reading rate becomes slower, with an increased number and duration of pauses which may relate to the difficulties in initiating the articulator movement. Perceptually, hypokinetic dysarthria is characterized by monopitch, monoloudness, reduced stress, imprecise consonants, and inappropriate silences. From a kinematic point of view, a reduction is observed in the movement peak velocity and amplitude of lips and jaw. This is evident from reduced vowel formant transition extents, reduced vowel spaces and reduced consonant spectral distinctiveness (Tjaden 2008). Besides reduction, incoordination has also been observed, and different gesture coordination relations can imply changes in syllabic affiliation too.

Research in speech production in Italian dysarthric speech by Parkinsonian subjects has been performed by adopting the Laboratory Phonology approach (Pierrehumbert et al. 2000). Specifically, the impact of the disease on speech production has been investigated with reference to its effect on phonological features, rather than by itself. Specifically, most of the analyses performed so far aimed at investigating phonological contrasts involving vowels and consonants, with specific attention to the realization of the geminate vs. singleton differences, as produced by mild-to-severe Parkinson’s Disease patients (Gili Fivela et al. 2014; Iraci et al. 2016, 2017a, 2017b; Iraci 2017; Gili Fivela et al., submitted). Besides the obvious interest in the realization of single vowels and consonants, gemination was chosen in order to get information also on the realization of syllables, as the presence of a geminate rather than a singleton involves a change in syllabic affiliation and a change in the duration of the preceding vowel.

The mail goals in investigating Italian dysarthric speech have been 1) to verify if pathological speakers were able to produce the articulatory correlates of a given phonological contrast, and 2) to identify possible compensatory phenomena speakers may adopt in order to overcoming the motor deficit.

Following the Articulatory Phonology framework’s predictions, the hypothesis behind 1) and 2) has been that distinctiveness is not threatened since the vocal tract (which is seen as a dynamical system) would re-organise as a function of the linguistic contrasts to maintain. In order to reach the above mentioned goals and check the main hypothesis, a corpus of acoustic and articulatory—AG501—data has been collected, by asking mild-to-severe Parkinson’s Disease patients, who developed hypokinetic dysarthria, to read aloud highly controlled sentences. They include minimal pairs differing as for the medial consonant—singleton vs. geminate—and the vocalic composition.

Results of analyses performed so far (acoustics by means of PRAAT and articulatory by means of MAYDAY—Sigona et al. 2015) show that Parkinson’s dysarthric speakers show spatial alterations (amplitude of movements) that not necessarily involve a reduction of the range of motion. Specifically, they show even gestures of systematically increased amplitude in the case of horizontal, antero-posterior, displacement of the tongue (Gili Fivela et al. 2014; Iraci 2017). However, intra-speaker analyses showed that phonological distinctions are preserved as much as possible, through a re-organisation of the vocal tract movements, consisting in re-adjustments of surrounding/secondary articulatory gestures (Iraci 2017; Gili Fivela et al., submitted).

The analysis of the abovementioned corpus is still on-going, and it will be deepened thanks to a funded National project (PRIN 2017JNKCYZ), within which prosodic correlates will also be investigated.