Tactile Speech Communication: Reception of Words and Two-Way Messages through a Phoneme-Based Display
Abstract
:1. Introduction
1.1. Background
1.2. Motivation for Current Study
2. General Methods
2.1. Participants
2.2. TActile Phonemic Sleeve (TAPS) System
2.3. Calibration of Perceived Intensity and Tactor Equalization
2.4. Haptic Symbols for Phonemes and Words
3. Study I: Acquisition of Four-Phoneme Words
3.1. Learning Materials
3.2. Procedures
- Day 1: Practice with List #1 (20 min).
- Days 2 to 4: Practice with List #1 (10 min) and test (25 trials/block × 2 blocks).
- Days 5 to 10: Practice with List 1 (10 min) and test (25 trials/block × 3 blocks).
- Days 11 & 12: Test with List #2 (25 trials/block × 4 blocks) without practice.
3.3. Data Analysis
3.4. Results
3.4.1. Word PC Scores
3.4.2. Number of Phoneme Errors per Word
3.4.3. Phoneme Errors vs. Position in a Word
3.4.4. Reaction Time
3.5. Discussion
4. Study II: Two-Way Messaging via TAPS
4.1. Experimental Setup
4.2. Procedures
4.3. Data Analysis
4.3.1. PC for Messages ()
4.3.2. PC for Words ()
4.3.3. Communication Rates in Messages per Minute () and Words per Minute ()
4.4. Results
4.5. Discussion
4.5.1. Word Reception
4.5.2. Message Reception
4.5.3. Communication Rates
4.5.4. Future Research
5. Concluding Remarks
- Following training on the order of 3 to 5 h, participants could successfully identify a trained set of 4-phoneme words at a rate of 87% correct. This training generalized to a novel set of 4-phoneme words where performance dropped by only 10 percentage points.
- A two-way messaging task was carried out by two experienced users of TAPS. Over a 45-day period encompassing 13 h, 73% of the messages were identified accurately in conjunction with 82% of the words, based on an average of 2.6 presentations per message. Rates of communication using this ecologically valid setup, however, were slow, averaging of the order of 1 message per minute.
- The two-way messaging results with TAPS contribute to the literature on the use of tactile devices for speech communication in a realistic environment, adding to the work of de Vargas et al. [7] with WhatsHap. These studies represent an advancement over much of the previous research that has employed closed sets of word stimuli with one-way communication in a well-controlled laboratory setting. Our research has demonstrated the potential for real-life communication using spontaneous messages composed of an open vocabulary between two users of the TAPS tactile system.
- Future research will be focused on increasing the rates of communication achieved with TAPS. The time needed for communication can be reduced by a reduction in temporal factors, including phoneme duration, inter-phoneme interval, and inter-word interval, as well a further practice with the TAPS system.
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Zhao, S.; Israr, A.; Lau, F.; Abnousi, F. Coding tactile symbols for phonemic communication. In Proceedings of the CHI Conference on Human Factors in Computing Systems, Montreal, QC, Canada, 21–26 April 2018; pp. 1–13. [Google Scholar]
- Turcott, R.; Chen, J.; Castillo, P.; Knott, B.; Setiawan, W.; Briggs, F.; Klumb, K.; Abnousi, F.; Chakka, P.; Lau, F.; et al. Efficient evaluation of coding strategies for transcutaneous language communication. In Proceedings of the International Conference on Human Haptic Sensing and Touch Enabled Computer Applications, Pisa, Italy, 13–16 June 2018; pp. 600–611. [Google Scholar]
- Chen, J.; Turcott, R.; Castillo, P.; Setiawan, W.; Lau, F.; Israr, A. Learning to feel words: A comparison of learning approaches to acquire haptic words. In Proceedings of the 15th ACM Symposium on Applied Perception, Vancouver, BC, Canada, 10–11 August 2018; pp. 1–7. [Google Scholar]
- Dunkelberger, N.; Sullivan, J.; Bradley, J.; Walling, N.P.; Manickam, I.; Dasarathy, G.; Israr, A.; Lau, F.W.; Klumb, K.; Knott, B.; et al. Conveying language through haptics: A multi-sensory approach. In Proceedings of the 2018 ACM International Symposium on Wearable Computers, Singapore, 8–12 October 2018; pp. 25–32. [Google Scholar]
- Dunkelberger, N.; Sullivan, J.L.; Bradley, J.; Manickam, I.; Dasarathy, G.; Baraniuk, R.; O’Malley, M.K. A multisensory approach to present phonemes as language through a wearable haptic device. IEEE Trans. Haptics 2020, 14, 188–199. [Google Scholar] [CrossRef] [PubMed]
- de Vargas, M.F.; Weill-Duflos, A.; Cooperstock, J.R. Haptic speech communication using stimuli evocative of phoneme production. In Proceedings of the 2019 IEEE World Haptics Conference (WHC), Tokyo, Japan, 9–12 July 2019; pp. 610–615. [Google Scholar]
- de Vargas, M.F.; Marino, D.; Weill, A.; Cooperstock, J.R. Speaking haptically: From phonemes to phrases with a mobile haptic communication system. IEEE Trans. Haptics 2021, 14, 479–490. [Google Scholar] [CrossRef] [PubMed]
- Marino, D.; de Vargas, M.F.; Weill, A.; Cooperstock, J.R. Conversing Using WhatsHap: A Phoneme Based Vibrotactile Messaging Platform. In Proceedings of the IEEE World Haptics Conference (WHC), Montreal, QC, Canada, 6–9 July 2021; pp. 943–948. [Google Scholar]
- Jung, J.; Jiao, Y.; Severgnini, F.M.; Tan, H.Z.; Reed, C.M.; Israr, A.; Lau, F.; Abnousi, F. Speech communication through the skin: Design of learning protocols and initial findings. In Proceedings of the International Conference of Design, User Experience, and Usability, Las Vegas, NV, USA, 15–20 July 2018; pp. 447–460. [Google Scholar]
- Jiao, Y.; M Severgnini, F.; Martinez, J.S.; Jung, J.; Tan, H.Z.; Reed, C.M.; Wilson, E.C.; Lau, F.; Israr, A.; Turcott, R.; et al. A comparative study of phoneme-and word-based learning of English words presented to the skin. In Proceedings of the International Conference on Human Haptic Sensing and Touch Enabled Computer Applications, Pisa, Italy, 13–16 June 2018; pp. 623–635. [Google Scholar]
- Reed, C.M.; Tan, H.Z.; Perez, Z.D.; Wilson, E.C.; Severgnini, F.M.; Jung, J.; Martinez, J.S.; Jiao, Y.; Israr, A.; Lau, F.; et al. A phonemic-based tactile display for speech communication. IEEE Trans. Haptics 2019, 12, 2–17. [Google Scholar] [CrossRef] [PubMed]
- Tan, H.Z.; Reed, C.M.; Jiao, Y.; Perez, Z.D.; Wilson, E.C.; Jung, J.; Martinez, J.S.; Severgnini, F.M. Acquisition of 500 English words through a TActile phonemic sleeve (TAPS). IEEE Trans. Haptics 2020, 13, 745–760. [Google Scholar] [CrossRef] [PubMed]
- Martinez, J.S.; Tan, H.Z.; Reed, C.M. Improving tactile codes for increased speech communication rates in a phonemic-based tactile display. IEEE Trans. Haptics 2021, 14, 200–211. [Google Scholar] [CrossRef] [PubMed]
- Reed, C.M.; Tan, H.Z.; Jiao, Y.; Perez, Z.D.; Wilson, E.C. Identification of words and phrases through a phonemic-based haptic display: Effects of inter-phoneme and inter-word interval durations. ACM Trans. Appl. Percept. 2021, 18, 1–22. [Google Scholar] [CrossRef]
- Kirman, J.H. Tactile communication of speech: A review and an analysis. Psychol. Bull. 1973, 80, 54–74. [Google Scholar] [CrossRef] [PubMed]
- Reed, C.M.; Durlach, N.I.; Braida, L.D. Research on tactile communication of speech: A review. ASHA Monogr. 1982, 80, 1–23. [Google Scholar]
- Reed, C.M.; Durlach, N.I.; Delhorne, L.A.; Rabinowitz, W.M. Research on tactual communication of speech: Ideas, issues, and findings. Volta Rev. 1989, 91, 65–78. [Google Scholar]
- Kappers, A.M.; Plaisier, M.A. Hands-Free Devices for Displaying Speech and Language in the Tactile Modality—Methods and Approaches. IEEE Trans. Haptics 2021, 14, 465–478. [Google Scholar] [CrossRef]
- Uchanski, R.M.; Braida, L.D. Effects of token variability on our ability to distinguish between vowels. Percept. Psychophys. 1998, 60, 533–543. [Google Scholar] [CrossRef] [PubMed]
- Dunkelberger, N.; Bradley, J.; Sullivan, J.L.; Israr, A.; Lau, F.; Klumb, K.; Abnousi, F.; O’Malley, M.K. Improving perception accuracy with multi-sensory haptic cue delivery. In Proceedings of the International Conference on Human Haptic Sensing and Touch Enabled Computer Applications, Pisa, Italy, 13–16 June 2018; pp. 289–301. [Google Scholar]
- Brooks, P.; Frost, B.J. Evaluation of a tactile vocoder for word recognition. J. Acoust. Soc. Am. 1983, 74, 34–39. [Google Scholar] [CrossRef]
- Brooks, P.; Frost, B.; Mason, J.; Chung, K. Acquisition of a 250-word vocabulary through a tactile vocoder. J. Acoust. Soc. Am. 1985, 77, 1576–1579. [Google Scholar] [CrossRef] [PubMed]
- Brooks, P.; Frost, B.; Mason, J.; Gibson, D. Continuing evaluation of the Queen’s University tactile vocoder. I: Identification of open set words. J. Rehabil. Res. Dev. 1986, 23, 119–128. [Google Scholar]
- Galvin, K.L.; Blamey, P.J.; Cowan, R.S.; Oerlemans, M.; Clark, G.M. Generalization of tactile perceptual skills to new context following tactile-alone word recognition training with the Tickle Talker™. J. Acoust. Soc. Am. 2000, 108, 2969–2979. [Google Scholar] [CrossRef] [PubMed]
- Norton, S.J.; Schultz, M.C.; Reed, C.M.; Braida, L.D.; Durlach, N.I.; Rabinowitz, W.M.; Chomsky, C. Analytic study of the Tadoma method: Background and preliminary results. J. Speech Hear. Res. 1977, 20, 574–595. [Google Scholar] [CrossRef]
- Reed, C.M.; Rabinowitz, W.M.; Durlach, N.I.; Braida, L.D.; Conway-Fithian, S.; Schultz, M.C. Research on the Tadoma method of speech communication. J. Acoust. Soc. Am. 1985, 77, 247–257. [Google Scholar] [CrossRef]
- Reed, C.M. Tadoma: An Overview of Research. In Profound Deafness and Speech Communication; Whurr Publishers: London, UK, 1995; pp. 40–55. [Google Scholar]
- Reed, C.M.; Durlach, N.I. Note on information transfer rates in human communication. Presence 1998, 7, 509–518. [Google Scholar] [CrossRef]
- Jones, L.A.; Tan, H.Z. Application of psychophysical techniques to haptic research. IEEE Trans. Haptics 2013, 6, 268–284. [Google Scholar] [CrossRef]
- Boothroyd, A.; Hanin, L.; Hnath, T. A Sentence Test of Speech Perception: Reliability, Set-Equivalence, and Short-Term Learning; RCI10; City University of New York, Speech and Hearing Sciences Research Center: New York, NY, USA, 1985. [Google Scholar]
- Dudai, Y.; Karni, A.; Born, J. The consolidation and transformation of memory. Neuron 2015, 88, 20–32. [Google Scholar] [CrossRef]
- Black, A.W.; Lenzo, K. Flite: A Small, Fast Speech Synthesis Engine. 2005. Available online: http://cmuflite.org/doc/flite.pdf (accessed on 5 February 2024).
- Shim, S.; Tan, H.Z. PalmScape: Calm and pleasant vibrotactile signals. In Proceedings of the Design, User Experience, and Usability, Interaction Design: 9th International Conference, DUXU 2020, Copenhagen, Denmark, 19–24 July 2020; Chapter 37. pp. 1–17. [Google Scholar]
- Sherbecoe, R.L.; Studebaker, G.A. Supplementary formulas and tables for calculating and interconverting speech recognition scores in transformed arcsine units. Int. J. Audiol. 2004, 43, 442–448. [Google Scholar] [CrossRef] [PubMed]
- Studebaker, G.A. A “rationalized” arcsine transform. J. Speech Lang. Hear. Res. 1985, 28, 455–462. [Google Scholar] [CrossRef] [PubMed]
- Geldard, F.A. Adventures in tactile literacy. Am. Psychol. 1957, 12, 115–124. [Google Scholar] [CrossRef]
- Reed, C.M.; Tan, H.Z.; Jones, L.A. Haptic Communication of Language. IEEE Trans. Haptics 2023, 6, 134–153. [Google Scholar] [CrossRef]
- Rantala, J.; Raisamo, R.; Lylykangas, J.; Surakka, V.; Raisamo, J.; Salminen, K.; Pakkanen, T.; Hippula, A. Methods for presenting braille characters on a mobile device with a touchscreen and tactile feedback. IEEE Trans. Haptics 2009, 2, 28–39. [Google Scholar] [CrossRef]
- Amemiya, T.; Yamashita, J.; Hirota, K.; Hirose, M. Virtual leading blocks for the deaf-blind: A real-time way-finder by verbal-nonverbal hybrid interface and high-density RFID tag space. In Proceedings of the IEEE Virtual Reality, Chicago, IL, USA, 27–31 March 2004; pp. 165–287. [Google Scholar]
- Chang, C.M.; Sanches, F.; Gao, G.; Johnson, S.; Liarokapis, M. An Adaptive, Affordable, Humanlike Arm Hand System for Deaf and DeafBlind Communication with the American Sign Language. In Proceedings of the 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Kyoto, Japan, 23–27 October 2022; pp. 871–878. [Google Scholar]
References | Learning Time (Hour) | No. of Tactors | No. of Phonemes | No. of Words Tested | Word Scores |
---|---|---|---|---|---|
[9] | 1 | 24 | 10 | 51 | ≈100% |
[11] | 1.5−4 | 24 | 39 | − | 85% * |
[10,12] | 1.7 | 24 | 39 | 100 | >90% (best) |
[12] | 4.5−8 | 24 | 39 | 500 | 71.9% |
[14] | 1.5−9.3 | 24 | 39 | phrases | 75% |
[4,5,20] | 1.7−2.7 | 4(+2) | 23 | 50 | 67−87% |
[6,7,8] | 1.7 | 2 | 25 | 150 | 51% |
List #1 | ||||
---|---|---|---|---|
able | clean | first | mirage | steak |
after | clothe | flower | moved | stock |
apple | color | fourth | movie | store |
aunts | cooked | fried | often | summer |
baby | cost | from | only | sweet |
band | cover | glove | oven | tailor |
belt | crack | golf | paper | tank |
berry | cream | green | park | teacher |
black | cruise | guest | plan | tent |
block | diet | hard | puppy | throat |
bond | dollar | honey | quit | train |
borrow | dress | huge | ranch | trick |
break | drop | kids | rest | trip |
bring | ducks | last | sand | turkey |
brush | easier | later | silk | under |
butter | Easter | lend | singer | water |
card | eaten | lion | skirt | weeks |
change | even | lunch | slice | wind |
chimp | fever | major | slope | years |
city | field | milk | snake | zipper |
List #2 | ||||
---|---|---|---|---|
about | cloth | fresh | neighbor | soccer |
alter | coffee | fries | office | sold |
asked | coins | fruit | onto | stall |
avoid | cookie | garage | open | still |
ballet | crash | gift | paint | stop |
bank | cross | great | pairs | style |
bark | cute | grill | parade | sunny |
best | desk | guest | past | swim |
better | dinner | heavy | pepper | tax |
bleach | doing | horse | place | think |
blood | drive | into | press | toast |
boots | dryer | July | ready | today |
breathe | eating | least | rent | track |
bridge | enjoy | letter | risk | treat |
broke | enough | lights | shirts | trim |
busy | every | liver | shutter | twice |
cats | fast | lost | skate | washer |
child | fatty | money | sleeve | weather |
chilly | find | month | slip | wrist |
clear | fix | must | smoke | yard |
Sender | Message | No. of Presentations |
---|---|---|
P1 | how is your project | 1 |
P3 | should start testing | 3 |
P1 | how many participants | 2 |
P3 | I think five | 2 |
P1 | how long does it take | 1 |
P3 | I think three days | 4 |
P1 | starting this week | 3 |
P3 | yes how about you | 1 |
P1 | next week probably | 2 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Jung, J.; Reed, C.M.; Martinez, J.S.; Tan, H.Z. Tactile Speech Communication: Reception of Words and Two-Way Messages through a Phoneme-Based Display. Virtual Worlds 2024, 3, 184-207. https://doi.org/10.3390/virtualworlds3020010
Jung J, Reed CM, Martinez JS, Tan HZ. Tactile Speech Communication: Reception of Words and Two-Way Messages through a Phoneme-Based Display. Virtual Worlds. 2024; 3(2):184-207. https://doi.org/10.3390/virtualworlds3020010
Chicago/Turabian StyleJung, Jaehong, Charlotte M. Reed, Juan S. Martinez, and Hong Z. Tan. 2024. "Tactile Speech Communication: Reception of Words and Two-Way Messages through a Phoneme-Based Display" Virtual Worlds 3, no. 2: 184-207. https://doi.org/10.3390/virtualworlds3020010
APA StyleJung, J., Reed, C. M., Martinez, J. S., & Tan, H. Z. (2024). Tactile Speech Communication: Reception of Words and Two-Way Messages through a Phoneme-Based Display. Virtual Worlds, 3(2), 184-207. https://doi.org/10.3390/virtualworlds3020010