The Dual-Coding and Multimedia Learning Theories: Film Subtitles as a Vocabulary Teaching Tool
Abstract
:1. Introduction
2. The Dual-Coding Theory
3. The Bilingual Dual-Coding Theory
- The image systems function independently from the two verbal systems, which is what the original monolingual dual-coding theory supports.
- The three systems are interconnected at the referential level and as a result, the imagery system can influence the verbal system and vice versa.
- The two verbal systems are referentially connected to the imagery system, which allows them to be partly shared and partly independent, with two major implications (a) there is a variable degree of referential overlap between languages, depending on the specific pair of languages each time and (b) the fact that the imagery system becomes a means of indirect access from one language to the other.
- Bilinguals have access to two verbal systems which are independent, however, also interconnected to a certain degree, which means that a change in the language input may result in code-switching.
4. The Cognitive Theory of Multimedia Learning
- Dual channels: As in dual-coding theory, there are two separate channels that process information from sensory memory, namely the auditory/verbal and the visual/pictorial channel. The auditory/verbal channel processes information in the form of spoken words, narrations, or sounds via the ears; and the visual/pictorial channel processes information in the form of pictures, graphs, videos/animation, etc., received through the eyes.
- Limited capacity: As in the cognitive load theory, the information that learners can process in their working memory, in each channel at any one time, is not unlimited; therefore, when a learning task demands more cognitive processing than the capacity of a learner’s processing system, then cognitive overload occurs. Sounds are stored in a verbal mental model and images are stored in a visual mental model, as Mayer [25] explains, “when an illustration or animation is presented, the learner is able to hold only a few images in the visual channel of working memory at any one time, reflecting portions of the presented material” (p. 49). The same is also true when learners are presented with a narration in that they can only hold a few words in the verbal channel of working memory at any one time [25].
- Active processing: learning based on the CTML is an active process where the learners select and organize words and images, and then integrate them together with prior knowledge drawn from their long-term memory which is brought back into their short-term memory to facilitate processing. The integration of new information into the learner’s existing knowledge may include all or some of the following cognitive processes:
- selecting relevant words for processing in verbal working memory,
- selecting relevant images for processing in visual working memory,
- organizing selected words into a verbal model,
- organizing selected images into a pictorial model, and
- integrating the verbal and pictorial representations with each other and with relevant prior knowledge activated from long-term memory. [25]
5. Teaching Vocabulary through Multimedia; Film Subtitles as a Tool
- L2 audio with L1 subtitles (standard subtitling),
- L2 audio with L2 subtitles (captions or bimodal subtitling),
- L2 audio without subtitles (no subtitling),
- L1 audio with L2 subtitles (reserved subtitles).
6. Interactive Uses of Subtitling as a Multimedia Tool
7. Conclusions
Author Contributions
Funding
Conflicts of Interest
References
- Cuban, L. Teachers and Machines: The Classroom Use of Technology Since 1920; Teachers College Press: New York, NY, USA, 1986. [Google Scholar]
- Jimoyiannis, A. Factors Determining Teachers’ Beliefs and Perceptions of ICT in Education. In Encyclopedia of Information Communication Technology; Palma, A.C.M., Ed.; IGI Global: Hershey, PA, USA, 2008; pp. 321–334. [Google Scholar]
- Βοσνιάδου, Σ. Εισαγωγή στην Ψυχολογία; Gutenberg: Αθήνα, Greece, 2001. [Google Scholar]
- Russell, M.; Bebell, D.; O’Dwyer, L.; O’Connor, K. Examining Teacher Technology Use: Implications for Preservice and Inservice Teacher Preparation. J. Teach. Educ. 2003, 54, 297–310. [Google Scholar] [CrossRef]
- Τάσση, Ό. Οι σχέσεις των εκπαιδευτικών με τις Τεχνολογίες της Πληροφορίας και Επικοινωνιών στο σχολείο. Έρκυνα, Επιθεώρηση Εκπαιδευτικών–Επιστημονικών Θεμάτων 2015, 1, 200–215. [Google Scholar]
- Mayer, R.E.; Moreno, R. A Cognitive Theory of Multimedia Learning: Implications for Design Principles. J. Educ. Psychol. 1998, 91, 358–368. [Google Scholar]
- Alzahrani, S. Linking Multimedia Vocabulary CALL Research to SLA Cognitive Theories. Am. J. Educ. Res. 2017, 5, 821–827. [Google Scholar]
- Paivio, A. Imagery and Verbal Processes; Psychology Press: New York, NY, USA, 1971. [Google Scholar]
- Paivio, A. Mental Representations: A Dual Coding Approach; Oxford University Press: Oxford, UK, 1986; Volume 9. [Google Scholar]
- Paivio, A. Dual Coding Theory: Retrospect and Current Status. Can. J. Psychol. Rev. Can. Psychol. 1991, 45, 255–287. [Google Scholar] [CrossRef]
- Reed, S.K. Cognition: Theories and Applications, 8th ed.; Wadsworth, Cengage Learning: Belmont, CA, USA, 2010. [Google Scholar]
- Sternberg, R.J. Cognitive Psychology, 3rd ed.; Thomson/Wadsworth: Belmont, CA, USA, 2003; p. 596. [Google Scholar]
- Al-Seghayer, K. The effect of multimedia annotation modes on L2 vocabulary acquisition: A comparative study. Lang. Learn. Technol. 2001, 5, 202–232. [Google Scholar]
- Paivio, A. Intelligence; Elsevier: Amsterdam, The Netherlands, 2014; Volume 47, pp. 141–158. [Google Scholar]
- Clark, J.M.; Paivio, A. Dual coding theory and education. Educ. Psychol. Rev. 1991, 3, 149–210. [Google Scholar] [CrossRef] [Green Version]
- Moeser, S.D.; Bregman, A. Imagery and language acquisition 1. J. Verbal Learn. Verbal Behav. 1973, 12, 8. [Google Scholar] [CrossRef]
- Paivio, A.; Begg, I. Psychology of Language; Prentice-Hall: Englewood Cliffs, NJ, USA, 1981. [Google Scholar]
- Rieber, L.P. Computers, Graphics, and Learning; Brown and Benchmark: Madison, WI, USA, 1994. [Google Scholar]
- Lin, C.-C. Learning Action Verbs with Animation. JALT Call J. 2009, 5, 23–40. [Google Scholar]
- Al-Seghayer, K. Celebrating the story of my first contribution to call. Lang. Learn. Technol. 2016, 20, 179–184. [Google Scholar]
- Paivio, A.; Walsh, M.; Bons, T. Concreteness effects on memory: When and why? J. Exp. Psychol. Learn. Mem. Cogn. 1994, 20, 1196–1204. [Google Scholar] [CrossRef]
- Pylyshyn, Z.W. What the mind’s eye tells the mind’s brain: A critique of mental imagery. Psychol. Bull. 1973, 80, 1. [Google Scholar] [CrossRef]
- Paivio, A.; Desrochers, A. A dual-coding approach to bilingual memory. Can. J. Psychol. Rev. Can. Psychol. 1980, 34, 388–399. [Google Scholar] [CrossRef]
- Soh, K.C. Bilingual dual-coding and code-switching: Implications for the L1 in L2 learning. J. Linguist. Lang. Teach. 2010, 1, 271–296. [Google Scholar]
- Mayer, R.E. The Cambridge Handbook of Multimedia Learning, 2nd ed.; Cambridge Handbooks in Psychology; Cambridge University Press: Cambridge, UK, 2014; p. 930. [Google Scholar]
- Mayer, R.E. Multimedia Learning; Cambridge University Press: Cambridge, UK, 2001. [Google Scholar]
- Mayer, R.E. Cognitive Theory of Multimedia Learning. In The Cambridge Handbook of Multimedia Learning; Mayer, R., Ed.; Cambridge University Press: Cambridge, UK, 2005; pp. 31–48. [Google Scholar]
- Mayer, R.E. Multimedia learning: Are we asking the right questions? Educ. Psychol. 1997, 32, 1–19. [Google Scholar] [CrossRef] [Green Version]
- Wittrock, M.C. Learning as a generative process. Educ. Psychol. 1974, 11, 87–95. [Google Scholar] [CrossRef]
- Butcher, K.R. Learning from text with diagrams: Promoting mental model development and inference generation. J. Educ. Psychol. 2006, 98, 182–197. [Google Scholar] [CrossRef]
- Butcher, K.R. The Multimedia Principle. In The Cambridge Handbook of Multimedia Learning, 2nd ed.; Cambridge University Press: Cambridge, UK, 2014; pp. 174–205. [Google Scholar]
- Sweller, J. Implications of Cognitive Load Theory for Multimedia Learning. In The Cambridge Handbook of Multimedia Learning (Cambridge Handbooks in Psychology); Mayer, R., Ed.; Cambridge University Press: Cambridge, UK, 2005; pp. 19–30. [Google Scholar]
- Mayer, R.; Fiorella, L. Principles for Reducing Extraneous Processing in Multimedia Learning: Coherence, Signaling, Redundancy, Spatial Contiguity, and Temporal Contiguity Principles. In The Cambridge Handbook of Multimedia Learning; Mayer, R.E., Ed.; Cambridge University Press: Cambridge, UK, 2014; pp. 279–315. [Google Scholar]
- Mayer, R.E. Applying the science of learning: Evidence-based principles for the design of multimedia instruction. Am. Psychol. 2008, 63, 760–769. [Google Scholar] [CrossRef]
- Rusanganwa, J.A. Developing a multimedia instrument for technical vocabulary learning: A case of EFL undergraduate physics education. Comput. Assist. Lang. Learn. 2015, 28, 97–111. [Google Scholar] [CrossRef]
- Koscianski, A.; Zanotto, D.D.C.F. A Design Model for Educational Multimedia Software. Creat. Educ. 2014, 5, 2003–2016. [Google Scholar] [CrossRef] [Green Version]
- Park, I.; Hannafin, M. Empirically-based guidelines for the design of interactive multimedia. Educ. Technol. Res. Dev. 1993, 41, 63–85. [Google Scholar] [CrossRef]
- Herrington, J.; Oliver, R. An Instructional Design Framework for Authentic Learning Environments. Educ. Technol. Res. Dev. 2000, 48, 23–48. [Google Scholar] [CrossRef]
- Schmitt, N. Review article: Instructed second language vocabulary learning. Lang. Teach. Res. 2008, 12, 329–363. [Google Scholar] [CrossRef]
- Schmitt, N. Researching Vocabulary: A Vocabulary Research Manual; Palgrave Press: Basingstoke, UK, 2010. [Google Scholar]
- Huang, B.G.; Yang, J.C. A Multiplayer Online Role-Playing Game for Incidental Vocabulary Learning. In Proceedings of the 20th International Conference on Computers in Education (ICCE 2012), Singapore, 26–30 November 2012. [Google Scholar]
- Nation, P.; Meara, P. Vocabulary, in an Introduction to Applied Linguistics; Schmitt, N., Ed.; Arnold: London, UK, 2002; pp. 35–54. [Google Scholar]
- Ansarin, A.A.; Khojasteh, M.R.B. Retention in Meaning-Based Vocabulary Instruction. SKY J. Linguist. 2013, 26, 7–19. [Google Scholar]
- Hulstijn, J.H. Psycholinguistic perspectives on second language acquisition. In The International Handbook on English Language Teaching; Cummings, J., Davison, C., Eds.; Springer: Norwell, MA, USA, 2006. [Google Scholar]
- Sadiku, A. The Role of Subtitled Movies on Students’ Vocabulary Development. Int. J. Sci. Basic Appl. Res. 2018, 42, 212–221. [Google Scholar]
- Kirschner, P.A.; Neelen, M. Double-Barrelled Learning for Young and Old. 3-Start Learning Experiences 2017. Available online: https://3starlearningexperiences.wordpress.com (accessed on 26 May 2019).
- Baddeley, A.D. Is working memory still working? Eur. Psychol. 2002, 7, 85–97. [Google Scholar] [CrossRef]
- Mayer, R.; Sims, V. For Whom Is a Picture Worth a Thousand Words? Extensions of a Dual-Coding Theory of Multimedia Learning. J. Educ. Psychol. 1994, 86, 389–401. [Google Scholar] [CrossRef]
- Talavan, N. Subtitling as a Task and Subtitles as Support: Pedagogical Applications. In New Insights into Audiovisual Translation and Media Accessibility; Díaz-Cintas, A.M.J., Neves, J., Eds.; Rodopi: Brill, Amsterdam, 2010; pp. 285–299. [Google Scholar]
- Bates, A.W. Using Video in Higher Education. Paper on Broadcasting. In 4th Science Fair; Institute of Educational Technology: Vienna, Austria, 1985; p. 15. [Google Scholar]
- Lertola, J. The effect of the subtitling task on vocabulary learning. Transl. Res. Proj. 2012, 4, 61–70. [Google Scholar]
- Price, K. Closed-Captioned TV: An Untapped Resource. Matsol Newsl. 1983, 12, 1–8. [Google Scholar]
- Rokni, S.J.A.; Ataee, A.J. Movies in EFL Classrooms: With or wihtour subtitles. Daw. J. 2014, 3, 715–726. [Google Scholar]
- Baltova, I. Multisensory Language Teaching in a Multidimensional Curriculum: The Use of Authentic Bimodal Video in Core French. Can. Mod. Lang. Rev. 1999, 56, 31–48. [Google Scholar] [CrossRef]
- Borrás, I.; Lafayette, R.C. Effects of Multimedia Courseware Subtitling on the Speaking Performance of College Students of French. Mod. Lang. J. 1994, 78, 61–75. [Google Scholar] [CrossRef]
- Koskinen, P.S.; Wilson, R.M.; Jensema, C.J. Closed-Captioned Television: A New Tool for Reading Instruction. Read. World 1985, 24, 1–7. [Google Scholar] [CrossRef]
- Garza, T.J. Evaluating the Use of Captioned Video Materials in Advanced Foreign Language Learning. Foreign Lang. Ann. 1991, 24, 239–258. [Google Scholar] [CrossRef]
- Huang, H.-C.; Eskey, D.E. The effects of closed-captioned television on the listening comprehension of intermediate English as a second language (ESL) students. J. Educ. Technol. Syst. 2000, 28, 75–96. [Google Scholar] [CrossRef]
- Markham, P.; Peter, L. The Influence of English Language and Spanish Language Captions on Foreign Language Listening/Reading Comprehension. J. Educ. Technol. Syst. 2003, 31, 331–341. [Google Scholar] [CrossRef]
- Birulés-Muntané, J.; Soto-Faraco, S. Watching subtitled films can help learning foreign languages. PLoS ONE 2016, 11, e0158409. [Google Scholar] [CrossRef]
- Bravo, M.D.C.C. Putting the reader in the picture. Screen translation and foreign-language learning. In Departament de Filologia Anglogermànica; Universitat Rovira i Virgili: Tarragona, Spain, 2009; p. 246. [Google Scholar]
- Bird, S.A.; Williams, J.N. The effect of bimodal input on implicit and explicit memory: An investigation into the benefits of within-language subtitling. Appl. Psycholinguist. 2002, 23, 509–533. [Google Scholar] [CrossRef] [Green Version]
- Talavan, N. Learning Vocabulary through Authentic Video and Subtitles. Tesol Spain Newsl. 2007, 31, 5–8. [Google Scholar]
- Bellalem, F.; Neddar, B.A.; Bouagada, H.; Benelhadj Djelloul, D. The Use of Subtitled Movies for Vocabulary Acquisition in ESP Settings: Insights from an Experimental Study in Algeria. Arab World Engl. J. 2018, 9, 14. [Google Scholar]
- Faqe, C.K. The Effectiveness of English Movie Subtitles in Vocabulary Learning Among Iraqi Kurdistan Efl Learners: Soran University Efl Learners as an Example. Int. J. Curr. Adv. Res. 2017, 6, 2590–2594. [Google Scholar]
- Koolstra, C.M.; Beentjes, J.W.J. Children’s Vocabulary Acquisition in a Foreign Language through Watching Subtitled Television Programs at Home. Educ. Technol. Res. Dev. 1999, 47, 51–60. [Google Scholar] [CrossRef]
- Yildiz, T.A. The Effect of Videos with Subtitles on Vocabulary Learning of EFL Learners. Int. J. Humanit. Soc. Sci. 2017, 7, 125–130. [Google Scholar]
- Gorjian, B. The Effect of Movie Subtitling on Incidental Vocabulary Learning Among Efl Learners. Int. J. Asian Soc. Sci. 2014, 4, 1013–1026. [Google Scholar]
- Etemadi, A. Effects of Bimodal Subtitling of English Movies on Content Comprehension and Vocabulary Recognition. Int. J. Engl. Linguist. 2012, 2, 239. [Google Scholar] [CrossRef] [Green Version]
- Perego, E.; Del Missier, F.; Porta, M.; Mosconi, M. The Cognitive Effectiveness of Subtitle Processing. Media Psychol. 2010, 13, 243–272. [Google Scholar] [CrossRef]
- Park, D.C.; Huang, C.-M. Culture Wires the Brain: A Cognitive Neuroscience Perspective. Perspect. Psychol. Sci. 2010, 5, 391–400. [Google Scholar] [CrossRef]
- Aidinlou, N.A.; Moradinejad, A. Short-Term and Long-Term Retention of Vocabulary through Authentic Subtitled Videos. Adv. Lang. Lit. Stud. 2016, 7, 14–22. [Google Scholar]
- Hsien, C.-H. The Study of DVD Subtitle on EFL Students’ English Listening Comprehension and Vocabulary Acquisition. Int. J. Humanit. Soc. Stud. 2015, 3, 131–138. [Google Scholar]
- Hirose, K.; Kamei, S. Effects of English Captions in Relation to Learner Proficiency Level and Type of Information. Lang. Lab. 1993, 30, 1–16. [Google Scholar]
- Zarei, A.A. The Effect of Bimodal, Standard, and Reversed Subtitling on L2 Vocabulary Recognition and Recall. Pazhuhesh E Zabanha Ye Khareji 2009, 49, 65–85. [Google Scholar]
- Holobow, N.E.; Lambert, W.E.; Sayegh, L. Pairing Script and Dialogue: Combinations That Show Promise for Second or Foreign Language Learning. Lang. Learn. 1984, 34, 59–74. [Google Scholar] [CrossRef]
- Neuman, S.B.; Koskinen, P. Captioned television as comprehensible input: Effects of incidental word learning from context for language minority students. Read. Res. Q. 1992, 27, 94–106. [Google Scholar] [CrossRef]
- Danan, M. Reversed subtitling and dual coding theory: New directions for foreign language instruction. Lang. Learn. 1992, 42, 497–527. [Google Scholar] [CrossRef]
- Goldman, M.; Goldman, S. Reading with closed-captioned TV. J. Read. 1988, 31, 458–461. [Google Scholar]
- Fazilatfar, A.M.; Ghorbani, S.; Samavarchi, L. The effect of standard and reversed subtitling versus no subtitling mode on l2 vocabulary learning. J. Teach. Lang. Ski. JTLS 2011, 3, 43–64. [Google Scholar]
- Yoshida, H.; Uematsu, S.; Yoshida, S.; Takeuchi, O. Modalities of Subtitling and Foreign Language Learning. Lang. Lab. Assoc. J. Kansai 1998, 7, 49–63. [Google Scholar]
- Lee, D. This Chrome Extension Lets you Learn a New Language by Watching Netflix. The Verge 2019. Available online: https://www.theverge.com/2019/2/12/18220289/language-learning-netflix-chrome-extension-two-subtitles (accessed on 26 May 2019).
- Kovacs, G.; Miller, R.C. Smart Subtitles for Vocabulary Learning, in CHI’14 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems; AMC Digital Library: Toronto, ON, Canada, 2014; pp. 853–862. [Google Scholar]
- Canning-Wilson, C. Practical Aspects of Using Video in the Foreign Language Classroom. Internet TESL J. 2000, 6, 36. [Google Scholar]
- Sokoli, S. Audiovisual Translation Scenarios; Advanced Translation Research Center (ATRC): Copenhagen, Denmark, 2006. [Google Scholar]
- Zanon, N.T. Using Subtitles to Enhance Foreign Language Learning. Porta Ling. 2006, 6, 41–52. [Google Scholar]
- Vanderplank, R. ‘Effects of’ and ‘effects with’ captions: How exactly does watching a TV programme with same-language subtitles make a difference to language learners? Lang. Teach. 2013, 49, 235–250. [Google Scholar] [CrossRef]
Principle | Category | Description |
---|---|---|
Coherence effect | Reduction of extraneous load | The material presented to the student should avoid including information that is not part of the contents being studied. |
Signaling effect | A presentation should give clues to students to guide their attention towards the main points, by emphasizing or repeating information. | |
Redundancy effect | The narrated text should not be accompanied by written text since this can distract students from observing pictorial information. | |
Spatial Contiguity effect | The close placement between texts and pictures reduces the effort of students to inspect the material and favors learning. | |
Temporal Contiguity effect | The presentation of verbal and non-verbal pieces of information should occur simultaneously instead of sequentially. | |
Segmenting effect | Management of essential processing | The presentation of information should use separable units whenever possible, instead of fusing several concepts into complex texts and pictures. |
Pre-Training effect | The introductory material at the beginning of a presentation may reduce the cognitive load associated with complex information that forms the core of the learning material. | |
Modality effect | When pictorial and verbal information is combined, the use of narrated (spoken) text is preferable over written text. | |
Multimedia effect | Fostering generative processing | Explanations with text and pictures are more efficient than those presenting information using only one of these possibilities. |
Personalization effect | The presentation of material should preferably make students feel part of the narration, for example, using the second person instead of the third person conjugation. |
© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Kanellopoulou, C.; Kermanidis, K.L.; Giannakoulopoulos, A. The Dual-Coding and Multimedia Learning Theories: Film Subtitles as a Vocabulary Teaching Tool. Educ. Sci. 2019, 9, 210. https://doi.org/10.3390/educsci9030210
Kanellopoulou C, Kermanidis KL, Giannakoulopoulos A. The Dual-Coding and Multimedia Learning Theories: Film Subtitles as a Vocabulary Teaching Tool. Education Sciences. 2019; 9(3):210. https://doi.org/10.3390/educsci9030210
Chicago/Turabian StyleKanellopoulou, Catherine, Katia Lida Kermanidis, and Andreas Giannakoulopoulos. 2019. "The Dual-Coding and Multimedia Learning Theories: Film Subtitles as a Vocabulary Teaching Tool" Education Sciences 9, no. 3: 210. https://doi.org/10.3390/educsci9030210
APA StyleKanellopoulou, C., Kermanidis, K. L., & Giannakoulopoulos, A. (2019). The Dual-Coding and Multimedia Learning Theories: Film Subtitles as a Vocabulary Teaching Tool. Education Sciences, 9(3), 210. https://doi.org/10.3390/educsci9030210