The Dual-Coding and Multimedia Learning Theories: Film Subtitles as a Vocabulary Teaching Tool
Department of Audio and Visual Arts, Ionian University, 49100 Corfu, Greece
Department of Informatics, Ionian University, 49100 Corfu, Greece
Authors to whom correspondence should be addressed.
Educ. Sci. 2019, 9(3), 210; https://doi.org/10.3390/educsci9030210
Received: 9 June 2019 / Revised: 3 August 2019 / Accepted: 6 August 2019 / Published: 8 August 2019
(This article belongs to the Special Issue Technology-enhanced Learning in Media Studies)
The use of multimedia has often been suggested as a teaching tool in foreign language teaching and learning. In foreign language education, exciting new multimedia applications have appeared over the last years, especially for young learners, but many of these do not seem to produce the desired effect in language development. This article looks into the theories of dual-coding (DCT) and multimedia learning (CTML) as the theoretical basis for the development of more effective digital tools with the use of films and subtitling. Bilingual dual-coding is also presented as a means of indirect access from one language to another and the different types of subtitling are explored regarding their effectiveness, especially in the field of short-term and long-term vocabulary recall and development. Finally, the article looks into some new alternative audiovisual tools that actively engage learners with films and subtitling, tailored towards vocabulary learning.
Over the last decades, technological advances have reformed the modern world. Along those lines, educational technologies promised to revolutionize the learning environment with new techniques introducing a wide range of tools, from computer-based systems to using films in the classroom. In 1922, Thomas Edison himself said that “the motion picture is destined to revolutionize our educational system and that in a few years it will supplant ...the use of textbooks” . However, over the following years, the use of films in the classroom was rare, if any at all. Over the years, most computer-assisted tools did not produce the desired effects teachers had dreamt of . In Greece, specifically, despite the fact that technology has been introduced in most schools by now, teachers are still skeptical about its regular use [2,3]. Computers in the Greek state schools are mostly used by teachers for low-level academic goals, such as searching the internet for information or preparing teaching plans, rather than as cognitive tools , mainly because of the lack of effective cognitive tools that teachers are confident in using in the classroom . It seems that new educational technologies cannot achieve extraordinary results on their own and it is necessary to develop new tools for the language classroom that stem from research-based theories of how students learn . Fortunately, developments in cognitive psychology offer a solid starting point for composing such theories and researchers should focus on developing technological tools which incorporate activities that facilitate cognitive processes . For the development of such tools, it is essential to determine the principles of how technology, and multimedia specifically, can be used to promote student learning in line with those theories.
2. The Dual-Coding Theory
The dual-coding theory (DCT) is a general cognition theory that has been directly applied to literacy and language learning. The theory was proposed by Allan Paivio in 1971 and explains the powerful effects of mental imagery on the mind and memory. In his theory, Paivio originally accounted for verbal and nonverbal influences in memory, but researchers soon started applying it in other areas of cognition [8,9,10]. According to this theory, a person can learn new materials using verbal associations or visual imagery but the combination of both is more successful in learning . The dual-coding theory states that the brain uses both visual and verbal information to represent information , but this information is processed differently along two distinct channels in the human mind, creating different representations for information that each channel processes. The existing two coding systems are the verbal system and the nonverbal/visual system. These two coding systems interact, and this interaction results in better recall [10,13]. The verbal system stores linguistic information/units (such as text, sound, or even motor experience such as sign language) in sequential units called “logogens.” The non-verbal/visual system processes visual information/units (such as symbols, pictures, or videos) and keeps them in units called “imagens.” The terms “logogen” and “imagen” refer respectively to representational units of verbal and nonverbal information that produce already existing mental words and images and can function unconsciously to improve cognitive performance . According to Paivio  and Clark and Paivio , there are three different processing levels that take place within or between verbal and nonverbal/visual systems: representational, referential, and associative processing. The two systems are linked together through referential connections (Figure 1).
Representational processing is the activation of a verbal or visual representation by a stimulus which may be a word or a picture; —that is, a word will activate a verbal representation, and a picture will activate a visual representation respectively. Processing begins with the direct activation of imagens and logogens from stimulus patterns, a kind of “template matching process involved in stimulus recognition” .
Referential processing is the activation of either system by the other one, that is, images or objects activate words, and vice versa. Referential processing is the special kind of associative mutualism that involves cross-system activation of logogens by activated imagens or vice versa. Referential associations between logogens and imagens permit objects to be named and names to activate images that represent world knowledge. Referential processing occurs in a wide range of domains including language acquisition, education, technology, and creative thinking . It occurs early in language acquisition when children begin to learn names of objects, and soon extends to learning grammar, which initially depends on experiencing correlated relations between language and object structures .
Associative processing refers to the activation of additional information in representational or referential systems. At this level, the associative connections between words (i.e., linguistic units) in the verbal system and images in the visual system are activated.
Verbal and visual codes, which correspond to these verbal and non-verbal representations respectively, are used to organize incoming information that can be acted upon, stored, and retrieved for subsequent use. Both visual and verbal codes can be used when recalling information .
What is also notable is that DCT applies to both verbal and nonverbal knowledge even when language is the object of study  where what is learned continually develops as a result of new experiences. It also offers some outstanding insight into vocabulary learning. It purports that nonverbal and verbal codes, because they are functionally independent, may have increasingly positive effects on recall. When foreign language learners associate new words with sounds or images in both modes at the same time, they are likely to learn the words better and retain them more easily than when they use only one mode . According to Reiber  “use of pictures and illustrations associated with unknown words are effective instructional devices that are superior to words alone for memory tasks and will help L2) learners remember the words sooner and retain them longer” (p. 141). With the term L2, we refer to any language learners have acquired apart from their mother tongue (L1). The effectiveness of the dual association of verbal and visual modes might stem from the fact that “when one memory trace is lost, the other remains and is accessible” probably because “pictures are more likely to activate the image-to-word referential connections” . Consequently, the dual-coding theory is best applied to L2 vocabulary learning environments where text, audio, and video can be used. In other words, multimedia environments where learners are exposed to multiple modalities of presentation including printed texts, sounds, pictures, and/or videos make for a language-learning environment with a significant effect on vocabulary learning [13,20].
Undoubtedly, there are certain limitations to the effectiveness of DCT. According to the concreteness effect, abstract ideas are difficult to image so they are least likely to be dually coded . “Concrete language is remembered better than abstract language in a wide variety of tasks”  because concrete words can be both processed as images as well as words, whereas, most of the times, abstract material can only be verbally processed. However, despite the cases where associations between words and images cannot be formed—hence making the recall of items significantly more difficult—DCT is still valid over a wide range of cases and empirical studies indicate it can be used to improve memory . Finally, one possibility that dual-coding theory does not consider is that cognition might be assisted by something other than words and images. More research needs to be done in this domain to determine if words and images are the only way we remember items since the theory might not hold true if another form of codes were discovered .
3. The Bilingual Dual-Coding Theory
While Paivio’s  original theory refers to one language, Paivio—Desrochers’s  work extends the theory to two or more languages. In the bilingual DCT, there are now two independent verbal systems which are linked to a common imagery system. In this case, researchers have developed several assumptions regarding how independent and/or interconnected these three systems are. Obviously, studies using the bilingual dual-coding need to focus both on the imagery–verbal connections, as well as on the L1–L2 connections.
Assumptions mostly relevant to language learning are as follows :
- The image systems function independently from the two verbal systems, which is what the original monolingual dual-coding theory supports.
- The three systems are interconnected at the referential level and as a result, the imagery system can influence the verbal system and vice versa.
- The two verbal systems are referentially connected to the imagery system, which allows them to be partly shared and partly independent, with two major implications (a) there is a variable degree of referential overlap between languages, depending on the specific pair of languages each time and (b) the fact that the imagery system becomes a means of indirect access from one language to the other.
- Bilinguals have access to two verbal systems which are independent, however, also interconnected to a certain degree, which means that a change in the language input may result in code-switching.
What is of the essence here is that when students call upon lexical items that have already been learned in one verbal system, when coding them into another verbal system, this results in enriching those lexical items and giving them new meaning. This is similar to the traditional process of vocabulary acquisition, but now the words are made into translation equivalents. Figure 2 presents a simplified picture of the process:
As a theory, bilingual DCT is a relatively young cognition theory as compared to its origin, and more research is required to test its validity. More studies on the L1–L2 link are necessary to investigate the connections between the two verbal systems. One of the first obvious constraints of this theory is when what an L2 learner has learned in L1 is not available in his L2, or vice versa (when a learner is attempting to learn something in L2 which is not available in L1). In which cases, the theory cannot apply since there is no L1–L2 link. But when the link does exist, then the respective effect can be positive (help) or negative (hindrance). It rests upon research to discover the conditions or factors which lead to either of the two outcomes.
4. The Cognitive Theory of Multimedia Learning
The cognitive theory of multimedia learning (CTML) looks into how people learn from multimedia presentations [25,26,27,28]. This theory is based on Wittrock’s  generative theory and Paivio’s  dual-coding theory. However, it goes beyond these two theories to give the learner the role of “knowledge constructor who actively selects and connects pieces of visual and verbal knowledge” . The theory supports the idea that learners build meaningful connections between words and pictures and process them actively in long-term memory [27,28]. The main difference between CTML and DCT is that the learner assumes a more active role in knowledge construction within the CTML framework.
Mayer’s [25,27] CTML tries to explain the process of learning based on the following three assumptions:
- Dual channels: As in dual-coding theory, there are two separate channels that process information from sensory memory, namely the auditory/verbal and the visual/pictorial channel. The auditory/verbal channel processes information in the form of spoken words, narrations, or sounds via the ears; and the visual/pictorial channel processes information in the form of pictures, graphs, videos/animation, etc., received through the eyes.
- Limited capacity: As in the cognitive load theory, the information that learners can process in their working memory, in each channel at any one time, is not unlimited; therefore, when a learning task demands more cognitive processing than the capacity of a learner’s processing system, then cognitive overload occurs. Sounds are stored in a verbal mental model and images are stored in a visual mental model, as Mayer  explains, “when an illustration or animation is presented, the learner is able to hold only a few images in the visual channel of working memory at any one time, reflecting portions of the presented material” (p. 49). The same is also true when learners are presented with a narration in that they can only hold a few words in the verbal channel of working memory at any one time .
- Active processing: learning based on the CTML is an active process where the learners select and organize words and images, and then integrate them together with prior knowledge drawn from their long-term memory which is brought back into their short-term memory to facilitate processing. The integration of new information into the learner’s existing knowledge may include all or some of the following cognitive processes:
- selecting relevant words for processing in verbal working memory,
- selecting relevant images for processing in visual working memory,
- organizing selected words into a verbal model,
- organizing selected images into a pictorial model, and
- integrating the verbal and pictorial representations with each other and with relevant prior knowledge activated from long-term memory. 
Figure 3 illustrates how learners learn L2 vocabulary in a multimedia-based setting.
According to Figure 3, words and pictures, as the two multimedia presentation modes, come to the sensory memory from the outside world through the ears and eyes. Written words and pictures are kept in the visual sensory memory (the green box), and spoken words and sounds are held in the auditory sensory memory (the blue box) for a short while. Learners select materials by paying attention to certain words and images. When the relevant materials have been selected, structural relations are built among the elements in the working memory. The information is then transferred further into the working memory where materials are temporarily held and manipulated. Working memory, therefore, consists of two sides: the left side, which represents “the raw material” such as “visual images of pictures and sound images of words”; and the right side, which represents “the knowledge constructed in working memory” such as “pictorial and verbal models and the links between them” . The last box on the right side of Figure 3 shows long-term memory, which can hold large amounts of information over long periods of time. However, in order for the materials to stay in long-term memory, they should be actively moved back and forth from long-term memory to working memory [25,27]. In this way, knowledge in the long-term memory is activated and brought into working memory if there is a connection between any new materials and the learners’ prior knowledge .
Two concepts that are central to the cognitive theory of multimedia learning are the multimedia principle and the contiguity principle.
The multimedia principle: The multimedia principle suggests, just like the DCT, that learners can learn more effectively if they are presented with words and pictures rather than words alone . However, the multimedia principle goes further than DCT as it is not limited to words and pictures alone but refers to a broader term encompassing different forms of visual and verbal representations when presented together [30,31]. Visual components of the multimedia instruction include illustrations, pictures, graphs and charts, photographs, and especially videos and animations . The verbal components are texts, spoken words/sounds, and narrations . The use of both words and pictures allows the brain to process more information in working memory  and can be recalled from long-term memory when required.
The temporal contiguity principle. This principle describes the potential extraneous overload of multimedia materials. It suggests that learners can learn more deeply from learning tools when the text, audio, pictures, and video/animation are presented simultaneously rather than successively or sequentially . According to CTML, “learners must have corresponding words and images in working memory at the same time in order to make connections between them” , meaning simultaneous presentation is expected to enhance learning more than successive and/or separate presentation .
Apart from the two principles explained above, there are more concepts which have been further developed into principles of CTML and that should be used when designing instructional multimedia.
A comprehensive list of those principles is presented in Table 1, according to Mayer’s categorization . The principles listed in this table have been extensively tested  and have been used to guide the design of educational materials [37,38] and to predict learning results .
The theoretical constructs of dual-coding (DCT), bilingual DCT, and cognitive theory of multimedia learning (CTML) imply that learning in a multimedia learning environment is best facilitated when the new information is presented through both verbal and visual representational modes rather than just a single mode [9,25]. For optimal learning results, the chances of overloading a learners’ cognitive system should be minimized by displaying both verbal and visual materials simultaneously rather than successively . In view of the above, research into L2 vocabulary learning has recently turned into films as an authentic source of multimedia materials and the use of subtitles as a teaching tool.
5. Teaching Vocabulary through Multimedia; Film Subtitles as a Tool
Vocabulary knowledge plays a prominent role in learning a foreign language . Over the years, there have been debates on the most effective ways for developing learners’ vocabulary knowledge and several theories point us to one or another direction for best results. While many researchers believe incidental learning is slow and random, they also support that it can supplement the “contextual” types of word knowledge . In their study, Huang and Yang  stressed the significance of incidental vocabulary learning as the main source of learner-centered vocabulary acquisition in authentic situations. On the other hand, there are several empirical studies which favor intentional acquisition over incidental acquisition , even though, when it comes to determining the effectiveness of specific methods within the context of intentional vocabulary learning, limited research exists .
Despite the debate over the issue of incidental vs. intentional vocabulary acquisition, one thing all researchers agree on is that a key element of fluent language use is the ability of the user to automatically recognize words in listening and/or reading, as well as being able to easily recall words when speaking or writing. Words cannot be used if they are not known; therefore, the acquisition of a large vocabulary and the automatic retrieval of this vocabulary are key elements in learning a foreign language . Another important issue is that vocabulary tends to be forgotten if it is not used through the right methods that will provide learners with language inputs in a genuine target language environment which leads to its acquisition. In this regard, the increasing access to different multimedia and technology resources facilitates spontaneous vocabulary acquisition for the contemporary age learners. In particular, films with subtitles can be a valuable tool in bringing students closer to authentic real-life vocabulary.
Ever since Karen Price’s ground-breaking work in 1983, consequent studies have found several benefits of using subtitled movies by confirming that subtitles indeed improve vocabulary learning . Films are one of the most prominent examples of multimedia tools for the foreign language classroom, especially with the addition of subtitles or captions. Images leave double and therefore stronger traces in our brain because written and spoken words only get coded once, whereas images of words get coded twice, first visually, then verbally . Using captions or subtitles enables the learner to access more working memory capacity  and, additionally, since the materials (captions or subtitles and audio text) appear almost simultaneously, there is reduced risk of overloading the learners’ cognitive capacity . Therefore, captions or subtitles are more likely to contribute to better word recall and incidental vocabulary acquisition .
Moreover, films can be beneficial even to weaker or more elementary students because despite the fact that “authentic videos can be challenging for the average student; the language can be understood with the help of subtitles, either by having them already visible on the screen or by creating them” . Αs audiovisual materials, films can provide engaging context involving action, not just still images, and several authors have described the specific features that make video a powerful medium for learning, such as “abstracting information, narrative visualization, recognition, and identification by the student” . Films as authentic audiovisual materials are not form-focused and they promote visual information in a real context, natural language at normal conversational speed, and different accents from various ages, genders, and socio-cultural backgrounds . Furthermore, in one of the first studies about the effectiveness of subtitling, Price  as cited in Rokni and Ataee  concluded that subtitling resulted in comprehending a great deal of cultural background, linguistic aspects, and social behaviors.
The potential of subtitles as a teaching tool has become apparent over the years; however, more investigation is necessary so that we can determine what types of subtitling are more suitable for each of the learners’ needs and skills development, as it seems that different types of subtitling can enhance different areas of learning a foreign language. Baltova  supports that subtitled audiovisual materials enhance vocabulary learning and improve content comprehension even when learners are relatively inexperienced. According to some researchers, subtitles may bridge the gap between reading and listening skills [55,56,57,58,59,60]. Bravo  found subtitles beneficial with regards to reading comprehension and other scholars support that subtitling may enhance vocabulary recall [62,63].
In film subtitling, the most common types are as follows:
- L2 audio with L1 subtitles (standard subtitling),
- L2 audio with L2 subtitles (captions or bimodal subtitling),
- L2 audio without subtitles (no subtitling),
- L1 audio with L2 subtitles (reserved subtitles).
Research is more extensive in the first three, but more empirical results are being presented over the last years on the last one. We will mainly focus on the first three and their effectiveness in foreign language vocabulary learning.
With respect to vocabulary, research in the field of foreign language acquisition indicates that exposure to subtitled films as comprehensible input generally enhances the acquisition of vocabulary [45,64,65,66,67,68] be it incidental or intentional; and also taking into consideration other factors, such as repetition of viewings , the quality of subtitles segmentation , or even the students’ own cultural backgrounds .
In experiments comparing standard subtitling with no subtitling, several researchers have found that standard subtitles are beneficial for vocabulary acquisition [64,72,73]. In the experiment of Aidinlou and Moradinejad, the control group (no subtitles) did better in short-term retention, but the experimental group outperformed the control group in long-term retention, which indicates the superiority of learning vocabulary through watching subtitled videos in long-term retention of vocabulary. In an empirical study in Algeria , findings indicate that using subtitled films clearly enhance the learning of new vocabulary. In the case study of Rokni and Ataee  subtitles led to the students’ improved speaking ability, which presupposes an improvement in vocabulary retrieval. Finally, Hsien’s study  of 107 students of English as a foreign language (EFL) in Taiwan looked into three types of subtitling: standard, bimodal, and reversed, and concluded that using standard and bimodal subtitling as well as making use of different multimedia have a positive effect on students’ listening and vocabulary abilities; with standard subtitling being more effective than bimodal subtitling.
Taking a closer look at studies on captions (intralingual subtitles), Hirose and Kamei , as well as Zanon  contend that text in the form of subtitles draw learners’ attention to a speech that would otherwise be lost and, thus, limits the level of insecurity and anxiety of learners when watching a film in terms of unknown vocabulary. Baltova , based on her empirical study, supports captions as a means of enhancing L2 learners’ understanding of vocabulary and authentic texts, while Zarei  indicates captions as the most effective type of subtitling for vocabulary recall. Further support also comes from Sadiku’s recent work  whose results indicate that captions facilitate better vocabulary acquisition for short- and long-term memory, specifically when they are shown to students in an intentional learning environment, for example, with the assistance and instructions from a teacher. More research findings support captions as a factor that facilitates comprehension and memory  and more specifically vocabulary recall [62,77,78]. In the same vein, Goldman and Goldman  favor captions as especially beneficial for slow readers.
As for reversed subtitling, there has been some research recently that indicates this type as more beneficial for improvement in L2 general comprehension or listening skills [59,76,78,80,81]. Danan (1992), specifically, attributed the success of reversed subtitling to translation facilitating language encoding. What was most interesting in her work was the fact that although her study involved two languages (English and French), the focus was not on the L1–L2 connections, but on how the features of the two languages might enhance learning. So far, there are not many empirical studies that compare captions and reversed subtitles while focusing on short-term or long-term vocabulary retention.
6. Interactive Uses of Subtitling as a Multimedia Tool
In recent years, there has been increasing interest in the field of multimedia and audiovisual technology in education. Technology media can assist learners in acquiring vocabulary incidentally or intentionally , especially using films as a tool. An example is the recent David Wilkinson and Ognjen Apic’s “Language Learning with Netflix”  “which is a Chrome extension that lets you watch shows with two subtitles on at the same time so you can visually pair translations with dialogue and learn some new vocabulary in the process.” The vocabulary-highlighting tool changes the color of words depending on their frequency and can be adjusted to match the viewer’s vocabulary level. There’s also a tool for producing a pop-up dictionary, and when you click on the words you can hear them. Many recent studies report more encouraging results on digital tools, particularly in film subtitling, and their pedagogical benefits. Nevertheless, more research is necessary to determine the types of subtitles that enhance specific cognitive skills. From the studies presented in this paper, it seems that standard subtitles tend to facilitate general comprehension , reversed subtitles improve language encoding , whereas captions enhance vocabulary recall and long-term retention . However, it should be questioned whether this is the only use that subtitles may have in the foreign language classroom. Attempts have been made by scholars to overcome the shortcoming of students’ passive film-watching; be it with or without subtitles, and engage them more actively with subtitling. Canning-Wilson  suggests that “images contextualized in video […] can help to reinforce language learning, provided the learner can see immediate meaning in terms of vocabulary recognition” and one of the best ways of “assuring this immediacy of meaning is with the help of subtitles”, which is may be further enhanced when subtitles are not used passively for example when students create the subtitles themselves. A case in point is ‘Smart Subtitles’ of Kovacs and Miller  which are interactive subtitles aiming at vocabulary learning. ‘Smart Subtitles’ have appealing features for students, such as vocabulary definitions on hover, and dialog-based video navigation. Another example of a more interactive use of subtitles is the SvL project (Learning via Subtitles)  which is a tool developed by the Laboratory of Educational Material of the Hellenic Open University, simulating a professional activity, that of film subtitling, mainly for students in translation courses. Multimedia is the core of the activity in this case and provides hands-on experience and result. The software seems a promising tool where films/videos are combined with subtitles in a more engaging activity. Finally, Zanon  suggested a multimedia platform where film clips are loaded by the teacher, who also decides on various parameters and degrees of support (e.g., whether students are given the corresponding script transcripts, vocabulary assistance, etc.) and subtitle type (standard, bimodal, and/or reversed). Students actively engage in subtitling the clips; an activity which could easily be applied in a classroom environment or in any distance learning program.
Acquiring new vocabulary is crucial in language learning, especially in improving students’ communication skills. Despite the fact that researchers agree on the importance of vocabulary acquisition in second language acquisition (SLA), the question of long-term vocabulary development has still not been fully researched . Cognitive theories, such as the dual-coding theory and the cognitive theory of multimedia learning offer a sound basis for the development of educational multimedia tools; but further research is necessary to continue exploring the issue of vocabulary acquisition with the use of subtitling. It seems that experimental findings point to different types of subtitling enhancing specific areas of students’ skills and vocabulary recall, however, on many occasions the findings of those studies are contradictory, or too general, and serve more as indicators rather than evidence. This may be because most studies involve a small number of participants (<50) and empirical methodologies and participants’ characteristics are so different that they do not offer comparable groups among the studies. Broadly speaking, however, findings so far indicate that standard subtitles could enhance general comprehension and short-term vocabulary recall and bimodal subtitles could improve long-term vocabulary acquisition and listening skills, whereas reversed subtitles could be beneficial to general comprehension and language encoding. Finally, what should also be stressed is the need for research in activities where subtitles are not passively used, but where learners are actively involved with them. Audiovisual digital technology offers a vast spectrum of possibilities in our pursuit of more sophisticated tools involving subtitles as an engaging medium for language development.
C.K.: writing; K.L.K.: editing; A.G.: supervising
This research received no external funding.
Conflicts of Interest
The authors declare no conflict of interest.
- Cuban, L. Teachers and Machines: The Classroom Use of Technology Since 1920; Teachers College Press: New York, NY, USA, 1986. [Google Scholar]
- Jimoyiannis, A. Factors Determining Teachers’ Beliefs and Perceptions of ICT in Education. In Encyclopedia of Information Communication Technology; Palma, A.C.M., Ed.; IGI Global: Hershey, PA, USA, 2008; pp. 321–334. [Google Scholar]
- Βοσνιάδου, Σ. Εισαγωγή στην Ψυχολογία; Gutenberg: Αθήνα, Greece, 2001. [Google Scholar]
- Russell, M.; Bebell, D.; O’Dwyer, L.; O’Connor, K. Examining Teacher Technology Use: Implications for Preservice and Inservice Teacher Preparation. J. Teach. Educ. 2003, 54, 297–310. [Google Scholar] [CrossRef]
- Τάσση, Ό. Οι σχέσεις των εκπαιδευτικών με τις Τεχνολογίες της Πληροφορίας και Επικοινωνιών στο σχολείο. Έρκυνα, Επιθεώρηση Εκπαιδευτικών–Επιστημονικών Θεμάτων 2015, 1, 200–215. [Google Scholar]
- Mayer, R.E.; Moreno, R. A Cognitive Theory of Multimedia Learning: Implications for Design Principles. J. Educ. Psychol. 1998, 91, 358–368. [Google Scholar]
- Alzahrani, S. Linking Multimedia Vocabulary CALL Research to SLA Cognitive Theories. Am. J. Educ. Res. 2017, 5, 821–827. [Google Scholar]
- Paivio, A. Imagery and Verbal Processes; Psychology Press: New York, NY, USA, 1971. [Google Scholar]
- Paivio, A. Mental Representations: A Dual Coding Approach; Oxford University Press: Oxford, UK, 1986; Volume 9. [Google Scholar]
- Paivio, A. Dual Coding Theory: Retrospect and Current Status. Can. J. Psychol. Rev. Can. Psychol. 1991, 45, 255–287. [Google Scholar] [CrossRef]
- Reed, S.K. Cognition: Theories and Applications, 8th ed.; Wadsworth, Cengage Learning: Belmont, CA, USA, 2010. [Google Scholar]
- Sternberg, R.J. Cognitive Psychology, 3rd ed.; Thomson/Wadsworth: Belmont, CA, USA, 2003; p. 596. [Google Scholar]
- Al-Seghayer, K. The effect of multimedia annotation modes on L2 vocabulary acquisition: A comparative study. Lang. Learn. Technol. 2001, 5, 202–232. [Google Scholar]
- Paivio, A. Intelligence; Elsevier: Amsterdam, The Netherlands, 2014; Volume 47, pp. 141–158. [Google Scholar]
- Clark, J.M.; Paivio, A. Dual coding theory and education. Educ. Psychol. Rev. 1991, 3, 149–210. [Google Scholar] [CrossRef]
- Moeser, S.D.; Bregman, A. Imagery and language acquisition 1. J. Verbal Learn. Verbal Behav. 1973, 12, 8. [Google Scholar] [CrossRef]
- Paivio, A.; Begg, I. Psychology of Language; Prentice-Hall: Englewood Cliffs, NJ, USA, 1981. [Google Scholar]
- Rieber, L.P. Computers, Graphics, and Learning; Brown and Benchmark: Madison, WI, USA, 1994. [Google Scholar]
- Lin, C.-C. Learning Action Verbs with Animation. JALT Call J. 2009, 5, 23–40. [Google Scholar]
- Al-Seghayer, K. Celebrating the story of my first contribution to call. Lang. Learn. Technol. 2016, 20, 179–184. [Google Scholar]
- Paivio, A.; Walsh, M.; Bons, T. Concreteness effects on memory: When and why? J. Exp. Psychol. Learn. Mem. Cogn. 1994, 20, 1196–1204. [Google Scholar] [CrossRef]
- Pylyshyn, Z.W. What the mind’s eye tells the mind’s brain: A critique of mental imagery. Psychol. Bull. 1973, 80, 1. [Google Scholar] [CrossRef]
- Paivio, A.; Desrochers, A. A dual-coding approach to bilingual memory. Can. J. Psychol. Rev. Can. Psychol. 1980, 34, 388–399. [Google Scholar] [CrossRef]
- Soh, K.C. Bilingual dual-coding and code-switching: Implications for the L1 in L2 learning. J. Linguist. Lang. Teach. 2010, 1, 271–296. [Google Scholar]
- Mayer, R.E. The Cambridge Handbook of Multimedia Learning, 2nd ed.; Cambridge Handbooks in Psychology; Cambridge University Press: Cambridge, UK, 2014; p. 930. [Google Scholar]
- Mayer, R.E. Multimedia Learning; Cambridge University Press: Cambridge, UK, 2001. [Google Scholar]
- Mayer, R.E. Cognitive Theory of Multimedia Learning. In The Cambridge Handbook of Multimedia Learning; Mayer, R., Ed.; Cambridge University Press: Cambridge, UK, 2005; pp. 31–48. [Google Scholar]
- Mayer, R.E. Multimedia learning: Are we asking the right questions? Educ. Psychol. 1997, 32, 1–19. [Google Scholar] [CrossRef]
- Wittrock, M.C. Learning as a generative process. Educ. Psychol. 1974, 11, 87–95. [Google Scholar] [CrossRef]
- Butcher, K.R. Learning from text with diagrams: Promoting mental model development and inference generation. J. Educ. Psychol. 2006, 98, 182–197. [Google Scholar] [CrossRef]
- Butcher, K.R. The Multimedia Principle. In The Cambridge Handbook of Multimedia Learning, 2nd ed.; Cambridge University Press: Cambridge, UK, 2014; pp. 174–205. [Google Scholar]
- Sweller, J. Implications of Cognitive Load Theory for Multimedia Learning. In The Cambridge Handbook of Multimedia Learning (Cambridge Handbooks in Psychology); Mayer, R., Ed.; Cambridge University Press: Cambridge, UK, 2005; pp. 19–30. [Google Scholar]
- Mayer, R.; Fiorella, L. Principles for Reducing Extraneous Processing in Multimedia Learning: Coherence, Signaling, Redundancy, Spatial Contiguity, and Temporal Contiguity Principles. In The Cambridge Handbook of Multimedia Learning; Mayer, R.E., Ed.; Cambridge University Press: Cambridge, UK, 2014; pp. 279–315. [Google Scholar]
- Mayer, R.E. Applying the science of learning: Evidence-based principles for the design of multimedia instruction. Am. Psychol. 2008, 63, 760–769. [Google Scholar] [CrossRef]
- Rusanganwa, J.A. Developing a multimedia instrument for technical vocabulary learning: A case of EFL undergraduate physics education. Comput. Assist. Lang. Learn. 2015, 28, 97–111. [Google Scholar] [CrossRef]
- Koscianski, A.; Zanotto, D.D.C.F. A Design Model for Educational Multimedia Software. Creat. Educ. 2014, 5, 2003–2016. [Google Scholar] [CrossRef]
- Park, I.; Hannafin, M. Empirically-based guidelines for the design of interactive multimedia. Educ. Technol. Res. Dev. 1993, 41, 63–85. [Google Scholar] [CrossRef]
- Herrington, J.; Oliver, R. An Instructional Design Framework for Authentic Learning Environments. Educ. Technol. Res. Dev. 2000, 48, 23–48. [Google Scholar] [CrossRef]
- Schmitt, N. Review article: Instructed second language vocabulary learning. Lang. Teach. Res. 2008, 12, 329–363. [Google Scholar] [CrossRef]
- Schmitt, N. Researching Vocabulary: A Vocabulary Research Manual; Palgrave Press: Basingstoke, UK, 2010. [Google Scholar]
- Huang, B.G.; Yang, J.C. A Multiplayer Online Role-Playing Game for Incidental Vocabulary Learning. In Proceedings of the 20th International Conference on Computers in Education (ICCE 2012), Singapore, 26–30 November 2012. [Google Scholar]
- Nation, P.; Meara, P. Vocabulary, in an Introduction to Applied Linguistics; Schmitt, N., Ed.; Arnold: London, UK, 2002; pp. 35–54. [Google Scholar]
- Ansarin, A.A.; Khojasteh, M.R.B. Retention in Meaning-Based Vocabulary Instruction. SKY J. Linguist. 2013, 26, 7–19. [Google Scholar]
- Hulstijn, J.H. Psycholinguistic perspectives on second language acquisition. In The International Handbook on English Language Teaching; Cummings, J., Davison, C., Eds.; Springer: Norwell, MA, USA, 2006. [Google Scholar]
- Sadiku, A. The Role of Subtitled Movies on Students’ Vocabulary Development. Int. J. Sci. Basic Appl. Res. 2018, 42, 212–221. [Google Scholar]
- Kirschner, P.A.; Neelen, M. Double-Barrelled Learning for Young and Old. 3-Start Learning Experiences 2017. Available online: https://3starlearningexperiences.wordpress.com (accessed on 26 May 2019).
- Baddeley, A.D. Is working memory still working? Eur. Psychol. 2002, 7, 85–97. [Google Scholar] [CrossRef]
- Mayer, R.; Sims, V. For Whom Is a Picture Worth a Thousand Words? Extensions of a Dual-Coding Theory of Multimedia Learning. J. Educ. Psychol. 1994, 86, 389–401. [Google Scholar] [CrossRef]
- Talavan, N. Subtitling as a Task and Subtitles as Support: Pedagogical Applications. In New Insights into Audiovisual Translation and Media Accessibility; Díaz-Cintas, A.M.J., Neves, J., Eds.; Rodopi: Brill, Amsterdam, 2010; pp. 285–299. [Google Scholar]
- Bates, A.W. Using Video in Higher Education. Paper on Broadcasting. In 4th Science Fair; Institute of Educational Technology: Vienna, Austria, 1985; p. 15. [Google Scholar]
- Lertola, J. The effect of the subtitling task on vocabulary learning. Transl. Res. Proj. 2012, 4, 61–70. [Google Scholar]
- Price, K. Closed-Captioned TV: An Untapped Resource. Matsol Newsl. 1983, 12, 1–8. [Google Scholar]
- Rokni, S.J.A.; Ataee, A.J. Movies in EFL Classrooms: With or wihtour subtitles. Daw. J. 2014, 3, 715–726. [Google Scholar]
- Baltova, I. Multisensory Language Teaching in a Multidimensional Curriculum: The Use of Authentic Bimodal Video in Core French. Can. Mod. Lang. Rev. 1999, 56, 31–48. [Google Scholar] [CrossRef]
- Borrás, I.; Lafayette, R.C. Effects of Multimedia Courseware Subtitling on the Speaking Performance of College Students of French. Mod. Lang. J. 1994, 78, 61–75. [Google Scholar] [CrossRef]
- Koskinen, P.S.; Wilson, R.M.; Jensema, C.J. Closed-Captioned Television: A New Tool for Reading Instruction. Read. World 1985, 24, 1–7. [Google Scholar] [CrossRef]
- Garza, T.J. Evaluating the Use of Captioned Video Materials in Advanced Foreign Language Learning. Foreign Lang. Ann. 1991, 24, 239–258. [Google Scholar] [CrossRef]
- Huang, H.-C.; Eskey, D.E. The effects of closed-captioned television on the listening comprehension of intermediate English as a second language (ESL) students. J. Educ. Technol. Syst. 2000, 28, 75–96. [Google Scholar] [CrossRef]
- Markham, P.; Peter, L. The Influence of English Language and Spanish Language Captions on Foreign Language Listening/Reading Comprehension. J. Educ. Technol. Syst. 2003, 31, 331–341. [Google Scholar] [CrossRef]
- Birulés-Muntané, J.; Soto-Faraco, S. Watching subtitled films can help learning foreign languages. PLoS ONE 2016, 11, e0158409. [Google Scholar] [CrossRef]
- Bravo, M.D.C.C. Putting the reader in the picture. Screen translation and foreign-language learning. In Departament de Filologia Anglogermànica; Universitat Rovira i Virgili: Tarragona, Spain, 2009; p. 246. [Google Scholar]
- Bird, S.A.; Williams, J.N. The effect of bimodal input on implicit and explicit memory: An investigation into the benefits of within-language subtitling. Appl. Psycholinguist. 2002, 23, 509–533. [Google Scholar] [CrossRef]
- Talavan, N. Learning Vocabulary through Authentic Video and Subtitles. Tesol Spain Newsl. 2007, 31, 5–8. [Google Scholar]
- Bellalem, F.; Neddar, B.A.; Bouagada, H.; Benelhadj Djelloul, D. The Use of Subtitled Movies for Vocabulary Acquisition in ESP Settings: Insights from an Experimental Study in Algeria. Arab World Engl. J. 2018, 9, 14. [Google Scholar]
- Faqe, C.K. The Effectiveness of English Movie Subtitles in Vocabulary Learning Among Iraqi Kurdistan Efl Learners: Soran University Efl Learners as an Example. Int. J. Curr. Adv. Res. 2017, 6, 2590–2594. [Google Scholar]
- Koolstra, C.M.; Beentjes, J.W.J. Children’s Vocabulary Acquisition in a Foreign Language through Watching Subtitled Television Programs at Home. Educ. Technol. Res. Dev. 1999, 47, 51–60. [Google Scholar] [CrossRef]
- Yildiz, T.A. The Effect of Videos with Subtitles on Vocabulary Learning of EFL Learners. Int. J. Humanit. Soc. Sci. 2017, 7, 125–130. [Google Scholar]
- Gorjian, B. The Effect of Movie Subtitling on Incidental Vocabulary Learning Among Efl Learners. Int. J. Asian Soc. Sci. 2014, 4, 1013–1026. [Google Scholar]
- Etemadi, A. Effects of Bimodal Subtitling of English Movies on Content Comprehension and Vocabulary Recognition. Int. J. Engl. Linguist. 2012, 2, 239. [Google Scholar] [CrossRef]
- Perego, E.; Del Missier, F.; Porta, M.; Mosconi, M. The Cognitive Effectiveness of Subtitle Processing. Media Psychol. 2010, 13, 243–272. [Google Scholar] [CrossRef]
- Park, D.C.; Huang, C.-M. Culture Wires the Brain: A Cognitive Neuroscience Perspective. Perspect. Psychol. Sci. 2010, 5, 391–400. [Google Scholar] [CrossRef]
- Aidinlou, N.A.; Moradinejad, A. Short-Term and Long-Term Retention of Vocabulary through Authentic Subtitled Videos. Adv. Lang. Lit. Stud. 2016, 7, 14–22. [Google Scholar]
- Hsien, C.-H. The Study of DVD Subtitle on EFL Students’ English Listening Comprehension and Vocabulary Acquisition. Int. J. Humanit. Soc. Stud. 2015, 3, 131–138. [Google Scholar]
- Hirose, K.; Kamei, S. Effects of English Captions in Relation to Learner Proficiency Level and Type of Information. Lang. Lab. 1993, 30, 1–16. [Google Scholar]
- Zarei, A.A. The Effect of Bimodal, Standard, and Reversed Subtitling on L2 Vocabulary Recognition and Recall. Pazhuhesh E Zabanha Ye Khareji 2009, 49, 65–85. [Google Scholar]
- Holobow, N.E.; Lambert, W.E.; Sayegh, L. Pairing Script and Dialogue: Combinations That Show Promise for Second or Foreign Language Learning. Lang. Learn. 1984, 34, 59–74. [Google Scholar] [CrossRef]
- Neuman, S.B.; Koskinen, P. Captioned television as comprehensible input: Effects of incidental word learning from context for language minority students. Read. Res. Q. 1992, 27, 94–106. [Google Scholar] [CrossRef]
- Danan, M. Reversed subtitling and dual coding theory: New directions for foreign language instruction. Lang. Learn. 1992, 42, 497–527. [Google Scholar] [CrossRef]
- Goldman, M.; Goldman, S. Reading with closed-captioned TV. J. Read. 1988, 31, 458–461. [Google Scholar]
- Fazilatfar, A.M.; Ghorbani, S.; Samavarchi, L. The effect of standard and reversed subtitling versus no subtitling mode on l2 vocabulary learning. J. Teach. Lang. Ski. JTLS 2011, 3, 43–64. [Google Scholar]
- Yoshida, H.; Uematsu, S.; Yoshida, S.; Takeuchi, O. Modalities of Subtitling and Foreign Language Learning. Lang. Lab. Assoc. J. Kansai 1998, 7, 49–63. [Google Scholar]
- Lee, D. This Chrome Extension Lets you Learn a New Language by Watching Netflix. The Verge 2019. Available online: https://www.theverge.com/2019/2/12/18220289/language-learning-netflix-chrome-extension-two-subtitles (accessed on 26 May 2019).
- Kovacs, G.; Miller, R.C. Smart Subtitles for Vocabulary Learning, in CHI’14 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems; AMC Digital Library: Toronto, ON, Canada, 2014; pp. 853–862. [Google Scholar]
- Canning-Wilson, C. Practical Aspects of Using Video in the Foreign Language Classroom. Internet TESL J. 2000, 6, 36. [Google Scholar]
- Sokoli, S. Audiovisual Translation Scenarios; Advanced Translation Research Center (ATRC): Copenhagen, Denmark, 2006. [Google Scholar]
- Zanon, N.T. Using Subtitles to Enhance Foreign Language Learning. Porta Ling. 2006, 6, 41–52. [Google Scholar]
- Vanderplank, R. ‘Effects of’ and ‘effects with’ captions: How exactly does watching a TV programme with same-language subtitles make a difference to language learners? Lang. Teach. 2013, 49, 235–250. [Google Scholar] [CrossRef]
Figure 1. The dual-coding theory (DCT) model of multimodal logogen and imagen units, connections, and implied processes .
Figure 2. Sharing meanings between two languages through code-switching (adapted from Soh, 2010).
Figure 3. Adapted from Mayer’s cognitive theory of multimedia learning (2014, p. 52).
Table 1. Categorization of the cognitive theory of multimedia learning (CTML) principles (as cited in Koscianski and Farago Zanotto, 2014).
|Coherence effect||Reduction of extraneous load||The material presented to the student should avoid including information that is not part of the contents being studied.|
|Signaling effect||A presentation should give clues to students to guide their attention towards the main points, by emphasizing or repeating information.|
|Redundancy effect||The narrated text should not be accompanied by written text since this can distract students from observing pictorial information.|
|Spatial Contiguity effect||The close placement between texts and pictures reduces the effort of students to inspect the material and favors learning.|
|Temporal Contiguity effect||The presentation of verbal and non-verbal pieces of information should occur simultaneously instead of sequentially.|
|Segmenting effect||Management of essential processing||The presentation of information should use separable units whenever possible, instead of fusing several concepts into complex texts and pictures.|
|Pre-Training effect||The introductory material at the beginning of a presentation may reduce the cognitive load associated with complex information that forms the core of the learning material.|
|Modality effect||When pictorial and verbal information is combined, the use of narrated (spoken) text is preferable over written text.|
|Multimedia effect||Fostering generative processing||Explanations with text and pictures are more efficient than those presenting information using only one of these possibilities.|
|Personalization effect||The presentation of material should preferably make students feel part of the narration, for example, using the second person instead of the third person conjugation.|
© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).