A Comparison of the Uptake of Two Research Models in Mobile Learning: The FRAME Model and the 3-Level Evaluation Framework

Abstract

This paper discusses the diffusion of two models of mobile learning within the educational research literature: The Framework for the Rational Analysis of Mobile Learning (FRAME) model and the 3-Level Evaluation Framework (3-LEF). The main purpose is to analyse how the two models, now over 10 years old, have been referenced in the literature and applied in research. The authors conducted a systematic review of publications that referenced the seminal papers that originally introduced the models. The research team summarized the publications by recording the abstracts and documenting how the models were cited, described, interpreted, selected, rejected, and/or modified. The summaries were then coded according to criteria such as fields of study, reasons for use, criticisms and modifications. In total, 208 publications referencing the FRAME model and 97 publications referencing the 3-LEF were included. Of these, 55 publications applied the FRAME model and 10 applied the 3-LEF in research projects. The paper concludes that these two models/frameworks were likely chosen for reasons other than philosophical commensurability. Additional studies of the uptake of other mobile learning models is recommended in order to develop an understanding of how mobile learning, as a field, is progressing theoretically.

1. Introduction

Mobile learning came into focus in the 1990s as personal digital assistants (PDAs) and, later, mobile phones began to facilitate learning [1]. In 2005, m-learning became an accepted term [1], although the definition of the term remained problematic. Within this relatively short time span, researchers and practitioners have grappled with defining, understanding, designing, applying, and evaluating mobile learning. They question who and what is mobile as well as how to integrate mobile tools into pedagogical practices. Conceptual models and frameworks play a significant role in answering these questions because they explain “either graphically or in narrative form, the main things to be studied—the key factors, concepts, or variables—and the presumed relationships among them” [2]. Evaluation models are also significant tools; they can be formative (information about the mobile intervention/tool is fed back to the researchers for improvement during the learning experience), summative (information is used to judge the usefulness of the intervention/tool after the learning experience), or both [3].

The purpose of this paper is to examine the uptake of two m-learning frameworks: Koole’s Framework for the Rational Analysis of Mobile Learning (FRAME) conceptual framework [4,5] and Vavoula and Sharples’ 3-level evaluation framework (3-LEF) as designed for the MyArt Space project [6,7]. A preliminary examination of citation numbers in Google Scholar suggests that both models have been referenced extensively. As of writing this paper, the number of Google Scholar citations for the FRAME model [4,5] is over 500; the number for the 3-LEF [6,7] is over 300. Upon closer examination, these numbers are somewhat misleading because they include any kind of reference—even references that appear in reference lists without having cited the original papers within the body text. Furthermore, the numbers from sources such as Google Scholar and other common indices fail to provide information about how the models have been used, if they have been understood and/or interpreted accurately, if they have offered a springboard to innovation in mobile learning, or if they have stimulated the emergence of other models.

A closer examination of how and why models proliferate through the field of mobile learning (and beyond) may help us gain a sense of the assumptions and perspectives of the researchers who have referenced the models. A general criticism of the field of educational technology is that there is insufficient evidence of critical thinking in the development of new perspectives, paradigms, methodologies, and reflective practice [8]. According to Yanchar, Gibbons, Gabbitas, and Matthews [8], critical thinking refers to “a cloud of intellectual processes by which ideas and processes are formulated, expressed, examined, questioned, tested, proven, discussed, and used within a field.” So, if authors have selected or rejected a model, did they have any underlying philosophical, practical, or operational reasons? If researchers fail to base their selection upon critically-thought-out criteria, what are the implications for the field? How can we determine “the conditions for progress” in the field [9]?

In order to examine the uptake of the FRAME and 3-LEF, the researchers conducted a systematic review of publications that cited the seminal articles in which Koole, Vavoula, and Sharples introduced their frameworks. The publications were summarized, the reference and citation information were documented, the abstracts were recorded, and any specific comments pertaining to the frameworks were recorded. These notes were then coded according to a list of criteria such as fields of study and/or researchers that have cited the models and their reasons for adoption or rejection.

This paper will first describe the two models. Then, the authors will explain the rationale for the study. The methodology section describes the databases, search terms, inclusion criteria, and exclusion criteria. The results section provides both numeric data of the included publications (number, types, geographic reach, topics of research, and research methods) and qualitative data (on reasons for use, critiques, and modifications of the models). The paper concludes by noting that there is need to trace the diffusion of theoretical models in order to understand the extent to which epistemological and ontological positions are guiding research in mobile learning, and more broadly, educational technology.

2. The Models

As mentioned above, the primary articles in which the models were introduced were published in between 2006 and 2009 [4,5,6,7]. These two models were chosen because they were developed and published at approximately the same time. These models are also over 10 years old; therefore, sufficient time has elapsed, and our research team can trace referencing patterns. These two models were also selected because the researchers are most familiar with them, which helped the team examine how the models were being used and (mis-)understood. Although, both the FRAME and the 3-LEF can be used outside of mobile learning, they can be applied to other educational technologies (including non-digital technologies). The 3-LEF is specifically an evaluative framework that outlines phases in the evaluation process. The FRAME can be used in an evaluative way, but it is more of a conceptual model in that it aids in the conceptualization of how phenomena are articulated.

2.1. The FRAME Model

The FRAME model was developed for a master’s thesis [4] at a time when mobile learning was first entering the mainstream of educational research. The purpose of the thesis was to examine key characteristics of a collection of mobile devices within the context of distance education at the post-secondary level. The study began in 2004 when few other mobile-learning models and frameworks were available. The master’s thesis was made available online in 2006. A chapter was published in 2009 in a free, open-access book, which likely aided in the proliferation of the model worldwide [10]; this open-access chapter is the document that is primarily referenced in the literature.

Originally, Koole worked primarily within a constructivist perspective, but also drew upon cognitivist theories (such as [11,12,13,14]). Special interest was placed upon situated learning (such as [15,16]), human-computer interaction (such as [17,18,19]), and classics in distance education (such as [20,21]). Koole has since moved to a sociomaterialist view of the FRAME model [22], which takes an ontological perspective in which the human and the material are equally important.

The FRAME Venn diagram depicts the three main aspects that influence and co-produce mobile learning (Figure 1). Within a researcher-delimited context, the key aspects of the FRAME model comprise the device, learner, and social aspects. The FRAME model requires a basic understanding of set theory, a branch of mathematical logic. In a Venn diagram, no one part is more or less important than any other. In the FRAME model, each circle is the same size symbolizing that they are equally important. Even the center of the circle (which is labelled “mobile learning:) is no more or less important than the parts that come together to create it nor is any other part of the diagram more important than the context within which it is situated (the “information context”). The reason the Venn diagram was used was to depict the pieces that come together co-construct the phenomenon of mobile learning. Logically, then, it is not possible to suggest that the FRAME model is device-centric, technologically determinist, or socially determinist. Similar to a jigsaw puzzle, all the pieces are necessary to create the finished picture, but no piece is more important than any other. The whole is just as important as the parts.

The device, learner, and social aspects overlap with each other creating the interaction-learning, social-technology, and device-usability intersections. The center of the Venn diagram shows the overlapping of all three aspects where mobile learning emerges. The overlapping of the circles guided the development of a list of key questions [5] that designers could use in developing mobile learning environments and interventions.

Table 1. Elements of the FRAME Model.

Set/Intersection	Characteristics
Information context	Where learning occurs
Device aspect	Physical components
	Input/output capabilities
	File storage and retrieval
	Processor speed
	Error rates
Learner aspect	Prior knowledge
	Memory
	Context and transfer
	Learning proclivities
	Emotions
	Motivation
Social Aspect	Conversation
	Cooperation
	Interaction
Device usability	Portability
	Information availability
	Psychological comfort
	Satisfaction
Social technology	Networking
	System connectivity
	Collaboration/interaction tools
Interaction learning	Interaction (learners, instructors, content, computers)
	Situated cognition
	Learning communities
	Pedagogical practices
	Curriculum
Mobile learning	Mediation, mediators, translators
	Information access and selection
	Knowledge navigation

summarizes the key (but partial) characteristics of each part of the FRAME model.

2.2. The 3-Level Evaluation Framework (3-LEF)

The 3-LEF model emerged as research was being done on the Myartspace project [7]. Myartspace was a project in which mobile phones were used by children on a field trip to a museum. Using their phones, learners could collect information and send it to a website. The students could view, share and present the information they gathered; in this way, they could digitally connect their field trip artifacts to their classroom and home environments. Pedagogically, the project involved an inquiry learning approach in which the learners would formulate their own questions, design their own plans, and determine what they learned.

The potential and effectiveness of project was evaluated using the 3-LEF. Whilst maintaining a focus on the socio-cultural context of learning [6], the framework is highly structured and is based upon Meek’s “lifecycle approach” [23]. The lifecycle approach was originally developed for a PhD dissertation in the field of software engineering. In this approach, evaluation occurs through the processes of conception, requirements analysis, design, implementation and deployment. The 3-LEF is sometimes referred to as the 3M framework because it comprises three levels: micro, meso, and macro. The micro level concerns the actual behaviours, interactions, and activities of the users. The meso level examines patterns in learning experiences across individuals and focuses on critical incidents—inclusive of both “breakthroughs and breakdowns” [6]. To identify gaps between what is expected and what actually occurs, the data collection occurs in three stages. Stage one involves collecting information through interviews with users and document analysis regarding expected and desired behaviours (of students and tools). In stage two, the researchers collect data through live observation or audio/video recordings, about what actually occurred. And, finally, in stage three, the researchers conduct reflective interviews with the users and analyse the data collected in stages one and two. To summarize, the 3-LEF combines three processes.

• Development process phases: requirements analysis, design, implementation, and deployment.
• Levels of granularity: micro, meso, and macro.
• Stages of data collection and analysis: stage 1 documentation of expectations, stage 2 documentation of actual activities, and stage 3 of gap analysis.

In order to help visualize the process, the authors provide an analysis-and-evaluation diagram (Figure 2).

The original and more detailed version of Figure 2 appears in the Myartspace article [7]. Although the Myartspace article provides an example of how the framework was applied, the article on “meeting the challenges” [6] is more cited in the literature. The article [6] outlines six challenges in evaluating mobile learning, introduces the 3-LEF, how the framework was applied to the Myartspace project, and discusses how the six challenges were address through application of the 3-LEF.

2.3. Brief Comparison of the Two Models

Table 2. Summary of the main features of both models (FRAME and 3-LEF).

	FRAME	3-LEF
Philosophy	Originally, social constructivist; shifted to sociomaterialist.	Socio-cultural.
Framework	Conceptual model; descriptive.	Evaluative model; procedural.
Implementation	Descriptive approach: Researcher can start at any section (context, aspects, intersections) in the Venn diagram; collect data on components that exist in each section; analyse how sections intersect.	Lifecycle approach to evaluation: researcher systematically studies the development process of a mobile tool at the micro, meso, and macro level of granularity.
Data collection	First order observation (direct observation/experience) or second order observation (interviews with users or examination of written documentation) of mobile learning situation, context, activities, and tools.	During each stage of development (conception, requirements analysis, design, implementation, and deployment), the researcher documents expectations and actual activities at each level of granularity.
Analysis	Analysis involves documentation of observations. Researcher writes a rich description of the mobile learning phenomenon.	Researcher conducts a gap analysis by comparing expectations to actual activities.

provides a summary of the main steps as originally intended by the authors of the FRAME and the 3-LEF models. (As will be discussed later, the models have been used in ways unpredicted by the authors.)

3. Motivation and Rationale for the Study

In their review of models and frameworks for designing mobile learning experiences and environments, Hsu and Ching [24] documented and categorized 17 notable models published between 2007 and 2015. (For the purposes of this paper, the terms model (conceptual model) and framework are used interchangeably. The differences between models and frameworks are outside the scope of this study.) A simple Google search will reveal that there are far more than 17; an exhaustive list needs yet to be compiled and published. Noting the proliferation of models and frameworks in the field of mobile learning, our research team began to question how existing models are being selected, extended, and/or rejected. We hoped that exploring the implementation and/or critiques of current models facilitate our understanding of how and why new models continue to emerge. Ultimately, what tools do mobile learning researchers require in order to answer their research questions? For these reasons, the main goals of this paper are to

• determine the number of references to the model/framework;
• examine the geographic and temporal reach of the model/framework;
• determine the number of times the model/framework has been used to guide research projects;
• locate the reasons why the model/framework was chosen;
• locate and analyse critiques of the model/framework (i.e., why the model/framework was rejected); and
• examine how the model/framework may have been modified.

4. Research Questions

Based on the aforementioned rationale and goals, the primary research questions are

• How have the seminal articles introducing the FRAME model and the 3-LEF been referenced in the education literature (including but not limited to the field of mobile learning)?
• How has the FRAME/3-LEF been used within the field of education?

5. Research Methodology

The research team for this project searched for any publication that cited the original (“seminal”) articles introducing the FRAME model [4,5] and the 3-LEF [6,7], which were published since June 2006 and May 2018. The inclusion criteria were broad because our team was interested in the range of study designs, subject areas, and geographic range in which the models have been used. The publications included spanned those that referred to the seminal articles only in passing to those that discussed and/or applied the model/framework in depth. Publications were rejected if they made no substantive comments about the seminal articles or the model/framework (failed to cite the seminal articles within the body text). We excluded articles by Koole, Sharples, and Vavoula in which they self-cited the introductory articles.

Each member of the research team received training from a research librarian on selecting the databases and formulating search strings. The following databases and search mechanisms were accessed:

• Eric;
• USearch;
• Proquest;
• Web of Science;
• Google Scholar;
• Research Gate;
• Academia.edu;
• Google search engine.

Because the search was so specific, the list of key words was kept simple.

Table 3. Keywords used in database searches.

FRAME Model	3-LEF
Koole	Vavoula and Sharples
FRAME model	3-level evaluation
Mobile learning	3-LEF
	Myartspace

lists the key words used in the database searches.

The author names were necessary in all searches; otherwise, the number of results returned was overwhelming. For example, without “Koole” in a search for FRAME model, the results yielded were in excess of 17,000. Similarly, “Vavoula” and “Sharples” were used in all the searches for the 3-LEF.

As the publications were located, they were summarized, the reference and citation information were documented, the abstracts were recorded, and any specific comments pertaining to the frameworks were documented.

Table 4. Template for summarizing publications.

Category	Notes
Author, date	Last name, date
Reference	APA citation
Country of author	Country
Country of research	Country
Type of research	Methodology, methods, approach
Applied	Yes/No
Summary	Description/abstract
Note	Notes about application, quality of references, observations, significant quotations.

lists the information documented for each paper located.

The summaries were then coded in Nvivo according to a list of criteria (a priori coding) such as fields of study, and/or researchers that have cited the models and their reasons for adoption or rejection. (See Appendix A for a complete list of codes/nodes).

The “applied” code was important in allowing the researchers to run queries in Nvivo separating those studies in which the FRAME or 3-LEF were applied in research. This helped the research team to determine the extent to which the seminal articles were merely cited (code: “literature review”) in contrast to the number of times the model/framework was actually used.

6. Results

6.1. Number of Publications Included in the Study

In total, 208 publications cited the FRAME model and 97 publications cited the 3-LEF. As can be seen in Figure 3, the FRAME model experienced its highest uptake in 2015. The 3-LEF diagram shows 2013 and 2015 as its years of highest uptake.

6.2. Types of Publications

The majority of the papers were published in journals followed by conference papers (

Table 5. Publication types.

	FRAME		3-LEF
Publication Type	All *	Applied **	All *	Applied **
Blog	1	1	0	0
Book	5	1	0	0
Book chapter	10	0	11	0
Conference paper	46	15	19	5
Conference poster	1	1	0	0
Doctoral dissertation	41	8	4	0
Journal article	87	24	57	2
Master’s thesis	5	2	5	3
Report	7	3	0	0
Unknown type	2	0	1	0
Wiki entry	1	0	0	0
Total	208	55	97	10

* Note: “All” includes all publications regardless of whether or not the model/framework was applied in research. ** Note: “Applied’ includes only publications in which the model/framework was applied to conduct research.

). Both the 3-LEF and FRAME model were also mentioned in a significant number of dissertations and master’s theses.

6.3. Geographic Reach

Our team was interested in geographic reach; that is, to what extent researchers in other countries are aware of the 3-LEF and the FRAME model. For each paper, we considered the location of each author (i.e., where s/he works) and, if indicated, the country in which the research was conducted.

Table 6. Number of countries.

Country of	FRAME (All Publications)	FRAME (Applied Only)	3-LEF (All Publications)	3-LEF (Applied Only)
Author	45	26	38	4
Research	39	22	26	5

shows the number of countries (authors’ location and country of research) for all publications and the number of countries (authors’ location and country of research) for those publications in which the FRAME or 3-LEF was applied.

6.4. Areas of Research

The 3-LEF and the FRAME model were referenced within publications belonging to a variety of fields and contexts. During this project, our team documented whenever the authors of the papers indicated areas of their research. We collapsed the areas into five major categories:

• Education levels;
• School subjects;
• Learning activities and skills development;
• Uptake, support, design of mobile systems; and
• Issues, challenges, and potentials of mobile learning.

Appendix B lists the areas of research within the five categories. It appears that the greatest number of studies were in the area of higher education. Within the learning activities category, gamification, contextual learning, field trips, and the use of social networks are the greatest in number. There was a wide variety of school subject areas across all the publications, but language learning and health-related subjects were very strongly represented. In the uptake, design, and support category, studies of learner uptake, attitudes, and support were most numerous, followed by the design and evaluation of learning environments and studies of pedagogical practices.

6.5. Research Methods and Methodologies

Table 7. Methods and methodologies as named by the authors.

Method/Methodology	FRAME	FRAME Applied	3-LEF	3-LEF Applied
Action research	2	1	0	0
Actor Network Theory	1	0	0	0
Artifact collection	1	0	0	0
Case study	19	6	8	2
Content analysis	1	1	0	0
Conversation analysis	0	0	1	0
Cooperative inquiry	0	0	1	0
Delphi study	1	0	2	0
Descriptive study	1	1	0	0
Design-based research	2	0	2	0
Diary	0	0	2	0
Ethnography	1	1	0	0
Experimental/quasi-experimental	10	5	8	3
Explanatory study	1	1	1	0
Exploratory study	1	0	0	0
Evaluation study	5	3	7	3
Focus groups	5	2	3	1
Grounded theory	1	0	0	0
Interaction analysis	0	0	1	0
Interviews	9	1	5	2
Mixed methods	15	4	13	5
Observation	0	0	3	0
Phenomenography	1	1	0	0
Phenomenology	2	0	0	0
Survey/questionnaire	24	8	7	4
Systematic review	20	0	13	0
Task analysis	0	0	1	0
Testing knowledge	2	0	0	0
Visual methodology	1	0	0	0

lists the methods and methodologies using the nomenclature of the original authors for all studies and only those that applied the FRAME model and 3-LEF in their research.

6.6. Contributions of the Seminal Articles

As our research team read through the publications that cited the 3-LEF, we began to note some patterns in how the seminal articles were being referenced. As such, it became clear that the works that initially introduced the FRAME model and 3-LEF contributed to the field in more general ways—that is, outside of evaluation and conceptual work. In the following section, the numbers in the tables offer a sense of proportion of the topics mentioned.

6.6.1. FRAME Model Comments in the Literature

Out of 65 references (outside of papers that explicitly used the model in their research), 42 publications made minor references to the FRAME model. Of those that more explicitly cited the two seminal articles [4,5],

Table 8. Most referenced topics (n = 65).

Topic *	References
Anytime anywhere access	5
Cognitive issues (especially load)	7
Context	6
Convergence of aspects (device, learner, social)	41
Definition of mobile learning	3
Knowledge navigation	6
Minor mention **	42
Social emphasis of the model	14
Technological limitations & characteristics	18
Utility in designing learning tools and practices	8

* Note: Some articles mentioned more than one topic. ** Minor mentions refer to those that mention the model/framework and/or seminal articles in non-substantial ways; a “nod” of acknowledgement.

summarizes the topics that were the most mentioned.

We observed that many authors were attracted by the notion that mobile learning comprises multiple aspects. At the same time, some authors focused their research on a particular aspect such as the social. Others made explicit reference to technological characteristics of mobile devices.

6.6.2. 3-LEF Comments in the Literature

We coded 59 instances of authors citing the two seminal articles on the 3-LEF [6,7]. Of these, we coded 35 comments as minor mentions.

Table 9. Most referenced topics (n = 59).

Topic *	3-LEF
Challenges in evaluation	8
Definition clarification	4
Evaluation methods	5
Examples of evaluation studies	12
In/formal learning and context	9
Micro, meso, macro levels	6
Minor mentions **	35
The need for evaluation studies	11
Mobile as social rather than technological	4

* Note: Some articles mentioned more than one topic. ** Minor mentions refer to those that mention the model/framework and/or seminal articles in non-substantial ways; a “nod” of acknowledgement.

summarizes the topics that were the most mentioned.

Vavoula and Sharples’ work was very often mentioned in a list of examples of evaluation (12 instances) studies in mobile learning. There were 11 instances where researchers explicitly argued for additional evaluation studies. There were four incidents in which the authors referred to the 3-LEF articles in order to clarify definitions such as mobile learning, location-based learning, and microsites. Six authors explicitly mentioned the micro, meso, and macro perspectives. Authors also drew upon the 3-LEF publications to champion the view that mobile learning is primarily a social rather than a technological phenomenon.

6.7. Reasons for Use

Our team searched for statements that explicitly stated why the researchers chose to use the model/framework to conduct their research.

6.7.1. Reasons for Using the FRAME Model

Seven of the eleven references coded for “reasons for use” indicated that the main appeal of the FRAME model is that is gives equal footing to the learner, the social and the technological aspects of mobile learning [25,26]. One author suggested that the model not only outlines the relationship between the three aspects but also addresses contemporary pedagogical issues of information overload, knowledge navigation, and collaborative learning [27]. Another work found that the choice of what to evaluate was alleviated, as all three aspects are equally important and should, therefore, all be evaluated [28]. Sandpearl [29] comments that maximum learning potential is achieved when the interactions between the social, the device, and learner converge.

Some authors showed interest in the social aspects of mobile learning [30]. Others commented that the FRAME model supports a socio-cultural view of learning [31,32]. Others appreciated the constructivist approach underlying the model and that it is an appropriate tool for examining learning and collaboration [25,26].

Several authors stated that the FRAME model was intuitive and easier to use than other models [26,33]. The accompanying checklist of key items and questions to guide the development of mobile learning applications [5] appears to have contributed to the perceived ease of use [24,34,35].

6.7.2. Reasons for Using the 3-LEF Framework

Three publications provided some insight into why they used the 3-LEF. One suggested that the framework, with the embedded micro, meso, and macro levels, acknowledges the ambiguity in the relationship between learning and the role of the institution [36]. The second indicated that the ability to evaluate at all stages along the development trajectory was key in the choice to adopt the 3-LEF [37]. And, the third paper indicated that the 3-LEF was selected because there were no other models specifically designed for wearable technology (a head-mounted display) [38].

6.8. Critiques of Model/Framework

Critiques of the model/framework were much easier to locate than rationales for their use. Our analysis revealed some astute observations and as well as some misconceptions of the models and the field of mobile learning overall. Although expressing appreciation for the 3-LEF and the FRAME model, some authors suggested that there is a lack of frameworks and models and, of the existing models, none of them are sufficient for guiding the design of mobile learning in varying contexts [35,39].

There were many critiques of mobile learning models in general. In these cases, the FRAME model and/or the 3-LEF were listed amongst other models:

• They have a limited perspective on context as they do not include the role of media at producing learning contexts. No methodologies or tools are available yet that treat the virtualization of context in an explicit way [40].
• They are limited in their practical applicability because they have no defined guidelines that consider the stages for the deployment of m-learning, but they do serve as starting points for the development of a sustainable M-learning model [41]. As such, there is need to bridge the gap between pre- and post-implementation phases in order to ensure sustainability.
• Do not address the question of how best to implement mobile learning in formal education [42].
• These frameworks are not learning theories per se. Rather, they offer ways to evaluate and frame mobile learning activities within the ubiquitous landscape of mobile learning [43]. It was also noted that the models lacked investigation into some macro-level factors including cultural and social barriers. Therefore, some papers highlighted the need for consideration and integration of broader social contexts when examining the efficacy of specific mobile learning contexts and research [44].

6.8.1. Critiques of the FRAME Model

Critiques specifically focusing on the FRAME model referred to missing criteria. Observing that the FRAME model defines mobile learning in terms of the interactions between learners, their devices, and other people [45], Wishart, cautions that Koole’s conceptual model does not acknowledge the potential mobility of the learner whose technology enables them to use information and data from one context to another. Khaddage et al. state that the FRAME model fails to address factors such as policies, pedagogy, technology, and innovative research in the field [46]; therefore, more guidance about how to utilize emerging mobile technologies and integrate them seamlessly into teaching and learning is still needed [46].

Power proposed an instructional-design related model that explicitly guides pedagogical issues [47]. He posited that although the FRAME model presents a holistic picture of the domains to be considered when designing or redesigning mobile learning initiatives, it does not provide guidance on the pedagogical design considerations needed for creating an effective collaborative learning experience for the learner [47].

Some critiques were more general in nature. For example, one paper argued that the FRAME model is a conceptual proposal and, hence, does not explore whether it would be suitable in real scenarios for supporting the development and adoption of its approaches [48]. Another researcher stated that the FRAME model is not yet sufficient given that further understanding of the highly dynamic emerging field of mobile learning is required [49]. Other authors indicated that Koole’s FRAME model is not applicable in primary education because it was “constructed in a higher education context, [which is] quite different from that of primary education” [50].

6.8.2. Critiques of the 3-LEF Model

We coded 11 instances of “critique” amongst the 97 publications. Some authors suggest that there is a general lack of systematic evaluation studies in the field of mobile learning. One paper includes a discussion of the 3-LEF but subsequently claims that the field of mobile learning lacks evaluation models/frameworks that have been “systematically and rigorously applied and fleshed out” [39]. Having listed a number of studies “conceptualizing” mobile learning, one paper indicated that no prior publications had adequately or rigorously measured success, scalability, and replicability of mobile learning initiatives. However, the author(s) offer no specific critiques of the 3-LEF (or other frameworks) were offered [51].

Farley and Murphy suggest that the 3-LEF was overly focused on the social characteristics of mobile learning to the detriment of the technical [39]. Another author praises the 3-LEF for its complex gap analysis and design heuristics but rejects the framework because it does not provide evaluation criteria [52].

Finally, we found an interesting critique of the micro-meso-macro approach: that it would require researchers to switch perspectives during the research project [53]. The authors suggested that perspective switching could result in communication issues and confusion because people working at one level would lack understanding of the impact [of the mobile intervention] on those working at other levels.

6.9. Extensions and Modifications

Our review of the literature revealed that some researchers suggested modifications to the FRAME model and the 3-LEF. Our team also noticed instances in which the FRAME and the 3-LEF were blended with other models.

6.9.1. Modifications to the FRAME Model

Some authors claim to have extended the FRAME model. For example, Norman, Din, and Nordin altered the model so that it would be based on four aspects: web 3.0 technology, learner context-awareness, learner cognition, and learner social skills [54]. Boyinbode, Ng’ambi, and Bagula integrated Anderson’s six types of educational interactions, something that they thought had not been addressed directly in the original FRAME model [55]. Meanwhile, Levene and Seabury combined the FRAME model with Park’s transactional distance theory model in an effort to inform instructional design practices [56].

The relational structure of the FRAME model was also modified. For example, the “augmented FRAME” was designed in order to differentiate K-2 learning methods based on characteristics such as targeted grade level, specific devices, necessary infrastructure, mobility, cost per student, and type of learning [50]. Pani and Mishra offer a modified view in which there are four aspects (social, learner, device, and context) and four intersections (device usability, interaction learning, mobility interaction, and pervasiveness) [57]. Finally, Wong recommended a new mobile learning model where he has embedded a curriculum aspect into the Koole’s FRAME model and places the learner aspect into the center of the FRAME model [27]. He suggests that the curriculum aspect should be evaluated in future research studies to find out how mobile informal learning experiences or activities can assist students in formal learning contexts.

Some authors offered less extensive modifications such as the addition of “mobile pedagogy” considerations that complement the socio-cultural characteristics of the FRAME model [32]. Another author suggested that instead of mobile learning, ubiquitous learning or pervasive learning could occupy the center of the Venn diagram [57]. Lefrere also recommended that the FRAME model include networking and networked services. In this way, researchers would acknowledge the surface functionality of a device as well as the functionality it gains through networking [58].

6.9.2. Modifications to the 3-LEF

There was no evidence of major modifications proposed for the 3-LEF. We documented only one minor modification. In one study, the author(s) examined the micro and meso levels in depth [53]. At the micro level, they added a mobile quality piece to the evaluation.

Our Nvivo code “extends other model” captured some evidence that elements of the 3-LEF were used to enhance other evaluation models/frameworks. In one dissertation, the author combined the 3-LEF with other models in order to create his/her own evaluation criteria [59]. Another study focused on the meso level (learner experience) whilst proposing a new model developed in order to highlight socio-cultural characteristics of mobile learning [32].

7. Discussion

7.1. Number of References

As mentioned, at the time of writing, Google Scholar indicated that the FRAME model had been referenced over 500 times and the 3-LEF over 200 times. Our team searched eight databases for any publications referencing the seminal articles that introduced the models [4,5,6,7]. Our team removed from analysis articles that referenced the seminal articles without actually discussing or citing them in their body text. Our final analysis was left with 208 articles citing the FRAME model and 97 for the 3-LEF.

One of our main goals was to explore the impact that the FRAME model and the 3-LEF have had in educational research. A preliminary look at numeric data in reference indices suggests that both models are well represented in conference presentations and journal articles. Surprisingly, the FRAME model has been mentioned in an unusually high number of doctoral dissertations (see Table 5). Overall, the FRAME model appears to have been more referenced than the 3-LEF. This might be due to the more general, conceptual nature of the FRAME model. It might also be related to the publication of the 2009 article [5] in a free, open-access publication.

7.2. Reasons for Use

As alluded to earlier, the both models were applied in ways that were not originally predicted by the Koole, Vavoula, and Sharples (summarized in Table 2). Although originally designed for qualitative analysis, the FRAME model has been used in quantitative studies in which researchers have attempted to develop numeric measures of mobile characteristics [60,61]. In the case of the 3-LEF, some researchers chose to implement it at only one level of granularity such as the micro or the meso [62,63]. Both of these examples suggest healthy innovation.

Although there were a number of topics that were mentioned in the literature, our team took special note that both the FRAME and the 3-LEF were recognized for emphasizing the social characteristics of learning (See Table 8 and Table 9). We would argue that both models, while having strongly supporting a social view of learning, also place significance on other factors within the mobile learning milieu. To a greater or lesser extent, they both involve examining the physical environment and technology.

Both models recognized the interrelationship of multiple components that are often treated separately in research such as

• The social, learner, and technical aspects;
• The micro, meso, and macro levels;
• The phases of development; and
• The stages of evaluation.

In our analysis, we gained a sense that the multi-components approach was a drawing feature of both the FRAME and the 3-LEF.

7.3. Critiques

Naturally, critiques were often used by authors in order to argue in favour of a new model they were proposing. Models such as the FRAME and 3-LEF were viewed by some as a starting point, but that they were lacking in some ways. Some authors appeared to require evaluation criteria or guidelines that were customized to their study contexts and phenomena such as virtualization of context, in/formal learning situations, and ubiquitous landscapes. Similarly, one paper suggested that neither model allowed for use in formal education settings. However, both the FRAME and the 3-LEF are deliberately designed to accommodate a wide variety of contexts and phenomena. And, the authors of both the FRAME model and the 3-LEF have acknowledged their applicability beyond mobile learning. As one writer correctly observed, they are conceptual/evaluative tools [44]. As such, they were designed with some flexibility and openness permitting customization for specific contexts and phenomena.

The critique that the FRAME and the 3-LEF (1) do not include the role of media in producing learning contexts [40] and (2) that factors such as cultural and social barriers are not considered lacks consistency with our reading of the FRAME model and the 3-LEF. The role of media and social barriers fit neatly within the models. For example, media can be represented in the technology aspect of the FRAME model (see Figure 1) and is embedded throughout the design processes in the 3-LEF (see Figure 2). Furthermore, social barriers can be discussed within the social aspect and the social-interaction intersection of the FRAME model. The 3-LEF was developed within a socio-cultural perspective; therefore, social issues can be examined through the ongoing interviews with users embedded within the evaluation process.

Interestingly, another critique of the 3-LEF was that it is overly focused on the social to the detriment of the technical [50]. If the scope of a research project is to examine only the technological aspects of a mobile learning initiative, then it is logical that the researcher would modify it so as to focus on a particular characteristic.

Finally, there was a criticism that neither model is a learning theory per se. That is true, these models were not designed as learning theories. One would choose to use the FRAME model to gain a conceptual understanding of a mobile learning situation or to guide the design of a mobile learning application. One would use the 3-LEF to evaluate a mobile learning situation or application. Nonetheless, both models were conceptualized and developed within perspectives such as social constructivism, situated-learning, and socio-cultural theories. As we have discovered through this research, both have been applied in the design of learning applications. Therefore, we argue that models can be used creatively and flexibly.

7.4. Modifications

The information context in the FRAME model is often overlooked (See Figure 1). The information context asks researchers to define the scope; all aspects and intersections fit within each, defined context. The authors of the “augmented FRAME” did not so much create a new version of the model but changed the scope appropriately. Rejecting the model for contextual reasons, requires careful thought. For example, suggesting that it cannot be applied to a K-12 context can be contested. Although the FRAME model was originally developed within a higher education context, the original Master’s thesis by Koole [4] includes learning theories that connect various levels of education. Koole, for example, refers to Vygotsky’s zone of proximal development [15], which is most often associated with childhood learning processes. Furthermore, if the context is K-12 or in/formal learning, it can be defined in the information context. The characteristics of all the aspects would be described within the specified context. More thought should also be given to the addition of an extra circle for context [57]; because the context is already a part of the FRAME model, the extra circle seems redundant.

As educators, we strive to ensure that learners occupy a significant place in education. Wong reconfigured the FRAME model so as to place the learner at the center of the mobile learning process. However, this upsets logic of set theory. When circles overlap, there are shared elements. The overlapping of all the circles cannot result in a learner aspect; the center must contain element of all the circles. In addition, in the original FRAME model, the curriculum is part of the social learning intersection (See Figure 1). Currently, as Koole moves toward a more socio-materialist philosophy, the co-creation of the social and the material is jeopardized when the human is positioned as more significant than the other elements [22]. Wong’s work, however, is highly valuable. In future descriptions of the social-learning intersection, curriculum should be explicitly listed. Similarly, Pani and Mishra’s work has led us to consider whether the social-learning intersection could be more aptly named “mobile pedagogy” [57].

Lefrere’s suggestion that ubiquitous or pervasive learning could occupy the center of the FRAME model (see Figure 1) is an astute observation [58]. This suggests that there is some thinking about how the model(s) can be successfully transitioned to different technologies. However, it is unclear if Lefrere is suggesting that mobile learning and ubiquitous or pervasive learning are the same phenomenon. And, this brings us to questions of definitions and nomenclature. There may be a philosophical piece here that requires further deliberation. Does the different nomenclature suggest a shift in semantics surrounding mobile learning or a shift in the ontology of mobile learning?

8. Conclusions and Future Research

This paper has only examined two of many models in the field of mobile learning. The FRAME and the 3-LEF models were selected for this study because they have amassed more than 10 years of references. It is encouraging that the models have been implemented across a wide range of topics, fields of study, and geographic contexts. As researchers and practitioners, our team had hoped to gain some insights into how researchers selected and rejected models/frameworks. The critiques, reasons for use, and modifications have provided us with additional ideas of how we might “tweak” the FRAME and 3-LEF in our own work.

At a more general level, we have begun thinking about how researchers in the mobile learning field are approaching model selection and evaluation. While some authors indicated that they were drawn to the constructivist approach of the FRAME model (Koole’s original approach) and the socio-cultural emphasis of both the FRAME and the 3-LEF, there was little evidence for conscious selection of models/frameworks based on ontological or epistemological concerns. We are left with some perplexing questions: Are mobile learning specialists making logistical decisions in which models/frameworks are chosen for criteria such as ease of use? Are they basing their decisions on a particular philosophical position, or are we seeing the emergence of new models because there is something yet unforeseen, and therefore unaddressed, in the nature of mobile learning? Similar reviews of other models would help us understand how mobile learning researchers and practitioners evaluate and choose their research tools.