The Challenges of Conducting Research in Diverse Classrooms: Reflections on a Pragmatics Teaching Experiment

Abstract

For researchers, the typical way of determining whether a pedagogical innovation works is by conducting an experiment. In migrant settings, however, experiments are more challenging to carry out due to the diversity of the learner population. Unfortunately, how to deal with these challenges is not addressed in a practical way in research methods textbooks, which typically provide a normative view of the research process. This paper aims to draw attention to the realities of classroom research carried out in the Language Instruction for Newcomers to Canada (LINC) setting. These classes consist of adult immigrants and refugees from a wide range of cultural, linguistic and educational backgrounds. We illustrate how this diversity along with other characteristics of LINC programs impact the decision-making of the researcher with respect to a pedagogical experiment focused on pragmatics. The study compared a formula-enhanced approach to teaching speech acts to the more mainstream approach aimed at raising learners’ meta-pragmatic awareness about speech act behaviour. The pre-post-delayed-post-test gains appear to favour the Formula group, but the interpretability of these results is compromised by the fact that the composition of the two classes was very different. Discussion of the limitations of this case study feeds into a broader consideration of the implications for classroom research of linguistic and cultural diversity typical of L2 educational contexts like LINC.

1. Introduction

In his pioneering book on second language (L2) classroom research, Chaudron (1988) articulated the goal of classroom research as being ‘to identify those characteristics of classrooms that lead to efficient learning of the instructional content, so that empirically supported L2 teacher training and program development can be implemented’ (p. 1). To achieve such a goal, research needs to be of good quality, which means that researchers need to be trained in the design and implementation of studies according to the principles and conventions found in research methodology textbooks. For applied linguists, there are many available textbooks with a focus on educational or language teaching research. All of these textbooks tell novice researchers what they should do, thereby presenting what Bryman (2007, p. 7) describes as a “normative” picture of the research process that is often at odds with the messiness of actually carrying out an empirical investigation. The gap between the theory of research methodology and the real-world experience of researchers is a particular concern for those working in the context of L2 programs for adult immigrants. In this paper, our aim is to draw attention to the characteristics of Language Instruction for Newcomers to Canada (LINC) programs, the need for classroom research to support the goals of these programs, and the challenges that a researcher of LINC faces. The methodological discussion is anchored in our reflections on a particular classroom study, which we refer to as an illustrative case study. It is hoped that these reflections may contribute to future efforts to articulate methodological guidelines for research in complex teaching and learning contexts such as LINC.

To begin, we provide some background information about the LINC context, the importance of teaching pragmatics to newcomers to Canada, and then briefly review some concepts relating to instructional pragmatics. The third section presents a compressed summary of the research design of the case study and its findings. Moving from the micro to the macro, in Section 4, we address the gap between the normative view of research methods and the real world of doing research in LINC classrooms. Section 5 takes the form of some recommended strategies for classroom researchers in LINC and other similar contexts.

2. Background

2.1. The LINC Context

Immigration has traditionally been one of the main sources of population and economic growth in Canada (IRCC 2020). According to Statistics Canada, approximately one in four workers are immigrants, and by 2036, immigrants are expected to account for up to 30% of the country’s population (Yssaad and Fields 2018). Currently, the top five countries that supply immigrants to Canada are India, China, the Philippines, Nigeria, and Pakistan (IRCC 2020). The integration of newcomers into Canadian society and the workforce is heavily dependent on language proficiency in at least one of the official languages, English or French. Formal language training in English funded by the federal government has been delivered through the LINC program since 1992.1 These classes are offered by a wide variety of providers including boards of education, private language schools, colleges, churches, and community centres (Gormley and Gill 2007). Teaching in LINC is typically multilevel as a result of the diversity of adult newcomers who come to class with, for example, different linguistic and cultural backgrounds, levels of previous education, and types of occupation prior to emigration (Cray 1997; Derwing et al. 2009; Haque and Cray 2007). The multilevel nature of LINC classrooms is further increased by the administrative practices of continuously enrolling students and of combining different LINC levels in a single class (e.g., LINC 4 and 5 together). Although teachers complain about the negative pedagogical consequences of these programming decisions, administrators argue that these practices are necessary to provide learners with flexibility, on the one hand, and to maintain class sizes, on the other (IRCC 2020). Yet another administrative practice typical of many LINC providers is hiring teachers on short-term and mostly part-time teaching contracts with limited compensation for work done outside of contact hours (IRCC 2020; Valeo and Faez 2013). For many teachers, this means teaching at more than one school.

One of the main reasons that newcomers enroll in a LINC program is to enhance their employability (Hajer et al. 2007). Regardless of professional and/or educational background and the level of technical skills, most newcomers struggle to find employment in Canada. Many newcomers are forced to look for jobs outside of their professional field and usually end up in unskilled labour positions (Derwing 2016). This tendency has a negative impact on the Canadian labour market since employers are not able to efficiently recruit and retain international talent (Derwing et al. 2009). One possible reason for this is that so-called ‘soft skills’ are valued over technical skills by Canadian employers (Laroche and Rutherford 2007). As opposed to technical skills, soft skills are directly related to the underlying sociocultural norms of the host country and the ‘unspoken rules’ of interaction and may, therefore, be harder to acquire.

2.2. Pragmatics Concepts

From an applied linguistics perspective, soft skills fall under the category of pragmatic competence. In contrast to linguistic competence (i.e., phonology, morphology, syntax, and vocabulary), pragmatic competence is a socially co-constructed phenomenon which consists of knowledge of linguistic forms and their functional meanings, sociocultural knowledge, and the ability to use this knowledge to create a communicative act in interaction (Taguchi 2018). Linguists distinguish between two aspects of pragmatic competence: pragmalinguistic and sociopragmatic knowledge (Leech 1983). Pragmalinguistic knowledge refers to the linguistic resources available in the target language for performing a specific speech act whereas the choice of a pragmalinguistic form appropriate for a given context requires sociopragmatic knowledge. For example, if a university student wants to submit an assignment after the deadline, she will probably say something like ‘I was wondering if it would be possible to get an extension for my assignment’ rather than ‘Can you give me an extension?’. Both reflect pragmalinguistic resources available to the English speaker for making requests, but the first is more sociopragmatically appropriate for the situation due to the social distance between the interlocutors and the perceived imposition level of the request. In order to achieve pragmatic fluency, second language (L2) learners need to acquire both types of knowledge and be able to control them efficiently under the pressure of real-time communication (Taguchi 2011). Pragmatic formulas (or routine formulae) serve an important role as pragmalinguistic resources for performing various speech acts such as apologizing, making requests, and many others (e.g., ‘I am sorry to hear about + [X]’ is conventionally used to express regret) (Nattinger and DeCarrico 1992). Therefore, acquiring pragmatic formulas is seen as particularly beneficial for language learners.

2.3. Instructional Pragmatics

Pragmatics researchers have addressed a broad range of issues (see reviews in Taguchi 2011, 2018; Taguchi and Roever 2017). Here we are most interested in what is referred to as instructional pragmatics. This is a growing area of empirical investigation from which it has been concluded that pragmatics instruction has a positive effect on learners’ L2 pragmatic development (Taguchi and Roever 2017). For our purposes, the salient characteristic of many of the existing pedagogical intervention studies with English as a target language is their research context. Many have taken place in university-level foreign language classes where all (or most) of the learners shared a first language (L1) and home culture and had similar previous educational experiences. Those pragmatics studies that have been carried out in English-speaking countries have also typically dealt with university-level students having relatively more advanced levels of English proficiency (e.g., Bardovi-Harlig and Bastos 2011; Bardovi-Harlig and Vellenga 2012; Bardovi-Harlig et al. 2015).

In contrast, there appear to be very few available pragmatics-focused intervention studies that have been conducted in educational settings involving recent immigrants in Canada (Zavialova 2016, 2017; Zeldenrust 2017), although studies involving immigrants in other English-speaking countries are logically relevant (e.g., Holmes and Riddiford 2011; Riddiford and Joe 2010; Wigglesworth and Yates 2007). Thus, evidence-based advice for teaching pragmatics in LINC will necessarily come from research conducted elsewhere. For example, one Canadian curricular guide for adult ESL (Kaskens 2012) suggests using the ‘6Rs Approach’ by Martínez-Flor and Usó-Juan (2006). This approach consists of six steps: (1) researching; (2) reflecting; (3) receiving; (4) reasoning; (5) rehearsing; and (6) revising (see below in Section 3). This is a pedagogical sequence supported by theory and empirical findings from the pragmatics literature but developed for Spanish English as a foreign language (EFL) classes. Implicitly, the authors of the Canadian guidelines assume that ideas that have been shown to be sound elsewhere will apply to the context of the diverse LINC classroom. In our view, it is important to move beyond assumptions and actually establish empirically that a given pedagogical approach is ecologically valid for a different context. Not only do we need to investigate whether or not it works for diverse learner settings, but also whether it works better than other pedagogical approaches. However, undertaking a classroom-based study in LINC involves dealing with a range of practical constraints that influence decisions about research design, and data collection and analysis. As an illustration of the issues, we describe the decision-making process of a novice researcher as she navigated the constraints imposed by the LINC context during the implementation of a pragmatics teaching experiment. In doing so, we are following the example set by Reid (1990), who aired the ‘dirty laundry’ of her own research process in the design of a learning style questionnaire. In the spirit of Reid, we aim to draw attention to methodological dilemmas that inhibit efforts to obtain sound empirical evidence of relevance to LINC educators and stakeholders.

3. Case-Study: A Pragmatics Experiment in LINC

The study described in this section is based on the second author’s unpublished doctoral research (Zavialova 2020). The impetus for the study was her experience as a substitute LINC instructor and volunteer teacher at several LINC schools. From this experience, she came to recognize the necessity of incorporating pragmatics into the day-to-day teaching practices in LINC, as well as the lack of adequate resources for doing so. In what follows, the decision-making involved in designing and conducting her pragmatics experiment and the key findings are described.

3.1. Participants

To find participants for her study, Zavialova relied on her familiarity with different LINC schools in the Ottawa area and personal connections within the LINC community. She first approached the manager of a LINC program whom she knew, explained the purpose and details of her research project, and asked her to share the call for participation with her teachers; two teachers from that school volunteered to participate. Given that each LINC class normally enrolled up to 25 students, it seemed possible that there would be an adequate number of participants for quantitative analysis. However, when the researcher visited each class to invite students to take part in the study, only about 60% were present and not every student was willing to participate. Consequently, Zavialova was forced to choose the LINC class with the larger number of students who had confirmed their ability to attend all sessions of the intervention and then recruit additional participants from a different school. Unfortunately, irregular attendance was an issue in both LINC locations, so the study proceeded based on the data from the number of students who had attended all of the instructional sessions and had completed all of the measures.

In the end, the convenience sample consisted of students from two LINC schools and with a total of 13 ESL students. Based on a test, they had been placed in LINC levels 4/5, and their proficiency levels ranged from lower- to upper-intermediate (corresponding to Canadian Language Benchmark levels 5–7). One intact class was assigned to the Formula treatment (n = 7) and the other to the Mainstream treatment (n = 6) (see description below). The student participants came from diverse linguistic, cultural, educational and employment backgrounds (

Table 1. Characteristics of the participants in the study by Zavialova (2020).

	Formula Group (n = 7)	Mainstream Group (n = 6)
First language	Arabic; French; French & Kirundi; Mandarin (2); Swahili; Tigrinya	Arabic (3); Farsi; Nepali; Somali
Country of origin	Burundi; China (2); Congo; Eritrea; Ivory Coast; Syria	Iran; Nepal; Somalia; Syria (3)
Occupation prior to emigration	accountant; communications engineer; mechanical engineer; manager; teacher	small business owner; stay-at-home parent (5)
Work experience in Canada	car mechanic; clerk in clothing store; gas station operator; teacher; unemployed (3)	seasonal farm worker; unemployed (5)

). There were seven male and six female students who spoke the following first languages: Mandarin Chinese, Swahili, Arabic, Tigrinya, French, Kirundi, Farsi, Nepali, and Somali. Each treatment group included both economic immigrants (who entered Canada as permanent residents) and refugees. The two participating teachers had qualifications in teaching English to speakers of other languages and had been teaching ESL for one to two years.

3.2. Instructional Treatments

In designing the instructional treatments, Zavialova wanted to compare the effectiveness of providing explicit metapragmatic explanations, a mainstream approach to teaching pragmatics (e.g., Cohen 2005), to a more innovative approach that draws learners’ attention to pragmatic formulas (i.e., recurring multiword combinations that possess functional meanings).

Originally, the Mainstream treatment was to be based on the 6Rs approach that had been proposed by Martínez-Flor and Usó-Juan (2006) for university-level EFL learners in a particular academic discipline (i.e., tourism). As mentioned above, this approach consists of six steps or stages: researching, reflecting, receiving, reasoning, rehearsing, and revising. At each stage students receive worksheets with awareness-raising questions about various pragmalinguistic and sociopragmatic aspects of a given speech act. Students are also encouraged to become data-collectors and evaluators; they are presented with conversation scenarios and then asked to review them and evaluate their appropriateness. However, since the 6Rs approach was designed primarily for university students and prescribed extensive metapragmatic discussions, it was not feasible to implement it ‘off the shelf’ in the LINC classroom, given the nature of the LINC program and the proficiency level of the students. Consequently, a customized version of the 6Rs approach that also incorporated some features from other recommendations from the instructional pragmatics literature (Usó-Juan and Martínez-Flor 2007; Cohen 2005, 2008; Cruz 2013) was developed for this study.

The teaching sequence in the Mainstream treatment consisted of two phases: Exploration and Awareness-raising and Practice. During the first phase, the teacher introduced a scenario and asked the students to brainstorm and write down what they would say if they were in that situation. The students first wrote these responses in their first language and then translated them into English. At the same time, the teacher encouraged the students to think about the levels of social distance, power, and imposition that exist between them and their imaginary interlocutors. Scripts with simplified explanations of these three factors were provided for the teachers in a teacher’s guide. At the end of this activity, the students were to share their translated English versions with the rest of the class in a discussion facilitated by the teacher. Next, the students were asked to compare their L1-based responses to those produced by native English speakers and identify similarities and/or differences between them. To this end, the students answered four reflection questions in a whole-class discussion facilitated by the teacher. Finally, the students identified and analyzed pragmatic patterns (i.e., speaker’s communicative strategies) in the native speaker examples. During the Practice phase, the students role-played the same scenario in pairs while the teacher circulated around the class and provided corrective feedback, as needed.

The Formula-enhanced treatment designed for this study built upon a pedagogical sequence developed by Zavialova (2017). The activities in this treatment were designed to promote what Boers and Lindstromberg (2009) refer to as ‘deep cognitive engagement’. The teaching sequence in this treatment consisted of three phases: Guided Noticing; Deep Cognitive Engagement and Stimulating Retention; and Practice. During the first phase, the students completed two written exercises that aimed to draw their attention to pragmatic formulas associated with different scenarios. In the second phase, the students received lists of 8–10 pragmatic formulas (from the researcher’s corpus) and explored the implied meaning of the target formulas. The first part focused on formula recognition and pragmalinguistic forms. Hints under each formula were added to avoid ‘blind guessing’ and stimulate conscious learning (Boers and Lindstromberg 2009). The second part focused on interpreting pragmatic implicature (directness/indirectness) and targeted the sociopragmatic meaning. Here, the students reviewed the same formulas again and evaluated their degree of directness. The final exercise of this phase was similar to the one in the Mainstream treatment where the students analyzed pragmatic patterns in native speaker examples. The Practice phase activity was identical to the role-plays in the Mainstream treatment.

Table 2. Key characteristics of the two pedagogical treatment programs in Zavialova (2020).

Formula Treatment	Mainstream Treatment
Explicit focus on pragmatic formulas	Explicit focus on factors of social distance, power & imposition
Equal emphasis on pragmalinguistics and sociopragmatics	More emphasis placed on sociopragmatics
Pragmatic formulas reviewed multiple times	Pragmatic formulas reviewed occasionally
No metapragmatic terminology used	Metapragmatic terminology used

summarizes the key pedagogical characteristics of the two treatments.

3.3. Measuring and Analyzing the Effects of the Interventions

Following standard experimental research design, pre-, post- and delayed post-test measures were administered to measure the impact of the experimental interventions. Role-plays were chosen as the most appropriate way of determining the effectiveness of the pragmatics interventions. The four role-plays used were based on scenarios mentioned in curricular documents; they evoked social, customer service, and workplace situations and involved the following speech acts: inviting, requesting (the rescheduling of an appointment), inquiring (about service-related issues), and refusing (managerial requests) (see example in the Appendix A). The researcher role-played the scenarios with each individual student, whose performance was audio-recorded and transcribed for analysis.

Three trained expert judges evaluated each student’s oral role-play performance using a rubric. This rubric, developed for this study, was composed of a pragmalinguistic component that included five criteria dealing with pragmatic formulas and a sociopragmatic component that included five criteria dealing with the levels of formality, directness, politeness, amount of information provided, and overall appropriateness of the speech act in each target situation. To complement the quantitative ratings of the judges, the researcher also conducted a qualitative utterance analysis of all of the student transcripts in order to identify signs of interlanguage development in the short term (immediate post-test) and the longer term (delayed post-test). The utterance analysis made use of a pre-defined list of categories based on Taguchi et al. (2013) and a native speaker corpus. Learner utterances were coded as being target-like, target-like slot-and-frame, repetitive use, or non-target.

3.4. Procedure

The following list summarizes the researcher’s steps in the implementation of this teaching experiment:

• Initial briefings with the participating teachers;
• Administration of consent forms and pre-test;
• Model lesson taught by the researcher to each treatment class; discussion of the model lesson with participating teachers afterwards;
• Four-week treatment delivery by the regular teacher and observed by the researcher;
• Administration of the immediate post-test and the delayed post-test two weeks later;
• Final debriefing with the teachers.

More details about the procedures can be found in Zavialova (2020).

3.5. Findings

Results from the ratings of the learners’ role plays by the expert judges are presented in

Table 3. Average median scores across four speech acts for Formula and Mainstream groups: Percentages in parentheses.

Component		Pretest	Post-Test	Delayed Post-Test
Pragmalinguistics (max. = 24)	Formula	14 (0.58)	18 (0.75)	19 (0.79)
	Mainstream	12 (0.50)	15 (0.63)	16 (0.67)
Sociopragmatics (max. = 30)	Formula	16 (0.53)	20 (0.67)	22 (0.73)
	Mainstream	15 (0.50)	18 (0.60)	20 (0.67)

Note: Scores have been rounded up.

. The table displays the average of the median scores across the four targeted speech acts for each treatment group. The overall pattern for the two groups is the same: the ratings for both the pragmalinguistic and sociopragmatic aspects of the learners’ performance increased from the pretest to the immediate post-test and the gains are maintained at the delayed post-test. The percentages in parentheses suggest no pattern of difference between the pragmalinguistic and the sociopragmatic aspects of their performance. The table also suggests a between-groups difference such that the Formula group had slightly higher scores across the board. However, this small difference in favour of the Formula group is evident not only after the intervention but also before it. Unfortunately, due to the small sample size it was not possible to determine whether the apparent differences are statistically significant.

The results of the qualitative utterance analysis revealed a greater tendency of the Formula group students to show improvement in terms of both grammatical accuracy and contextual appropriateness. In contrast, learners in the Mainstream group did not demonstrate any visible shift towards nativelikeness in either area. At the same time, students in both treatment groups experienced similar pragmalinguistic and sociopragmatic difficulties when trying to perform the target speech acts. The analysis also revealed that the utterances produced by students in both treatment groups contained pragmalinguistic form errors (i.e., inaccurate grammatical or lexical choices) and sociopragmatic norm violations (i.e., inappropriate tone or speech act).

3.6. Interpretation of the Findings

The ratings around 50% for both groups at the pretest indicate that prior to their respective interventions, these learners lacked adequate pragmalinguistic resources needed to perform the four speech acts in a target-like way. The improvement in scores over time suggests that both groups benefited from instruction focused on developing learners’ pragmatic awareness, as has been found in other studies (Taguchi and Roever 2017). These results support the argument that formulas have the potential to help L2 users to be perceived as proficient target language users and, thereby, facilitate newcomers’ acceptance into the target language speech community (House 1996; Bardovi-Harlig 2012). The relatively weaker performance of the Mainstream group may indicate that explicit metapragmatic explanations about social distance, power, and imposition (as prescribed by the 6Rs Approach) did not help the students notice the link between these factors and language choices during social interaction (see Takahashi 2010 for a similar viewpoint). Indeed, the researcher’s impression from observing the Mainstream class was that the abstract theoretical constructs of social distance, power, and imposition were not well understood. The comprehensibility of metapragmatic explanations is an issue that deserves to be further investigated in future research.

The results from this quasi-experimental study could be interpreted as evidence in support of a formula-oriented approach to pragmatics instruction. However, the findings need to be interpreted with a grain of salt due to the limitations that follow from the use of intact LINC classes. Given that these limitations are typical of LINC as well as other multilingual, multicultural L2 classrooms, a closer examination of these limitations is warranted.

4. Discussion of Limitations

A limitation of an empirical study is defined as ‘the systematic bias that the researcher did not or could not control and which could inappropriately affect the results’ (Price and Murnan 2004, p. 66). It is quite typical for empirical researchers to acknowledge only briefly the limitations of their studies at the end of their report. In contrast, in this section, we scrutinize the limitations of our case study in the spirit of the ‘dirty laundry’ revelations of Reid (1990). Specifically, we consider the researcher’s decisions in comparison to the conventional recommendations of research methodologists. The following issues are discussed: intact classes, student diversity, student participation, and the teacher effect.

4.1. Intact Classes

In our illustrative case study, Zavialova chose to use intact classes to represent the pedagogical treatments in her pragmatics experiment. This is referred to as a quasi-experiment because participants were not randomly assigned to experimental groups. In the so-called true experiment, random assignment to treatments distributes extraneous variables across all experimental groups, which means that the groups become equated (Cresswell and Guetterman 2019). It is then assumed that any differences between experimental groups are due to the manipulated variables (i.e., the intervention(s)) and, in this way, causality can be established. Using the standard technical jargon, Cresswell and Guetterman (2019) list the threats to internal validity associated with quasi-experiments as relating to participant history, maturation, regression, selection, mortality, and the interaction of these participant characteristics with instrumentation. In order to control for the impact of extraneous factors introduced by the use of intact classes, researchers are advised to equate groups artificially through covariate analysis (e.g., adjusting group means on the basis of pretest scores). These techniques vary from the more familiar ANCOVA to sophisticated statistical procedures for dealing with random effects such as multilevel analyses or mixed effect models (Barr et al. 2013). Another strategy is to match individuals in one treatment group with another individual in the comparison group having similar characteristics. According to Smith (1997, p. 327), finding good matches is difficult because ‘as the number of covariates increases linearly, the data demands increase geometrically’, thus requiring a large sample size to draw from.

As is the case in most educational contexts, random assignment of learners to treatment groups is usually not an option for the LINC researcher for practical reasons such as adult learners’ limited availability outside of class time and lack of space for data-gathering during class time. The use of intact classes would appear to be a necessary evil. Unfortunately, it is unlikely that the effects of non-equivalent classes can be readily compensated for through sophisticated statistics or after-the-fact matching due to the small numbers of students available and willing to participate in research at any given level. These standard solutions may only be viable in large urban centres, such as Toronto (e.g., Spada et al. 2014 was able to use multilevel modelling to analyze data from 109 participants in four intact classes).

4.2. Student Diversity

As evident in Table 1, the participants in our research case study were diverse in many ways both within and between classes, even though the students were placed at levels 5–7 on the 12-level Canadian Language Benchmarks (i.e., lower to upper-intermediate proficiency). In the Formula group, there were more participants with a white-collar background whereas the Mainstream group had a higher percentage of refugees, many of whom were stay-at-home mothers with no prior work experience or career aspirations. From a research methodology perspective, group differences that result from using intact classes introduce extraneous variables that are likely to interact with the experimental materials (Cresswell and Guetterman 2019). Such an interaction was evident in the observations made by the researcher during the Mainstream intervention. The students were to imagine being asked to work overtime by their manager. Since many of the students did not have any work experience in Canada or in their home country prior to emigration, they did not have any basis for formulating potential answers. Thus, it is likely that the employment profile of a student influenced whether that particular instructional activity was beneficial or not.

The reality of LINC classrooms is that there is usually a high degree of diversity among members of any given class with respect to their linguistic, cultural and educational backgrounds as well as employment history and immigration status. Research methods manuals point out the threat to internal validity when extraneous variables are involved, but are silent on the issue of what researchers should do when the population they are interested in is highly diverse.

4.3. Student Participation

In our case study, while the diversity of the participants can be said to reflect the kind of diversity found in the target population of LINC learners, only 13 students from about 20–25 students in each class volunteered to participate in the study. This number is far below the conventional target for experiments of at least 30 participants per treatment group, which is needed for statistical power (Fraenkel and Wallen 2003). This low participation rate by students is likely due to a number of factors. The most important of these is the LINC programming practice of continual enrolment, which causes class composition to vary from week to week. In addition, adult learners’ situational factors may lead to irregular attendance because of employment or family responsibilities; individuals may be very willing to participate but not able to attend all sessions of an intervention study. Furthermore, some adult immigrants from non-Western cultural contexts may choose not to volunteer for a study because they are unfamiliar with the idea of research or distrustful of the purpose of gathering students’ test scores. For these reasons, recruiting and retaining research participants from LINC programs is inherently challenging. Unfortunately, the consequence of the small number of participants in our case study was that it was not possible to do any kind of inferential statistics to determine whether apparent differences were statistically significant.

4.4. The Teacher Effect

In our case study, the researcher chose to have the regular teacher of each of the intact classes deliver the instructional treatment. This decision can be considered positive in that it contributed to the ecological validity of the study; but it can also be considered a negative because teachers vary greatly, and teacher characteristics are likely to influence the success of any teaching intervention (Taber 2019). The standard solution to the variability problem is to have several teachers assigned to each experimental treatment in order to neutralize bias in any particular direction (Taber 2019).

In our pragmatics case study, the teachers were both qualified, but relatively new to teaching in LINC. The Formula teacher had worked as a LINC teacher for about two years at the time of the study and had previous experience teaching English overseas as well as teaching English for academic purposes. The Mainstream group teacher had been teaching LINC for about one year. Before that, she had worked as a pronunciation and writing tutor. From this information, it seems that the Formula teacher had somewhat more L2 classroom experience than the other teacher and had herself lived in another country. It is possible that the Mainstream teacher was less well prepared to implement the activities prescribed by the researcher, which involved managing what can be characterized as ‘instructional conversation’ (Goldenberg 1991). For example, the Mainstream treatment’s instructional guide prompted the teacher to elicit from the students their ideas about similarities and differences in realizing speech acts in their L1 and in English. To implement this activity, a teacher needs to think quickly on her feet in order to incorporate students’ contributions about cross-cultural differences into the planned discussion. As a result, it might be that Zavialova’s study was actually a comparison between relatively more effective to less effective teaching rather than between two different pedagogical approaches. Unfortunately, the recommendation by Taber (2019) to increase the number of participating teachers for each treatment condition may not be a practical option in LINC research. As noted above, many teachers are relatively poorly paid and not adequately compensated for non-classroom work such as course preparation (IRCC 2020; Valeo and Faez 2013). Nonetheless, teachers are expected to invest time and energy into developing course content that meets the needs of their multilevel students and to assess their progress using portfolios (Desyatova 2020). Given such working conditions, it is likely that only exceptional teachers would welcome a researcher into their classrooms.

The discussion of the limitations of our case study suggests that ‘the strictly codified sets of values and procedures’ (Nunan and Bailey 2009, p. 84) espoused in research methods manuals set a standard that is unlikely to be attained by experimental researchers working in the context of LINC. Our reflections lead us to ponder the implications of the gap between what is considered desirable in research and what the real-world researcher has to deal with. This is the focus of the next section.

5. Implications for Classroom Researchers

From the discussion above, it follows that quasi-experimental designs using intact classes are problematic in contexts like LINC where there is typically a high level of diversity among students. Paradoxically, even though conducting an experiment in an actual LINC classroom is logically more ecologically valid than applying practices proven in other types of L2 classes (e.g., university EFL in Spain), the diversity of LINC classes is likely to lead to non-significant results, which cannot be interpreted. This is due to the effect of both systematic bias and random noise in measurement (Kahneman et al. 2021). However, research that takes the ecology of LINC into account is needed. What is the researcher to do? In the following subsections, we consider three alternative strategies that seem most appropriate for doing classroom-related research in this setting.

5.1. Strategy 1: Proof-of-Concept Perspective

The pre-to-post-to-delayed-post results from our case study support the proposal that raising awareness of pragmatic formulas is pedagogically viable. As such, our illustrative case-study offers what we might call a ‘proof of concept’. This descriptor is defined by the Oxford English Dictionary as ‘evidence (usually deriving from an experiment or pilot project) demonstrating that a design concept, business idea, etc., is feasible’. The label is used in different fields and generally ‘describes research in the beginning stages, at the cutting edge of new applications or technologies’ (Kendig 2016, p. 737). There are other studies in the pragmatics literature involving a small number of participants and lacking a control group, which could be considered proof-of-concept studies (e.g., Riddiford and Joe 2010), even though the authors do not label them as such. Although such studies are not definitive tests of L2 learning theories or pedagogy, proof-of-concept studies that are supported by relevant theory and related research findings can offer teachers concrete ideas for pedagogical experimentation. Indeed, even an evidence-based innovation that has been proven in a highly controlled lab study or in a large-scale, multi-site quasi-experiment will always have to be adapted by teachers to suit their own context and learners. Teachers necessarily filter pedagogical recommendations through the ‘sociocognitive reality of their classroom’ (Toth and Moranski 2018, p. 82).

5.2. Strategy 2: Descriptive Research

As part of her study, Zavialova (2020) carried out an interlanguage analysis of the utterances produced by the students during role-plays in order to document patterns in L2 pragmatic development. This type of qualitative analysis has been a cornerstone of second language acquisition research from its earliest days and still has much to offer. Other descriptive studies with a focus on pragmatics teaching in Canadian adult ESL have been quantitative, highlighting the pragmatic content of commercial textbooks (Diepenbroek and Derwing 2013; Ross 2018) and ESL teachers’ self-reported teaching practices (Diepenbroek 2010). Whether quantitative or qualitative, descriptive studies serve an important role in informing us about the status quo and helping to identify gaps in instructional content with respect to learners’ needs. What seems to be lacking in the literature, as far as we are aware, are systematic classroom observation studies in LINC. We need published classroom research describing teacher–student or student–student pragmatics-focused episodes in naturally occurring LINC or other diverse ESL contexts. Possible models of this kind of study are provided by the interaction analysis in EFL by Alcón-Soler (2002) and the conversation analysis of incidental discussions of culture topics in EAP classes by Lazarton (2003).

The reason why descriptive studies of LINC are needed is that research conducted in environments outside of LINC (e.g., in Spanish university level English classes) do not reflect the ecologies of highly diverse L2 classrooms. To illustrate this point, consider the technique of having students compare L1 and L2 requests, one aspect of the 6Rs approach to pragmatics instruction. In Spain it would be a fairly straightforward process for the teacher to plan and implement a cross-cultural comparison. The students and the teacher can be assumed to have shared knowledge of first language pragmatic behaviour and their previous educational experiences would be largely known to the teacher. It would also be easy for the bilingual teacher to find resources in which Spanish and English speakers’ typical pragmatic behaviour is exemplified and explained in a comprehensible way. However, to implement the same kind of lesson in the multilingual, multicultural, and multilevel LINC classroom is a more challenging instructional conversation. First of all, the teacher is unlikely to know very much about the pragmatic patterns of all of the cultural groups represented in her class. Consequently, he would have to rely on the students themselves to report L1 forms and rules for making requests. However, given their diverse educational and occupational backgrounds and variability in the individual learner’s metalinguistic awareness, the content of the lesson cannot be predicted in advance. Furthermore, L2 proficiency may limit learners’ ability to discuss their L1 social interaction behaviour. Thus, the cross-cultural comparisons that are likely to be elicited, may be difficult to organize and build upon satisfactorily for classroom discussion. However, another complexity is that the classroom meta-pragmatic discussion, whether in small groups or in the class as a whole, necessarily involves intercultural communication among students and with the teacher, which in itself can be difficult for learners to navigate. How can an ESL teacher manage such complexity? Clearly classroom research that describes expert teachers’ implementation of such lessons, with follow-up interviews using stimulated recall, would be very useful for L2 teacher education and professional development. This could also be part of the process data in a process-product intervention study and possibly the starting point for a new strand of research in instructional pragmatics.

5.3. Strategy 3: Design-Based Research

For her pragmatics study, Zavialova created and adapted pedagogical ideas based on her experience as a LINC teacher and knowledge of the instructional pragmatics literature. This combination of the practical and the theoretical is a defining feature of a relatively new idea in the educational research literature, design-based research. Also referred to as design experiments or design research, this approach asks researchers to shift their perspective and view themselves as educational engineers rather than natural scientists and study learning phenomena in the real world rather than in the laboratory (Brown 1992; Collins et al. 2004). Rather than treating educational design as the implementation of a theory, design research begins with an instructional design and studies the design in practice, progressively refining the design ‘until all the bugs are worked out’. Some of the characteristics of design research discussed in Collins et al. (2004) are:

• The goal is to refine both theory and practice;
• It takes place in messy real-life learning contexts;
• Variables are not necessarily kept constant;
• Procedures and materials are revised depending on their success in practice;
• Different participants are involved in order to contribute their expertise to the development and analysis of the design.

The examples of design experiments in Brown (1992) and Collins et al. (2004) describe investigations of instructional interventions, such as ‘fostering a community of learners’ and ‘interest-centred curriculum’. These examples are impressive in their complexity and scope, involving multiple phases of intervention, collection of both quantitative and qualitative data, and a variety of evaluation techniques. With respect to the possible relevance to the present discussion of teaching in LINC, we value the ethos of design research in its insistence on real-world learning contexts, the collaboration of experts and practitioners, and the progressive reworking of the original instructional design. This could be a game-changing methodological re-orientation for pedagogically oriented L2 research. However, whether applied linguists can access the resources needed for design-based research and whether this innovative approach to pedagogically-oriented research can fulfill its promise in diverse adult ESL settings remains to be seen.

6. Conclusions

We believe in the importance of classroom research as an essential tool for improving the effectiveness of teaching in LINC and to address the needs of adult newcomers. Indeed, it should be considered an ethical obligation given ‘the human and financial investments’ that go into language education (Stern 1983, p. 57). Stern also argued (in the same paragraph) that teachers should have access to knowledge based on ‘rational enquiry, systematic investigation, and, if possible, controlled experiment’. In this paper, we have described one attempt at conducting a systematic investigation of L2 pragmatics teaching. Although the case study results are far from conclusive, they do suggest that further pedagogical exploration of pragmatic formulas might be fruitful. The purpose of this paper has been to retrospectively scrutinize the implementation of that quasi-experiment as a case study that illuminates some of the challenges of conducting classroom research, given the diversity typical of the second language, as opposed to the foreign language, context. As such, this paper makes a contribution to what we might call ‘the how-to-do-research-in-the-real-world’ literature, where researchers recount the hidden back story of their research process (see articles by Reid 1990; Rossiter 2001; Spada et al. 2014; edited book by McKinley and Rose 2016; and Research Methods Cases available on the SAGE website https://methods.sagepub.com/Cases; accessed on 24 May 2022). This kind of scholarly reflection fulfills a need expressed by Reid (1990, p. 335):

merely reading published research does not allow the new researcher insight into the extent to which most research […] is cleaned up and manicured for publication. A new researcher might well think, as did I, that simply deciding on a research design will result in clear, clean implementation of the methodology and in unambiguous results.

We have argued that many of the ‘sanitized’ (Bryman 2007) recommendations by research methodologists are not realistically viable for LINC classroom research and have suggested some alternatives. The only way to shed light on the range of constraints and affordances that are specific to classroom life in LINC programs is to conduct research in actual LINC classes, which means confronting problems associated with multilingual, multicultural and multilevel L2 instructional settings. Our reflections here are intended to draw attention to the methodological needs of classroom researchers and to invite the instructed SLA research community to undertake the framing of suitable conceptual and practical guidelines.