Redesigning a Foreign Language Learning Task Using Mobile Devices: A Comparative Analysis between the Digital and Paper-Based Outcomes

Abstract

The introduction of mobile devices such as smartphones provides new opportunities to enhance the teaching–learning process of a foreign language. However, its use can affect not only the students’ type and form of writing but also their ability to achieve the academic competencies targeted. Thus, aspects such as the development of linguistic and communicative competencies could be affected. In this paper, a comparative analysis of two different versions (a paper and a digital version) of the same learning task is carried out in order to analyze the impact of each on the development of students’ linguistic and communicative competencies. The study is conducted with undergraduate students enrolled in a course called German II. Modern Language I (level A1.2, Common European Framework of Reference for Languages). The results obtained illustrate, on the one hand, the advantages of the paper version for developing students’ linguistic competence and, on the other hand, that the digital version, despite facilitating students’ interaction amongst each other, does not seem to be more suitable for developing students’ communicative competence. Future studies will, therefore, focus on identifying those features that might increase the potential of the digital version for fostering the targeted competencies.

1. Introduction

The widespread use in the last two decades of Information and Communication Technologies (ICTs) among students, teachers and educational institutions has triggered a continuing interest in exploring its potential, paving the way for the development of new teaching-learning environments [1,2,3,4,5,6,7].

Language teaching has not remained oblivious to this trend, moving from a face-to-face-centered teaching approach to a student-centered, blended teaching approach enriched with emerging technologies [8,9,10,11,12,13]. This change has led to the development of new teaching-learning models such as mobile-assisted language learning (MALL) that facilitate students’ interaction and negotiation with each other [14,15,16,17,18].

Bearing in mind that meaningful interaction and negotiation in the target language are paramount for language acquisition and the development of language competencies, MALL systems focused on peer-to-peer interaction and negotiation provide the ideal environment for enhancing foreign language learning (FLL) processes [19,20,21,22,23,24]. However, despite the advances made in recent years, the monitorization of teaching–learning processes remains a challenge [25,26] since many of the MALL systems currently available permit only limited access to students’ logs and thus make the analysis of students’ interaction extremely difficult.

In addition, several studies underline that the use of digital communication tools often affects both the educational objective as well as the users’ type and form of writing [27,28,29]. This becomes especially evident in the case of instant messaging, which encourages the interaction by means of short and often informal messages, the objective being rapid interaction while at the same time facilitating and prioritizing aspects such as communication and negotiation [14,30,31]. Therefore, aspects such as the development of formal language may be left unattended and thus negatively affect the development of students’ linguistic competence [29,32]. In this context, researchers have stressed the added value of the combined application of tools that encourage the use and development of formal language along with others that favor the use of informal language, prioritizing interaction and the rapid communication of a message or information [23,28,33].

With regard to language acquisition and the use of computer-mediated communication (CMC), several researchers have underlined the positive impact of both asynchronous and synchronous CMC. Asynchronous CMC enables nonsimultaneous communication as well as collaboration between different people (e.g., via e-mail, blogs, social media or wikis) [5,34]. Synchronous CMC enables the exchange of information in real-time (e.g., via chat). In a study carried out with English students, Sotillo [23] concludes that while asynchronous CMC usually enhances the production of more complex answers and utterances together with higher attention towards formal aspects [35,36], synchronous CMC favors a more personalized and fluid interaction, allowing students to improve their language learning thanks to the instant peer feedback. Regarding synchronous CMC, research has highlighted especially its potential to enhance interaction in the target language and to make more flexible use of linguistic structures [37].

Linguistic competence refers to the set of metalinguistic knowledge of a speaker, which includes lexical, grammatical and morphosyntactic knowledge and which allows the speakers of a language to express themselves correctly [38,39,40,41]. In contrast, communicative competence refers to the ability of a speaker to communicate effectively with other native or nonnative speakers [38,42,43,44,45].

With the purpose of strengthening both competencies of their German language students, the authors of the present study decided to digitize a learning task called Terminkalender that was initially designed to be performed in paper format (students had to use a pen and paper to perform the given learning task).

Terminkalender is based on a students’ exchange of written messages to negotiate and agree on various appointments regarding different leisure activities they would like to do together. Although Terminkalender had been successfully implemented in its paper format for several years with German-language learners, its paper format made it difficult to analyze in detail students’ interaction and task performance, especially when the number of course participants and, hence, messages generated was very high.

Apart from the difficulties in terms of learning analytics (LA), the authors detected other drawbacks that affected students’ task performance more directly: (1) the exchange of messages amongst students was, in general, very tedious and convoluted due to the continuous delivery of papers to arrange the different appointments, and (2) it was not easy to ensure that communication between students took place just by means of written messages and solely using the target language since they knew each other and were physically grouped together.

To address such drawbacks of Terminkalender in its original paper-based format, the authors decided to design a multiuser system that allows performing the learning task through mobile devices, allowing students to synchronously communicate with each other by sending instant messages anonymously while at the same time facilitating teachers’ the access and, hence, the analysis of students’ interaction.

On the aforementioned background, the current study aims to analyze the pros and cons of digitizing a concrete learning task (Terminkalender) that intends to help students in developing their linguistic and communicative competencies. The knowledge gathered during the study can serve to inform the design and, specifically, the digitization of effective learning tasks that focus on the development of such competencies.

For their analysis, the authors first compare the results obtained from the implementation of both the paper and the digital versions of Terminkalender and then examine the advantages and disadvantages of each tool for developing students’ linguistic and communicative competencies.

2. Method

2.1. Research Questions

For the analysis, the following research questions (RQs) will be addressed:

• RQ1: What pros and cons does the digitization of the learning task have for the development of students’ linguistic competence?

To answer this first RQ, we will examine important linguistic factors such as the correct use of greetings and farewells (e.g., Hallo, Hi, Tschüss, Bis bald! Bis dann! Bis später!), of interrogative pronouns (e.g., Wann? Was? Wo? Wohin? Mit wem? Um wieviel Uhr?), temporal adverbs (e.g., morgens, mittags, nachmittags, abends, nachts), personal pronouns with preposition (e.g., mit mir, mit uns, mit …), adjectives (genial, super, schlecht, so lala), affirmative expressions (e.g., Genial! Alles klar! Ja, natürlich! Ja, perfekt! Kein Problem!), expressions of denial (e.g., Nein, Ne, Keine Ahnung!), and the names of different places (e.g., Park, Disco, Tennisplatz, Kino), etc., in the messages created by the students and compare these factors from the paper-based learning task against the digital-based learning task.

2.

RQ2: What pros and cons does the digitization of the learning task have for the development of students’ communicative competence?

To answer this second RQ we will examine aspects such as the number of turns taken by each student, the number of messages written as well as the number of activities agreed upon either in pairs or groups of three.

2.2. Type of Research

A quantitative research approach has been used for the study since the RQs were answered based on the numerical analysis of the data obtained. Its scope was exploratory, taking into consideration that the impact of digitizing a peer-to-peer-based language learning task on students’ development of linguistic and communicative competencies has not yet been sufficiently studied. The research design was experimental and student participants were randomly divided into an experimental group (ExpGr) and a control group (ContrGr). In addition, the empirical study was carried out following an action research methodology [46,47]. According to Oates [48], the main characteristics of this methodology are:

• Concentration on practical issues: in the context of the current study, it means facilitating the analysis and accomplishment of the learning task (Terminkalender) with the idea of enhancing the development of students’ linguistic and communicative competencies.
• Iterative cycle of planning–action–reflection: in the first iteration (before developing the digital version of Terminkalender), students’ conversations on paper had to be digitized by hand for its posterior analysis. Moreover, the continuous exchange of messages on paper during the implementation of this learning task did not yield a fluid interaction and communication amongst students, leaving messages from peers sometimes unattended. This led to the planning of a second iteration and the final design of the first digital version (app) of Terminkalender. The idea was to facilitate (1) the accomplishment of the learning task and (2) its posterior analysis, thanks to the exchange of instant messaging and the automatic registration of all the information generated on the server. In subsequent iterations, this first version of the app was refined.
• Emphasis on change: the digitization of the learning task.
• Collaboration with practitioners: apart from the teaching staff and students who accomplished the learning task during the experiment, there were several student assistants involved in the analysis, design and improvement of the digitization of Terminkalender.
• Multiple data-generation methods: in addition to automatic data registration and numerical analysis, several focus-group interviews were carried out to refine the definition and fullfilment of the requirements and improve the usability of the app.
• Action results plus research results:
  • Action: the practical achievement obtained was the digitization of the learning task to facilitate its accomplishment and the analysis of the results obtained.
  • Research: the advantages and disadvantages of using each version (paper and digital) to accomplish the learning task and develop students’ linguistic and communicative competencies.

2.3. Participants and Context

The participants in the study were first- and second-year university students who were enrolled in a four-month A1.2 level (CEFR) German language course (48 h of face-to-face teaching and 109 h of non-contact hours), and the prerequisite for taking the course was to have passed the previous level (the A1.1 level). The learning task that was part of the current experiment was mandatory for all course participants. Although the initial number of participants was 52 students, 26 of which were randomly assigned to the ExpGr and another 26 to the ContrGr, there were finally only 43 students participating in the experiment (60% female and 40% male, aged between 18 and 21). This explains the difference regarding the number of participants in each group (21 students in the ExpGr and 22 students in the ContrGr). The teaching approach was based on a foreign language acquisition approach, allowing German as the only vehicle for communication in class [19].

2.4. Materials

2.4.1. Terminkalender

For the study, the authors used Terminkalender, a learning task that requires students to negotiate, agree on and record (in their respective personal agendas) a number of appointments to jointly plan several leisure time activities (e.g., going to the cinema and watching a movie, etc.).

The fact that students had to agree on a series of appointments and fit them into their agenda required them to negotiate with their peers. This way, students’ interaction in the target language is increased by favoring not only its acquisition but also the development of students’ communicative competence.

To allow students to organize their personal appointments, each student was provided with a personal agenda and a list of possible activities to perform. In the case of accomplishing the learning task using the paper version (ContrGr), each student received two worksheets, one that contained a personal agenda to complete (Figure 1) and another one with a list of activities to perform (Figure 2).

In the case of accomplishing the learning task using the digital version (ExpGr), students were provided with the same materials, which have previously been implemented in the app itself (Figure 3 and Figure 4).

In both the digital as well as the paper versions, the agenda was divided into five days (Wednesday to Sunday) and four time slots (morning, noon, afternoon, night) so that students could indicate in each time frame the following information: (1) the kind of activity they want to do, (2) the person with whom they intend to do the activity and (3) the place/city and time at which they intend to do the activity.

To arrange the respective appointments, those students who used the digital version (ExpGr) were provided with a chat-function (integrated into the same app) that allowed them to interact with the other members of their group (members were randomly assigned by the students’ language teacher). Students’ identities were previously anonymized by the system itself using random names so that they could not communicate by any other means than the integrated chat function (Figure 5).

To allow students to interact and communicate with each other via text chat, a cloud server was configured that had to run the platform (that was developed to implement the digital version of Terminkalender) and store the student data. Moreover, the platform provided the language teacher with a web interface through which she could configure the content of the learning task (i.e., type and number of activities) as well as the different task groups. Once students had completed the learning task, the language teacher had access to a log file (in plain text) that contained various kinds of information regarding the activities agreed upon by each student and the interaction that took place to negotiate and agree on each of the activities. The information gathered for each activity was the following: (1) the name of the student who proposed the activity, (2) the day and time when the activity would take place, (3) the kind of activity, place and city where it would be performed and (4) the time at which students registered the respective activity in their agenda. Regarding the chats, the list of messages exchanged between each pair of students was stored, indicating the sender, recipient, time and content of each message.

Unlike the digital version, the paper version required students to exchange their messages on paper, using as many papers as necessary to reach an agreement with their group members (Figure 6). In addition, students’ identities were not anonymous, but since the experiment was monitored by the language teacher and several assistant students, participants could not communicate with each other, except via text messages.

In the case of both versions, students had to register each of the activities agreed upon in their respective agendas. However, unlike the paper version, the digital version included a ‘check button’ (Figure 7) that allowed students to check whether the information stored in their agendas coincided with the one recorded by their peers. This way, students received immediate feedback on their task performance, allowing them to eventually revise and correct the respective information.

Terminkalender is free and available on a public forge [49,50]. It is available with an open-source license, making it easy to adapt the software according to the teachers’ needs. The software source code can be found at the following forge: https://github.com/javosuher/Terminkalender (accessed on 2 June 2022).

2.4.2. TKStats

To analyze the logs registered by the platform, which had been developed to run the digital version, the authors implemented TKStats, a tool that allows the automatic extraction of quantitative information from each group. Such information included (1) the number of words used by classifying them according to total words, number of words in the dictionary and number of words studied in class, (2) the number of sentences and turns used by each student and (3) the number of tasks that were negotiated as well as those that were successfully agreed upon.

The TKStats software is available at the URL https://github.com/luisrozo/tk-stats (accessed on 1 June 2022).

2.5. Procedure

To conduct the current experiment, the students of the ContrGr and those of the ExpGr were distributed on the same day and at the same time in different classrooms. Likewise, in each group, randomized task groups with four to six members were formed. To accomplish the given learning task, both groups were assigned the same time (76 min) and had to employ the same kind of materials, although provided in different formats (text messages, a personal agenda and a list of different activities to choose from).

The task groups included in the ContrGr (paper version) were physically grouped, and their identities were known to facilitate the exchange of messages. In addition, the different messages on paper were stacked next to each recipient who had to reply to each of the messages received by either accepting the proposal made by its sender or suggesting some kind of change regarding the activity, place or time proposed for doing such. Unlike the students of the ContrGr, those of the ExpGr (digital version) were randomly distributed by the language teacher throughout the classroom, keeping their identities anonymous.

Prior to the experiment, the participants of each group (ContrGr and ExpGr) were asked to take part in a 15-min pilot test to familiarize themselves with the materials and tools provided.

2.6. Statistical Analysis

To determine if there is a statistically significant difference between the means of the two groups (ContrGr and ExpGr) the parametric independent Student’s t-test was used. Given the size of the groups (n₁ = 22; n₂ = 21), the normality of the dependent variables was checked using the Shapiro–Wilk test. The Levene’s test was applied to test the equality of variances. An adjusted t-statistic based on the Welch method was used when the group variances were unequal.

Additionally, when the normality assumption of the dependent variable was violated, the nonparametric equivalent Mann–Whitney U test was used.

The hypotheses were tested using a significance level (α) of 0.05. Furthermore, an effect size was calculated to measure the magnitude of the difference between groups. To measure the effect size, Hedges’s g statistic was used. For the Mann–Whitney U test, the rank-biserial correlation (r_rb) was considered as an effect size.

The data were obtained from the interaction files using TKStats. Moreover, statistical analysis was performed using IBM SPSS Statistics 24 and JASP 0.14.

3. Results

To better understand the results obtained, the data are divided into different categories. The latter is used for the analysis of students’ conversations, taking into account (1) the information conveyed, (2) the intensity of the interactions that took place and (3) the object of negotiation. The categories grouping dependent variables are the following: words, sentences (phrases), messages, turns, activities negotiated in pairs, activities negotiated in groups of three and linguistic aspects.

3.1. RQ1: Linguistic Competence

In order to answer RQ1, the analysis focused on those aspects that permit assessing students’ language proficiency according to the parameters set by the CEFR for level A1.2 [51].

Especially interesting for the analysis were (1) the correct use of specific vocabulary and expressions related to the field of leisure time and everyday situations (e.g., expressions used to plan to go with another person to the cinema or a concert; expressions used to ask for and express an opinion of a film or a music group; expressions used to ask for and indicate the time, day and place of an appointment and jointly perform a certain activity, etc.) and (2) the type and complexity of the syntactic structures that students used (noun and verbal phrases, adjectival phrases, interjective phrases, etc.).

Considering that Terminkalender required students to negotiate and agree amongst peers on a series of activities to be carried out jointly (expressing preferences, disagreements or even the lack of availability regarding a certain activity, place or time slot), the analysis especially focused on linguistic aspects such as the following: the correct use of adjectives and adverbs, personal pronouns, local, temporal and conceptual prepositions that are used in combination with personal pronouns (e.g., mit mir, mit uns, etc.); the use of interjections and idiomatic expressions (e.g., Neee!, Keine Ahnung!, Tut mir leid!, Alles klar!, Cool!, Perfekt!, etc.); and finally, the type and complexity of the structures used (sentences and questions with a single word, sentences and more complex questions).

Table 1. Descriptive statistics and hypothesis testing results in order to answer RQ1.

		ExpGr (Digital) n1 = 21		ContrGr (Paper) n2 = 22
Category	Variable	m	sd	m	sd	Hypothesis Testing
Words	Total Words (W)	321.19	73.472	290.27	69.387	t (41) = 1.42, p = 0.163, g = 0.425 *
	Different Total Words (D_W)	148.67	33.356	147.00	35.813	U = 257.5, p = 0.527, rrb = 0.115 ***
	Correct Words (CW)	265.57	59.932	262.32	63.091	t (41) = 1.17, p = 0.863, g = 0.052 *
	Different Correct Words (D_CW)	109.48	21.044	126.05	32.628	U = 168.5, p = 0.132, rrb = −0.271 ***
	Incorrect Words (IW)	55.62	32.185	27.95	7.749	U = 364, p = 0.001, rrb = 0.576 ***
	Different Incorrect Words (D_IW)	39.19	22.402	20.95	5.038	U = 361, p = 0.002, rrb = 0.563 ***
Sentences	Total Sentences (S)	142.38	42.545	83.95	17.089	t (26) = 5.86, p < 0.001, g = 1.769 **
	Sentences of a Single word (S1W)	70.33	33.081	13.09	7.578	t (22) = 7.74, p < 0.001, g = 2.341 **
Linguistic aspects	Interrogative Pronouns (IP)	13.19	6.104	13.82	5.662	t (41) = −0.35, p = 0.728, g = −0.105 *
	Different Interrogative Pronouns (D_IP)	3.71	0.717	5.27	1.956	U = 109.5, p = 0.002, rrb = −0.526 ***
	Adjectives (AD)	1.24	1.578	3.59	2.404	U = 90.5, p < 0.001, rrb = −0.608 ***
	Different Adjectives (D_AD)	0.62	0.669	1.45	0.858	U = 106.5, p = 0.001, rrb = −0.539 ***
	Greetings or Farewells (GOF)	9.43	4.044	9.41	4.194	U = 230, p = 0.990, rrb = −0.004 ***
	Different Greetings or Farewells (D_GOF)	1.90	0.889	3.91	2.245	U = 96, p < 0.001, rrb = −0.584 ***
	Idioms (ID)	0.43	0.811	1.73	1.518	U = 112.5, p = 0.002, rrb = −0.513 ***
	Different Idioms (D_ID)	0.29	0.561	1.23	1.066	U = 112, p = 0.002, rrb = −0.515 ***
	Affirmative Expressions (AE)	20.57	6.079	14.05	6.418	t(41) = 3.42, p = 0.001, g = 1.024 *
	Different Affirmative Expressions (D_AE)	3.71	0.845	5.55	2.632	U = 128.5, p = 0.011, rrb = −0.444 ***
	Expressions of Denial/Doubt (EOD)	8.52	4.332	7.86	4.853	U = 261, p = 0.472, rrb = −0.130 ***
	Different Expressions of Denial/Doubt (D_EOD)	2.19	0.814	3.32	1.673	U = 129.5, p = 0.010, rrb = −0.439 ***
	Places (PL)	7.00	5.282	7.14	3.299	t (41) = −0.10, p = 0.919, g = −0.031 *
	Different Places (D_PL)	3.67	1.906	4.73	2.004	t (41) = −1.78, p = 0.083, g = −0.532 *
	Days of the Week (DOW)	2.00	2.191	2.95	3.154	U = 199.5, p = 0.441, rrb = −0.136 ***
	Different Days of the Week (D_DOW)	1.43	1.363	2.14	2.077	U = 194, p = 0.363, rrb = −0.160 ***
	Temporal Adverbs (TA)	8.76	5.029	9.64	5.678	t (41) = −0.53, p = 0.596, g = −0.160 *
	Different Temporal Adverbs (D_TA)	6.57	3.091	7.59	3.750	t (41) = −0.97, p = 0.338, g = −0.291 *
	Time (TI)	3.81	4.008	8.95	5.103	U = 76, p < 0.001, rrb = −0.671 ***
	Different Time (D_TI)	2.05	2.747	5.27	2.694	U = 65, p < 0.001, rrb = −0.719 ***
	Preposition ‘mit’ (PM)	6.48	3.473	7.27	3.667	t (41) = −0.731, p = 0.469, g = −0.219 *
	Preposition ‘mit’ with Pronoun (PMP)	1.00	1.183	1.36	1.840	U = 229, p = 0.969, rrb = −0.009 ***
	Different Prepositions with Pronoun (D_PMP)	0.67	0.658	0.59	0.796	U = 255, p = 0.526, rrb = 0.104 ***

Note. * Student t-test, ** Welch t-test, *** Mann–Whitney U test. Significant results are in bold. Note. For the U test, effect size is given by the rank biserial correlation. Note. For the t-test, effect size is given by the Hedges’s g.

shows descriptive statistics and parametric/nonparametric independent test results that help to answer RQ1. The results revealed significant differences between the ExpGr and ContrGr.

The most remarkable result is that the ContrGr employed a greater variety of words/expressions in 7 of 11 linguistic aspects considered, with no significant difference between both groups for the remaining four aspects. Thus, the Mann–Whitney test showed that the following linguistic aspects were significantly higher for the ContrGr than for the ExpGr: a variety of interrogative pronouns employed by the learners to require a piece of information (D_IP), number of different adjectives (D_AD), ways of greeting or saying farewell to a person (D_GOF), number of different idioms (D_ID), diversity of affirmative expressions (D_AE), variety of expressions of denial/doubt (D_EOD) and the total of different expressions used to ask for and indicate the time (D_TI).

However, there were no statistically significant differences between both groups regarding the diversity of words/expressions used when indicating a place (D_PL), day of the week (D_DOW), using temporal adverbs (D_TA) or the to go preposition ‘mit’ with a pronoun (D_PMP).

Extending the scope of analysis to include the number of words/expressions used, including repetitions, there were also some statistically significant differences in favor of the ContrGr. In this sense, the Mann–Whitney test showed that the number of adjectives employed (AD), the number of idioms employed (ID) and the sum of expressions employed both to ask for as well as indicate the time (TI) were greater for the ContrGr than for the ExprGr.

Only one linguistic aspect was significantly higher in the ExpGr. Therefore, an independent t-test showed that the number of affirmative expressions (AE) was much higher for the ExpGr than for the ContrGr. Hedges’ g suggests that this is a large effect.

Figure 8 shows significant differences between groups regarding the variables discussed above.

Concerning the correct language use (Figure 9), the Mann–Whitney test indicated that the number of incorrect words (IW) and the diversity of incorrect words (D_IW) was significantly higher for the ExpGr than for the ContrGr.

Concerning the complexity of the sentences used, based on the number of words employed, there were statistically significant differences between both groups (Figure 10). An independent t-test showed that the number of sentences of a single word (S1W) was much higher for the ExpGr than for the ContrGr. Hedges’ g suggests that this is a huge effect. Levene’s test indicated unequal variances (F = 21.88, p < 0.001), so degrees of freedom were adjusted from 41 to 22.

3.2. RQ2: Communicative Competence

To answer RQ2, both the intensity of the communication and its effectiveness in successfully agreeing on an appointment were analyzed.

Table 2. Descriptive statistics and hypothesis testing results in order to answer RQ2.

		ExpGr (Digital) n1 = 21		ContrGr (Paper) n2 = 22
Category	Variable	m	sd	m	sd	Hypothesis Testing
Messages	Total Messages (M)	138.48	41.916	81.23	15.823	t (25) = 5.87, p < 0.001, g = 1.774 **
Turns	Total Turns (T)	84.19	17.679	37.41	7.417	t (27) = 11.22, p < 0.001, g = 3.387 **
Activities (in pairs)	Total Activities in Pairs (AP)	8.19	1.436	6.86	2.315	t (35) = 2.27, p = 0.029, g = 0.676 **
	Pending Activities in Pairs (PAP)	0.67	0.913	1.09	0.921	U = 166, p = 0.095, rrb = −0.281 ***
	Agreed Activities in Pairs (AAP)	3.05	1.802	3.41	1.790	U = 205.5, p = 0.536, rrb = −0.110 ***
	Not Correctly Agreed (NCAAP)	4.48	2.015	2.36	1.432	t (41) = 3.98, p < 0.001, g = 1.191 *
Activities (in trio)	Total Activities in Groups of Three (AT)	1.95	1.024	3.32	1.555	U = 104.5, p < 0.001, rrb = −0.548 ***
	Pending Activities in Groups of Three (PAT)	0.38	0.590	0.73	.827	U = 180.5, p = 0.169, rrb = −0.219 ***
	Agreed Activities in Groups of Three (AAT)	0.57	0.746	0.45	0.671	U = 249, p = 0.625, rrb = 0.078 ***
	Not Correctly Agreed Activities in Groups of Three (NCAAT)	1.00	0.949	2.14	1.424	t (41) = −3.06, p = 0.004, g = −0.918 *

Note. * Student t-test, ** Welch t-test, *** Mann–Whitney U test. Significant results are in bold. Note. For the U test, effect size is given by the rank biserial correlation. Note. For the t-test, effect size is given by Hedges’s g.

shows the results obtained in the experimental test that help to answer this research question.

With regard to the first aspect, an independent t-test showed that the number of messages sent (M) was much higher for the ExpGr than for the ContrGr. Hedges’ g suggests that this is a large effect. Levene’s test indicated unequal variances (F = 19.26, p < 0.001), so degrees of freedom were adjusted from 41 to 25. Similarly, the number of turns (T) was greater for the ExpGr than for the ContrGr. Hedges’ g suggests that this is a huge effect and Levene’s test indicated unequal variances (F = 6.82, p = 0.013), so degrees of freedom were adjusted from 41 to 27. Figure 11 shows the differences between both groups regarding these variables.

An independent t-test showed that the total number of activities negotiated in pairs (AP) was greater for the ExpGr than for the ContrGr. Hedges’ g suggests that this is a medium effect. Levene’s test indicated unequal variances (F = 6.84, p = 0.012), so degrees of freedom were adjusted from 41 to 35. Conversely, a Mann–Whitney test showed that the total number of activities negotiated in groups of three (AT) was higher for the ContrGr than for the ExpGr.

A Mann–Whitney test showed that there was no statistically significant difference in the number of activities agreed upon in pairs (AAP) between the ExpGr and the ContrGr. There was also no significant difference regarding the number of activities agreed upon in groups of three (AAT) between the ExpGr and the ContrGr.

The single most striking observation to emerge from the data comparison was that an independent t-test showed that the number of activities that were not correctly agreed upon in pairs (NCAAP) was much higher for the ExpGr than for the ContrGr. Hedges’ g suggests that this is a large effect. By contrast, the number of activities that were not correctly agreed upon in groups of three (NCAAT) was much higher for the ContrGr than for the ExpGr. Hedges’ g suggests that this is a large effect.

4. Discussion

4.1. RQ1: Linguistic Competence

The current study results indicate that those students who accomplished the learning task using the digital version (ExpGr) made more writing mistakes than those who accomplished it using the paper version (ContrGr). This leads to the assumption that making more mistakes could be related to the added difficulty of using a virtual keyboard, the lack of experience with using the digital version, and the use of a more informal language (usually linked to communications via text chat). Another factor that might have caused the higher number of mistakes in the ExpGr could be the pressure students felt when there were several threads of open conversation in which activities were negotiated.

Terminkalender in its digital format encourages faster and, therefore, less reflective communication during which language learners make more mistakes, paying less attention to formal language aspects, and use fewer complex structures (e.g., nominal phrases, adjectival phrases, prepositional phrases, etc.). This finding suggests that the digital version is less appropriate for developing students’ linguistic competence than the paper version.

Likewise, the results of the study seem to reinforce the claims of previous studies [52,53,54,55] that observed differences at the cognitive level between students who accomplished a learning task by handwriting and those who used a virtual keyboard. Handwriting seems to help those who write the words themselves to focus more attentively on the words they write, while writing by means of a virtual keyboard often divides the attention of those who write between controlling and correctly choosing the letters and words from the keyboard to compose a sentence. The latter can finally affect students’ cognitive process and result in worse retention of words [52].

In line with the analyzed data, the authors of the current study can confirm that the ContrGr employed a greater variety of words and expressions from those previously used in class than the ExpGr. This means that the ContrGr employed a much richer and more diversified language compared to the ExpGr. The results, therefore, suggest that the paper version could be more adequate than the digital version for applying and practicing the lexical, grammatical and morphosyntactic knowledge previously acquired in class.

Regarding the complexity of students’ writing, it stands out that students employed shorter sentences when they accomplished the task using the digital version. The fact that the digital version invites students to use simpler syntactic structures (based on nominal, verbal, or adjectival phrases, etc.) seems to indicate that the digital version encourages its users to prioritize fast and immediate communication while, at the same time, paying less attention to formal language aspects. This can be an argument against (cons) the digital version, since excessive simplification of the syntactic structures used could be an obstacle to the development of students’ linguistic competence.

4.2. RQ2: Communicative Competence

The analysis of students’ logs has further shown that students’ communication was more intense and fluid when they used the digital version to accomplish the given learning task. The number of turns and messages exchanged between students from the ExpGr was considerably higher than that from the ContrGr. Consequently, students from the ExpGr were exposed to more significant interaction with their peers and thus obtained more opportunities to negotiate and exchange information in the target language than those from the ContrGr. Therefore, the data suggest that the digital version provides more suitable conditions for developing students’ communicative competence.

Nonetheless, despite being exposed to more significant interaction with their peers, students from the ExpGr were not more successful in arranging their appointments than those from the ContrGr. Bearing in mind that communicative competence refers to the ability of a speaker to communicate effectively with other native or nonnative speakers [56], the results suggest that the digital version is not more effective than the paper version for solving the given learning task. This implies that the digital version of Terminkalender—against the authors’ initial assumption—does not enhance students’ communicative competence more than the paper version does.

4.3. Critical Reflections and Future Research

The results from the study generate several questions that need to be addressed in future research. Those questions are: (1) When and in which learning scenarios should digital interventions be implemented?; (2) what would be the best outcome, one where all interventions are digitized or one where only some of them are?; (3) how do we know exactly in which conditions we should exploit the advantages of the digital interventions and in which we should better use the analogous ones?; (4) can we improve digital interventions so that we can overcome their weaknesses?; (5) what type of studies are required to answer these questions? and finally, (6) based on the results and observations obtained from the present study, how could we improve the current digital version of Terminkalender to guarantee better learning outcomes?

The current research has stressed once again the importance of thoroughly analyzing, prior to any kind of digital intervention, the possible strengths and weaknesses of the targeted learning scenarios and tools. A key factor in this process is to follow an iterative cycle as proposed by action research-based methodologies [48,57,58]. An iterative cycle consists in doing research based on different phases that are (1) exploring, (2) identifying, (3) planning, (4) collecting data, (5) analyzing/reflecting, (6) hypothesizing/speculating, (7) intervening, (8) observing, (9) reporting, (10) writing and finally, (11) presenting [57,58]. The idea behind this iterative cycle is to explore and analyze both users’ needs as well as requirements to meet such needs by means of a user-centered design process.

Further steps to evaluate whether a digital intervention and concrete software meet the requirements to trigger better learning outcomes might consist in gathering feedback from the users themselves. A valuable method for such purposes is the one proposed by the Technology Acceptance Model (TAM) [59], which aims to analyze aspects such as the perceived ease of use, the usefulness of a concrete technology, and the intention to use it.

With regard to Terminkalender and the intention to improve its digital version, we aim to analyze next the results from the TAM handed out to our German language learners once they have participated in the learning experience. This way, we aim to gather valuable feedback on their learning experience and possible difficulties with the activity and the learning tool. At the same time, we intend to identify the need to refine the system and software and thus to ensure that in future interventions, learners might be better supported by the system and feedback provided to develop their language competencies.

4.4. Limitations of the Study

Finally, it is worth mentioning that although the results obtained are based on objective data extracted from the students’ logs, several aspects could affect the conclusions of the study: (1) the sample of the study was limited to the students from a very concrete university and foreign language course, hence hindering the extrapolation of the results obtained; (2) the average age of the sample was limited to students around 20, and thus students felt immediately comfortable with using the digital version; (3) the design of the digital version itself. The fact that Terminkalender was developed following popular instant messaging applications such as WhatsApp or Telegram [60] does not imply that the interaction mode chosen is the best for developing the different competencies targeted by the current study; (4) students’ linguistic and communicative competencies were solely analyzed in a very limited context, that is, the analysis was based on students’ performance of solely one concrete learning task. To draw stronger conclusions from the study, a more longitudinal intervention focusing on one or more learning tasks is needed. Moreover, an additional exam testing both competencies separately, some days after the intervention, might have allowed us to obtain a more precise picture of each of the competencies developed and to draw some stronger conclusions on the pros and cons of the digital version when targeting concrete competencies.

5. Conclusions

Learning a foreign language through mobile devices offers the possibility of implementing learning dynamics focused on peer-to-peer interaction and negotiation that can hardly be carried out in conventional classroom settings. This also applies to Terminkalender, a learning task that was initially designed to be performed using conventional tools such as a pen and paper but which was later digitized to facilitate both the use of LA and the teaching-learning process itself. By digitizing and redesigning the learning task using mobile devices, language teachers can now access students’ logs and analyze them later [61], whereas students can easily interact with other peers in the target language and register the required information on their digital agendas. However, the results from the study have shown that the digitization of Terminkalender, apart from affecting students’ performance, could also affect, both positively and negatively, the development of the academic competencies that are intended to be achieved.

In this sense, the authors have analyzed the pros and cons of using the digital version of Terminkalender (ExpGr), as opposed to its paper version (ContrGr). For their analysis, the authors focused on two different research questions (RQs).

Regarding RQ1, the data obtained suggest that the paper version is more suitable for developing students’ linguistic competence, considering that the number of writing errors was significantly smaller when students accomplished the learning task using a pen and paper than when using a virtual keyboard. Likewise, the results indicate that those students who used the paper version generally employed more complex structures, writing a higher and more diversified number of words and expressions from those previously seen in class. The results, therefore, suggest, on the one hand, that the use of conventional tools, such as a pen and paper, are not necessarily less efficient than using digital tools for developing concrete language competencies and, on the other, that synchronous text chat interaction probably is not the best option for developing students’ linguistic competence. Unlike this, asynchronous learning tools and environments, such as wikis, forums, etc., might provide better opportunities for developing students’ linguistic competence since they allow more form-focused reflection and the use of more complex language structures. However, bearing in mind the advantages of digitizing conventional paper-based learning tasks both to facilitate ubiquitous learning as well as LA, and thus formative assessment, the current study reinforces the need to explore other software solutions than the ones used for the current study capable of providing students with instant feedback on their language use. This way, digital learning tasks such as the ones developed for the present study could help students better develop their linguistic as well as communicative competencies.

Concerning RQ2, contrary to the authors’ initial assumptions, the data of the present study indicate that the digital version of Terminkalender does not seem to be more suitable than the paper version for developing students’ communicative competence. This was especially noteworthy taking into account that students who used the digital version, despite interacting more with each other and thus initiating more negotiations in the target language, did not arrange more appointments compared to those students who used the paper version.

Although further research is necessary to dig deeper into the reasons for this phenomenon, the results of the study suggest that those students who used the digital version and thus interacted via text-chat might have struggled with attending many chat-conversations at the same time. In this sense, students might have encountered certain difficulties coordinating themselves efficiently with their peers when agreeing on an appointment. To minimize such difficulties and drawbacks, future interventions should probably have longer training sessions to allow students to become more familiar with the respective technology and its functionalities.

Furthermore, to identify the possible weaknesses of the digital version of Terminkalender against its paper version, a first step in exploring ways to improve the digital version is to identify how students perceive its use. Therefore, a user evaluation of Terminkalender such as the one described by the Technology Acceptance Model [59] is foreseen to be valuable.

Aspects that could be analyzed in this regard are those which refer to students’ perceptions in terms of the app’s utility and ease of use, as well as to students’ attitudes towards using the digital version for such purposes. Moreover, the authors intend to explore and design additional computer tools for the implementation of other learning tasks that might be keener to enhance students’ linguistic and communicative competencies while at the same time facilitating language teachers in the subsequent analysis of students’ learning results.

In conclusion, despite the limited results for developing students’ linguistic and communicative competencies and the need to further develop the software used, the implementation of the digital version undoubtedly favors several fundamental aspects that could enhance and facilitate the teaching and learning process: (1) the use of LA and scalability to a high number of students; (2) the possibility of implementing the learning task out of class, hence during the autonomous learning process; (3) the use of the target language as the only vehicle for communication with other peers and (4) the provision of instant peer feedback and thus the possibility of revising and renegotiating an appointment, if necessary.