Can Google Translate Rewire Your L2 English Processing?

Round 1

Reviewer 1 Report

The authors explore the extent to which the use of the output of Neural Machine Translation (NMT) affects the cognitive processing of English as a second language. This study expands the work presented by Resende, Cowan, and Way (2020) by involving more participants as well as by assessing both the implicit and explicit nature of learning. A third phase was added to the experiments in order to establish whether a learning effect takes place in addition to the priming effect.

The authors challenge current academic thought on the use of translation in L2 learning and teaching. The paper establishes that not only can an NMT system, namely Google Translate (GT), be used as a tool for language learning, but it can also facilitate learning by influencing the cognitive processing of L2 syntax.

The authors show through a well-designed experiment that syntactic priming also happens from exposure to MT output and Brazilian Portuguese speaking English L2 learners. It is the continuation of a former work and, as such, not totally novel, but more detailed and with a larger number of participants. The methodology is sound, mostly clearly described. Against the common view, machine translation can have a positive and long-term effect on L2 learners. The possible application of this finding is exciting.

The question about the frequency of MT usage should be made more precise. What does frequent usage mean? Once per day or once per week? This is a weakness in the design of the experiments.

Why were the recordings manually transcribed? How about automatic speech recognition?

Table 1 (Age range of respondents) is not relevant for the priming experiments.

Table 7: the columns should be reordered from left to right: Baseline (Pre-test) Priming Post-test; and the row headings "NO" and "'S" need to be described => Figure 6 uses labels "OTHER" and "S", as Figure 7 uses labels "NO", "OF" (instead of "PNP"?) and "S"
What is the difference between Table 7 and Table 8? Percentage of NP structures produced (by participants?) in table 7 does not correspond to table 8... This is not clear to me!
Table 9 is not clearly described, the final formula is not explained; and is the p-value at line 555 really p<.001? or p<0.01? what does the first row of Table 9 (Intercept - Baseline) mean?

I don't think you are observing a "cross-linguistic" priming effect. All the priming is in English. If the priming was in Portuguese and had an effect on English, I would consider this to be cross-linguistic.

Notes, comments, typos, etc.: Please see the attached file.

Reference

Resende, Natália, Benjamin Cowan, and Andy Way. 2020. “MT Syntactic Priming Effects on L2 English Speakers.” In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, 245–53.

Comments for author File: Comments.txt

Author Response

First of all, we would like to thank you for the complete review of our article. All of his comments were of great relevance to the overall improvement of the article. Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

* Summary

This paper investigates whether the use of MT can influence the learning of a second language (L2), focusing on the case of Brazilian Portuguese L2 English speakers.

The investigation consists of a survey followed by a syntactic priming experiment. The survey involved 90 subjects and aimed to collect information on whether, how and for what purposes they use MT systems. The priming experiment was  carried out on 32 subjects with the goal of assessing if Google Translate (GT) can influence the use of a specific syntactic alternative in subsequent speech, namely "noun phrase" (NP) instead of "prepositional noun phrases" (PNP), given that NP is the structure mostly used by GT while PNP is that initially preferred by subjects.

Results are presented in terms of percentage of use of NP/PNP in different stages of experiments; a statistical analysis based on a Linear Mixed-Effects Model is also presented.

* Clarity - In general, the paper is well written and clear. Apart some minor oversights listed below, I found some parts of Section 3.2 not totally clear; details are given in "General Comments" section.

* Originality - Authors admits that "results presented here replicated our previous findings as well as results of a number of studies in the field of Psycholinguistics" (lines 594-595). Anyway, a respectable amount of innovativeness is given by the observation that the influence of MT in L2 syntactic processing is long-lasting and then that MT-human interaction is effective in L2 learning. Moreover, according to what authors claim, "there are no studies to date that have used the syntactic priming methodology as a vehicle for investigating the processing and representation of Brazilian Portuguese syntax in the context of a human-artificial system interaction" (lines 282-285).

* Meaningful comparison - I'm not an expert of Syntactic Priming nor of Psycholinguistics, but it seems to me that the comparison with related work is precise and complete. On the other side, a discussion of limitations of the investigation is missing.

* Soundness - The collection of data from the survey and the organization of primining experiments are both correct. Rather, it is my opinion that one of the claims of the paper is weak, or at least not clearly supported, namely that on the correlation between the  use of "NP structures" and the English proficiency of participants. I discuss on that in "General Comments" section.

* General Comments

Above, I mentioned two main weaknesses of the paper, one related to the clarity of the presentation, one to the soundness of one claim.

Starting from the (un)clarity, I found some parts of Section 3.2 not totally understandable and others that could be easily improved. 

In the description of Baseline pre-test (lines 401-415), I'd suggest to underline better how and why experimental and filler trials differ: if I'm not wrong, the only difference is that the subject of the source sentence in one case can be translated in PNP/NP, in the other case is always a simple noun; is that correct? If I'm wrong, that's a warning about the fact that the description should be clearer. If it is correct, since the goal of fillers is to prevent "participants of guessing what the experiment was measuring" (line 405), I wonder why the fillers are presented after the experimental trials. Wouldn't it have been better to present filler and experimental trials in random order?

The same holds for the Primimg test phase: why didn't you present priming and filler trial triples in random order?

Let's know consider Section 3.2.5. First of all, I would prefer to read an initial explicit statement on what you are measuring: I think it is the use of PNP/NP in experimental trials of the post-test phase, right? Please, state it explicitly or whatever is correct.

That said, I have some questions/comments concerning Table 7:

- the sorting of columns, "Priming pre post", is a bit weird; wouldn't "pre priming post" be more natural?
- "NO" (and actually even "'S") is not defined
- why does not the sum on columns give 100%? Which are the other structures used by participants? A clarification is needed, and also a comment on the fact that the sum is << 100% 

The discussion in lines 489-491 compares only pre vs. priming, but in my opinion it is more informative to see what happened in post-test phase, don't you agree?

Finally, I would expect to find a link between numbers in Tables 7 and 8, while I do not see any correspondence. Please, could you add a comment on that?

Let's now consider the second weakness of the paper, that regarding the soundness. In Figure 5 it is shown a linear regression (do you confirm?); please, could you provide the "R squared" or whatever you want to measure the reliability of what you claiming? In fact, imagining to remove the first  and the last values (that could be considered as outliers), the claimed trend is not so evident. In addition: What does the darker gray area around the straight line represent?

Moreover, in my opinion what is stated in lines 499-505 is not clearly supported by Figures 6,7. In fact, I found those figures not easy to read; a description of what bars/areas represent would be desirable. In addition, why the companion figure for the pre-test phase is not provided?

A final remark regards the fact that nothing is said on the correspondence between the statistics collected from the survey participants and those from participants of priming experiment. Is the population of the experiment representative of the total (survey) population? Can you comment on that, please? Moreover, I wonder why some of the infos collected from participants are not considered at all in the analysis of the results (e.g. the age); I see that numbers would lose any statistical validity, but in a scientific paper if a value is shown it must provide useful info to the reader, otherwise it's better omit it, avoiding frustrated expectations.

* Recommendation

I think that the paper is interesting and all in all deserves to be published, provided the weaknesses I have mentioned are amended.

* Detailed/minor suggestions/comments

lines 39-40: In Way [6], he continues: -> In [6], he continues:

49: business^1. -> business.^1 [in general the number of footnotes goes after the punctuation marks]

94: the TM output -> the MT output

135-136 ... processing in both in the first and second language -> ... processing both in the first and second language

164:  cocaine).The images ->  cocaine). The images 

175: processing.This -> processing. This 

223: prime.Traxler -> prime. Traxler

233: of the amplitude of the P600 component  -> "P600 component" I do not know what it is and I think the same holds for the standard reader

270:  preference effect[26] ->  preference effect [26]

In Section 2, the same concept of "inverse preference effect" is mentioned and discussed twice:

   151-153: Some researchers call this effect the “inverse preference effect” and they claim that the most uncommon structures drive the priming effect [26].

   269-270: This effect is known in the literature as the inverse preference effect[26].

   -> I suggest to merge the two parts

305: Participantion  -> Participation 

305: voluntary.The -> voluntary. The

306: The first set of questions were created -> was created

306: demographic data 1 and -> what does "1" stand for?

Tables 2/4: I'd prefer a different sorting: "Adv Int Beg" (or the opposite)

330: dysplayed -> displayed

330-331: As dysplayed in Table 5, 50 respondents (55,5%) out of 90, answered Yes to this question. -> Table 5 shows other stuff and I do not find any table reporting those values. Please, fix.

Figure 2: At this point of the paper, I do not understand what the words "item" and "filler" outside the boxes stand for. If it is intendend to show just that there are actual trials and filler trials, the figure is not clear, especially considering that the meaning of the diagonal arrow is mysteryous. I understood better going on with the reading, but it would be desirable to allow the reader to understand it earlier.

416 Priming test.From -> Priming test. From 

469 experimental -> it is an adj, not a noun; use "experiment(s)"

541 were cantered. -> ???

549 shown in Table . -> 549 shown in Table 9.

Table 9: stars probably indicate the level of significance, but it would be desirable to explicit it

Table 9: Prime2 -> Prime 2

560-561 This interaction confirms boxplots (Figures 6 and Figure 7) -> actually, in Figgs 6 and 7 I didn't see anything

596 HCI literature (e.g.[30–32,35]. -> missing closed parenthesis

663 Dublin City University -> Dublin City University"

Author Response

First of all, we would like to thank you for the complete review of our article. All your comments and suggestions were of great relevance to the overall improvement of the article. Please see the attachment.

Author Response File: Author Response.pdf

Round 2

Reviewer 2 Report

Only minor suggestions this round:

• Table 7: separate better the two groups of phases (day 1/2), either by a line or a vertical space
• line 530: but the did not produced ->  but they did not produce
• Figure 6: ok as far as the contents, but the form could be improved: much blank room, no horizontal gridlines, uncentered...
• line 541: interestingly -> Interestingly
• line 709: a popular system, in future... -> a popular system; in future...

Author Response

Please see the attachment

Author Response File: Author Response.pdf