In Search of Alignment between Learning Analytics and Learning Design: A Multiple Case Study in a Higher Education Institution

Round 1

Reviewer 1 Report

Today, I had an opportunity to review the paper entitled 'In search of alignment between learning analytics and learning 2 design: A multiple case study in a higher education institution'. In this paper, the authors present a multiple case study at one higher education institution to unveil instructors' pilot efforts in using an open AF system to align LA and LD within their unique contexts, with the goal of delivering personalized feedback and tailored support. The study found that instructors successfully aligned LA with LD for personalized feedback through checkpoint analytics in highly structured courses. Moreover, teachers relied on check-point analytics as an evaluation mechanism for evaluating the impact.

Overall, this is a well-written paper and easy to follow. The authors start the paper with an introduction to learning analytics and learning design to provide context to the study. They later review the literature on personalised feedback with learning analytics and how learning analytics can be used to support student learning. The authors used this literature to justify knowledge gaps and the importance of their study. The authors also provide a detailed methodology section, and the use of 3 cases made the study's findings to be robust. 

However, there are some issues that the authors should consider before the paper is published.

Introduction: 

In the first two paragraphs, the authors define the concepts of learning analytics and learning design. However, the term feedback is used without a clear explanation. This is a very big concept, and to make it clear to the reader, please clarify what you mean by feedback or LA feedback in this paper. Does it mean the visualisations? metrics from dashboards? Please clarify. Also, when you talk about automated feedback, please be specific. Is this feedback pre-determined by the teachers and embedded in an LA system? or its just metrics provided by the dashboard eg number of papers submitted visualised in a table, etc..

In the introduction and some other sections of the paper, you interchangeably use the abbreviation of learning analytics (LA), while in other sections, you use (LA). Please revise for consistency.

The authors provide a list of LA and learning design frameworks. They later highlight that these were selected in the interest of the present research. Where all these frameworks relevant to the study? If so, the authors should be explicit about how they were all used; otherwise, just include a statement highlighting which one was used and why.  

Across the paper, the authors keep switching between LA and LD or LD and LA. The question is, what relationship are you interested in? How does LA inform LD? or how does LD inform LA? From what I read, it seems the focus is on how instructors use LA to inform the learning design, which also includes personalised feedback. If this is the case, the authors should be explicit and consistent.

Research questions: Following the previous comment, if the relationship is LD and LA, then RQ1 should possibly be revised to show this. Maybe how instructors leverage LA to inform different LD decisions e.g. personalised feedback,.....only a suggestion. 

The authors use learning design and instructional design interchangeably. The two terms are different and should not be combined unless explicitly defined.  

Methodology:

The authors mention interviews as one of the data collection methods. Similar to what you do for interviews, please include the actual questions asked other than statements. 

Also, no information about the participants and the way data from the three sources was analysed. For example, interviews, artefacts and surveys. Please add some details about the analysis. 

Figure 4 is not clear. Kindly provide a clear figure with a better resolution.

Findings and discussion

I am a bit confused about how the authors are organising the findings. While describing the cases, some results are presented. How do you classify these? From what I see, the section named results and discussion should be called discussion, and then split the methods section into two. The results from the three cases and then the methods. 

Overall, I congratulate the authors for a well-written article and for attending to a lot of details in this first draft. I look forward to reading the revised article.

Author Response

We are very grateful to Reviewer 1 for their feedback on our submission and thoughtful suggestions regarding specific aspects of our manuscript. We have now provided a point-by-point response to the reviewer's comments in the attached Word file. 

Author Response File: Author Response.pdf

Reviewer 2 Report

The paper tackles the issue of the relationship between learning design and learning analytics. 

By presenting a multiple case study carried out within a higher education institution, the authors would like to unveil the impact and reactions of teachers and students to the use of an open automatic feedback to align LA and LD.

The topic is interesting and the paper is well structured and well presented.

The authors might consider widening the spectrum of references cited regarding the alignment between LA and LD. For example, they might want to consider a couple of special issues on this topic published by the British Journal of Educational Technology in 2015 and 2018 respectively.

Moreover, the authors might consider the possibility to describe more extensively the way they selected the 3 cases and might elaborate on the possible risks of bias, depending on the profiles of the educators involved.  

Author Response

We would like to thank the Reviewer for their positive feedback about our submission. We also appreciate the suggestions to improve our manuscript, and respond to these point-by-point, in the attached Word document.

Author Response File: Author Response.pdf