Personas Design for Conversational Systems in Education

1

The paper is well-written and has a   clear structure.

Thank you very much. We have also went   through the paper again several times and proof checked it.

2

My concern with this paper is that I   do not see a clear contribution - is it the different analysis methods? is it   the application on the specific domain (students)? or? this needs to be   clarified.

We   went through the Conclusion (Section 6) and the Introduction (Section 1) and   presented the contributions of the paper in a clearer format.

3

You end with six personas, and as far   as I can read it you do not combine the 2 studies, but add them together -   and is this a sound way to create personas? Arguments for your decisions are   needed.

Thank   you for your comment. We have added a new Discussion section (5) where we   discuss the results from the two studies and justify how the two data   analysis are combined to create the personas.

4

On page 14 you write: 3) motivations   and interests, including motivations and interest in using new technology,   and 4) skills and experience, including skills and experience in using   chatbots and mobile applications. Where does data for these areas come from?   Please clarify.

Thank you for mentioning this important   point. We were going to collect these data in the second iteration. However,   to clarify and make the persona template easier to understand we removed the   motivation, interest and experience from the current version of the paper   (Section 4 was amended as appropriate). We kept the skills data as it can   currently be retrieved from the module outline. Thus, Table 7, Figure 18 and   Figure 19 were changed as appropriate. An explanation of the skills data are   added to Section 4 (just before Table 7). At the moment, all of the data for   the persona template are retrieved from the data analysis. We look to expand   into including and collecting data on motivation, interest and experience as   part of future work.

5

Please   describe what a persona-based conversational system is – in my view personas   are a design tool to create a conversational system, which does not make it   based on personas.

The   authors have added the following clarification to page 3 line 25 to 28

“Personas   are a design tool to create a conversational system. However, in this study,   the conversational system will interact differently based on the students’   persona type, which means that the conversational system is based on persona   and it is referred to as a persona-based conversational system.”

6

Page 4, line 25 “data-driven personas”   I guess this refer to quantitative data driven persona elicitation. Most   personas are based on data - but qualitative data. The quantitative   data-driven approach should by reflected in the literature review, see e.g.   Salminen et al.

On page 3 paragraph 4 (Section 1), we   have added more information on personas that are based on qualitative data.   We have also reflected from the literature on studies that are based on both   qualitative and quantitative personas, and how our study focuses on quantitative   data-driven personas.

7

The use of the singular and plural   form of persona(s) is confusing in the introduction it is as if you end with   one persona, when in reality you end with six. I suggest to call it   "personas" all the way through.

We went through the whole paper and made   sure that we changed the singular persona to plural personas unless there is   a need to keep it as singular such as definition or quotation.

8

Depending on the contribution the   paper would be highly improved by adding the evaluation by experts.

Due to   the scope of this study, we were not able to perform the persona evaluation   research as it requires few months’ work to collect and analyse the data and   the scope of this paper will be too large. However, we have highlighted in   the paper the need to perform the persona evaluation as part of future work   (both in Section 4 and Section 6).

1

This is a well written and presented paper. There   are some improvements that could be made however.

Thank you very much. We have also went through and proof checked the   paper several times.

2

In the introduction, you state that students   like engaging with chatbots more that learning F2F with their lecturers. It   would be beneficial to support any assertions similar to this with evidence   in the form of prior research from the literature. Please check for this   throughout the paper.

We went through the introduction section, in particular, this sentence   and added two references to support this claim. We also, went through the   paper and backed up any similar claims with references from the literature.

3

It would be useful (again early in the   introduction) to provide a 'lay man's' explanation of K means clustering   method and analysis. This would be useful for non-statisticians - as the   results of the research will be of wide interest to HE academics involved in   pedagogic practice. Student engagement is a significant and growing problem   (lack of). It would be useful to provide some more evidence for this in the   literature review of related work - brief summary of some significant studies   that highlight this problem - both in the UK HE sector and also worldwide.

In the introduction, where k-means was first mentioned we explained the   technique in ‘lay man’s’ terms. We have also went through the paper and   updated section 3.2 and added more information about k-means, and explained   the algorithm of k-means in more details.

In regards to the student engagement comments, we went through the   paper and updated the introduction to include more literature and   explanations on the latest UK and worldwide student engagement research.

4

Can you provide a few more examples of the use   of chatbots (more detailed) in other disciplines besides computer science -   i.e. business and management, or medicine perhaps?

We   went through the introduction section and provided few paragraphs about using   chatbots in other domain such as health and entertainment. Also, we explained   the uses of chatbots for information retrieval, customer services and   assistance.

5

In your methodology section - can you perhaps   link up the DSR method more to the K clustering technique used - in terms of   the persona elicitation and development. Who else has used this DSR and K   clustering together?

We went through the methodology section (Section 3) and added a paragraph   (paragraph above Figure 1) that links the design science approach to k-means   clustering.

Also, we added a paragraph about other study that uses DSR and k-means   clustering and included it in the related studies section (Section 2, the   last paragraph).

6

An appreciation or acknowledgement of the   limitations of the study with regards to only measuring one variable for   outcomes (grades) would be useful. How could this be made more representative   or convincing?

We went through the conclusion section and highlighted this limitation   (as well as other limitations) that we would like to cover as part of our   future research (5^th paragraph of Section 6).

7

Finally, it feels that there is a missing   discussion section - before the conclusions in section 5. A little more on   how this research now moves forward to link with student engagement factors   and then chatbot technology would be useful. It would seem that Chatbots are   highlighted at the start of the paper - but not really discussed again in the   discussion or conclusions.

We   went through the paper and added a discussion section (section 5) where we   discuss the results from the two studies and justify how the two data   analysis are combined to create the personas. We explained how we linked the   three iterations together. In addition, we went through the discussion and   conclusion sections and discussed/linked chatbots in more details.

8

Overall, a very interesting and relevant paper   - and it will be very useful for development of new forms of pedagogic   practice that use VLEs and Chabot technology - in addition or in place of F2F   traditional teaching.

Thank you very much for your feedback. We hope that this study   alongside our future improvements can help with the development of new forms   of pedagogical practices that relies on VLEs, Chatbots etc…instead of traditional   teaching approaches.