Research on the Development of a Proctoring System for Conducting Online Exams in Kazakhstan

Round 1

Reviewer 1 Report

The paper presents the implementation and testing of a proctoring system implemented in a University in Kazakhstan. The article does not sufficiently motivate the interest of the system. Regarding the commercial systems it is compared to, it seems that it has less functionality. So what is the point of such a system? What is the point of developing yet another proctoring system if it has no innovative features compared to the existing offer? The implicit argument seems to be that of developing one's own proctoring system for a University. If this is the case, it should be made explicit and justified and the article revised in that direction. The article could then have an interesting contribution to the state of the art. The argument of cost that is put forward remains to be demonstrated. The cost of a subscription to a commercial service should be compared with the cost of developing and maintaining a system and its operating costs in terms of material, resources and human costs. It seems to me that another interesting argument to exploit would be that of the digital sovereignty of a university. On the issue of proctoring, which involves sensitive data on students, this issue of digital sovereignty seems very relevant.

The comparative study with commercial systems should be found after Section 3. It is not very coherent to present the comparison of a system before having described its implementation and functioning.

A technical study should be added to analyze the hardware resource requirements: server, storage, cpu and especially the capacity to scale up according to the number of simultaneous exams and students.

Specific information on certain points should be provided and quantified. For example, what does "rather low Internet speed" mean precisely? At least an order of magnitude.

User testing with teachers needs to be presented in a more rigorous manner with all results.

The discussion and conclusion really need to be more elaborate and specific about the characteristics of the system developed. As it stands, the claims mentioned are those that can be applied to all the many existing proctoring systems, such as it "can increase the frequency of online tests by reducing staff costs and other costs associated with time allocation".

The tables of functions of the algorithms as well as the picture of the audio in figure 4 are useless and do not add anything to the understanding.

Use the bullets for lists. Otherwise, the reading is tedious.

Author Response

DEAR REVIEWER

Thank you for your comments, we have worked them out and strengthened the text description. Your given remark is important for our research and we have eliminated them according to the recommendations you have indicated.

Author Response File: Author Response.docx

Reviewer 2 Report

Accept as is

Author Response

Dear REVIEWER

Thank you for your comments.

Best regards Shakizada Niyazbekova

Author Response File: Author Response.docx

Reviewer 3 Report

This paper presents a surveillance system that is in its pilot phase. It seems to be a system that is being tested in a university. So, it is a good option for such university. However, as a product of scientific research, no major contribution to the field was detected.

 The title of this paper mentions the development and pilot implementation of a surveillance system. However, the development of the system is not observed and therefore it is recommended to change the title.

The introduction does not provide enough background on the main issue which is surveillance systems. There is too much redundant talk about distance learning and its benefits. This is most noticeable from line 93 onwards, where the advantages of distance learning are mentioned, but within each point there is no detailed explanation. For example, when explaining the progress of technology, the following paragraph does not explain this point in detail. There are also overly repeated points such as the limitations of time and space that are broken by distance education.

 In the literature review section, there is only a single reference to the main topic of the paper on line 189 and this is a self-reference. Furthermore, the citation on line 176 contains important information on the topic but is not adequately summarized. The other references cited only support the topics mentioned in the introduction which are mostly related to distance education.

 In section 3, three commercial systems are mentioned that should have been explained in detail in the literature review. The paragraph starting on line 211 is not clear about its purpose. Abruptly the Proctor SU system appears without explaining its background. In line 216 there is an allusion to an analysis but only the characteristics of the system are presented. The concept of functional features is presented, but this concept is not explained. From line 233 onwards it would be better to use a list of points. Line 239 is very badly worded. The section entitled on line 248 should have been explained earlier to understand what the Proctor SU system is. Also, it is mentioned that the image processing is done on the client side, but it is not justified why and if it is the best option. In line 259 it is implied that there are other requirements but it is not mentioned exactly what these are. In general, this section presents some predefined algorithms without any observable contribution.

The results section firstly presents the results of the system in a qualitative way with respect to the experience of the system. Within this result, it is concluded that the team that tested the system obtained good results with a low internet speed, but the number of the internet speed is not mentioned. It is also mentioned that the system is easy to integrate into Moodle, but it does not explain in detail the necessary features for such integration. In the subsection on line 440, it is mentioned that 4 teachers found several bugs but it is not mentioned which ones. Likewise, there is no mention of the testing conditions. The next subsection of line 467 is presented as the results of the face recognition but only the tests for face detection are observed. Furthermore, it mentions a conclusion already reached in the self-referenced work (line 476). Table 3 presents another two models but the features used are not presented. It is mentioned that the basic YOLO model processes images in real time at 45 frames per second but it is not specified whether on any particular computer architecture.

The discussion section is presented but there is no such discussion of the work. It looks like a recapitulation of the systems involved in distance education and a conclusion of the tested system.

The work presents several writing errors such as the repetition of a word in a statement, as in line 44, 102 The misuse of punctuation marks as in line 68, 83, 91. and statements that are not understood as in line 113, 242, 252, 259, 310.

Author Response

DEAR REVIEWER

Thank you for your comments, we have worked them out and strengthened the text description. Your given remark is important for our research and we have eliminated them according to the recommendations you have indicated.

Author Response File: Author Response.docx

Round 2

Reviewer 1 Report

By orienting the study as a feasibility proposal for the development of an open source solution for universities, the article takes on a more suitable orientation for publication. But there's still a lot of work to be done before it can be published. The title should be modified to reflect this change. The necessary maintenance costs should be clarified: hardware and personnel to maintain the solution, to manage it, to provide it as a service in a University. You also need to provide performance comparisons for commercial software, and not just comparisons of features and costs.

The article is still very messy and difficult to read and grasp the important elements. The article still needs editing to clarify the whole: reduce, avoid repetition, simplify, structure.  I don't see the point of providing the algorithms. Figure 4 is of no interest. Not sure figure 5 and 6 are useful. Figure 7 is not legible.

The Unit testing section is of no interest. It would be more interesting to provide the detailed and precise results of this evaluation phase: "A mock-up of an online test was prepared, and members of the research team tested and tried to cheat the system, for example: use a mobile phone, open a new browser tab, talk to someone in the room, view notes in a book, move away from the screen, etc. The reports on the violations were recorded and discussed with the developers." 

References to YOLO (1) and other models should be provided: CNN and R-CNN. Furthermore, the comparison does not seem useful; it has already been carried out and should simply be referenced (1). YOLO v3 dates from 2018, and there is a v4, v5, v6 and even a v7 (3). Why not use them? 

1) https://pjreddie.com/darknet/yolo/

2) https://medium.com/ibm-data-ai/faster-r-cnn-vs-yolo-vs-ssd-object-detection-algorithms-18badb0e02dc, https://iopscience.iop.org/article/10.1088/1742-6596/1453/1/012139/meta

3) https://www.mdpi.com/1424-8220/22/2/464, https://www.v7labs.com/blog/yolo-object-detection, https://learnopencv.com/performance-comparison-of-yolo-models/

Author Response

Dear REVIEWER
Thank you for your comments, we have worked them out and strengthened the text description.

Author Response File: Author Response.docx

Reviewer 3 Report

This revised manuscript lacks a review table where authors must submit a list of changes or a rebuttal against each point that is raised when the revised manuscript was submitted. Authors must highlight where the corrections were effected on the revised manuscript. A complete point-to-point responses must be included in the revised manuscript file within the online submission system. 

I think any revision submission can not be proccesed if point-to-point responses are missing or incomplete in the revised manuscript file within the online submission system.  

Authors must make a significant effort to satisfy the reviewers' concerns in the revision since it is most likely that a final determination about this paper. 

The writing errors have been corrected. 

Author Response

Dear REVIEWER

Thank you for your comments, we have worked them out and strengthened the text description.

Best regards Shakizada Niyazbekova

Author Response File: Author Response.docx