Research on the Development of a Proctoring System for Conducting Online Exams in Kazakhstan

Abstract

The demand for online education is gradually growing. Most universities and other institutions are faced with the fact that it is almost impossible to track how honestly test takers take exams remotely. In online formats, there are many simple opportunities that allow for cheating and using the use of outside help. Online proctoring based on artificial intelligence technologies in distance education is an effective technological solution to prevent academic dishonesty. This article explores the development and implementation of an online control proctoring system using artificial intelligence technology for conducting online exams. The article discusses the proctoring systems used in Kazakhstan, compares the functional features of the selected proctoring systems, and describes the architecture of Proctor SU. A prototype of the Proctor SU proctoring system has been developed. As a pilot program, the authors used this system during an online university exam and examined the results of the test. According to the author’s examination, students have a positive attitude towards the use of Proctor SU online proctoring. The proposed proctor system includes features of face detection, face tracking, audio capture, and the active capture of system windows. Models CNN, R-CNN, and YOLOv3 were used in the development process. The YOLOv3 model processed images in real time at 45 frames per second, and CNN and R-CNN processed images in real time at 30 and 38 frames per second. The YOLOv3 model showed better results in terms of real-time face recognition. Therefore, the YOLOv3 model was implemented into the Proctor SU proctoring system.

1. Introduction

The COVID-19 pandemic has changed the lives of large numbers of students, teachers, and parents around the world. Universities around the world are in need of distance learning tools as they battle the COVID-19 pandemic. The pandemic has been a catalyst for the rapid integration of distance assessment into education.

Distance education is a modern education based on the use of two-way interactive modern information and communication technologies; its technical basis is mainly built upon computers, multimedia, and modern communication technologies. Distance education not only uses the three main networks of computers, communications and digital satellites, and computer multimedia technologies, but also includes second- and even first-generation distance education technologies, such as audio recordings, videos, and printed materials. Its main technical characteristics and advantages are two-way interaction, that is, using digital devices, multimedia, and other information technology to achieve human–computer and interpersonal communication and interaction. It can enhance two-way communication between teachers and students and greatly promote the interaction between students personalized and cooperative learning [1,2,3]. Modern distance education is a new form of education brough about through the development of modern information technologies, and it is the main means of building a system of continuous learning for people in the era of the knowledge economy, going beyond the boundaries of time and space enabling people to take classes anytime and anywhere. By taking advantage of various educational resources, it provides convenient, fast, and comprehensive educational services for various learning objects.

Modern distance learning technologies are more widespread and include radio and television, video education, cable television, video conferencing, satellite television, teletext, computer multimedia, interactive network systems, etc. With the creation and improvement of national and global information infrastructure, as well as the growing popularity of information technology and the constant enrichment of online learning resources, the third generation of distance education has achieved a leap forward, and the quality and efficiency of education has improved significantly.

The main feature of online education is that teachers and students are geographically separated, yet they can fully communicate at the same time. Online education based on shared resources has created an unparalleled form of collaborative learning. Students work together to support each other in solving problems, sharing information, building knowledge, and socializing. Collaborative learning helps to stimulate students’ motivation, generate a variety of ideas and strategies, and increase their intelligence. Online education can realize global collaborative learning and contact with people from different cultural backgrounds, which promote mutual respect, mutual trust, and teamwork.

In the educational process, a proctoring system is used to control the completion of an exam by students. Proctoring is an effective solution to prevent cheating and other violations by students. Compliance with the principles of academic honesty is the most important factor in achieving the target learning outcomes of training programs for students. In the article, we considered the development and pilot implementation of an automated proctoring system for online exams.

To achieve the goal, the following tasks were set in the article:

(1)

describe the architecture of the Proctor SU proctoring system;

(2)

Describe the prototype of the Proctor proctoring system SU;

(3)

Review the proctoring systems used in Kazakhstan;

(4)

Compare the functional features of selected proctoring systems;

(5)

Show test results of Proctor proctoring system SU;

(6)

Conclusions on the development of the Proctor SU proctoring system.

2. Literature Review

Proctoring is the most effective method for protecting exam integrity [4]. Numerous studies [5,6] show that some subjects cheated on the online AP (Advanced Placement) exam, as did many college students on academic tests [7]. In addition, cheating occurred not only in online tests but also in the offline exam, even with the involvement of observers.

New information and communication technologies are gradually penetrating the educational field, but there are still some limitations in the use of these technologies due to the difficulties in terms of the accessibility of the tools used by students. As more students take online courses as part of college curricula, the integrity of online testing becomes more important. It is generally believed that the likelihood of cheating in exams is higher in an online environment. One approach to compensate for the absence of a physical observer is to use a remote proctoring service that electronically monitors the student during the exam period [8]. Proctoring is unlikely to result in a statistically significant difference in test scores, and thus it does not appear to cause any change in student performance. A proctored exam does not affect student academic achievement and grades [9]. Online exams have become widely used to assess student learning achievement in recent years, especially during the COVID-19 pandemic. Online exams are more vulnerable to various fraudulent activities, which can undermine their credibility [10,11].

Exams are important components of educational programs as well as online learning programs. In the exam, proctoring is a method of detecting and reducing the possibility of cheating is very important for students to learn this material. Distance learning allows teachers to use information technology to deliver the curriculum to students who are separated from their teachers. As higher education has evolved, distance learning enrollment continues to be fraught with issues of academic dishonesty. A proctoring system can provide educators with a practical and cost-effective approach to reducing academic dishonesty [12]. Remote proctoring with cutting-edge digital technology can help overcome pandemic-, geographic-, and resource-based constraints for mentoring and training. Proctoring or online supervision has experienced a significant rise in recent years in educational institutions in general, as well as in the university context. The worldwide trend is due to many reasons: the desire of universities to internationalize and provide international content leads to the need to offer exams for students living in different countries of the world. Another reason is students who are often absent, such as competitive athletes or students with musculoskeletal disabilities, who are sometimes unable to take the exam on campus. Exams from home—or anywhere in the world—make the testing process more flexible and personal and often create a more comfortable environment for students than the rigorous conditions in a lecture hall. However, there has been only limited experience with online proctoring in many aspects: in addition to technical requirements and the management of security-critical aspects of data transmission and storage, there are questions concerning the data protection of sensitive personal information, such as audio and video recordings during an exam. In addition, fraud prevention outside the relatively easily controlled on-campus lecture halls requires new, often technical solutions [13].

The COVID-19 pandemic has disrupted higher education in many ways, such as causing the shift to emergency distance online learning, often including the move to online assessments and exams. With evidence of growing academic dishonesty in unsupervised online assessments, institutions have been looking for ways to ensure academic and institutional integrity and their reputation. Thus, many institutions have chosen and implemented online proctoring solutions. In the midst of the COVID-19 pandemic, higher education institutions have been heavily impacted in terms of cost, usability, and efficiency when choosing an online proctoring solution to ensure academic and institutional integrity. Student privacy was either considered in terms of data protection and transparency or was not considered at all [14].

A new multimodal method of automatic online monitoring using a combination of image processing, sound processing and PC monitoring methods was proposed by the authors [15]. The system uses intelligent facial recognition algorithms to detect any suspicious movements or actions and report to the instructor in real time. This serves as a viable solution to reduce the likelihood of cheating and allows teachers to simultaneously monitor a large group of students during an online exam [16]. The authors investigated methods based on computer vision to offer a quintuple control mechanism for online tests. The features include authentication, head movement, eye movement tracking, and speech and object recognition. The overall accuracy of the system is 91% [17]. In the article, the authors propose an online proctoring system that ensures confidentiality. The proposed image caching system can detect excessive movement of the student’s face and body (i.e., anomalies) caused by the student’s attempt to cheat during the exam. Detection can be performed even if the student’s face is blurred or hidden in video frames [18]. The authors of the proposed system monitor the behavior of candidates and enter them into the database. The framework provides automated monitoring of candidates’ exams. The framework of the proposed system is built on a client–server system with HTML, CSS, and JavaScript in the external interface, and Flask processes the server part [19]. The proctoring system proposed by the authors includes six main components that continuously evaluate key behavior signals: user, text, and voice detection, as well as active window detection, gaze evaluation, and phone detection. By combining the components of continuous assessment and applying a time sliding window, the authors developed higher-level functions to classify whether the test taker is cheating at any time during the exam [20]. Proctoring technology includes facial recognition technology, which belongs to the field of artificial intelligence and biometric recognition. This is a very successful application of image analysis and understanding [21]. When developing AIP, it is necessary to consider various psychological, cultural, and technological parameters. This document systematically examines the existing proctoring systems based on artificial intelligence, as well as systems that are not based on artificial intelligence. The authors considered four questions of the main research, which focused on the existing MIPS architecture and the parameters that need to be taken into account for AIPS, trends, problems, and its future. The main issues include security and privacy concerns, ethical considerations, trust in artificial intelligence-based technologies, insufficient training of technology users, cost, and more. Students can be monitored using hardware settings, such as a gaze-tracking devices [22], which can also be achieved by teaching AIPS to implement when a user has taken a screenshot [23,24].

Educators have quickly moved to remote learning with the use of proctoring programs for their testing needs due to both the COVID-19 pandemic and the growing virtualization of the education sector. Three key issues arise when using these complex programs: exam honesty, the objectivity of the exam procedures, and the safety and confidentiality of the examinee [25].

Many universities in the Republic of Kazakhstan did not develop distance education and found themselves in a difficult situation during the pandemic. Each university or college has independently determined the format of the exams or decided to postpone them to a later date. The pandemic has shown that the Kazakhstani education system urgently needs modernization as well as readiness to switch to an online format. The last two years have become difficult for the education system of the Republic of Kazakhstan. The coronavirus and the subsequent transfer of training to a distance format demonstrated and confirmed the importance of the proctoring procedure for the university.

3. Materials and Methods

This study was conducted at the S. Seifullin Kazakh Agro Technical Research University, which is among the 10 essential universities of the Republic of Kazakhstan. Its current student population is 13,000 students (37 bachelor’s degree specialties, 31 master’s degree specialties, and 14 Ph.D. doctoral specialties).

Due to the transition to the distance learning format, Kazakh universities had to solve the issue of remote work with students. To do this, proctoring systems were used that tracked their actions. As practice shows, there may be vulnerabilities in the systems, the main of which is the safety of students’ personal data. The digital sovereignty of the university is an important factor, meaning the ability to independently determine and conduct its information policy and form and manage information resources and infrastructure, as well as ensure information security. Electronic sovereignty provides resistance to cyber warfare, i.e., protection from cyberattacks, network attacks, various viruses, leaks, and the theft of personal data, as well as from external influence on the critical infrastructure of the university [26,27]. In some cases, network attacks can lead to the almost complete destruction of websites; hence, electronic security is the basis of electronic sovereignty and includes the protection of information with the help of modern digital technologies.

3.1. Risks of Using Automated Proctoring Systems

Proctoring systems allow users to conduct exams in a remote form. In such systems, as a rule, it is required to initially pass an identity check and provide remote access to the user’s computer. The data collected during the exam in real time are sent to a remote server, which, in addition to storing information, processes it using artificial intelligence algorithms to look for violations. Like all systems involving the use of artificial intelligence, this form of organization of the educational process is subject to significant risks:

(1)

Academic risks: the use of proctoring forces a change in the procedures used when conducting exams, complicates their technical equipment and simplifies the tasks of the exam, which is caused by a reduction in the time to think about answers and compliance with restrictive measures. Additionally, not all exams can be carried out in a remote format, especially if the lack of teachers’ preparation for the introduction of remote forms of the educational process is taken into account;

(2)

Financial risks related to the budget of educational institutions associated with the use of commercial systems since there is no market for free proctoring systems;

(3)

Information security risks associated with the transfer of personal information and device data to external structures that manage proctoring systems, despite the documented guarantees regarding the protection of such information. A certain risk is the acquisition of remote access by the proctoring system to the student’s computer, which, in the conditions of using non-certified commercial systems, makes it possible to use broader-than-stated and undocumented opportunities;

(4)

Opposition on the part of students associated with the perception of proctoring systems as an invasion of artificial intelligence into personal life, which, together with certain negative features of the worldview, enhanced capabilities of modern information and communication technologies, provokes students to search for ways to circumvent the limitations of systems of this kind.

Thus, such systems used to monitor educational activities are far from ideal, requiring a responsible attitude in terms of their use and guaranteeing the rights of students.

We offer the Proctor SU system. Creating your own proctoring system reduces academic risks, financial risks, and information security risks. The Proctor SU system is less functional when compared to other commercial systems; however, the Proctor SU, developed on the basis of artificial intelligence, is very cost-effective for the university. The Proctor SU proctoring system provides the most necessary functionality; in the future—with help from university scientists and developers—it will be possible to improve this system. The university can use proctoring systems created by them not only in exams but also in online courses.

3.2. Architecture of Proctoring System: Proctor SU

This article presents the automated proctoring system: Proctor SU. The proctoring system uses a B/S architecture (browser/server). The server part was implemented using the Python programming language and Django framework, and it determines the identification of students and the state of change in their behavior, including their faces, mouths, and the sounds they make. The browser side sends the most important data related to changes in the behavioral state of the subject to the server. The browser side also collects and analyzes user data. Face and voice recognition are supported by open-source software (Versions 1–3 of YOLOv3 were created by Joseph Redman and Ali Farhadi, Brandenburg, Kentucky, United States of America).

The proctoring system works as follows: before the start of each exam, the communication and computer parameters are automatically checked for compliance with the minimum technical requirements, such as a working webcam and microphone, a network bandwidth of at least 1 Mbit/s, a network response time of no more than 1000 ms, etc.

The exam will not start until these requirements are met. During the exam itself, the student’s webcam, desktop, and additional camera are broadcast and recorded, and violations are automatically tracked. At the beginning of each session, a photo of the student’s face is automatically taken, which later goes through the identification procedure and the percentage of similarity of the face in the session with the photo from the database is determined.

Throughout the exam, continuous verification of a person’s identity when at their computer is carried out, which is based on automatic detection and facial recognition methods and analyzing keyboard input. The proctor can observe the students assigned to them, see their broadcasts, and see notifications of recorded violations. If necessary, the proctor can contact the student via video or chat. Additionally, the proctor can interrupt the exam at any time and indicate the reason in the comments of the session.

After the exam is completed, an assessment of the degree of confidence in the results of the exam is formed as a percentage, and an interactive protocol is formed with a video recording of all broadcasts, a history of correspondence, and the minute-by-minute details of the recorded violations. Figure 1 shows the architecture of the proposed Proctor SU proctoring system.

To create the Proctor SU system, the IID (Iterative and Incremental Development) method and solution strategies were chosen. The following stages of the IID method were used:

-

requirements analysis and design;

-

Coding, testing, and implementation.

The database was created using the Django framework and PostgreSQL. There are three types of user profiles in the database: administrator, student, and teacher. Each user is granted separate access to log in and register. In addition, this database has the full logic of the test system and the user ID and, consequently, performs actions to access and perform certain user functions.

The technologies used in the development of an automated proctoring system are shown below:

-

Frontend (client side): HTML5, CSS3, and Javascript;

-

Package version control: git. Frameworks: Vue and Vuetify;

-

Package managerNpm, Yarn;

-

Backend: Python, Django, and Django Rest framework;

-

Database: Sql and PostgresSql.

The facial recognition and student identification system includes the following blocks:

-

a block for building an object recognition model (searching for the coordinates of the human surface in the figure, determining the location of information, pre-processing, and normalization);

-

The object recognition authentication block (authentication algorithms for the object to be identified by controlling access to the photo recognition system of the user registered in the database);

-

A block for calculating information identification marks (convolutional neural networks, correlation indicators, Minkowski distance, etc.).

The proposed Proctor SU system architecture includes the functions of face recognition, face tracking, sound capture, and the active capture of system windows.

YOLO (You Only Look Once) is a real-time object detection algorithm, which is a single deep convolutional neural network that splits the input image into a set of cells forming a grid; therefore, unlike image classification or face detection, each grid cell in the YOLO algorithm in the output data will have an associated vector that tells us whether an object is present in a grid cell, the class of the object (i.e., the label), and the expected geometric characteristics of the object (location).

3.3. Prototype

Proctor SU is an automated system used for performing online exams, allowing users to conduct online exams using a comfortable, non-dangerous, and economical method. The program allows teachers to easily create and distribute prepared tests and for students to take them. In order for users to interact with the Proctor SU system, users can simply access the system in their web browsers. There is no need to install any part of the application. The architectural scheme of the automated proctoring system—Proctor SU—consists of 4 modules.

Module 1: Teacher (home page, list of students, create exam, and results). Exam results (test results, reports, and webcam recordings).

Module 2: Student (home page, available exams, exam results, and student profile).

Module 3: Proctor (home page, list of test results, webcam entries, and reports with violations).

Module 4: Administrative part (authorization and administrator’s personal account).

Proctor SU offers a user interface for creating, launching, completing, and monitoring online exams. To create an exam using Proctor Ex, teachers must create an account and log in. Figure 2 shows a student’s page in the Proctor SU system with a navigation panel for “Exams” and “Results”. The student can view the scheduled exams, completed exams, and exam results. On their page, the student can view the names of the subject, the duration of the exam, the start and end date of the exam, and the status of the exam (Figure 3).

If suspicious activity is detected, the system sends a warning to the proctor. Any form of identification is displayed in the table “Exam status in real time” on the teacher’s exam page. There are 7 different types of Proctor SU alerts (tab switching detection, screen splitting, student looking down, student looking to the right, student looking to the left, student absence on webcam, and the detection of several people on the webcam), which can be marked by the proctors. If any of the above actions are detected, the application immediately informs the proctor about it.

3.4. Comparison of Kazakhstan’s Proctoring Systems

Currently, there are 3 commercial proctoring systems in Kazakhstan (OES, Oqylyq, and Aero). None of the 120 universities in Kazakhstan have its own proctoring system; therefore, currently, Kazakh universities use these 3 commercial proctoring systems to organize and monitor the progress of exams.

These commercial proctoring systems help educational institutions, enterprises, and businesses conduct reliable online testing, reduce organizational costs, and save time by providing an advanced online proctoring system (

Table 1. Kazakhstan proctoring systems.

Proctoring System	Official Website, E-Mail	Date of Creation	Founders	Integration with Various LMS Platforms
OES	https://oes.kz/ (accessed on 15 May 2023). E-mail: [email protected]	2018	Oes * Company	– Univer 2.0 – Canvas – Sirius – Moodle – Indigo – Platonus
Oqylyq	https://oqylyq.kz/ (accessed on 15 May 2023). E-mail: [email protected]	2018	Alumni of Astana Hub 7.0 **	– Moodle – Canvas – Univer – Platonus
Aero	https://proctoring.aero.kz/ (accessed on 15 May 2023). E-mail: [email protected]	2019	Alumni of Nazarbayev University ***	– Moodle – Platonus

OES * is a private company in Kazakhstan. Astana Hub 7.0 ** is the largest international technopark of IT startups in Central Asia. The main audience of Astana Hub is representatives of the IT market: startups, investors, programmers, and students of IT specialties. Nazarbayev University *** is an international English-speaking research university located in Astana. The activity of Nazarbayev University is based on the principles of autonomy and academic freedom. The autonomous status was assigned to the University in accordance with the Law of the Republic of Kazakhstan dated 19 January 2011 “On the status of “Nazarbayev University”, “Nazarbayev Intellectual Schools”, and “Nazarbayev Foundation”.

).

During the exam, the OES, Oqylyq, and Aero systems automatically fix violations, such as checking the serviceability of equipment, fixation of the head turning to the sides, fixation of attempts to copy and paste text, the absence and replacement of the user, blocking the right mouse button, fixing the presence of third-party software, fixing extraneous noise and voice, browser and desktop control, and face recognition in the process of testing with face fixation and voice detection. All 3 systems have a proprietary license. The features and technologies of Kazakhstan’s proctoring systems are shown in

Table 2. Features and technologies of Kazakhstani’s proctoring systems.

Proctoring System	Delivery Options	Features	Number of Simultaneous Exams	Payment for the Exam
Oes	(1) Cloud Software as a service (SaaS) is provided. The Inga proctor system works in the cloud with internet access. Monitoring, maintenance, and updating are carried out by the vendor. Data are stored on cloud platforms: Google Cloud, AWS (Amazon Web Services), Microsoft Cloud, etc. They comply with all GDPR (General Data Protection Regulation) standards and use complex encryption with transmission over a secure channel with SSL certificates. (2) Box The proctoring system is deployed on the customer’s servers using their infrastructure. It provides full control over data and can work inside a local network without Internet access.	Working with a low Internet speed (256 kB/s); Convenient and effective fathers; Support for proctoring on Android and iOS mobile devices; Double proctoring: connecting an additional mobile camera for 360° viewing; Live proctoring.	10,000 exams simultaneously	KZT 1500 (Approximately EUR 3)/1 h
Oqylyq	Cloud The inga proctor system operates on the cloud, which is accessible via the Internet. Updates, maintenance, and all technical complexity are hidden and can only get the necessary service. The data are stored on the following platforms: Google Cloud, AWS (Amazon Web Services), Microsoft Cloud, etc. They comply with all GDPR (General Data Protection Regulation) standards and use complex encryption with transmission over a secure channel with SSL certificates.	- Automatic assessment of trust in testing results with convenient monitoring and analytics; - Support for proctoring on Android and iOS mobile devices; - Dual proctoring: connecting an additional mobile camera for 360° view; - The monitoring of users in real time.	10,000 exams simultaneously	KZT 1500 (approximately EUR 3)/1 h
Aero	Cloud The inga proctor system operates on the cloud, which is accessible via the Internet. Updates, maintenance, and all technical complexity are hidden and can only get the necessary service. The data are stored on the following platforms: Google Cloud, AWS (Amazon Web Services), Microsoft Cloud, etc. They comply with all GDPR (General Data Protection Regulation) standards and use complex encryption with transmission over a secure channel with SSL certificates.	- Automatic assessment of trust in testing results with convenient monitoring and analytics; - Support for proctoring on Android and iOS mobile devices; - Monitoring of users in real time.	5000 exams simultaneously	KZT 1000 (approximately EUR 2)/1 h

. The OES proctoring system works on the cloud with Internet access and is also deployed on university servers in their infrastructure. Oqylyq and Aero proctoring systems work on the cloud with internet access. The features of the 3 systems are almost the same, with the exception that in the Aero proctoring system, there is no dual proctoring function: connecting an additional mobile camera for 360° viewing. The number of simultaneous exams and the payment required for 1 h of the exam are shown in Table 2. Depending on server capacity, it is planned that proctoring Proctor SU can perform with a simultaneous number of 5000 participants.

The OES proctoring system supports two integration options.

(1)

Integration is based on SDK and API, which is suitable for most seamless integration cases. Integration consists of connecting a special JS library to the test page, which contains functions for managing the proctoring session. The results are transmitted via the API from the proctoring system. The communication between the student and the server is protected by TLS encryption. A valid SSL certificate is installed on the Moodle web server so that the pages of the Moodle system open over HTTPS. This is a requirement of the browser’s security policy, which does not allow access to the camera and micro-background over the unsecured HTTP protocol.

(2)

Advanced integration, according to the IMS LTI standard, allows for transparent authorization and transmission of user and exam data to the proctoring system via tokens, according to the JSON Web Token standard. Integration consists of the implementation of the JWT token generation mechanism on the testing system side and the formation of links, which can then be placed on the page or sent by e-mail. All of the content is displayed in an IFRAME.

The Oqylyq and Aero proctoring systems support one integration option: SDK-based and API-based integration.

The technical requirements for testing through the desktop application are shown in

Table 3. Technical requirements OES, Oqylyq, Aero proctoring system.

Parameter	Minimum Requirements
Web-browser	Chrome 72, Opera 59, Firefox 66, Edge 79, Yandex. Browser 19.3, Safari 12.x
Operating system	Windows 7, macOS 10.12, “Sierra”, Linux
Mobile version	Android 4.4+, Chrome 72+, iOS 13+ Safari
Web-camera	640 × 480, 15 frames/s
Microphone	Required
Speakers	Optional (for connection with the proctor)
Network (Internet)	256 Kbit/s
Network connection speed:	3 Mbps
Free disk space	100 MB
Free RAM	1 GB

. The technical requirements for mobile devices are as follows: Android 8.0 or higher, iOS 13.4 or higher, and 1 GB of RAM (recommended—3 GB or higher).

If the total number of hours in all sessions for the required storage interval is 25,000, and the number of exams held simultaneously is 5000, then the peak load of the server will be as shown in

Table 4. Equipment calculator.

Displaying the Predicted Load to Estimate Which Server Will Be Required for the Box Version	Peak Load
Number of simultaneous exams	5000
The total number of hours for all sessions for the required storage interval	25,000
Number of processor cores (CPU)	82
CPU Mark (www.cpubenchmark.net)	110,700
The amount of RAM	82 GB
Network Bandwidth (LAN)	1000 Mbit/s
Data storage capacity (HDD)	765 GB

. If the total number of hours in all sessions is 100,000 h for the required storage interval, then at maximum load, only the data storage (HDD) will change, that is, 15,000 GB. The other parameters (number of processor cores (CPU), CPU Mark, amount of RAM, network bandwidth (LAN)) remain unchanged.

The subscription price for the commercial proctoring service is shown in

Table 5. Subscription price for a commercial service.

Name	Description	Proctoring Mode	Cost (KZT)
Small University	Takes exams twice a year with 3 thousand exams, a total of 6 thousand exams	Automatic Asynchronous	KZT 7 million (EUR 14,000, approximately)/year
Average University	Takes exams twice a year, up to 30 thousand exams a year	Automatic Asynchronous	KZT 15 million (EUR 30,000, approximately)/year
Large University	Takes exams, up to 60 thousand exams a year	Automatic Asynchronous	KZT 18 million (EUR 36,000, approximately)/year

. S. Seifullin Kazakh Agro Technical Research University accepts exams twice a year, with up to thirty-thousand exams. There are also services and additional services, i.e., integration and technical support of the server part. These services include the following: SaaS work on the contractor’s cloud infrastructure (including resources and support), which is equal to KZT 300,000 per month, for installation on the university server, KZT 500,000, and for technical support in relation to the university servers, KZT 75,000 per month. When performing the installation on the server, you will need to call specialists, in which case, the university will pay for the business trip expenses.

The business processes of developing a proctoring system are determined by three main indicators: the duration of development, the required labor costs (when calculating the nominal salary value), and the number of developers. The total development time required for a proctoring system is 12 months, the required resources for the development of the project are equal to 8 man-months, and the total cost of the project is KZT 35,000,000 (approximately EUR 70,000). In addition, it is possible to determine the following time assessment of the stages of the project life cycle: analysis of system requirements, 1.2 months; the identification of features, 1.2 months; design, 1.8 months; coding, 2.9 months; and testing, 5.4 months. The average annual profit is KZT 14,000,000, and the payback period of the project is 3.5 years.

The source code of the Proctor SU proctoring system is open (open source), which allows the user to easily configure the software for the needs of the university, simplify the development of the system, reduce the dependence of the university on the software developer company, and facilitate program maintenance.

The Department of Information Technology is part of S. Seifullin Kazakh Agro Technical University. The department’s employees are specialists in the field of information technology (programmers/developers, web designers, video editors, system administrators, and testers). After the introduction of the proctoring system, the proctoring program will be administered by the Department of Information Technology. The Department of Information Technology will be responsible for the maintenance of the proctoring system and the management of both activities at the university.

The Department of Information Technology is a structural production unit of the University.

The main tasks of the department are as follows:

• Introduction, development, and support of advanced information technologies in the educational process, scientific research, and university management.
• Developing and implementing a unified technical policy in the field of development and improving the hardware, software, information, and telecommunications infrastructure of the university.
• The implementation of digital transformation processes in terms of developing the software infrastructure of the university, which provides support for the digital transformation of the educational process, scientific research, university management, the electronic interaction of participants in the educational process and digital interaction with external organizations in order to expand the electronic information and educational environment of the university, and implement the concept of “digital university”
• Providing students and employees with modern WEB services, including services of collective digital communications and services of access to the information resources of the university.
• Ensuring the smooth functioning of the information and telecommunications infrastructure of the university, the hardware and software of centralized departments, the information systems used to solve university management tasks, IT services and Internet representation, and access to the global Internet.
• The provision of technical measures to maintain the information security of the university’s computer network and the protection of the corporate data of information systems.

Four online control systems were selected in the study: OES, Oqylyq, Aero, and Proctor SU (Proctor Seifullin University). The selected systems were analyzed in terms of the following main points: infrastructure, the system used, licensing, end-user support, user authentication, the frequency of updates, cost calculation models, record confidentiality policy, type of services offered, and integration with Moodle. The capabilities of each of these systems are analyzed as follows:

• Aero (cloud, proprietary license, live/recorded/automated proctoring, online processing, authentication required, can be integrated with Moodle, face recognition function, and the ability to be used on all devices, including mobile).
• Oqylyq (cloud-based, proprietary license, real-time verification, authentication required, can be integrated with Moodle, face recognition function, the connection of an additional mobile camera for 360° viewing, and the ability to be used on all devices, including mobile).
• OES (cloud, proprietary license, automated proctoring, can be integrated with Moodle, regular updates, face recognition function, the connection of an additional mobile camera for 360° viewing, and the ability to be used on all devices, including mobile).
• Proctor SU (cloud-based, authentication required, recorded/automated proctoring, can be integrated with Moodle, face recognition function, regular updates, in the licensing process).

Table 6. Functional features of proctoring systems.

Proctoring Features	Proctor SU	OES	Oqylyq	Aero
Live human proctors available	Yes	Yes	Yes	Yes
Internet required	Yes	Yes	Yes	Yes
Secure/encrypted transferring of data	Yes	Yes	Yes	Yes
Student able to book exam time	Yes	Yes	Yes	Yes
Proctoring provider certified	No	Yes	Yes	Yes
Student can message issues to proctors	Yes	Yes	Yes	Yes
Students get live exam instructions	Yes	Yes	Yes	Yes
Proctor able to see students screen	No (only record after the exam)	Yes	Yes	Yes
Recorded video reviewing option	Yes	Yes	Yes	Yes
Pause test/cancel test	No	No	No	No
Automated proctoring	Yes	Yes	Yes	Yes
Keystroke checking	Yes	Yes	Yes	Yes
Audio recording	Yes	Yes	Yes	Yes
Browser lockdown	Yes	Yes	Yes	Yes
Authentication option	Yes	Yes	Yes	Yes
Web camera needed	Yes	Yes	Yes	Yes
Log reports	Yes	Yes	Yes	Yes
Recording storage option	Yes	Yes	Yes	Yes
Test review option	No	No	No	No
Incident logs with date and time	Yes	Yes	Yes	Yes
Customizing options for institution	Yes	Yes	Yes	Yes
Lockdown Features
Available on both Windows and Mac	Yes	Yes	Yes	Yes
Plugin for browser	Yes	Yes	Yes	Yes
Avoids control options on the browser	Yes	Yes	Yes	Yes
Stops navigation (forward/back)	Yes	Yes	Yes	Yes
Stops right clicks using mouse	Yes	Yes	Yes	Yes
Stops printing	No	No	No	No
Proctoring Features
Hides taskbar	Yes	Yes	Yes	Yes
Hides desktop	Yes	Yes	Yes	Yes
Stops copying and pasting	Yes	Yes	Yes	Yes
Stops other applications	No	No	No	No
Stops starting of other applications	No	No	No	No
Authentication options
User required to authenticate	Yes	Yes	Yes	Yes
Username provided/required	Yes	Yes	Yes	Yes
Password provided/required	Yes	Yes	Yes	Yes
Ability to do facial recognition	Yes	Yes	Yes	Yes
Ability to do voice recognition	Yes	Yes	Yes	Yes
Fingerprint scanning required	No	No	No	No
Iris scanner required/available	No	No	No	No
Webcam Features
Web camera required	Yes	Yes	Yes	Yes
Room panning allowed	No	Yes	Yes	No

presents the functional features of OES, Oqylyq, Aero, and the Proctor SU proctoring systems.

The essential characteristics of the proctoring system include the following: new technology in terms of knowledge control, beneficial economic effect, educational and social effects, positive effect on the educational process, the presence of specially trained employees, and the possibility of self-sustaining.

As a result of considering the functional features of four proctoring systems (OES, Oqylyq, Aero, and Proctor SU), the following conclusions were obtained:

• OES, Oqylyq, and Aero proctoring systems are commercial systems (charging an hourly fee for each exam), whereas the Proctor SU proctoring system is a product of S.Seifullin Kazakh Agro Technical Research University, which does not belong to a commercial product; therefore, the development of its own program for the university is very profitable from a financial point of view;
• Information security is preserved in all four proctoring systems. The communication between the student and the server is protected by TLS encryption;
• In the OES and Oqylyq proctoring systems, one can control the room in a 360° mode by turning on an additional mobile camera. The other two proctoring systems do not have this function;
• The authentication parameters of all four proctoring systems include face recognition. Fingerprint and iris scanning are not available in the proctoring systems. According to the developers, these biometric security systems (fingerprint scanning and iris scanning) are too expensive to implement in a proctoring system.

4. Results

4.1. Testing the Proctor SU System

A mock-up of an online test was prepared, and members of the research team tested and tried to cheat the system, for example, through the use a mobile phone, opening a new browser tab, talking to someone in the room, viewing notes in a book, moving away from the screen, etc. The reports of the violations were recorded and discussed with the developers.

The program, using artificial intelligence, broadcasting information from three sources: a camera, a microphone, and a computer desktop.

The proctoring system recorded cases when:

-

the user covers their mouth or other parts of their face;

-

Other people appear in the frame;

-

Other human voices are heard near the person being tested;

-

The person uses other gadgets;

-

Books or records are visible in the frame;

-

The user switches tabs in the browser;

-

Communication takes place through gestures.

Additionally, as a result, the researcher provided the test organizer with a video recording of the process and an assessment of the reliability of the results obtained.

In some cases, the system warned the user about the detected violation and allowed them to continue working; in some cases, it completed testing immediately. The confidence level ranges from 0 to 100%. If the overall confidence level falls below a certain level (30%), the test ends automatically (the university can also raise or lower the confidence level at its discretion).

One of the main components of the software package for proctoring is a module for collecting data related to user actions. The data collection module is a developed browser extension. Browser extensions allow a user to extend the functionality of the browser. With their help, a user can change the contents of the browser, access information about the browser, as well as some information about the user’s computer. Standard web technologies, such as JavaScript, HTML, and CSS, are used to develop extensions. With the help of a special extension, you can collect data and send it to the server. Before its implementation, a list of features will be compiled that will be as informative as possible and will be able to show, in the most detail, the actions that the user performed during testing. The list of events and the information collected is as follows:

• Opening and updating information in a browser tab: Page URL, tab name, number of tabs, date and time of the event, the flag of what is running now with the user tab, flag that the tab is open in “incognito” mode;
• Switching to any browser tab: URL address of the page, tab name, tab serial number, and the date and time of switching;
• Completion of the test: date and time of completion, the name of the test, and the user data (tested name).

One of the main goals of testing is to check the compliance of the Proctor SU proctoring system and its individual modules. Software testing makes it possible to identify code defects in a timely manner, improve the reliability and fault tolerance of the system, and avoid financial and expensive fees associated with the unstable operation of the finished solution.

4.2. Test Trial with Students

Using a convenient selection method, tests were conducted with 2nd- and 3rd-year students at S. Seifullin Kazakh Agro Technical Research University (n = 120). After the students completed the test, they were asked a set of questions and offered to evaluate their experience.

After the students passed the trial test, discussions were held. The students were asked the following questions: were they able to pass the test; what was their ability to easily navigate the system; the clarity of instructions within the system; was it convenient for them to take the test (rating scale: 1, not at all, and 5, very much) (

Table 7. Questions asked to respondents and the results.

Questions Asked to Respondents	1	2	3	4	5
Did you like your experience with Proctor SU?	0%	8%	7%	70%	15%
Have you successfully passed the test with Proctor SU?	13%	5%	36%	23%	23%
Did you navigate the system easily?	0%	0%	10%	10%	80%
Are the instructions provided by Proctor SU clear?	0%	0%	11%	15%	74%
How would you rate your comfort level from the presence of a proctoring system during your exam?	10%	12%	12%	30%	36%

).

In total, 85% of students stated that the experience with Prostor SU was positive. Most of the students passed the Proctor SU test with a positive result, and only 13% of the students failed the test. This was mainly due to technical problems concerning Internet connection that students encountered during testing. Clear and detailed instructions were explained to the students before the actual testing. Before that, the students were able to take online tests in Moodle. Proctor SU has not made significant changes to the test apart from identity verification (students must be assigned the test before taking it). Students were nervous when they started checking webcams and their IDs (10%). Some students shared that they felt uncomfortable because the camera was recording their every move. The students were also concerned about the confidentiality of the videos and their use. Some students (about 10%) warned that they do not have a personal computer or laptop, a neat room with sufficient lighting, an Internet connection, and other technical devices. These students were allocated an office in the university and were provided with technical devices and an Internet connection.

Main conclusions:

-

the team tested the proctoring system at a fairly low Internet speeds (256 Kbit/s) and obtained good results;

-

Online proctoring can be easily integrated into Moodle without additional infrastructure;

-

Students are positive about the use of online proctoring.

4.3. Testing of the Proctor SU System

Two types of tests were used to evaluate Proctor SU: unit testing and acceptance user testing. Unit testing is a type of software testing in which individual modules or software components are tested. Its purpose is to check that every piece of program code is working properly. This type of testing is performed by developers at the coding stage of the application. Unit tests isolate a part of the code and check its operability. The unit of measurement can be a particular function, method, procedure, module, or object. Acceptance testing (UAT—user acceptance testing) is testing that is carried out by the end users of the system in order to make a decision concerning its implementation. Any software development or refinement goes through the final stage of UAT testing.

4.4. Unit Testing

Unit testing allows you to check whether individual parts of the program meet the stated requirements and work correctly. We used the built-in standard Python library unit test. We checked each block (recognition block, audio block, and the browser block), most often functions, separately. We checked how all of the components of the code fit together. We performed the following actions:

• Imported unit test as a standard.
• Created the Test Add class, which inherits from the Test Case class.
• We used the self.assertEqual() method in the Test Case class.

By executing these commands, the developers checked all of the blocks and individual parts of the program.

4.5. Acceptance User Testing

Using the following questions, 10 teachers were asked to evaluate the functioning of the proctoring system:

-

interface design;

-

Ease of use;

-

Checking the input data;

-

Accuracy (object detection and tracking);

-

General satisfaction. Four users found several errors. Minor changes were made to the system’s external interface based on feedback from two users to improve the overall usability of the system. Two users who tested the system showed that the system can work slower when the quality of the Internet connection is lower than expected. The solution to this feedback was to reduce the frame rate of the webcam needed for processing in order to reduce computational costs and increase the overall efficiency of the system.

The main elements of the graphical interface are as follows:

• window (browser window, dialog box, modal window, and floating windows);
• menu (main, pop-up, context);
• Widgets/controls/controls (button, radio button, icon, list, toolbar, tree, scroll bar, slider, status bar, and tooltip);
• Tab;
• Interaction elements: mouse cursor and text cursor.

The general checks reviewed by teachers are as follows:

• the appearance of elements when the browser window is reduced and the appearance of scrolling;
• Correct spelling and alignment of the text;
• Correct focus movement in the window (Tab/Tab+Shift);
• Selected items are highlighted;
• Immutable fields look the same and differ from editable ones;
• It is advisable not to use a double click;
• Unification of design (color, font, and size);
• Switch buttons using the keyboard;
• There must be a scroll function;
• If the forms are duplicated, then there must be the same names.

The teachers checked the main elements of the graphical interface. The following recommendations were proposed by the teachers:

• Font. In the system interface, the Times New Roman font (serif font) size 14 has been increased to size 18. The instructions were left in the font Times New Roman, size 14.
• Mathematical formulas. When teachers entered math test questions into the system, they were displayed in hieroglyphs, incomprehensible symbols. The developers have fixed these errors.
• Authorization. Authorization is carried out via a login and password. There is also an additional forgotten password feature. If the student has forgotten their password, it is reset via email. For the convenience of students, a password reset function was offered by SMS over the phone, not by mail.
• Scroll function. When testing the interface, the scrolling function did not work in some places. The developers have fixed this error.

Users are crucial to software development as they are the ones who will use the software. Consequently, it is important to consider their thoughts and opinions and make adjustments in accordance with their preferences. In this project, UAT was performed by selecting people at the end of each iteration in order to obtain feedback.

4.6. Student’s Face Recognition Results

We photographed 100 university students’ faces from different angles (up and down and right and left) and uploaded them into the database. We also looked at five different facial expressions of students during a photo shoot in a video stream, eight types of accessories (five types of glasses and three types of caps), and ten lighting directions. In the front image (FI), the students were looking directly into the camera. The database included images of students looking up, down, and horizontally, and the direction of head rotation in these images was within [≈−750; +750] degrees. The detection results are shown in

Table 8. False-positive results and detection frequency from the dataset.

Data Set	Faces	False Alarms	Detection Rates
Front image (FI)	1674	36	97.84%
Image looking down (ILD)	1024	9	94.23%
Image looking up (ILU)	1254	11	96.12%
Image horizontally (IH)	1048	26	97.51%
Total	5000	144	97.12%

. Table 8 shows that the system successfully identified 5000 facial features; out of 5000 images (only one person in each picture), there were only 144 false ones. Analyzing certain cases of errors, we can conclude that the accuracy is reduced primarily due to images with noise and poor lighting quality.

The highest percentage of hits falls on the front image (FI) dataset since all faces are located in the front. In the image looking down (ILD), the image looking up (ILU), and the image horizontally (IH), there are facial expressions, accessories, and various lighting options. These changes lead to more false-positive and fewer hits in the front image (FI) dataset. It can be concluded that the search for images where the user is looking down is somewhat more difficult for the system than images where the user is looking up. The main idea is to consider the statistical relationships between the location of the anthropometric points of the face. The faces are compared according to their relative position.

Due to the speed of image processing, ease of implementation, and the minimal cost of the technical requirements for the system, we chose the YOLOv3 model [28,29]. Using the YOLOv3 model allows us to choose algorithms and methods that ensure high accuracy and minimize false positives when solving face recognition tasks. YOLOv3 is an improved version of the YOLO architecture. It consists of 106 convolutional layers and detects small objects better than its predecessor, YOLOv2. The main feature of YOLOv3 is that there are three layers at the output, each of which is designed to detect objects of different sizes [30,31,32,33,34,35] (

Table 9. Model recognition efficiency of CNN, R-CNN, and YOLOv3.

Model	Accuracy	Sensitivity (Recall)	Specificity	Precision	F1-Score
CNN	0.9395	0.7368	0.8421	0.8235	0.7777
R-CNN	0.9695	0.7762	0.8926	0.8637	0.8877
YOLOv3 (our model)	0.9895	0.8865	0.9429	0.9235	0.9046

).

The YOLO model processes images in real time at 45 frames per second, and CNN and R-CNN process images in real time at 30 and 38 frames per second, respectively.

5. Discussion

Proctoring is used in distance learning technologies to organize and monitor the progress of exams. The proctoring system fully controls the user’s camera, microphone, and screen during the exam with the help of AI, computer vision, and other modules, and provides the proctor with a fractional report that analyzes the user’s behavior. Controlled exams reduce the risks associated with organizational costs and increase the validity of the qualifications of employees and students.

The research that we collected during the analysis of commercial proctoring systems shows that many systems combine innovative technologies and capabilities that provide an idea of the effectiveness of conducting online exams.

It is difficult to understand whether the benefits of these online proctoring technologies outweigh the risks. The most reasonable conclusion we can draw at the moment is the ethical justification of these technologies and their various capabilities, which require us to strictly maintain a balance between fears and possible benefits.

Compliance with the principles of academic integrity is the most important factor for students to achieve the target learning outcomes of the programs. Respect for the principles of academic integrity at the university teaches time management, proper prioritization, organization, and efficiency, as well as providing solid knowledge and skills that cannot be obtained through cheating. It is the absorption of the traditions of academic honesty during a student’s time at university, and preferably earlier, which helps to establish ethical standards for life and gives a sense of achievement. Controlled exams reduce the risks associated with reputation and organizational costs as well as improve the quality of competence of employees and students.

It is more convenient for teachers to work in the Proctor SU system, it is easy to upload tests, and students do not need a lengthy verification process before undertaking an online exam.

6. Conclusions

Proctor SU is a piece of software designed to be used for online monitoring, logging, and evaluation of user behavior during important online events (exams, tests). Proctoring functions are connected to testing platforms, which allow a user to conduct controlled remote exams both manually, with the participation of proctors, and automatically.

Proctor SU can increase the frequency of online tests by reducing staff costs and other fees associated with time allocation. The system also improves employee productivity by eliminating the need for an additional number of supervisors to conduct online tests. It is expected that reliable online exams will increase the income of educational institutions, and the reliability of these institutions will increase since the system provides the opportunity to conduct closed exams. The long-term use of Proctor SU is expected to increase the reliability of online tests and promote academic integrity.