Modern society has continued technological advances in telecommunications and equipment, such as artificial intelligence (AI), ubiquitous computing, the Internet of Things (IoT), smart cities, machine learning, big data and satellites, street advertisements, smart devices, and the surrounding environment, so it can provide personalized videos. These videos may or may not be desired by the individual. Videos began a long time ago as movies or TV programs but now include all videos produced using computer technology as a generic term for moving pictures [1
]. Videos are actively watched for one’s own interest and satisfaction, such as dramas, movies, news, information, privacy of others, and YouTube. However, even though you may not want to view online learning or advertising videos, occasionally you may have to watch them for your own goals or for your company’s purposes [2
]; furthermore, there are even cases where watching is essential for safety, such as school violence, corporate violence, fire, industrial safety, and disasters. However, unwanted viewing places pressure on people and causes them to play videos without watching.
Especially in education areas such as e-learning, smart learning, flipped learning, and blended learning, videos are being used very actively [3
]. Due to COVID-19 in 2020, most schools around the world have been ordered to close and classes are being conducted through online remote learning [5
]. However, the form of online learning, which requires self-directed learning, makes it difficult to identify false viewers [6
]. Currently, used online remote learning is marked as “Viewing Complete” in the system, but it is difficult to determine whether it was actually watched. In actual online training experiments, a lack of consistency in learning effectiveness is largely related to video learning [9
]. However, teachers cannot know exactly whether video learning has faithfully been done, and there is little research related to video watching judgement.
Common online video watching judgement techniques, which have been used for a long time and are now commonly used, are examined below. First, if the video is played from beginning to end, the system recognizes it as watching completion. Second, if in the middle of the video, the viewer solves the quizzes related to the content he or she watched, the system leads to the next video, and if that plays to the end, the system recognizes it as watching completion. Third, if another window on the computer is over the video while the video is played, it stops playing. However, even if you just play it until the end, the system recognizes it as watching completion. Fourth, when the play stops in the middle of the video, if the viewer clicks, it moves to the next screen, and even if the viewer just repeats these tasks to the end, the system recognizes it as watching completion. These methods can easily incorporate simple obstacles, but even if the video stops, it then plays continuously as long as the obstacle is cleared, so it is eventually misjudged as watching completion. These instances are difficult to see as honest watching completion because even if students do other activities without watching videos, the system misjudges watching completion. Therefore, it is necessary to accurately determine whether the online video was watched normally.
This paper proposes the time interval different answer popup quiz (TIDAPQ) model to judge video watching. TIDAPQ is a model that presents two interval popup quizzes in a video of approximately 10 min. This model calculates the time difference of answer submissions (RESULT 1) and the individual/different correct answers (RESULT 2). Then, if both of them are TRUE, the TIDAPQ will judge completion as normal watching; otherwise, it will judge completion as abnormal watching. After developing the TIDAPQ, 100 students at engineering universities were recruited as participants for the experiments, and the model was verified. In this paper’s experiment, the video was content of a learning character, and two popup quizzes were used in the videos of approximately 10 min long. However, TIDAPQ was not developed to judge only learning video watching. Depending on the purpose of watching a video, the length of the video and the number of popup quizzes may vary; thus, this paper describes the most basic of the TIDAPQ’s various forms.
We hope that the results of this study will be used in various fields, including online learning, reward advertising, and announcements from institutions or companies that require judging the watching of videos.
The composition of this paper is as follows. Section 2
(Background) explains the existing video watching judgement methods and describes their weaknesses. Section 3
describes the TIDAPQ model; of the three watching judgement approaches, time difference pass (RESULT 1) is explained in Section 3.3
, individual/different correct answers (RESULT 2) in Section 3.4
, and comprehensive judgement, which makes a final judgement using these two, is explained in Section 3.5
. Section 4
shows the verification experiments of the TIDAPQ model, and Section 5
presents the conclusions and suggestions.
The first area to try video watching judgement was in education due to the expansion of the online education market with the development of telecommunications and technology. In particular, video watching judgement systems emerged in 2015. Video learning methods were not absent previously in education; however, while previous video learning was watched based on the learners’ needs in an auxiliary learning mode, current video learning consists of diversified forms of learning using video, such as e-learning and smart learning. This diversification has occurred because one can take online video classes such as edX, Coursera, and Udacity and obtain a certificate, and one can obtain a degree with only online classes such as the Academic Credit Bank System [1
]. In particular, education has also appeared in which classroom classes can be conducted only when you do the prior video learning, such as flipped learning [11
]. Currently, flipped learning using online video is a great means of educating students in schools where talented people are gathered, such as Harvard, MIT, Seoul National University (SNU), and Korea Advanced Institute of Science and Technology (KAIST) [13
]. However, flipped learning is difficult to expand into groups that lack self-directed learning skills, such as elementary, middle, and high schools, because it is difficult to determine whether they have normally done prior learning via online video watching [9
]. In addition, due to the influence of COVID-19, online video learning has already entered the realm of elementary, middle, and high school public education around the world [5
]; however, educational materials are only being distributed online and it is difficult to know whether young students who lack self-directed learning have honestly watched the online learning videos [6
]. The form of learning that was expected in the future has been moved forward to the present. Accordingly, ancillary technological developments will have to be achieved.
In addition to the education sector, there are records that have been studied in the reward advertisement field for the purpose of judging the viewing of advertising videos [16
], but they have not been used. Since the use of smartphones has become more active, advertising apps have appeared where you can obtain points when you look at advertising videos for approximately 2~3 min. During these advertisements, it was hard to identify false viewers who engaged in different activities as soon as the video started playing. Because the advertiser had to pay points for customers who did not watch the advertisements, currently, the method of paying points for watching advertising videos has almost disappeared. Instead of video advertising, advertising apps now offer points for photography, writing, web pages, trying, touching, and signing up.
The following Section 2.1
, Section 2.2
and Section 2.3
describe the viewing judgement method attempted in the fields of education and advertising that generated the need for video viewing judgement.
2.1. The Appearance of the Video Watching Judgement System
Education is the first area to feel the need for video viewing judgement, and Zaption, which provides learner analysis, appeared in 2015 [17
], followed by Educannon and Workday [19
], but the service was terminated without a long duration. They carefully analyzed learners to help online video learning judgements, but the function was too complicated and the teacher had to make the final decision on watching completion, so it was ineffective. The URLs for Zaption and Educannon remain, but the pages have been deleted; the URLs for Workday, which acquired Zaption, have been changed from a learner analysis system to a system that helps the business.
Zaption of Figure 1
, a representative company in learning video viewing judgement, shows viewers by date, average viewing time, question completion, stars, average skip forward, and average skip backward with graphs. Zaption can also check each student’s submitted responses, last submission date, last viewed date, total viewing time, and total views and check the answers to the video quizzes completed by the student [18
]. Very detailed analysis was possible, but it was difficult to judge watching completion by compiling this information.
2.2. Current Video Watching Judgement System
There are currently no systems to make a video watching judgement after this video watching learning system was discontinued between 2015 and 2020, but similar systems include Playposit and Office Mix [7
Playposit chose “Inducing View through Interactive Video” as its signature feature, but just when the quizzes, related to the content of the video, are solved in the middle of a video, the video can simply play into the next screen [7
]. This technique is difficult to call a special interaction and it is difficult to determine whether the video was watched.
Office Mix is a function supported in MS Office 2013 and above and can create lecture videos by placing the teacher’s voice and face on PowerPoint slides. The provided analysis menus show the quiz answer rate, slide view frequency, and slide view time for each slide, but not enough information to analyze each user; it only shows a rough analysis [8
]. The learner watching analysis through Office Mix expired in 2018 [8
]; existing MS Office buyers can continue to use it, but new buyers cannot use it for free. Microsoft is currently operating it for a fee by converting Office Mix’s extended capabilities to Microsoft Stream. Microsoft Stream said, “We have online intelligent video, so it induces learners to watch” [21
]. However, this approach is only different in that the video includes lecture content and it is released to a designated group, but there is no special difference from YouTube. While watching the video, a user can ask questions through chatting and have opportunities to interact with the teacher, but the user cannot see the watching analysis data, such as quiz answer rate, slide view frequency, or slide watch time provided by Office Mix.
2.3. Khan Academy
Khan Academy, famous for free online lecture services, has been growing steadily since 2008. Khan Academy is not a service run for learner analysis, but it shows “How much did they invest in studying per day?”, “What video did they see?”, “When did they stop the video and what did they look at?”, “What exercise did they use?”, and “Where did they focus?” [22
]. It also shows the exercises and videos that many students focused on [23
]. It is not a detailed analysis, such as Zaption and Educannon, but it is enough to grasp the student’s learning status. However, Khan Academy also has difficulties for teachers to judge whether video watching has been completed by analyzing the data provided by Khan Academy. There are Edmodo [24
], Moodle [25
], Blackboard [26
], Schoology [27
], Brightspace [28
], Litmos [29
], and TalentLMS [30
] as learning management systems (LMS), which give or manage points for watching videos and prior learning, but like Khan Academy, they also have difficulties judging video watching.
2.4. Advertising Video Viewing Surveillance System
In addition to education, a field that perceives the necessity of watching judgements is advertising. The content viewing monitoring system of mobile reward advertising was developed to determine whether customers watched a provided commercial video [31
]. For video advertisements that pay money or point rewards for watching, it is important to determine whether sincere viewing is occurring. The content viewing monitoring system of Figure 2
can detect facial areas from images acquired by cameras in Android smartphone environments; it then can monitor the location of eyes and the opening of eyes, so it can determine whether clients are looking at the screen. The eye detection method of the system uses block contrast between the right central block and the surrounding block to detect the eyebrow and then, looks for the eye location using geometric properties between the eyebrows and eyes to determine whether the eyes are open or closed [31
]. In the integrated image of the eyebrow area, the eyebrows are extracted using the characteristic that the area corresponding to the eyebrows is relatively dark compared to the surrounding blocks. At the same time, from the integral image of each eye seeking area, the eye candidate areas are extracted using the characteristics that the pupil blocks are relatively dark and symmetrical compared to the rest of the surrounding blocks. Then, the central pixel of the block with the maximum block contrast is taken as the pupil candidate point.
The content viewing monitoring system was developed to determine the viewing of reward advertisements, but viewing judgement, which uses eye color contrast, determines watching completion regardless of whether a face photo or a teddy bear is placed in front of the camera, so it is difficult to apply to actual reward advertisements. This technology needs to be supplemented slightly more.
However, this technology has the great advantage of being simple to use to judge viewing. The viewing learning judgement systems, described in Section 2.1
, Section 2.2
, and Section 2.3
, were ambiguous in the criteria for viewing judgement and difficult to use to judge because they provided incidental data collected after viewing rather than judging while viewing. On the other hand, the content viewing monitoring system can perform the viewing judgement immediately after viewing using only videos, without incidental data that occur after viewing. It has great advantages of being simple to use and immediate judgement. It is expected that if this technology develops, it will be a means of effective viewing judgement.
The abovementioned methods of video watching judgement have a common weakness—false viewers may occur when playing videos and performing other activities. When you find a video that has been stopped while doing other activities, you can just answer the quizzes. Even if it is a difficult quiz, it is possible to ask your colleague for answers through a chat, allowing you to write the answers any time you want. The video, which has been displayed on the monitor for a long time, becomes a video that the student has focused on and points will pile up even if there is a face photo in front of the camera. Learner analysis has been well done, but teachers have difficulty determining whether they watched the video through extensive data analysis. To achieve smooth learning progression, it is necessary to accurately determine whether videos are being watched. This paper uses only video and immediately calculates the watching judgement based on the viewer’s events occurring while watching the video, clarifies the criteria for watching judgement, and presents a simple method of watching judgement.
This study presents the time interval different answer popup quiz (TIDAPQ) model to determine whether viewers watched an online video honestly. One-hundred students at engineering universities were recruited as research subjects and participated in the model verification.
Modern society has drawn attention to the technological development of telecommunications and equipment, and the fourth industrial revolution, street advertising, and the surrounding environment provide individual videos through smart devices. The form of videos has also diversified. Videos have long provided us with much information through TV and video players; however, viewer ratings have only been investigated through surveys that showed no interest in individual watching judgements. Since then, videos uploaded online, including news, movies, dramas, YouTube, and even personal privacy have become virtually infinite, but this situation also did not require individual watching judgement. However, since 2015, several companies and studies have attempted to judge video watching [18
]. These attempts appeared for the purpose of judging honest watching when viewers had to make unwanted watching mandatory. Education was the first area to attempt video watching judgement. The reasons are as follows: because of the development of technology, the field of online education has also developed, so an education program has been developed that can bestow a degree just through online classes such as the Academic Credit Banking System [1
]. There have also appeared education programs in which classroom classes are possible only when video learning has taken precedence, such as flipped learning [11
]. Especially in groups in which self-directed learning abilities are excellent, such as Harvard and MIT, flipped learning using online video watching is going well [13
], but in groups in which self-directed learning ability has not been verified, especially in younger grades, it is difficult to proceed smoothly due to false watching [6
]. Online video watching judgements which are currently being used have difficulty in determining honest watching [7
]. In addition, due to the influence of COVID-19 in 2020, online video learning has reached the realm of elementary, middle, and high school public education around the world [5
]; however, educational materials are only distributed online and it is difficult to know whether students have watched online learning videos [6
]. Teachers want to know accurately whether video learning has faithfully been done to design quality online classes, and in the current global situation, there is an urgent need to judge online video watching. In the area of reward advertising, which paid points for watching an advertisement, viewer judgement was necessary [31
]. However, there was no particularly successful method used and it was difficult to identify false viewers, so the point payment method through advertising videos has almost disappeared; instead, point payment is currently made based on photos, web pages, trying to use, or signing up, etc. If it is possible to judge video watching, it is expected that the reward advertising using video clips will also be expanded. Similarly, judgement of video watching is a necessary technology, but it is difficult to find research related to the judgement of video watching.
The online video watching judgement method, which is currently widely used, needs to solve obstacles even if the video stops. Additionally, even if viewers have been doing other activities, if the video has just been played from start to finish, the system misjudges watching completion. Similarly, because these methods misjudge abnormal watching as normal watching, it is difficult to identify false viewers. Therefore, a technology that clearly determines whether the video has been viewed honestly is needed.
TIDAPQ, developed in this study, is a model that inserts two popup quizzes in videos, makes watching judgements with time interval pass (RESULT 1) and individual/different correct answers (RESULT 2), and then, makes comprehensive judgements on whether viewers were normally watching or abnormally watching using these two. First, TIDAPQ calculates the allowed time range using various timepoints in the video, including popping up timepoint of the quiz and disappeared timepoint of the quiz, and if viewers’ event time comes within the calculated time range, it judges time interval pass (RESULT 1) as TRUE. Second, the quizzes are randomly taken from the database and shown for a certain period on the video screen. It is difficult to share the correct answer with colleagues because the quiz has been created using the unique number given to each viewer, and the correct answer is different depending on the unique number. At this time, if you write the correct answer to all popup quizzes, individual/different correct answers (RESULT 2) is determined by TRUE. Finally, comprehensive judgement uses the results of RESULT 1 and RESULT 2 to determine whether normal watching is completed, and it automatically informs the viewers immediately after they watch the video. After being judged as watching complete, viewers can watch the parts they want intensively and repeatedly without interference from the popup quizzes.
To measure the accuracy of TIDAPQ, the research subjects were recruited among freshmen in domestic engineering universities, and 100 students without prior video knowledge were selected to participate in the verification of the TIDAPQ model. The video used in the experiment contains content of approximately 10 min of learning, but TIDAPQ is not a watching judgement model exclusively for the education sector; the length and content of the video can be different depending on the watching purpose. In addition, the number of popup quizzes used to validate the model is two, but this form is the most basic type of TIDAPQ; the number of popup quizzes can be different depending on the length of the video. As a verification result of TIDAPQ’s model, the performance of normal watching judgement was 93.31%, and the performance of abnormal watching judgement was 85.71%. These results show that TIDAPQ has high performance in video watching judgement.
The existing methods of video watching judgement have a common weakness; false viewers may occur when playing videos and performing other activities [6
]. When you find a video that has been stopped while doing other activities, you can just answer the quizzes [17
]. Even if there is a difficult quiz, it is possible to ask your colleague for answers through a chat, allowing you to write the answers any time you want. The video, which has been displayed on the monitor for a long time, becomes a video that the student has focused on [17
], and points will pile up even if there is a face photo in front of the camera [24
]. Learner analysis has been well done, but teachers have difficulty determining whether they watched the video through extensive data analysis [17
]. On the other hand, the study justifies developing a TIDAPQ model that can judge the honest watching completion of online videos. This would differentiate it from previous research—to set the allowable time range using the viewer’s event time and to judge watching completion using individual/different correct answers even though it is the same problem. TIDAPQ uses only video and immediately calculates the watching judgement based on the viewer’s events occurring while watching the video, clarifies the criteria for watching judgement, and presents a simple method of watching judgement.
TIDAPQ is a model developed to monitor watching by groups that must watch unwanted videos, but researchers should be aware that applying TIDAPQ to videos that are too long can cause great stress to viewers due to the frequent popup quizzes. This paper has limitations in that the number of participants in the samples was small and that there have not been experiments on the actual video watching monitoring site. We plan to verify the effectiveness of TIDAPQ by applying it to sites where more participants and compulsory watching are required in the future. Through this study, we expect that TIDAPQ will be used properly in areas where watching completion judgements are needed, and we hope there will be more research related to video watching judgement.