1. Introduction
The first use of Unmanned Aerial Vehicles (UAVs) was defense systems, but nowadays it has become widespread for civilian purposes. Today, drones are used for entertainment and media [
1], search and rescue missions [
2], surveillance [
3], mapping [
4], agriculture [
5], traffic monitoring [
6] and disaster management [
7]. Today, UAVs used for civilian purposes are often called as drones. When talking about drones, ordinary people first think of small-scale quadcopters. There are several definitions for drones, but Clarke, who defines devices that are heavier than air, capable of continuous and reliable flight, do not carry people and can be controlled enough to perform useful functions, tried to provide a more comprehensive definition [
8]. Drone piloting is expected to be a popular profession in the near future, according to the 2018 report of the World Economic Forum [
9]. As in all areas where there is a human factor, there are risks of material and moral accidents during remote drone control [
8,
10]. To minimize these risks, various institutions and organizations offer many regulations and drone-pilot training. In fact, many countries require pilots to have pilot certification for use of drones and subject their journey to various legal restrictions [
11]. Drone training can be done in both physical and virtual environments, but pilot candidates are more likely to have an accident, so the first method is training in a virtual environment [
12].
Computer-based simulation programs are preferred for operator and pilot training because they can minimize costly, complex, and risky real-life situations. Scenario-based training environments allow processes to be easily changed, improved, and repeated. Many studies show that working in such environments has a positive effect on student learning performance [
13,
14]. It is also possible to change the training scenarios used in education to increase motivation in desired tasks and activities. The main reasons behind this high motivation and increased pleasure are human emotions felt during games such as scoring, gaining status, competition, puzzle solving, and social proof [
15,
16]. Points or ranking lists are used to increase player motivation, improve competition between players and groups of players, and make games more enjoyable [
16].
It is called flow theory to adjust the most appropriate cognitive load by providing an engaging experience at an optimum level in teaching environment and game designs [
17]. Regarding the changes caused by video games in the brain, researchers mostly focused on neuro-cognitive issues related to attention, visual-spatial skills, cognitive workload, cognitive control, skill acquisition, and reward processing [
18]. In an experimental study on the relationship between game and EEG, it was concluded that increasing game experience can provide faster learning and game designs for player performance and can be performed with neuro measurements [
19].
When the studies on the performance evaluation of simulation environments are examined, it is seen that research methods such as meta-analysis [
20], questionnaire [
21,
22], content analysis [
23], observation and interview [
24] are used more. Meta-analysis is the re-analysis of the results by estimating the effect size in order to obtain a general result by examining the current study results related to the research subject in the literature [
25,
26]. The survey method is carried out by collecting the answers to the questions determined about the research from the participants. This method has some difficulties, such as the fact that the answers of the participants may vary according to their current mental state and the participants cannot enter the research in depth. This method, which is often criticized, is widely used to determine the characteristics and relationships of sociological and psychological variables [
27]. Content analysis method is to summarize, compare and interpret all data in sources such as communication materials, written materials, images, visual and audio recordings by making a diagram [
28]. The observation method is the collection and analysis of comprehensive and detailed research data from the experience of the participants in a specific field, without time limitation [
29]. The interview method is carried out by asking questions to the participants to collect the data required for the research and taking note of the collected data and then analyzing these data [
30].
In addition to traditional methods of observing changes in the educational environment, more objective methods have gained popularity in recent years. The most popular is the observation and understanding of variables in an educational environment with a variety of sensors. In a study conducted in 2015, it was observed that the brain activity of a group of students was collected simultaneously for the first time using EEG titles and the effect of various teaching methods on attention was measured [
31]. In another study they conducted as a continuation of this study, the researchers found a relationship between student participation and attention [
32]. In a study examining the effect of colors on the heart rhythm of students in the classroom, it was found that light colors have a more positive effect [
33]. In a different study using MRI and fMRI, the positive effect of game-based learning on the learning process was demonstrated [
34]. In another study, it was revealed that video games increase player processing speed, hand-eye coordination, reduce reaction time and player stress level [
35]. In a study on the detection of emotion intensities in games, it is seen that it is estimated by heart rate (HR) and facial expressions (FE) [
36]. In a study on virtual reality games, the emotional states of individuals playing the game were examined using Electroencephalography (EEG), Galvanic skin response (GSR), and heart rate [
37]. Information about these studies is presented in
Table 1.
When studies examining the relationship between EEG and simulation environments were examined, it was stated that the alpha band power gradually decreased in the explorations after the first visual exploration [
38]. Repetitive behaviors have been reported to reduce stress and anxiety [
39,
40]. It is stated that decreasing stress and anxiety increases attention [
41].
In this study, it is aimed to investigate whether objective methods are possible to identify successful individuals during education and to create an experimental setup for this purpose. In the study, it was aimed to provide simulator training as an objective measurement method and to collect time, number of keystrokes and EEG signal data simultaneously during training. Unlike existing empirical studies, this study is expected to provide the following contributions to the literature with multimodal and objective applications.
Is it possible to identify successful people in the educational environment by simultaneously recording and analyzing the number of keystrokes or brain signals?
What effect does the number of training trials have on human performance?
Is there a certain relationship in the performance and mental activities of the participants during the training repetitions in the simulator?
Is there a relationship between performance in training trials and attention and meditation metrics?
3. Experiments and Discussion
Participants of the study were randomly selected on a voluntary basis among individuals aged 18–32. Before the experiments, the participants were informed about the registration system and the experimental procedure, and informed consent was obtained from the participants. The experiments were carried out in an environment free of noise as much as possible, at room temperature, and the participants could keep their mobile phones turned off during the experiments. In the experiments, the participants are expected to repeat the task defined in GDTS three times. The task involves participants taking photos of the computers in the building via the sub-camera and the occupants via the front camera. During the training, participants were asked to take photos of the targets without causing the drone to fall or stop. The trials were completed with each participant repeating the same task 3 times. The information about the age, education level and game experience of the participants was obtained. This information is presented in
Table 4.
Game experience indicates whether the participants played games for at least 1 h a week in the last 1 month. T-scores are presented in
Figure 5 according to how long the participants completed the course and the duration of the tour completion between participants. The time users spend to complete the course usually decreases after each repetition. It is the V9 that completes the course the fastest. Realizing the shortest completion time, the V9 achieved the record of 154 s on the third trail. Task Completion Times of the participants were used for analysis. The Z-score given in Equation (1) was calculated initially to complete the course as quickly as possible. This means higher performance. The x value is the time for a participant to complete that round, µ is the average time for all participants to complete that round, and σ is the standard deviation of the round completion time for all participants. The T-score given in Equation (2) was calculated using the Z-score. In the research findings, the T-Score value of each training repetition was expressed as t (1), t (2) and t (3).
The reward points that the participants earned by repeating the course and the awards they received in return for these points are shown in
Table 5. After the game, V5 earned 2 Colonel Badge points and V9 earned 3 Colonel Badge points.
The relationship between T-score, Roundtime and KeyStroke was expressed by calculating the Pearson correlation between the means of these values (tsmean, rtmean, ksmean) and presented in
Table 6. It shows that there is a high negative correlation between T-score and Roundtime and Keystroke values. Each replay causes a decrease in the time spent per round in the game and a decrease in the need for keystrokes. In addition, there is a high positive correlation between Keystroke and Roundtime values.
The number of keystrokes of the commands given by the participants for drone control in each training session is shown in
Figure 6. It is observed that as the game performance of the participants increases, the keystrokes decrease. This finding suggests that high-performance players have shorter game completion times on average and require fewer keystrokes in parallel. The minimum number of keystrokes belongs to the V5, which is 145 in the 3rd training, and the average of the V5 is 172. The average number of keystrokes of the V9 with the highest game performance average is 166.
EEG signals collected during the training process were analyzed for frequency bands and changes in alpha, beta-low, beta-medium, beta-high, delta and theta bands recorded for each repeated GDTS track training of each subject, respectively
Figure 7,
Figure 8 and
Figure 9, is shown in
Figure 10,
Figure 11 and
Figure 12. When the changes related to the bands are examined, it is seen that there is a decrease in the strength of the alpha band and all beta bands in parallel with the increase in the number of repetitions of the tracks.
It is observed that the alpha band power of the participants who complete the task quickly is higher than the other participants, and the participants who complete the task slowly, on the contrary, have low alpha band power. It is seen that Beta-Low band power is higher in V1 and V10. This shows that these participants are more relaxed and focused than other participants. Increases in Alpha and Beta-Low bands indicate relaxation and formerly relaxed state, respectively [
47]. Accordingly, it can be inferred that V1 is not comfortable compared to other participants, and V9 is more comfortable. In addition, it has been determined that the Beta-Mid and Beta-High band power of V9 and V5 with high GDTS track performances are higher than the others.
Figure 13 and
Figure 14 show the changes in the GDTS tracks of attention and meditation emotion measurements generated from the participants’ brain signals. These values show that attention and meditation increase with the increase in the number of trainings in GDTS tracks. This can be explained as the greater number of training sessions, the more focused the participants on the game and their attention.
The absolute values of the Pearson cross correlation coefficient for Delta, Theta, Alpha, Beta-Low, Beta-Mid, Beta-High and Keystroke values are shown in
Figure 15. The correlation values of Theta, Beta-Low and Keystroke values are negative, but the values in
Figure 15 are absolute values. The highest correlation among the EEG bands is the Alpha and Beta-Mid bands with an average value of 0.69. The delta band appears to be unrelated to score. Keystroke values appear to be highly correlated with T-score. The delta band appears to be slightly correlated and the Theta band negatively related. In the study, it is seen that as the number of trials increases, the subjects complete their lessons in a shorter time and the number of keystrokes decreases slightly with the increase in the number of trials. This can be seen because of training sessions at GDTS. It can be thought that as the participants become more familiar with the commands, they start pressing fewer keys. In the research, as the number of repetitions of the track increases; it is observed that the number of keystrokes per task of the participants decreased in 70% of the participants. The reduction in the number of keystrokes can be seen because of the participants making fewer keystrokes as they become accustomed to commands, and the tracks in GDTS being unchanged.
4. Conclusions
In this study, an objective measurement system that can be used in computer aided education systems is proposed. For this purpose, a multimodal recording system that can simultaneously collect keyboard and brain activity data during simulator use has been developed to analyze the cognitive and physical development of students during drone pilot training. Using this system, it has been determined that the number of keystrokes is generally lower in participants with high gaming performance than other participants. In addition, it is seen that attention and meditation values are higher with alpha, beta mid and beta high band strength. This result shows that game performance is related to the stated characteristics and revealed that the height of attention and meditation ensures that game performance and learning are more effective [
48].
As the number of trials increased, it was observed that the alpha and beta band power decreased, and the meditation and attention values increased as the participants got used to the cognitive load in visual/auditory tasks. The repetition of the game reduced the stress and anxiety levels of the participants and enabled them to play more carefully. As a result, gaming performance has been positively affected. In addition, as the number of track repetitions increased, the task completion time of the participants decreased. With the increase in the number of track repetitions, it is seen that the commands given by the participants for task completion generally decrease. This showed that participants were more familiar with commands using GDTS, and increased game repetition was associated with increased reaction speed, reduced stress levels, and improved hand-eye coordination [
35].
It is seen that this proposed system is an innovative system in the measurement of physical and mental activity. In order to find the ideal number of training in future studies, the number of participants, tasks and training should be increased. In addition, in future studies, each track will be differentiated from each other in terms of design (such as differentiating the objects on the track in terms of location, color and mobility) and difficulty, and the tracks will be randomly assigned. Thus, it will be possible to evaluate the different game experiences of the players with these differences.