Research on the Use of Eye-Tracking in Marking Prohibited Items in Security Screening at the Airport

Abstract

This article presents the results of an experimental study that indicates a problem in assessing the effectiveness of security control operators, that is, the Hit Rate indicator. It was pointed out that there is a significant error in determining whether an operator correctly identifies prohibited items. This is a significant problem because the effectiveness of operators is expected to be very high, and the measurement error may interfere with their correct assessment. So far, there has yet to be any research on improving the accuracy of estimating this rate. This article examines whether eye-tracking technology can be used to eliminate this error. An experimental test of using a 120 Hz eye-tracking system was conducted to see what the effectiveness of identifying gaze location is on two 19″ monitors, which simulates the operator’s actual work environment. The tests were conducted under simulated conditions, using a replica of an actual security control operator’s station. As a contribution, it has been proven that it is possible to use eye-tracking technology to significantly increase the accuracy of screening operators’ assessments. This is important knowledge that can be the basis for further research in this area.

1. Introduction

The article aimed to determine the effectiveness of visual confirmation when pointing out prohibited items during the analysis of images overexposed with an X-ray device. Obtaining appropriate effectiveness will allow the system on which airport security control operators work every day to be expanded. As a result of the work, the accuracy of the determined evaluation indicators of security control operators is expected to increase.

In their day-to-day work, security control operators are tasked with identifying items not authorized for air transportation. The most significant restrictions apply to the carriage of cabin baggage. A detailed list of items that should be determined by screening operators is specified in ref. [1]. For example, items prohibited for carriage in cabin baggage are divided into six main groups:

• guns, firearms, and other devices that discharge projectiles;
• stunning devices;
• objects with a sharp point or sharp edge;
• workmen’s tools;
• blunt instruments;
• explosives and incendiary substances and devices.

Each group mentioned above has detailed descriptions that specify the scope of restricted items [1].

The security control operator is evaluated continuously by collecting statistics, which also evaluate the effectiveness of detecting prohibited items. A Threat Image Projection (TIP) system is built into X-ray devices. The concept of the TIP system is to create images with artificial threats that the security control operator must indicate. A library of FTIs (Fictional Threat Images) is used for cabin luggage. FTIs are superimposed by the IT system on actual luggage carried by passengers. When screening checked baggage, a library of CTIs (Combined Threat Images) is deployed. In this case, crafted whole images where a prohibited object is placed are woven into the stream of authentic images. Due to the need for a highly efficient security screening process at airports, X-ray machines only provide the ability to trigger a threat detection alarm by physically pressing a button on the machine’s console. Triggering an alarm triggers functions to check whether it is an image containing a CTI or FTI. The operator does not physically indicate the area in the image where he or she believes the prohibited object is located. This results in the operator’s effectiveness needing to be evaluated correctly. The operator may consider an image a threat with an object in mind that is not a prohibited object. If a TIP is superimposed on this image, the system will assume the operator’s knowledge to be correct. This adversely affects the accuracy of the operator’s assessment of the screening’s effectiveness. The scientific literature has clearly indicated that there is currently a problem with training and evaluating the work of operators. As a result, an operator for as long as the first 6 months of work may perform his duties without adequate knowledge [2].

Research in this area has not yet been conducted in the scientific literature. This article presented data to estimate the scale of the problem and point to a potential solution. The use of eye-tracking technology to eliminate this problem was proposed. The paper is a scientific contribution to improving the accuracy of screening operators’ assessments and was carried out based on the authors’ research using a developed proprietary training simulator.

The article is structured as follows. Section 2 presents the current state of knowledge about the evaluation models of security control operators and the systems that support their work. Section 3 presents the research results. Section 4 presents a summary of the study conducted.

2. State of the Art

The world literature [3] indicates that the activities carried out by security control operators are priority activities to meet civil aviation security standards. However, the articles that deal with the security control process at airports mainly deal with the problems related to the efficiency of this process rather than its effectiveness. A comprehensive review of the literature on the management of the performance of passenger service processes at the airport terminal is presented in ref. [4]. It also includes all the work that deals with the problems of the efficiency of the security control process at airports.

Scientific works aimed at ensuring an adequate level of security are a significant minority. In their case, two different areas of application can be distinguished. The first area relates to research that seeks automated support for detecting prohibited items. The deep convolutional neural network YOLO (You Only Look Once) was instantiated by Ponnusamy et al. [5] to classify X-ray baggage images and identify hazardous materials. Convolutional neural networks have been used to detect specific dangerous objects such as knives [6], sharp objects [6,7], and weapons hidden in luggage [8,9]. An analogous method was also used in [5,10,11,12,13]. The general conclusion of such work, however, is that the current development in this area does not provide a basis for abandoning the human element in the security control process. Hättenschwiler et al. [14], in a study on the automation of explosive detection systems for cabin baggage screening (EDSCB), verified that a human-machine system with automatic decision-making showed better performance than automation as a diagnostic aid. The proposed systems have also a computational performance problem, making them unsuitable for real-world deployment [15].

The second area of work is directly focused on the human factor. Few scientific works have addressed aspects related to ensuring an appropriate level of security. These works confirm that the human factor is the most important and directly affects the reliability of the detection of prohibited items in security control at airports [2]. Based on experimental data from a real system, various factors influencing the level of safety were evaluated. In the work [16], it was proven that the level of training of security control operators needs to be improved. Operators do not meet the required indicators on the real system after training that is not conducted on a replica of the workstation. Security control operators gain experience only after more than half a year of work experience. Analogous conclusions were drawn in ref. [17]. An essential factor in correctly assessing the image is the time it takes for the security control operator to decide on the contents of the baggage [18]. The correct detection of prohibited articles depends on how the prohibited article is placed in the baggage, the degree to which the baggage is full, and the extent to which other items in the baggage cover the prohibited article [19]. Work aimed at increasing the level of safety has yet to be dedicated to trying to directly influence the human factor by directly eliminating the causes of misjudgment. Systemic attempts have been made by increasing the number of barriers (e.g., two-stage screening) [9]. However, such solutions are not used in practice due to the throughput limitation of such a system. Studies have also been conducted using different detection methods [20]. However, the reconfiguration of the systems is associated with high costs without eliminating the human factor [21,22].

The work to date has focused on an attempt to assess the current state and increase the security and efficiency of the process. All the work mentioned above has yet to be dedicated to increasing the precision of assessing security control operators. However, this factor is crucial to know the operators’ weaknesses and train them better. The purpose of this work is to fill this gap, on the basis of which work will begin on building a system to accurately assess the effectiveness of security control operators in a further stage of research work.

3. Research Problem

One crucial indicator of the effectiveness of the security control operator is the Hit Rate indicator (1). This indicator is crucial in terms of the operator’s ability to perform his duties. Screening operators must be properly qualified to perform their duties. Operators must pass an exam, during which a certain value of the HR indicator must be achieved. The HR index is also evaluated periodically in subsequent years. Failure to achieve the appropriate HR index value in subsequent periodic tests results in the loss of the operator’s license to perform his duties. Therefore, it is crucial to correctly estimate the value of the HR indicator, which is not being properly done today. The Hit Rate refers to the number of correct indications of images that contain a prohibited item. The number of correct indications n_c and the number of incorrect indications n_nc are collected. The collected data determine the probability of indicating images with a threat (1).

$$HR={\displaystyle \frac{n_c}{n_c+n_{nc}}}$$

(1)

However, this indicator does not consider whether the operator indicated an image by seeing a threat among the objects visible on the screen; theoretically, if the operator indicated every image as a threat, it would achieve a 100% success rate. However, this is unacceptable because the security checkpoint’s capacity would be too small. Any baggage marked as a threat must be searched manually. Operators are, therefore, also assessed through other indicators. They consider, for example, the probability of baggage being labeled as a hazard when there are no prohibited items in the baggage. However, these factors are not investigated in this article. The article attempts to better determine the n_c parameter in evaluating the HR index. In the current system, an operator is credited with a positive decision (counted with the variable n_c) even when the operator decides to indicate a hazard with another object in mind that is not part of the group of prohibited objects or makes his decision randomly.

To test the relevance of this problem, a preliminary study was carried out, which involved surveying 10 operators completing their training as a security control operator by asking them to mark prohibited items on a sample of 100 test images, 30 of which contained a threat. TIP images collected from the real system were used for the study. The TIP system places prohibited items in random positions. Each operator was given an equal set of images to evaluate. An example of a TIP image is shown in Figure 1. Operators with little experience (up to 6 months of duties) were selected for the study. The results for this group of operators indicate that there is a large spread between the minimum and maximum error values. This highlights precisely the point of the problem. An incorrectly estimated HR index can result in the admission of people who do not meet the assumed level of the HR index. The study does not specify the required level due to the fact that it is classified information, determined by the institutions involved in certifying screening operators.

The following data were collected during the study:

• n_c—calculated in accordance with the applicable system;
• n_ct—number of true alarms (images with the threat actually identified);
• n_cf—number of false alarms (images with a threat where the operator has indicated another object as the threat).

The HR indicator determines the Hit Rate value according to the current system. The HR’ indicator determines the Hit Rate value with consideration of the eye-tracking system. The difference between the indices ΔHR was also determined as the absolute error of the index, expressed as a percentage. The results are shown in

Table 1. Results of HR validity tests HR.

No.	nc	nct	ncf	HR	HR’	ΔHR [%]
1	23	20	3	0.77	0.67	10
2	21	19	2	0.70	0.63	7
3	25	22	3	0.83	0.73	10
4	24	23	1	0.80	0.77	3
5	26	24	2	0.87	0.80	7
6	29	26	3	0.97	0.87	10
7	28	28	0	0.93	0.93	0
8	28	27	1	0.93	0.90	3
9	29	25	4	0.97	0.83	13
10	22	21	1	0.73	0.70	3

.

The study results indicate that the HR would have obtained the correct value for only 1 of the 10 subjects. For the others, the HR value error ranged from 3 to 13%. The average error is 7%.

As high effectiveness is required of security control operators, a 7% error in the screening operator’s assessment is essential. Section 4 explores eye-tracking technology, which can assist in the data collection process and correctly estimate the HR.

4. Results of Research on the Accuracy of Marking Objects with Eye-Tracking

The purpose of conducting research is to build a system that introduces an additional verification step for the correctness of the operator’s decision. Figure 2a shows the current state of the system. There is only an X-ray device and an operator who visually searches for prohibited objects in the screened images. In the proposed solution, there is a new element in the system (Figure 2b), a module that collects data on the location of the gaze and the location of the forbidden object (TIP). With these data, it is possible to verify if the operator is actually looking at the prohibited object when the alarm is triggered.

A self-built training simulator was used for the study (Figure 3). The simulator features a design that replicates the actual workstation of a safety counter operator. Since dual-view X-ray viewers are very often used at airports and have two 19″ monitors for image evaluation, an analogous arrangement was used in this study. The simulator was additionally equipped with an eye-tracking system mounted as a bar under the monitor screen. A commercially available measurement solution (eye-tracker) operating at 120 Hz and an average delay of <13 ms was used.

The research began by checking the device’s overall accuracy in different parts of the displayed image. At 170 locations on the screens (selected at random positions from a uniform distribution), points where the eye should be focused were displayed sequentially, one after the other, at fixed intervals. The subject was given 10 s to mark the displayed marker with their eyes. For each measurement, the coordinates of the displayed point on the screen and the coordinates of the gaze focus were recorded. The collected positions indicated the distance measured in pixels from the lower left corner of the screen. Then, for each point, the absolute error Δl was determined for each test person. The error represents the distance between the point displayed and the focus of the subject’s gaze. The set of points was drawn once, and each subject observed the points displayed in the same order. For each point, the average error value $\overline{\mathit{\Delta }}l$ was determined from a sample of 30 people. This brought the total sample to 5100 locations (170 locations × 30 participants). Due to the lack of need for knowledge of the screening operator in this study, non-certified individuals participated in the survey. The study included people between 20 and 60 years of age who had different eye conditions (without glasses and lenses, with glasses, with lenses) to take into account the applicability of the system for different groups of people. Average error values $\overline{\mathit{\Delta }}l$ (the error value is presented by the diameter of the circle) depending on the location on the screens are shown in Figure 4.

The results obtained (Figure 4) indicate that the error is not a constant value. A different error value was obtained at each of the analyzed points. A certain trend can be seen, which indicates higher error values in the middle part of the screen compared to the lower and upper parts of the screen. However, these values are not important enough to make separate calculations for different areas of the screens. The minimum value of the average error $\overline{\mathit{\Delta }}l$ is 40.7 px. The maximum value of the average error $\overline{\mathit{\Delta }}l$ is 105.7 px. The dispersion between these values is 65 px. Figure 5 shows a summary of the maximum, average and minimum values of the error with respect to the size of the example forbidden object. Figure 6 shows the distribution determined for the average error values $\overline{\mathit{\Delta }}l$ for all 170 analyzed points on the screen.

The distribution (Figure 6) confirms that unfavorable results were obtained. There is a low probability (<0.1) that the error value will be below 50 px. For this value (50 px), obtaining a precision with a probability of at least 0.95 would be desirable. This would guarantee a high solution efficiency. A marking efficiency of at least 95% would be obtained for at least 95% of the prohibited item types.

Analyzing the detailed data, it is essential to note that each person had a different accuracy of gaze focus points. The detailed data show that the range of error Δl for a single gaze is between 1.0 and 175.1 px among all subjects. The range of error Δl is, therefore, 174.1 px. This is even greater than when considering the average error values.

The results show that direct application of the eye-tracking method to the issue at hand is impossible and that an area of permissible error will be required. The value of the total error obtained is influenced by the error resulting from the accuracy of the measuring device as well as natural human movements performed during the test, e.g., body movement, head rotation.

The results obtained in the research are closely related to the size of the screens and the resolution displayed. This research is based on 19″ monitors and 1600 × 1200 px screen resolution. Changing the system configuration requires further research in this area.

5. Discussion

The security control operator must search for objects of different shapes. The set of prohibited items includes objects whose size on screen does not exceed 175.1 px, i.e., the maximum error value. Therefore, the operator may look at a prohibited item, but it will not be scored due to an error in the measuring device. The best results can be obtained when the area of the object is larger than the error that can occur. In this case, there are two situations in which a 100% image marking performance will occur (Figure 7a,b). This situation occurs when the operator focuses his or her gaze near the object’s center. The further the gaze is away from the center, the more likely it is that the error in the indication of the gaze position will not coincide with the object area (Figure 7c). On the other hand, if the operator concentrates his or her gaze outside the object area, but close to it (Figure 7d) then the system may work in his or her favor. An error in this case may indicate that the operator has identified the correct object. However, the greater the distance of gaze from the object is, the more likely it is the system will adequately judge that the operator has missed the object. The situation strives for a situation in which the system makes a 100% correct assessment for this case (Figure 7e).

When the object has a smaller area than the magnitude of the error occurring, cases according to Figure 8 are possible. Note the analogy with Figure 7 in cases (c)–(e). For cases (a) and (b), however, there is an area where the system may misinterpret because the magnitude of the error may occur outside the area of the object.

However, actual prohibited objects on the ICAO list have shapes that are much more complex than circles. If the common part of the error area and the actual area occupied by the object are considered, the probability of the system making the correct decision may be very low. This is shown in Figure 9a, where the scissors have a small part in common with a deviation in the interpretation of the point of gaze likely to occur. However, it is possible to simplify any shape by inscribing it in a figure surrounding its most protruding points (Figure 9b). Such a procedure, however, results in a two-sided effect. The probability of the system’s correct decision is increased if the operator looks at the object. However, suppose the operator does not see the object but pays attention to another object in the immediate vicinity (inside the figure in which the object is inscribed). In that case, the system will make an error. The situation will also worsen if the operator does not direct his attention to the center of the forbidden object (Figure 9c).

The experiment’s results open significant avenues for future research to optimize the technology used to evaluate airport security screening operators. One key area for further analysis is minimizing errors caused by the inaccuracy of eye-tracking technology and natural head movements, which can significantly impact assessment precision. Thus, it is recommended that future studies focus on enhancing eye-tracking technology’s accuracy, particularly in real-world operator conditions. Optimization of eye-tracking systems should involve technological improvements and analysis of various environmental factors affecting operators’ working conditions. Testing these systems under different configurations, such as multiple screen types and lighting conditions, will provide better insight into how these factors influence measurement accuracy and system performance. Simultaneously, adaptive eye-tracking models that account for individual operator characteristics, such as age or visual impairments, need to be developed. These systems could dynamically adjust to specific user needs, increasing their practicality and reliability.

Another critical aspect of future research is the development of more advanced detection algorithms. Current technologies cannot precisely recognize the various shapes and sizes of prohibited items, which impacts operators’ effectiveness in identifying potential threats. Developing more sophisticated algorithms that consider these variables could significantly enhance the accuracy of detecting dangerous objects.

Furthermore, integrating eye-tracking systems with artificial intelligence tools could represent the next step in advancing operator evaluation technology. AI-based systems could automatically classify and analyze eye-tracking data, identifying patterns and potential anomalies in operator behavior. Such integration could lead to more efficient and autonomous evaluation systems that support operators’ training and certification processes.

The proposed research directions aim to improve the accuracy of evaluation systems and enhance their practicality and adaptability to real operational conditions. Implementing these solutions could significantly improve the effectiveness of security screening operators, thereby contributing to a higher overall level of safety at airports.

6. Conclusions

Section 4 of the article estimates the magnitude of error due to the inaccuracy of eye-tracking devices and the natural head movements that occur when working with this technology. It was pointed out that discrepancies in determining the gaze focus position relative to the actual gaze focus position are significant. Section 5 of the article identifies the potential consequences of the error, affecting the ability of airport screening operators to assess their decisions correctly. It should be noted that due to the technology’s disadvantages, it is not possible to achieve 100% success in determining the Hit Rate using this technology. However, given the nature of the problem, i.e., a 7% average error in the assessment of security control operators, eye-tracking technology may contribute to a partial reduction in the size of this error. This will be a significant advance in the assessment process of security control operators. These assumptions are based on the fact that the situations described in the article in Figure 7 and Figure 8 in points (a), (b), and (e) occur where there is a probability equal to one or close to one in making a correct assessment by the system. This will allow for a more accurate assessment of the security control operators.

In connection with the above, further research will be undertaken to verify the effectiveness of the assumptions adopted during the training sessions of security control operators. The exact impact of eye-tracking technology on the effectiveness of the assessment of security control operators will be assessed. A test image database will be prepared, in which TIP images with various prohibited objects will be created. A sensitivity analysis will be carried out to estimate the margin of error in eye indication that needs to be taken into account in order to achieve the highest possible precision of the HR indicator. On this basis, a logic model will be built to develop the final version of the system. As a final step, tests will be conducted on the precision of the system.