A Human-Computer Control System Based on Intelligent Recognition of Eye Movements and Its Application in Wheelchair Driving

Abstract

This paper presents a practical human-computer interaction system for wheelchair motion through eye tracking and eye blink detection. In this system, the pupil in the eye image has been extracted after binarization, and the center of the pupil was localized to capture the trajectory of eye movement and determine the direction of eye gaze. Meanwhile, convolutional neural networks for feature extraction and classification of open-eye and closed-eye images have been built, and machine learning was performed by extracting features from multiple individual images of open-eye and closed-eye states for input to the system. As an application of this human-computer interaction control system, experimental validation was carried out on a modified wheelchair and the proposed method proved to be effective and reliable based on the experimental results.

1. Introduction

Human-computer interaction (HCI) has been widely studied since the 1960s with the rapid development of information systems, which aims to design a human-computer interface with ergonomic characteristics [1]. HCI systems in automated devices have been based on the traditional interface with the monitor, keyboard, and mouse for a long time. However, this manual input HCI was cumbersome to use, and to change this situation, HCI with gesture-controlled interfaces has been widely studied [2,3,4]. Nevertheless, there are many physically disabled people in real life who still are unable to use these devices or even to travel independently. These physically disabled people are completely dependent on others for their daily needs [5]. In order to improve the quality of life for people with disabilities, HCI systems without relying on hands and feet is particularly important.

In recent years, there has been much research on control systems for people with ALS (amyotrophic lateral sclerosis) and other severe physical disabilities that use biological information such as eye movements, EEG, and EMG directly for HCI, without the use of hands and feet [6,7,8,9,10,11,12]. As some particular interest research works are studies on eye gaze input and eye blink, etc., which transmit information through the human eye. Because the signal conveyed by the eye has greater stability and real-time compared to EEG and EMG, humans can communicate a great deal of information more quickly and directly through the eye [13,14,15,16,17,18,19,20,21,22,23,24,25]. Additionally, the problem of calibration in eye gaze regarding the difficulty recognition was obtained due to the method using smooth pursuit motion does not require calibration, which is of interest for eye gaze input studies because it makes the calibration of gaze input easy [21].

This paper presents a practical HCI system for wheelchair control. Eye tracking and eye blink detection based on image processing techniques have been integrated into the proposed system. Since subjects can convey incorrect information when they are slightly inattentive during eye gaze interactions, eye blink detection was introduced to assist in completing this study. In contrast to some commonly used methods [22,23,24,25], image processing techniques were applied as a new approach, which improved the reliability and convenience of the system. After the captured image was processed by image processing, its features were extracted, which made it easier to be recognized and the accuracy of recognition becomes higher. Similarly, the eye object detection module in the Haar Cascade (a Face Recognition Module) was used in the program, and as soon as the system turned on, it was able to quickly capture and locate the eye region from the images captured by the camera.

In this study, for eye gaze detection, pupil features were extracted from the eye images using image binarization. The eyeball movement trajectory was tracked by locating the pupil, based on the eyeball movement trajectory, the subject’s eye gaze area was determined so that the information that the subject wants to express through eye gaze can be obtained. For eye blink detection, some machine learning methods have been used to accomplish eye blink detection in order to solve the problems of low accuracy, lack of stability, and inconvenience of use of other methods. Machine learning techniques were used very successfully in the field of computer vision, where it was used to simulate human intelligence by learning the surrounding environment [26]. Machine learning was used in this study to extract image features of the open and closed states of multiple individual eyes to obtain a learning model so that the system could discriminate whether a subject the eye blink or not by take real-time eye images of the subject. Three methods have been used to determine the eye blink: pixel ratio [27], support vector machine (SVM) [28], and convolutional neural network (CNN) [29] to determine eye blink. In comparison, using CNN had the highest accuracy in detecting eye blink.

The HCI system of this study was mounted on a modified electric wheelchair for experimental validation. Experiments with many methods to drive wheelchairs for disabled people have been used in many HCI systematic studies, which gives us some references [30,31,32,33,34,35,36]. Based on the data obtained in the experiments, it can be demonstrated that this integrated multi-domain interaction system is effective. This also provides some basis for further research on gaze interaction and HCI.

2. HCI Control System of Wheelchairs

2.1. System Overview

The HCI system of this study was applied to control the movement of an electric wheelchair (see Figure 1). The testing system included a blink detection device, head-mounted eye gaze tracking device (see Figure 1). Wireless communication was used between the interactive system and the wheelchair, and the subject used eye gaze direction and eye blink to control the wheelchair movement. Throughout the process of controlling the movement of the wheelchair, the subject interacts with the computer, communicating information to the computer through the eyes, and the computer communicates commands to the wheelchair drive unit to control the wheelchair movement.

2.2. Hardware Systems for Wheelchairs

In this study, an aluminum AR-200 wheelchair manufactured by Matsunaga Seisakusho Ltd. was used. The hardware part of the wheelchair was improved and we designed a drive unit for the wheelchair. This allows us to control the travel of the electric wheelchair by eye gaze in the direction or by eye blink. In the modification, the rocker control unit of the wheelchair was still retained so that the wheelchair could still be controlled by the rocker when switching to manual operation. Figure 2 shows the hardware of the improved power wheelchair.

In order to be able to drive the wheelchair through the interactive system, a driver board was designed and developed for the wheelchair. Figure 3 shows the circuit schematic of the drive circuit board. This is very important as it makes our interactive system highly portable without having to consider the complex communication protocols of the controlled devices. A wireless communication serial port has been used in the driver board so that wireless communication could be made between the computer and the wheelchair.

3. Eye Movement Recognition Methods

3.1. Eye Gazes Detection Method

The eye gaze tracking device (see Figure 4) used contrast to locate the center of the pupil and used infrared non-columnar light to generate corneal reflections, capturing the black area of the pupil by illuminating the eye with infrared light [17]. In order to be able to use in a variety of light environments like sunlight, a filter was added to the lens, which is to be able to filter out light other than infrared light.

The image binarization method was used to process the captured eye images. Image binarization was the process of converting an image with shadows into two shades of black and white. A threshold was preset, and if the value of each pixel was above the threshold, it was replaced with white, and if it was below the threshold, it was replaced with black.

$$f^{\prime }(x, y)=\{\begin{array}{c}255 f\left(x,y\right)\ge \theta \\ \\ 0 f\left(x,y\right)<\theta \end{array}$$

(1)

Here, the threshold is θ, the coordinate system is (x, y), the concentration value is f(x, y), and the transformed concentration value is f′(x, y). The flow of the eye tracking technique is shown in Figure 5. The pupil in the eye image was extracted by binarization, and the pupil center coordinates were calculated. The gaze direction can be computed based on the motion trajectory of the pupil center coordinates.

The image of the eye’s moveable range was divided into five regions, and the five regions were labeled 2, 4, 5, 6, and 8 according to the numeric keypad. These 5 regions can be used to represent different commands for different driving devices, and of course more regions can be delineated depending on the driving device, so that more commands can be obtained. In this article, when controlling the wheelchair movement, the commands for these 5 zones are: backward, left turn, stop, right turn and forward. The five areas set up are shown in the Figure 6.

3.2. Eye Blink Detection Method

3.2.1. Pixel Ratio

The value of each pixel in the image is represented by 8-bit unsigned characters (value range: 0–255), which can be converted to 0 values by binarization if it is below the threshold, or to the highest value of 255 if it is above the threshold. The image is shown in Figure 7a shows a black-and-white image of the open-eye state and Figure 7b shows a black-and-white image of the closed-eye state. By comparing the two images below it can be seen that the binarised image varies according to the open/closed state of the eye. Thus, if different values are obtained from the two images, the images can be discriminated.

First, acquire the following eye image with a resolution of 64 × 64 pixels. The number of pixels will be 4096. Next, count the number of black pixels in the image. If there are 1024 pixels, the percentage of black pixels is (1024/4096) × 100, or 25%. This ratio was higher when the eye is open and lower when the eye is closed, so when the eye blink, this ratio changes. Figure 8 shows the Data waveform of eye blink image pixel ratio change.

3.2.2. Support Vector Machine

One of the oldest methods used in image classification is the SVM [28]. It is one of the pattern recognition models that uses supervised learning. It differs from ordinary pattern recognition models in that it performs margin maximization and kernel tricks.

A.

Margin maximization

Margin maximization is the shortest distance between the boundary and the data. Figure 9 shows the image of marginal maximization, the idea of margin maximization is to draw the boundary as far away as possible from the data that is closest to the boundary between the two classes.

B.

Mapping of feature space

In encountering nonlinear separation, it is necessary to use kernel tricks in SVM to map the data from the original space to the new space (see Figure 10, mapping 2-dimensional feature space to 3-dimensional feature space), and then the training data are used in the new space to get the learning model using linear methods.

To find a map to feature space, we need to find the inner product $\phi {\left(x\right)}^T\phi \left(y\right)$ on the feature space. The kernel trick allows us to calculate the inner product on the feature space without knowing what the feature space is and what φ is.

Discriminant function:

$$f\left(x\right)=sign[{\displaystyle \sum }_{i=1}^n{a}_i{y}_i\phi {\left(x_i\right)}^T\phi \left(x\right)]$$

(2)

(a: Weight; y: Label (1 or −1); x_i: i-th learning data; x: Input data)

Kernel function:

$$k\left(x,y\right)=\phi {\left(x\right)}^T\phi \left(y\right)$$

(3)

(Inner product in feature space: $\phi {\left(x\right)}^T\phi \left(y\right)$)

Replace the inner product of the discriminant function:

$$f\left(x\right)=sign\left[{\displaystyle \sum }_{i=1}^n{a}_i{y}_{i}k\left(x_i,x\right)\right]$$

(4)

3.2.3. Convolutional Neural Network

The Convolutional Neural Network (CNN) is an important method in the field related to pattern recognition [29]. This research constructed a learning model with CNN and used it to detect blinks of the subject. Figure 11 shows the CNN model constructed in this study. In the constructed CNN model input images of open and closed eyes with a resolution of 64 × 64 were subjected to convolutional operations, the feature values of the input images were extracted, and a recognition model was built based on these feature values and it was applied to the system.

In this experiment, the well-known VGG-16 discriminative model has been used as a reference. Since only two types of images (open-eye and closed-eye) need to be feature extracted and classified, the model was finally reduced from the original 16 layers to 10 layers and the convolutional layers from 13 layers to 8 layers after continuous experiments in order to balance high efficiency and high accuracy. The relevant parameters of each layer are shown in

Table 1. Relevant parameters of each layer.

Layer Name	Layer Type	Relate Parameters
Conv1_1	convolution	3 × 3, 8, relu, stride1
Conv1_2	convolution	3 × 3, 8, relu, stride1
Pool1	Pooling	2 × 2, 8, max pool, stride2
Conv2_1	convolution	3 × 3, 16, relu, stride1
Conv2_2	convolution	3 × 3, 16, relu, stride1
Pool2	Pooling	2 × 2, 16, max pool, stride2
Conv3_1	convolution	3 × 3, 32, relu, stride1
Conv3_2	convolution	3 × 3, 32, relu, stride1
Pool3	Pooling	2 × 2, 32, max pool, stride2
Conv4_1	convolution	3 × 3, 64, relu, stride1
Conv4_2	convolution	3 × 3,64, relu, stride1
Pool4	Pooling	2 × 2, 64, max pool, stride2
Fuc1	Fully-connected	512, sigmoid
Drop	Dropout	dropout-ratio 0.5
Fuc2	Fully-connected	2, softmax

.

3.2.4. The Eye Blink Detection Device and Its GUI

The graphical user interface of the eye blink detection system is shown in Figure 12, which displays the camera image, the eye status picture area, the eye blink waveform, and the number of consecutive eye blinks by the user. In the upper left area, the camera image was displayed. The eye status picture was displayed in a rectangular area, and the position of the eye area was updated every 10 frames. The two waveforms at the bottom represent the open and closed state of the eye, 1 when the eye is open and 0 when it is closed. The image at the top right shows the state of the eye in the form of an image. The text string at the bottom indicates the number of consecutive eye blinks. The symbol (*) increases with each blink of the eye. The face and eyes object detection module in a face recognition module (Haar Cascade) was used in the program in order to quickly capture the face and locate the eye area, so that the capture frame in the screen always firmly captures both eyes. During the control of the wheelchair, the wheelchair moves forward after the subject eye blinks three times rapidly. After the subject blink once with the right eye, the wheelchair turns right. After the subject blink left once, the wheelchair turns left. After the subject eye blinks four times quickly, the wheelchair moves backward. After the subject eye blinks twice quickly, the wheelchair stops moving.

4. Results and Discussion

4.1. Results of Eye Gaze Direction Recognition Experiments

Figure 13 Comparison of eye images in various states during the experiment. Figure 13a shows the eye image without infrared light irradiation, and Figure 13b shows the image of the eye under infrared illumination. A comparison between the two can be found in Figure 13b, where the pupil contour was more well-defined. Because corneal reflection is a way to detect pupils darker, the iris colours can be compared by separating the infrared light from the optical axis of the eye-tracking camera. Figure 13c shows the eye image after the binarization process. To highlight the binarization changes, the background color was set to blue instead of white. After binarization there were only two colors left in the plot, the next step was to adjust the threshold to make the pupil outline more prominent and obvious in the image.

4.2. Eye Blinks Recognition Experimental Results

A.

Pixel Ratio

In this experiment, a 5% change in the pixel ratio was used as an eye blink condition. In addition, some changes can be obtained from the image four frames ago, since the changes were not significant compared to the image one frame ago. Figure 14 shows the judgment waveforms of eye blink detection by the pixel ratio method in this study. During the experiment, a total of 20 blinks was made, only two blinks were detected in the left eye and 10 blinks were detected in the right eye.

B.

Support Vector Machine

In the experiment using the SVM approach to detect eye blink, the testers blinked both eyes the same 20 times. Figure 15 shows judgment waveforms for eye blink detection by SVM, and we can see that 15 blinks were detected in the left eye and 19 blinks were detected in the right eye.

C.

Convolutional Neural Network

Convolution layers can combine different local structures to present more useful features in a region. By convolving the input image, the input image is rendered with more feature maps. Figure 16 shows the convolutional layer feature map data of this study in the experiment. In the first and second layers, there does not seem to be much change. However, in the third layer, the brightness becomes more diversified, indicating that the network can adapt to changes in the brightness of the input image. In the fourth layer, the extracted features were different from those in the previous layers, and the contours were emphasized. Five to six layers show that the contours were greatly emphasized. In the seventh layer, more features were extracted. In the eighth and final layer, the features were extracted to the extent that the original form was no longer visible to the eye.

Figure 17 shows the judgment waveforms of eye blink detection experiment with CNN. The testers performed 20 blinks in both eyes. Based on the data waveform plot it can be seen that all 20 blinks in the left and right eyes were detected by the system.

D.

Comparative Discussion of Results

One hundred experiments were done for each of the three methods of identifying blinks.

Table 2. Number of detections per 100 blinks.

Methods	Detection Count	Undetected Count	False Positives Count
pixel ratio	28	72	0
SVM	74	26	11
CNN	99	1	0

shows the results of the number of detections for 100 blinks, and Figure 18 shows a comparison between the experimental data of the three detection methods. According to the experimental results, the recognition rate using CNN is the highest with 99% accuracy. This indicates that CNN method is still optimal, and the next step is just to adjust each parameter to further improve the accuracy rate. In comparison, the Pixel ratio method is the simplest of the three methods, which does not require the acquisition of human eye images of the subject for learning and runs relatively fast, however, the detection rate is the worst. Although a neural network was built in the SVM method, however it also does not perform image convolution and has the second fastest running speed and has the second lowest detection rate. Finally, the convolutional neural network has the highest detection rate among the three methods, and it is also the most complex of the three methods. In operation, subjects can use different methods depending on the conditions and are able to collect a wider range of data.

4.3. Drive Experiment Results and Discussion

A.

Drive Experiment Results

Figure 19 shows the roadmap for driving the wheelchair movement during the experience, and some obstacles were set up in the test site in order to increase the difficulty. Twelve subjects controlled the wheelchair movement by eye gaze and eye blink according to the path plan in Figure 19 and each experimented once. All 12 subjects successfully completed the experiment and were able to control their wheelchairs very smoothly to avoid obstacles during the experiment. The shortest time was 2 min and the longest time was 3 min for all 12 subjects to complete the experiment. Figure 19a shows the subjects driving the wheelchair clockwise, and after reaching that end point, the subjects turned around in the same place and moved counterclockwise according to the route in Figure 19b, and completed the whole process after reaching the end point. Figure 19c shows the subject’s remotely controlled wheelchair moving according to the route in the figure. The subject first controls the wheelchair straight ahead to reach the preset point, and then controls the wheelchair backward to the starting point, the subject then controls the wheelchair to turn 90° left, and then 180° right to complete the process.

Figure 20 shows a combined image of experimental data using eye gaze to drive the wheelchair movement, where subjects controlled the wheelchair to move along the route planned in Figure 19a,b. The subject sat in the wheelchair and controlled the wheelchair to move clockwise from the starting point by eye gaze. When the controlled wheelchair reachs the preset end point, it was controlled to stop, turn around in place, and continue to move counterclockwise to the preset end point, thus ending the process.

In the experiment using eye blink controlled wheelchair movement, in order to differentiate the experiments using eye gaze controlled wheelchair movement, the subject sat in front of a monitor with the eye blink detection device placed in front of the eyes, and remotely controlled the wheelchair to move along the route planned in Figure 19c. Figure 21 shows a data set plots of eye blink controlled wheelchair movement experiment. In the figure set, the first row of images (number: 1–4) shows the blink controlled wheelchair moving forward, the second row of images (number: 5–8) shows the blink controlled wheelchair moving backward to its original position, the third row of images (number: 9–12) shows the blink controlled wheelchair turning 90° to turn left, and the fourth row of images (number: 13–16) shows the blink controlled wheelchair turning 180° to turn right. This concludes an experimental cycle.

B.

Discussion of Results

The experimental results show that gaze and eye blink could effectively control the wheelchair to complete the related movements. However, the test subjects had to concentrate on controlling the wheelchair movement throughout the experiment, which was a problem. As a future research topic, the controlled device can provide some feedback to the human body so that it can be alerted in time when the operator is not focused enough.

5. Conclusions

In this paper, the extraction of eye movement information to control relevant mechanical movements was investigated. An eye-movement based HCI system that communicates information to the machine through the human eye was developed in this research. The binarization of the image was used to localize the pupil in this study. Since the difference between the two colors in the eye was apparent, binarization of the pupil image enables the localization of the black pupil, and based on the localization of the pupil, the direction of eye gaze could be determined. Similarly, pixel ratio modelling has been fully used in the eye blink detection. Through operational experiments, it could be confirmed that the eye blink state could be detected, however, the recognition rate could be further improved. Therefore, the machine learning methods SVM and CNN were used for eye blink detection and the accuracy was significantly improved, especially the eye blink detection using CNN method, was tested to reach 99% accuracy.

To further validate these, the system was ported to a power wheelchair. We first modified the hardware of the wheelchair and developed a hardware drive system for the electric wheelchair. This allowed the wheelchair to receive commands from the subject through eye movements. In the experiment, the subject sat in the wheelchair to control its movement, or controls the wheelchair movement remotely. The wheelchair was controlled and followed a set route, successfully avoiding obstacles along the way to the end point, and was tested repeatedly and successfully. The experiments proved that the method of obtaining information expressed by human eyes through image parsing and machine learning has been effective. Future research can optimize the model on this basis to obtain more information conveyed by the human eye, for example, the fatigue level and mental state of a person can be judged by the human eye.