Computational EEG Analysis Techniques When Playing Video Games: A Systematic Review ^†

Abstract

Video games and electroencephalography (EEG) can be used together in more than one way: cognitive analysis, mood analysis or Brain-Computer Interfaces (BCI), for instance. Nowadays, these two fields are gaining popularity when working together. We have consider that it is important to know what approaches are the most used when using video games and EEG, so we have performed a systematic review through the literature about these two fields together to find the most relevant techniques. Once identified a list of techniques, we briefly explained for what they can be used. Also, we have made a ranking of these techniques by their popularity to get some perspective about them all. After the process, we have concluded that the used techniques are really diverse and that approaches used to BCI and cognitve analysis are very similar between them.

1. Introduction

Electroencephalography is a discipline that studies the brain by the electric potential produced at the scalp. Computational methods can be used to analyse the data from EEG and these methods can get lots of information that cannot be detected just by looking exclusively at the graphs of the raw data. It is noticed that lately using video games alongside EEG is gaining popularity. That is because video games can be used to stimulate many parts of the brain. EEG and video games also are seen together when working with Brain-Computer Interfaces. EEG and video games are used in much more ways together, for example, using EEG for assessing how people feel when playing a video game.

The number of techniques of EEG analysis is really huge so the goal of this paper is to focus in some of the most important methods used when working with video games by conducting a systematic review and, afterwards, analysing the results.

In the next section the systematic review is described by talking about the process followed and also about the results of this review in which a table with the selected articles and a list with the techniques identified are shown. Later, in Section 3, we discuss about the results of Section 2 and about the future work derived from the knowledge obtained in this paper.

2. Systematic Review

Being aware of the increasing popularity of using video games along with EEG, it may be interesting to explore and try to discover different ways to analyse this data in this specific context. In our case, we are interested in computational techniques used to process EEG data. In this sense, we have developed our research question:

RQ1. What are the main tools, methods and techniques that are being used to analyse EEG data when using video games?

Thus, we have conducted a systematic review to answer the formulated RQ1. The protocol to perform the systematic review was the following: (a) To determine the electronic database(s) where exploring the state of art related to the objective of this study; (b) To clearly identify the target keywords and define the searching string; (c) To define inclusion/exclusion criteria, i.e. the mandatory eligibility factors to include documents into the current study; (d) To screen those documents that previously accomplished the eligibility factors by means of the title and abstract; (e) Based on the content of the papers, to select those documents that provide information about EEG analysis techniques; (f) To determine the metrics to characterize them and provide the results of this systematic review.

2.1. Searching Terms

As we have introduced previously, the goal of this systematic review is to identify the main EEG techniques or methods applied to video games. Thus, we have selected the terms “eeg” and “video*game*” or “game” as mandatory search words and, at least, one of the following terms: “techniqu*”, “meth*”, “machine learning” and “analy*”:

eeg AND ( video*gam* OR game ) AND ( techniqu* OR meth* OR ( machine AND learning ) OR analy* ) AND NOT ( ecg OR fmri )

In the search terms we exclude “ecg” and “fmri” because they usually appear along EEG and we want to focus on EEG techniques. We do not exclude “MEG” term because usually both EEG and MEG can be analysed using the same techniques.

2.2. Eligibility Criteria

This systematic review aims to identify recent high quality research works in English. Consequently, we define the following inclusion criteria:

• Impact: based on citations per year, to ensure certain quality. The criterion is to have a minimum of four citation per year from the date of publication.
• Publication year: we limited the search to publications from 1998 that was the year in which the popularity of this subject started to grow.
• Language: This criterion is to select the works understandable by the majority of the readers, being English the most standardized language for scientific publications.

So, the final search string that includes the eligibility criteria is the following:

TITLE-ABS-KEY ( eeg AND ( video*gam* OR game ) AND ( techniqu* OR meth* OR ( machine AND learning ) OR analy* ) AND NOT ( ecg OR fmri ) ) AND PUBYEAR > 1997 AND PUBYEAR < 2018 AND ( LIMIT-TO ( LANGUAGE , "English" ) )

2.3. Document Collection

Considering the domains related to this systematic review, being Computer Science primarily and Medicine secondarily, we have selected Scopus and ACM as electronic databases for this study. However, after searching documents with the previously mentioned search string, the database that provided the greatest number of papers was Scopus (n = 409) and most of the results given by ACM (n = 33) were duplicated records from the Scopus’s results. For this reason, we limited the electronic databases to Scopus.

2.4. Manual Screening

After screening the results through the eligibility criteria, 57 articles remained. The next step was to review the title and abstract of those documents to determine whether they fulfil our objectives based on the formulated research question (RQ1). This step was divided into two phases:

• First phase consisted in determining if the articles were using or not video games while recording the EEG data. This was made because there were some articles that used traditional games and there were some others that did not record the data while using a video game. At the end of this phase, the number of documents were 36.
• In the Second phase we selected the most interesting and relevant documents from the previous phase. After this phase we got the final list of 14 papers.

Figure 1 shows the flow diagram of the systematic review process.

2.5. Results

2.5.1. Initial Search String

The results obtained from the initial search string can provide an overview of the outputs and state of researching on EEG analysis using video games. Figure 2 shows the count of papers per country of first author affiliation. One can observe that United States is the main country followed by China, Germany and United Kingdom.

Regarding the evolution over the year of publication, this is shown in Figure 3 and it can be observed a clear increasing trend from 2006 to 2017.

2.5.2. Final Selected Documents

From the 14 finally selected documents we have identified up to 31 different algorithms and techniques to analyse or pre-process EEG data.

Table 1. Selected articles and summarized content. They are descendently ordered by the number of citations.

Publication	Summary	Techniques
chanel et al. [1]	In this article they find different difficulty levels in games affects players’s emotion and that playing several times in the same difficulty gives rise to boredom. Also they train classifiers to detect three emotional classes that they identify.	Laplacian filter, bandpass filter, LDA, QDA, SVM, EEG_W
Lalor et al. [2]	This paper presents a BCI for binary control with high precision in a 3D game.	Squared FFT, FFT of autocorrelation LDA
Huang et al. [3]	This article shows BCI for a 2D virtual wheelchair game with a multiclass control and with high hit rate.	Laplacian Filter, Hamming window, FFT, Mahalanobis distance
Scherer et al. [4]	It presents a BCI control (of three classes) for a 3D video game.	Bandpass, averaging, LDA, DSLVQ
Finke et al. [5]	They present a BCI for controlling a character on a three-dimensional game board by detecting P300 events.	PCA, LDA, ERP
Russoniello et al. [6]	In this paper, they study the effects of Casual Video Games (CVGs) on mood and stress through EEG and they find that CVGs increase mood and reduce stress.	FFT, LSD (Least Significant Difference)
Wang et al. [7]	It proposes a non linear fractal dimension based method quantify concentration level in neurofeedback games.	HFD, Box-Counting, ROC curve
Berta et al. [8]	They performed a spectral characterization of the video-gaming experience using a a four-electrode EEG.	PCA, FFT, ANOVA, SVM
Millan et al. [9]	This paper reviews the field of BCI, they develop three BCI systems and discuss about current research directions.	Canonical analysis, Gaussian Classifier
Zhang et al. [10]	They propose EEG-based expert system by using complexity features. The EEG-based expert system is tested by using a video game.	DWT, ICA, sample Entropy, Lempel-Ziv complexity, sLoreta, neural networks
Mu et al. [11]	They research about oxytocin (OT) effects on an individual’s brain activity during social interaction using a real-time coordination game and recording EEG in order to examine the OT effects.	Morlet Wavelet Transform, Phase Locking Value
Bai et al. [12]	In the article they develop a BCI system suitable for ALS and PLS patients that does not require too much training. They develop a video game in order to test the system.	Laplacian Filter, Bhattacharyya distance, Mahalanobis distance, Genetic algorithm, ROC curve, SVM
Johnny and Tan et al. [13]	This article describes two experiments in which they explore task classification through EEG. In the second experiment they use tasks that involve playing a video game.	FFT, averaging different values, phase coherence, Correlation based Feature Selection, Bayesian network classifiers
Mondejar et al. [14]	This paper explores cognitive skills and their training through the uses of serious games by using EEG to assess that relation.	Cross-Correlation Coefficient, Dynamic Time Warping

shows the selected articles along a summary and the techniques and algorithms identified. Exploring these documents we have observed that they show different approaches joining EEG and video games in these documents, but there is one that is the most recurrent: Brain Computer Interfaces (BCI). There are many articles talking about cognitive analysis too.

In the papers there are lots of techniques that we have identified and also studied to discover what they can be used for. The following list shows the main techniques identified, alternative names of them and their description:

• Fast Fourier Transform (FFT) [15]. FFT is an efficient algorithm that implements Discrete Fourier Transform (DFT), which transforms a signal from the time domain to the frequency domain.
• Discrete Wavelet Transform (DWT). DWT refers to a family of transformation techniques that decompose signals obtaining information about time and frequency domains, in contrast to FFT that only decomposes into the frequency domain.
• Morlet Wavelet Transform [16]. Morlet Wavelet Transform is a DWT that uses a complex exponential with a Gaussian envelop.
• Hamming Window [17]. Hamming Window is a window function which is a mathematical function used to avoid discontinuity at the beginning and at the end of a signal. It is a preprocessing technique useful when working with FFT in order to get better results.
• Bandpass Filter (Low-pass filter or high-pass filter). It filters the signal removing the parts over a low-pass frequency and bellow a high-pass frequency which are specified explicitly.
• Laplacian Filter (Spatial Laplacian Derivation, SLD). This preprocessing technique is used for improving the spatial resolution of EEG signals by using the Laplace operator.
• Independent Component Analysis (ICA) [18]. ICA is a feature extraction method that transforms multivariate random signal into a signal having components that are mutually independent.
• Principal Component Analysis (PCA) [19]. PCA is a dimensionality reduction technique by creating new uncorrelated features.
• Canonical Analysis (Canonical Variance Analysis). Canonical Analysis is a statistical technique that captures a relationship between a set of predictor variables and a set of criterion variables. It is commonly used in feature selection.
• Distinction Sensitive Learning Vector Quantization (DSLVQ) [20]. DSLVQ is a modification of the LVQ algorithm [21], but DSLVQ rates each feature depending on how informative it is for the classification. DSLVQ is usually used as a feature selection technique.
• Linear Discriminant Analysis (LDA) [22,23]. LDA can be either a classifier or a dimensionality reduction technique when the variable to predict is categorical. It is not a suitable technique in clustering.
• Quadratic Discriminant Analysis (QDA). QDA is a classifier related to LDA but it is more flexible than LDA to classify.
• Support Vector Machine (SVM) [24]. SVM is a supervised machine learning technique that allows classification and regression (SVR). It works by finding linear separators between classes, but it can perform non-linear classification by using the kernel trick.
• Naive Bayes Classifier [25]. This method classifies by giving to an element (defined by its features) a probability to belong to each of the classes. It can be used either for classification or regression.
• Artificial Neural Networks (ANN). Artificial Neural Networks are a powerful machine learning technique inspired by biological neural networks that can do either classification and regression.
• Bhattacharyya Distance [26]. The Bhattacharyya distance is a measure of the similarity between two probability distributions and can be used to know the degree of separability of classes in classification.
• Mahalanobis Distance [27]. The Mahalanobis distance is a measure of the distance between two multidimensional random variables that haves into account the correlation between each variable. It is used in some classifier algorithms as the distance. This is a particular case of Bhattacharyya Distance.
• Event Related Potential (ERP) [28]. ERP is an EEG analysis technique that consists in measuring the EEG of a signal that is result of a certain event or stimulus. At a given moment, the brain is working in many tasks and a single recording has usually a lot of noise, so, in order to deal with it, the ERP technique captures more than one recording of an event and averages them.
• Higuchi Fractal Dimension (HFD [29]. This algorithm returns a number that is related to the fractal dimension of a signal. The fractal dimension is a mathematical concept used to describe fractal objects[30]. Fractal dimension is usually applied in the context of signal processing by assuming that a signal is a fractal object. In EEG, fractal dimension is a feature that indicates the complexity of a signal.
• Box-Counting Dimension [31]. This is another algorithm to compute fractal dimension that uses a different approach than HFD, but returns similar results.
• Sample Entropy [32]. Sample Entropy is a feature that measures the complexity of a signal and it is calculated as the negative logarithm of the number of subseries of length m that have distance < r divided by the number of subseries of length m + 1 that also have distance < r.
• Lempel-Ziv Complexity (LZC)[33]. LZC is a feature that measures the complexity of a signal. It can be calculated by discretizing the signal, then counting the number of different sequences (c) and finally dividing c between the maximum possible number of different sequences in order to normalize it.
• Averaging. A common used method of creating features is by averaging another features, signals data or transformed signals between two limits.
• Cross-Correlation Coefficient (CCC). CCC is a method to determine the degree to which two signals or numerical series are correlated.
• Dynamic Time Warping (DTW) [34]. DTW is an algorithm for measuring similarity between two temporal sequences.
• Phase Locking Value (PLV) [35]. PLV is a feature that can be used to detect the synchronization of two signals.
• sLORETA [36]. sLORETA is a method that computes images of electric neuronal activity from EEG.
• ANOVA. ANOVA is a collection of statistical models and their associated procedures used to analyse the differences among more than two groups.
• Least Significant Difference (LSD). LSD is a statistical technique that consists in creating a value that indicates if two groups are different between them when the difference between their means is lesser than this value.
• Genetic Algorithms (GAs). GAs are a family of optimization algorithms that are inspired by the evolution theory.

Table 2. A classification of the identified techniques based on for what they can be used.

Preprocessing	Dimensionality reduction	ICA, PCA, canonical analysis, DSLVQ, LDA, QDA, GA
	Filtering	Hamming window, bandpass filter, Laplacian filter, ICA
	Transformations	FFT, DWT, Morlet Wavelet Transform
Feature extraction	One Channel	HFD, Box-Counting, sample entropy, LZC, Averaging, ERP
	Multi-channel	CCC, DTW, PLV, ERP
Classification		LDA, QDA, SVM, Naive-Bayes Classifier, ANN
Visualization		sLoreta
Statistical analysis		ANOVA, LSD

shows a classification of each of the above techniques by their usage. Notice that some techniques can be classified in more than one use.

We have also ranked these techniques by searching them in the title abstract and keywords in the first searching terms and also in a search that only includes the term “EEG”. This ranking must be understood as a popularity ranking, not as a ranking of how good is a technique. The ranking of the 10 more used techniques when using video games can be seen in Figure 4. In order to compare the results of the two searches, we have divided the number of occurrences between the total documents in the search, so we have got a percentage of articles that contains each technique.

3. Conclusions and Future Work

In this paper we have explored the state of art of EEG when using video games and we have identified the principal analysis techniques that are used in this specific context. Futhermore, we have ranked this techniques by their popularity to obtain the most relevant ones.

Even if we performed a search over the general techniques when using EEG and video games, our focus is on cognitive analysis and not BCI. By searching in the selected articles, we have noticed that the approaches of cognitive analysis and BCI are not much different between them. Both usually follow the same scheme: first, they preprocess the data, then they extract features and finally they train a classifier with the selected features. However, every BCI paper uses a classifier, but not every cognitive analysis paper uses it. Fractal dimension algorithms and others algorithms to compute complexity seem to be more used in cognitive analysis while Laplacian filter is more used in BCI. In conclusion, even if there are some differences between the two applications, they are similar enough to consider that they both follow the same approach.

In the ranking we can observe that some techniques have a similar popularity in the general search and in the video games search, but there are some other ones that differ significantly. FFT, which is one of the classic techniques and it is widely used in EEG is notoriously less used when working with video games. On the other hand, sample entropy and genetic algorithms are much more used when working with video games. Artificial Neural Networks are more than twice popular when using video games and this fact can be explained because many articles are about BCI and ANN are good classifiers for this purpose. Another important interpretation of the ranking is that even the most used techniques are not too much used (7%) and that means that there is a really big dispersion in the approaches in the articles. While we were elaborating the ranking, we realised that some techniques returns 0 results, even when they are inside the paper. This happens because some papers does not consider that the technique is relevant enough to include it in the keywords, the abstract or the title. The technique “averaging” has not been ranked because it’s a word used in a large number of contexts and not only as an analysis technique, so the results of the searching are not relevant.

In the future, we want to perform some experiments using some of the identified techniques. These experiments will be focused in the cognitive analysis and we will aim to obtain, not only a ranking of techniques based on popularity (the main contribution of this paper) but also a ranking in terms of efficacy and performance.