4.3. Classification Results
Each experimental stage corresponds to a different brain state and so to a different class. Thus, the alcohol-free EEG recordings acquired in the first stage formed the class A. Class B consists of the EEG recordings obtained during the second stage, meaning after the 1st dose and the 15-min interval. Accordingly, class C consists of the EEG signals of the third stage (after the 2nd dose and the 15-min interval) and class D contains the data acquired during the fourth stage (after the 3rd dose and the last 15-min interval). In Figure 4
, each experimental stage is shown and each class is clearly distinguished.
Spectral features were used individually for the classification (Scenario 1) and then also in combination with time-based features (Scenario 2). Data from all the 14 electrodes of the Emotiv EPOC+ were used in both scenarios. Thus, in Scenario 1, which comprises 5 different spectral features as presented in Table 1
, a total of 70 different features were extracted for all the 14 electrodes. Accordingly, in Scenario 2 that comprises both spectral and time-based features (11 features), a total of 154 different features were calculated from entire set of electrodes.
To examine the performance of the Grammatical Evolution, each scenario was also tested with 4 well-known classification algorithms (Decision Trees (DT), Linear Discriminant Analysis (LDA), MultiLayer Perceptron (MLP), and k-Nearest Neighbor (KNN)) combined with a dimensionality reduction stage. Principal Component Analysis (PCA) for dimensionality reduction and 3 feature selection techniques (Information Gain, Correlation Attribute Evaluation and Entropy-based) was initially applied in Scenario 1 and Scenario 2, and then each classifier was evaluated.
After applying Principal Component Analysis, 25 features were created for the moderate drinkers group and 37 features were created for the heavy drinkers of the Scenario 1. For Scenario 2, PCA resulted in 31 and 33 new features for moderate and heavy drinkers, respectively. For the application of the Information Gain, Correlation Attribute Evaluation and Entropy-based feature selection, the 10 best features were selected for each feature selection method for Scenario 1 and Scenario 2, according to their information gain, the correlation (Pearson’s) and the entropy regarding the class. Thus, different feature vectors were used to train the 4 different classifiers. Grammatical Evolution can deal with the huge number of features since it contains a feature selection step and thus, no further dimensionality reduction stage is needed. To compare our approach with a combination of dimension reduction and classic classifiers, Waikato Weka Software and IBM SPSS Statistics Software have been employed, using the same set of features.
To train and test the classification algorithms, the 10-fold cross-validation technique was employed. According to this technique, the entire dataset is divided in 10 equal folds. In the first division 9 of these folds are used to train the classifier and the remaining 1 to test it. The procedure is repeated 10 times and each time a different fold is used as test set. In the final stage, the procedure is repeated on the entire dataset for a last time and the reported results are an aggregation of the results of the 10 models.
To evaluate the performance of the Grammatical Evolution in discriminating 4 alcohol-related mental states, the four classifiers were statistically compared using the Paired-sample t
], aiming to examine whether the classification accuracy differences are statistically significant. For this reason, a pairwise comparison of each of the four classifiers was conducted with the Grammatical Evolution. Results in terms of accuracy for the 5 classifiers for both Scenario 1 and Scenario 2 are presented in Table 2
and Table 3
. Furthermore, the statistical t
-value and p
-value are reported in the parenthesis.
Classification accuracy for the group of moderate drinkers when applying the PCA feature reduction technique, ranges from 54.39–66.89% for the four classifiers. Heavy drinkers indicated better classification results for the PCA feature vector (63.33%, 65.19%, 80.06% and 73.97% for DT, LDA, MLP and KNN respectively). Concerning the feature selection methods, Information Gain did not show so promising results. LDA and MLP showed the worst classification accuracy (53.15% for moderate and 50.27% for heavy drinkers for LDA and 55.43% for moderate and 58.89% for MLP, respectively). DT and KNN performed slightly better with 58.99% for moderate drinkers and 59.51% for heavy drinkers when classified with DT and 62.92% for moderate and 60.33% for heavy drinkers when classified with MLP. The selected feature vector with the Correlation Attribute Evaluation obtained the best classification accuracy with KNN (63.22%) following by DT (58.99%), MLP (55.43%) and LDA (50.31%) for the moderate drinkers. For the same feature selection technique, heavy drinkers showed better classification accuracy reaching 76.60% with KNN and 72.57% with DT. LDA and MLP underperformed with 59.41% and 69.79% accuracy respectively. For the Entropy-based feature selection method, the group of heavy drinkers outperformed in the classification accuracy of all classifiers (70.15%, 58.21%, 71.49% and 73.35% for DT, LDA, MLP and KNN respectively), whereas the group of moderate drinkers did not show so good classification results (58.42%, 49.54%, 60.07% and 62.25% for DT, LDA, MLP and KNN respectively). Grammatical Evolution outperformed the previous mentioned classifiers with classification accuracy reaching 85.55% for moderate drinkers and 87.53% for heavy drinkers.
Regarding Scenario 2, which contains spectral and time-based features, features extracted with PCA for the group of moderate drinkers combined with DT, LDA, MLP and KNN showed good classification accuracy (54.96%, 55.07%, 68.28% and 76.35% respectively). Among feature selection methods and PCA, the best accuracy of Scenario 2 was obtained with this PCA, for heavy drinkers with KNN (84.04%) followed by MLP (80.89%), LDA (67.98%) and DT (60.90%). For the feature vector of Information Gain and the moderate drinkers the best accuracy was given by KNN (57.08%), whereas DT, MLP and LDA did not perform so well (50.31%, 48.91% and 44.68% respectively). Classification accuracy for the same feature selection method and the group of heavy drinkers, was slightly better (58.73%, 51.92%, 61% and 62.91% for DT, MLP, LDA and KNN, respectively). Moreover, not great classification results were obtained with the selected feature vector with the Correlation Attribute Evaluation combined with KNN (52.07%) following by DT (40.24%), MLP (48.35%) and LDA (56.30%) for the moderate drinkers. However, Correlation Attribute Evaluation in combination with DT and KNN outperformed for this technique (75.83% and 75.57% respectively) followed by MLP (68.24%) and LDA (59.35%). The Entropy-based feature selection showed the worst classification for both moderate and heavy drinkers. For moderate drinkers the best accuracy was given with MLP (53.98%) followed by KNN (53.61%), DT (50.98%) and LDA (49.28%). Likewise, for the group of heavy drinkers, accuracy values did not show great alterations (52.33%, 51.91%, 50.01% and 49.12% for LDA, DT, KNN and MLP, respectively). Nevertheless, Grammatical Evolution indicated the best classification accuracy for both groups (80.52% and 88.70% for moderate and heavy drinkers).