TCAV-Based Cross-Representation Explainability for Facial Emotion Recognition

Abstract

Deep learning has achieved strong predictive performance in facial emotion recognition (FER), but interpreting the internal representations underlying model decisions remains challenging. In this study, we propose a cross-representation concept-based explainability framework that extends TCAV analysis beyond a single visual representation by evaluating the same human-interpretable facial concepts in both latent image features and explicit landmark-derived geometric features. AffectNet and CK+ facial image databases were analyzed using a frozen pretrained EmoNet backbone with a task-specific classification head, and landmark-derived geometric features were classified with FLAML AutoML. We analyzed seven human-interpretable concepts, such as mouth width, mouth opening, bilateral eye openness, bilateral eyebrow raising, and facial asymmetry, through visual TCAV and P25/P75-based tabular TCAV. The image-based model outperforms the geometric model, with reported accuracies of 0.7111 and 0.9329 on AffectNet and CK+, respectively, versus 0.6640 and 0.8265 on tabular data. TCAV found strong concept–emotion relationships, but they depended on the concept representation. Visual concepts tended to have stronger sensitivities, while geometric concepts had more moderate relationships that were directly interpretable. Visual and geometric representations provide complementary explanations for FER decisions. Concept stability, dataset dependence, and concept-class confounding are still important considerations for reliable TCAV interpretation.

1. Introduction

Facial Expression Recognition (FER) is a multidisciplinary research area that combines concepts from computer vision, machine learning and deep learning, affective computing, psychology and behavioral science, and pattern recognition or signal processing fields, having the aim of feature extraction from facial regions, transfer learning, emotion modeling, facial action coding systems, classification, or feature extraction. Despite significant advances and applications, FER remains challenging due to variations in facial appearance, image quality, and subtle differences between emotional expressions [1,2].

A new approach to achieving correct classification of emotional faces is the hybrid or multimodal learning process. It refers to combining multiple modalities to improve emotion recognition performance. This process integrates signals, audio, physiological data, or handcrafted features to perform more robust and accurate predictions [3,4].

Socially and cognitively, facial expressions can be viewed as components of a larger system of social signaling whose meanings are not limited to classical basic-emotion categories. According to Poggi and D’Errico [5], social signals are communicative or informative cues that provide information about social actions, interactions, emotions, attitudes, evaluations and interpersonal relationships. It suggests that observable behavior may express not only affective states but also socially relevant intentions and relational information. This broader interpretation is exemplified by complex social emotions such as Schadenfreude. Cecconi et al. [6] showed that malicious joy is affected by cognitive and motivational factors like judgments of justice, social image, rivalry, aversion, and identification with ingroups and outgroups. In their socio-cognitive account, emotions involve beliefs, appraisals, goals, and motivational tendencies, and thus cannot always be reduced to one prototypical expressive pattern. Thus, the way FER systems interpret facial cues can encode broader affective and social meanings, supporting a cautious interpretation of concept–emotion relationships, rather than assuming a one-to-one correspondence between a facial configuration and a discrete emotion class.

Convolutional Neural Networks (CNNs) are the most popular way to accomplish FER classification because they can learn hierarchical and discriminative feature representations straight from raw images. Most new methods use pretrained CNNs with weights that have already been trained on large datasets like ImageNet or VGGFace2 [3]. This speeds up convergence to expected results and makes the model more general. Pretraining that is specific to faces is especially useful for FER tasks because it captures structural and identity-related facial features that are important for recognizing emotions [4,7]. To further enhance feature learning, attention mechanisms have been incorporated into pretrained CNN architectures, helping the model to focus on the most important features while ignoring irrelevant information [8].

Geometric features are still useful for getting extra information that CNNs could overlook. Local Binary Patterns (LBP) [3], Histogram of Oriented Gradients (HOG) [9], and geometric facial landmarks [10,11] are geometric descriptors that provide strong representations of texture and facial structure. In a hybrid learning framework, combining these handcrafted features with deep features lets the model use both low-level textural information and high-level semantic representations. This makes the FER model more robust and better at its intended function.

Testing with Concept Activation Vectors (TCAV) is an explainable AI technique that helps us understand why a deep learning model makes a prediction by measuring the influence of human-understandable concepts [12]. TCAV assesses the influence of concepts from each modality on model predictions by projecting the learned concept directions into the model’s latent space. TCAV provides a score that measures how sensitive the model is to a concept [13].

When considering facial emotion recognition, the psychological relevance of CAVs is particularly salient, as they enable the description of model behavior in terms of observable (rather than opaque) latent facial characteristics. The concepts of mouth opening, mouth width, eye openness, eyebrow raising, and facial asymmetry are associated with measurable variations in facial configuration, and thus they can be an interpretable interface between computational representations and psychological descriptions of facial expression [14]. TCAV allows us to test whether a trained model is using facial cues that are meaningful for human emotion perception by measuring the sensitivity of the model’s predictions of emotion to these concept directions. TCAV measures the degree to which concept-related information is represented and used internally by a predictive model rather than whether a facial movement is present or not [15]. Such analyses are important to understand the representational and decision processes of the model but must not be interpreted as evidence for causal relationships between specific facial movements and emotional states.

Among available facial-affect models, EmoNet [16] was selected because it learns high-level visual representations from facial images for emotion-related prediction tasks. This CNN is for automated facial affect analysis in naturalistic settings, enabling both discrete facial expression recognition and continuous valence-arousal estimation. In this study, the pretrained EmoNet backbone is fixed as a visual feature extractor for feeding the obtained latent representations into a task-specific classification head. This approach preserves the relevant features learned from large-scale facial data while reducing the number of trainable parameters and limiting overfitting, which is particularly useful when evaluating model explanations with TCAV on comparatively smaller datasets such as AffectNet and CK+.

We classified the geometric facial features generated from landmarks using FLAML (Fast and Lightweight AutoML) [17], an automated machine-learning framework to discover accurate models and appropriate hyperparameter settings. FLAML selects models and optimizes hyperparameters through the efficient exploration of candidate learners and their configurations, while balancing predictive performance with computational costs. The choice of this AutoML approach was made to avoid a dependence on the manually selected classifier, as well as to supply a data-driven determination of the most appropriate model for the geometric representation.

This study contributes a unified, concept-based explainability framework for FER that evaluates comparable, human-interpretable concepts across visual EmoNet representations and landmark-derived geometric features.

The framework is assessed on AffectNet and CK+, enabling both cross-representation and cross-dataset analysis of concept–emotion relationships.

For visual models, TCAV is integrated into stratified five-fold cross-validation using fold-specific training references and validation-only concept evaluation (wide mouth, open mouth, left eye open, right eye open, left eyebrow raised, right eyebrow raised, and high asymmetry), whereas tabular concepts are defined through training-only P25/P75 thresholds and evaluated using a model-agnostic finite-difference TCAV formulation. The geometric feature set was normalized and supplied to FLAML for multiclass facial emotion classification.

Beyond identifying influential concepts, the study analyses explanation stability, concept–class confounding, CAV separability, and disagreement between visual and geometric representations.

The resulting framework links computational concept sensitivity with psychologically interpretable facial characteristics while maintaining a clear distinction between model explanation and causal interpretation of emotional behavior.

Our main goal is to compare how semantically related facial concepts are represented and how they affect predictions in latent visual and explicit geometric representations in a unified TCAV-based approach. The study is therefore mainly methodological and interpretability-oriented. A second aim is to investigate whether these concept sensitivities relate consistently or represent dependently to the emotion classes of AffectNet and CK+. Our aim is not to propose a new general psychological theory of facial emotion recognition, nor to demonstrate that the examined facial features are causal determinants of emotional states. This distinction is emphasized in the introduction and conclusion.

2. Related Work

Hybrid FER methods [18,19,20,21,22,23,24,25,26,27,28,29,30] fused information from different sources such as facial expressions, deep [18,19,20,21,22], speech [18,20,26], temporal [24], metadata [20], text [19,23,24,25,26], visual [25,26,27,28], or handcrafted [21,22], and keypoint [29,30] features.

The advantage of multimodal learning is that the classification becomes more accurate and robust than single-modality approaches because each modality offers complementary information about a person’s emotional state.

In the paper proposed by Mamieva et al. [18], an attention-based multimodal framework is implemented that processes facial and speech features using separate encoders and then selectively focuses on the most informative elements through an attention mechanism. The proposed model captures both low-level and high-level facial features to enhance representation and effectively handles differences in feature size across modalities that are fused into a unified multimodal representation, which is then used for emotion classification.

Qui et al. [19] introduced multimodal ideology prediction, where a model determines political leaning using both text and images together instead of relying on a single modality.

In the paper published by Ye et al. [20], an emotion word reading experiment applied to the AVEC 2014 dataset is proposed, where features from facial expressions and audio features are extracted for depression recognition.

The multimodal feature fusion approach for 3D facial expression recognition, by combining handcrafted features and deep CNN features and representing them using covariance matrices in a manifold space, which improves recognition performance under challenging conditions, by Hariri and Farah [21] was implemented on the BU-3DFE and Bosphorus datasets.

The study proposed by Sun et al. [22] integrated a multimodal fusion framework for facial expression recognition in the wild, combining deep CNN features and handcrafted texture features from both static images and video sequences, leading to significantly improved recognition performance.

In recent research, Rishu and Kukreja [23] classified emotional faces with vision transformers in a multimodal process, with the text and visual features having the aim to improve emotion detection in comics, achieving significantly higher accuracy when the Multimodal Comic Emotion Dataset (MCED) was used.

The audio, visual, and textual features were integrated into a hybrid fusion strategy, and a Multimodal Emotion-Aware Interaction Framework (E-MXNet) was proposed by Ren [24] and a pipeline for detection of emotionally intelligent and context-sensitive interaction experiences. The study was based on IEMOCAP, RAVDESS, and CREMA-D datasets.

Feng et al. [25] employed a new concept called Dual-level Language-Driven Cross-Modal Emotion Alignment (DLCEA), composed of a fully unsupervised multimodal framework that uses text as a guiding anchor to align emotions across text, audio, and visual data at both global and local levels. The DLCEA presents a dual-level semantic alignment scheme and uses intrinsic emotion semantics in text to direct cross-modal alignment; experiments on multimodal datasets such as MAFW, MOSEI, and IEMOCAP were conducted.

The work performed by Wang et al. [26] highlights key limitations in multimodal emotion recognition, namely the lack of large-scale aligned datasets and domain gaps, and implements Synthetic-to-Real Multimodal Emotion Recognition (SRMER) on text, speech, and visual features; this framework generates synthetic multimodal emotional data and applies cross-domain adaptation techniques to improve generalization and performance on MELD and CMU-MOSEI real-world datasets.

Moorthy and Moon [27] developed a multimodal learning framework for emotion recognition dubbed the Hybrid Multi-Attention Network (HMATN) with the Hybrid Attention of Single and Parallel Cross-Modal (HASPCM) mechanism for examining the link between audio and visual inputs. The suggested method’s efficiency is illustrated via experiments on complex films from the AffWild2 and AFEW-VA datasets.

In the papers [10,29], the landmark indices for computing Euclidean distance between different elements of the face were used as geometric features. The Gated Recurrent Unit (GRU)-based model and Random Forest classifier leverage engineered features for classification of valuable emotional insights extracted from a new dataset collected from child–robot interactions [29]. The ResNet50 backbone and representative Keypoint Feature Attention (KFA) module are supplied with performance-affecting AffectNet, RAF-DB, and FERPlus datasets, improving FER classification [10].

3. Materials and Methods

3.1. Datasets and Hyperparameters

The original AffectNet dataset is available in the Kaggle repository at https://www.kaggle.com/datasets/mstjebashazida/affectnet, with the last access date set to 12 May 2026. The dataset includes the following images: anger (1222), contempt (1555), disgust (1223), fear (1505), happy (2337), neutral (2758), sad (3074), and surprise (2114) [30].

The Extended Cohn-Kanade (CK+) database [31] is a set of 593 facial expression sequences, collected from 123 subjects in controlled laboratory conditions. Each sequence begins with a neutral or near-neutral face configuration and ends with a peak expression. Among the 593 sequences, 327 sequences from 118 participants have validated labels of seven discrete emotion categories: anger, contempt, disgust, fear, happiness, sadness, and surprise. Under the common FER preprocessing protocol, the last three frames of each labeled sequence were selected as peak-expression samples, resulting in 981 images. This yielded a class distribution consisting of 135 anger, 54 contempt, 177 disgust, 75 fear, 207 happiness, 84 sadness, and 249 surprise images. It is probable that several selected frames are taken from the same expression sequence and that participants provide several labeled sequences. Thus, images are not statistically independent individual observations but are clustered by subject and sequence. The web address is https://app.hyper.ai/console/public/datasets/UVlM8LquLsL/1/overview?path=%2Fdata%2F, accessed on 18 September 2026. The five-fold cross-validation on the CK+ dataset was performed at the subject level to avoid identity and sequence leakage in the training and validation data. The 118 participants corresponding to the 327 labeled expression sequences were divided into five non-overlapping folds, with all images and expression sequences from the same participant in the same fold. In each fold of cross-validation, about 80% of the participants were used for training and the remaining 20% for validation. Depending on the fold size, this resulted in 94–95 subjects (79.7–80.5%) for training and 23–24 subjects (19.5–20.3%) for validation. The fold assignment was optimized to keep the emotion-class distribution as close as possible while ensuring subject independence. Therefore, no participant contributed images to both the training and validation sets, limiting the risk that model performance was overestimated due to subject-specific or sequence-specific similarities.

For the AffectNet dataset, the image-based classification pipeline used stratified five-fold cross-validation at the image level. The complete dataset was shuffled using a fixed random seed of 42 and divided into five approximately equal folds while preserving the emotion-class distribution. In each iteration, four folds, corresponding to approximately 80% of the images, were used to train a newly initialized classification head, while the remaining fold, approximately 20%, was used for validation. Each image therefore served as validation data once across the five folds. The pretrained EmoNet backbone remained frozen throughout training. Fold-specific performance metrics were summarized as mean ± standard deviation, and out-of-fold predictions were combined to generate the overall confusion matrix and ROC curves. TCAV random references were drawn only from the corresponding training fold, whereas TCAV sensitivity was evaluated only on the held-out validation fold.

The concepts were selected from the AffectNet dataset; these are wide mouth (50), open mouth, left eye open (50), right eye open (55), left eyebrow raised (52), right eyebrow raised (54) and high asymmetry (56).

The tabular data computed from each dataset are uploaded in the public repository https://github.com/simonamoldovanu/AI, with the last access date set to 21 August 2026.

Table 1 summarizes the main training and preprocessing configurations used for the two classification branches. The CNN branch uses a frozen pretrained EmoNet backbone with a task-specific fully connected classifier, whereas the geometric-feature branch uses FLAML AutoML with stratified cross-validation to automatically select the most suitable tabular classifier and its configuration.

Table 1. Hyperparameters and experimental settings for the CNN and AutoML classification models.

CNN		AutoML
Hyperparameters	Value	Hyperparameters	Value
Pretrained backbone	EmoNet	Split type	Stratified
Backbone training	Frozen	Missing-value imputation	Median
Input image size	256 × 256	Feature scaling	StandardScaler
Cross-validation	Stratified 5-fold AffectNet: image-level stratified 5-fold; CK+: subject-grouped 5-fold	AutoML framework	FLAML
Hidden layer size	256	Optimization metric	macro F1
Activation	ReLU	AutoML time budget	120 s
Dropout	0.35	Cross-validation	Stratified 5-fold
Optimizer	AdamW	Number of parallel jobs	−1
Learning rate	1 × 10−3	Auto augmentation	False
Weight decay	1 × 10−4	Random seed	42
Batch size	32	Split type	Stratified
Epochs	30	Train partition	70%
Loss function	CrossEntropyLoss	Test partition	30%
Random seed	42	-	-

Table 2 summarizes the main parameters used for concept-based explainability in the visual and geometric representations. Visual TCAV learns CAVs from concept images and random training-image references and computes sensitivity using analytical gradients, while tabular TCAV defines concepts from training-only P25/P75 thresholds and estimates directional sensitivity using a symmetric finite-difference approximation.

Table 2. Hyperparameters and experimental settings for visual and tabular TCAV analysis.

Hyperparameter	Visual TCAV	Tabular P25/P75 TCAV
Number of concepts	7	7
Number of TCAV/CAV experiments	10	10
Maximum samples per group	100	100
Minimum concept/group samples	10	10
Evaluation on correctly classified samples only	True	True
CAV classifier	Logistic Regression (LR)	LR
Logistic regression max iterations	5000	5000
Class weighting	Balanced	Balanced
Concept-positive definition	Concept images	Feature ≥ P75
Concept control definition	Random training images	Feature ≤ P25
P25 threshold	—	0.25
P75 threshold	—	0.75
Middle P25–P75 interval	—	Excluded
Directional sensitivity	Analytical gradient CAV	Symmetric finite difference
Finite-difference ε	—	0.05
Probability floor	—	1 × 10−12

The FLAML AutoML procedure selected Light Gradient Boosting Machine (LightGBM) as the best-performing classifier for the geometric feature representation. The optimal configuration consisted of 21 boosting estimators, 8 leaves per tree, a minimum of 11 samples per child node, and a learning rate of 0.5514. The histogram binning parameter was set to a log max bin of 5, while all available features were considered at each boosting step (colsample bytree of 1.0). This configuration reflects a relatively compact gradient-boosted tree model with moderate regularization, selected automatically by FLAML to optimize classification performance on the geometric facial features.

3.2. Hardware and Software

The experiments were implemented in Python 3.12 using FLAML 2.6.0, NumPy 2.5.1, pandas 3.0.5, scikit-learn 1.9.0, SciPy 1.18.0, Matplotlib 3.11.1, and Joblib 1.5.3. FLAML was used for automated model selection and hyperparameter optimization, while scikit-learn was used for preprocessing, stratified cross-validation, classification metrics, and logistic-regression-based concept activation vectors.

Experiments were conducted on an Apple Mac Studio (2023) equipped with an Apple M2 Ultra SoC (ARM64 architecture) and 64 GB of unified memory, running macOS Sequoia 15.1.1. PyTorch 2.11.0 computations were accelerated using Apple’s Metal Performance Shaders (MPS) backend.

3.3. Geometric Features

Psychology relies on landmark indices for detecting facial features and quantifying non-verbal communication, emotions, social cognition, and personality traits. Psychologists can detect emotional states by tracking mouth, eye aperture, ear, and eyebrow movements. For hybrid learning, the mouth width, mouth open, left eye aspect ratio, right eye aspect ratio, left eyebrow, right eyebrow, and asymmetry features were computed for eight FERs from the AffectNet and seven from CK+ datasets. For each feature, the main landmark indices were mentioned from the 468 proposed in the MediaPipe Face Mesh model [32,33]:

Let each landmark be the following:

p_i = (x_i, y_i), i = 1, …, 468

(1)

Distance between two points:

$$d(p_i,p_j)=\sqrt{(x_i-x_j{)}^2+(y_i-y_j{)}^2}$$

(2)

1. Mouth width computed for indices p₆₁ for left mouth corner and p₂₉₁ for right mouth corner:

Mouth_width = ∥p₆₁ − p₂₉₁∥

(3)

2. Mouth opening computed for indices p₁₃ for upper lip center and p₁₄ for lower lip center:

Mouth_open = ∥p₁₃ − p₁₄∥

(4)

3. Right eye aspect ratio (EAR_right) use p₃₃ and p₁₃₃ points for horizontal and p₁₅₉, p₁₄₅, p₁₅₈, and p₁₅₃ points for vertical.

$${\mathit{EAR}}_{\mathit{right}}={\displaystyle \frac{\parallel p_{159}-p_{145}\parallel +\parallel p_{158}-p_{153}\parallel }{2 ·\parallel p_{33}-p_{133}\parallel }}$$

(5)

4. Left eye aspect ratio (EAR_left) use p₃₆₂ and p₂₆₃ points for horizontal and p₃₈₆, p₃₇₄, p₃₈₅, and p₃₈₀ points for vertical.

$${\mathit{EAR}}_{\mathit{left}}={\displaystyle \frac{\parallel p_{386}-p_{374}\parallel +\parallel p_{385}-p_{380}\parallel }{2 ·\parallel p_{362}-p_{263}\parallel }}$$

(6)

5. Eyebrow distance is computed for the right eyebrow p₇₀, and p₃₃ point right eye corner. Right eyebrow and has the left eyebrow p₃₀₀, and left eye corner p₂₆₃ points

Eyebrow_right = ∥p₇₀ − p₃₃∥ 
Eyebrow_left = ∥p₃₀₀ − p₂₆₃∥

(7)

6. Facial asymmetry was assessed using ocular and eyebrow features on the left and right sides of the face. We consider two complementary components: eye openness, represented by the Eye Aspect Ratio (EAR), and eyebrow position. The absolute difference between left and right EAR values quantifies the unequal level of eye opening, and the absolute difference between left and right eyebrow measurements quantifies inconsistencies in eyebrow positioning. The overall asymmetry score is given by the following:

Asymmetry = ∣EAR_left − EAR_right∣ + ∣Eyebrow_left − Eyebrow_right∣

(8)

Final geometric features vector:

F = [Mouth_width, Mouth_open, EAR_left, EAR_right, Eyebrow_left, Eyebrow_right, Asymmetry]

(9)

Table 3 shows the geometric facial features extracted from MediaPipe landmarks and their corresponding emotional indicators used in multimodal facial expression recognition [32].

Table 3. Theoretical correspondence between selected geometric facial concepts, facial actions, and emotion categories.

Selected Concept	Theoretical Facial-Action Correspondence	Most Defensible Emotion Motivation
wide_mouth	Horizontal lip-corner displacement; related to corner puller, lip stretcher	happiness and fear
open_mouth	Lip separation/jaw lowering; related to lips part and jaw drop	disgust, surprise and fear
left_eye_open/ right_eye_open	Increased palpebral aperture.	anger, surprise and fear
left_eyebrow_raised/right_eyebrow_raised	Approximate eyebrow elevation; related to inner-brow raiser and outer-brow raiser.	sadness, surprise and fear
asymmetry	Left–right imbalance in facial configuration	contempt

The seven facial concepts were selected because they are simple, quantifiable geometric proxies for facial movements that are often involved in categorical facial expression configurations. The neutral class is best understood as a low-expression reference state. Characteristic lowering of the brow and tightening of the lips are not explicitly measured, so the eye aperture changes only partially reflect anger. Mouth opening may be a partial indicator of disgust, but important actions, such as brow lowering and lip-corner depression, as well as nose wrinkling and raising the upper lip, are not included in the geometric feature set. A more plausible relation to eyebrow displacement is through inner-brow raising in sadness, but other characteristic actions, such as brow lowering and lip-corner depression, are not directly encoded.

To minimize the impact of face scale and camera distance, distance-based geometric features were normalized by the interocular distance calculated between the outer eye landmarks prior to training the model. Eye aspect ratio features are already scale-invariant because they are defined as the ratio of distances between landmarks. Following geometric normalization, all seven tabular features were standardized using Standard Scaler fitted only on the training partition and subsequently applied to the validation/test data.

3.4. Convolutional Neural Network and AutoML

The image-based classification pipeline uses the pretrained EmoNet CNN as a fixed feature extractor rather than retraining it from scratch. EmoNet is a deep architecture for FER analysis that jointly models categorical facial expressions and continuous valence and arousal information. Input facial images were resized to 256 × 256 for EmoNet. The CNN parameters were frozen and the representation that is obtained directly after the avg_pool_2 layer was used as the image feature vector. This strategy is equivalent to transfer learning with a frozen CNN backbone, where the emotion-specific representation learned by EmoNet can be reused while reducing the number of trainable parameters and the risk of overfitting on the target dataset. The extracted features are then input to a newly initialized classification head which includes a fully connected layer with 256 hidden units, ReLU activation, a dropout rate of 0.35, and a final linear layer that outputs the target emotion logits. Note that this component is called CNNEmotionModel in the code. In each of the five stratified cross-validation folds, a new classification head is trained independently with cross-entropy loss and an AdamW optimizer, while keeping the EmoNet representation frozen. The suitability of EmoNet for this application is underscored by the fact that the original architecture was developed for facial affect recognition in naturalistic conditions, integrating convolutional facial representations with emotion-specific feature learning [16].

The tabular classifier was tuned with FLAML AutoML. The AutoML procedure performed model selection and hyperparameter optimization jointly, subject to a computational budget. Candidate estimators were LightGBM, XGBoost, depth-constrained XGBoost, Random Forest, Extra Trees, stochastic gradient descent classification, and L1-regularized logistic regression [17].

The datasets were divided into a stratified 70% training split and a separate 30% hold-out test split for the geometric representations with a fixed random seed of 42. Median imputation and standardization were performed on the full training partition prior to FLAML model selection. FLAML used stratified five-fold cross-validation on this preprocessed training data to perform model selection and hyperparameter optimization, optimizing for macro F1-score.

After the model was selected, we evaluated the selected estimator through an additional stratified five-fold cross-validation on the same 70% training partition. In this further evaluation, imputation and standardization were done independently on each fold’s training portion and applied to its validation portion. For each performance metric, the arithmetic mean and sample standard deviation were calculated over the five validation-fold scores, using n – 1 = 4 as a denominator for the variance. These are mean ± SD values and are shown in Table 4. They represent development-set cross-validation performance.

Table 4. Classification performance of the image-based and geometric models on AffectNet and CK+, with geometric-model results.

Metrics	Image Data		Tabular Data
	AffectNet	CK+	AffectNet	CK+
Accuracy	0.7111 ± 0.0094	0.9329 ± 0.0219	0.6640 ± 0.0082	0.8265 ± 0.0263
Precision macro	0.6561 ± 0.0161	0.9134 ± 0.0415	0.6301 ± 0.0040	0.7923 ± 0.0328
Precision weighted	0.7023 ± 0.0125	0.9374 ± 0.0217	0.6400 ± 0.0039	0.8348 ± 0.0294
Recall macro	0.6511 ± 0.0083	0.9185 ± 0.0246	0.6281 ± 0.0035	0.7650 ± 0.0349
Recall weighted	0.7111 ± 0.0094	0.9329 ± 0.0219	0.6640 ± 0.0082	0.8265 ± 0.0263
F1 macro	0.6408 ± 0.0144	0.9102 ± 0.0352	0.6291 ± 0.0027	0.7694 ± 0.0353
F1 weighted	0.6967 ± 0.0108	0.9331 ± 0.0214	0.6394 ± 0.0027	0.8251 ± 0.0295
MCC	0.6662 ± 0.0115	0.9182 ± 0.0273	0.4550 ± 0.0096	0.7902 ± 0.0321
AUC macro	0.9504 ± 0.0031	0.9920 ± 0.0121	0.7682 ± 0.0049	0.9687 ± 0.0137
AUC weighted	0.9556 ± 0.0033	0.9928 ± 0.0095	0.7698 ± 0.0049	0.9764 ± 0.0113

The model returned by FLAML was evaluated separately on the reserved 30% hold-out test partition, which was not used to fit preprocessing transformations or perform model selection and hyperparameter optimization. The evaluation resulted in a single value for each performance metric. The standard deviations reported in Table 4 were not calculated from the hold-out test set.

3.5. Testing with Concept Activation Vectors

TCAV is a post hoc explainability method that encodes each concept as a direction in the feature space of the model and measures the effect of human-interpretable concepts on model predictions. In image-based models, we feed examples of concepts to the CNN to obtain internal activations. We then learn a Concept Activation Vector (CAV) by training a linear classifier to separate activations associated with the target concept from those of a reference set. Then, the importance of a concept for a given class is estimated by the directional derivative of the class score in the CAV direction.

The TCAV score is defined as the fraction of evaluated samples for which the directional derivative is positive. In the tabular setting, a similar principle can be applied to semantically meaningful numerical variables by defining concept-positive and concept-negative groups from the data distribution. For example, samples above the 75th percentile (P75) can be used as a high-expression concept and samples below the 25th percentile (P25) as the control group. A CAV is learned in the standardized tabular feature space. If the tabular classifier is differentiable, we can directly compute directional derivatives. For non-differentiable models, we estimate the directional effect using finite differences along the CAV direction. Therefore, TCAV provides a common conceptual framework for comparing explanations between image and tabular models, allowing one to assess the impact of the same semantic facial concepts in both latent visual representation and explicit geometric feature spaces [15].

The FLAML analysis provides a complementary perspective because it operates directly on explicit facial geometry rather than on latent image features. The tabular representation contains Mouth Width, Mouth Open, Left EAR, Right EAR, Eyebrow Left, Eyebrow Right, and Asymmetry concepts. These variables correspond directly to the seven semantic concepts examined with the CNN. This P25 and P75 percentile-based strategy has an important advantage: it avoids imposing an arbitrary absolute threshold for what constitutes an “open”, “eye” or “mouth”. Concepts are defined relative to the distribution of facial measurements within the dataset. Consequently, the semantic interpretation becomes high versus low expression of a geometric characteristic, rather than the presence versus complete absence of a facial action.

For the image-based classifier, we ran TCAV on the latent representation extracted by the frozen EmoNet CNN backbone. We projected sets of concept-specific images that correspond to seven human-interpretable facial concepts (mouth width, mouth open, left EAR, right EAR, left eyebrow, right eyebrow, and asymmetry) into the feature space of EmoNet. For each concept, we learned a CAV using LR to separate concept activations from randomly selected reference activations, drawn only from the training set of each cross-validation fold. Next, we used the directional derivative of each target-class logit in relation to the feature representation of EmoNet along the CAV direction to measure the TCAV sensitivity on held-out validation samples. The TCAV score was calculated as the fraction of correctly classified validation samples from the target class with a positive directional derivative.

For the visual TCAV analysis, 95% confidence intervals were estimated with a hierarchical bootstrap accounting for the nested structure of the five cross-validation folds and the repeated CAV constructions within each fold. For each concept–emotion pair, folds were resampled with replacement, and, within each resampled fold, the run-level TCAV scores from the 10 CAV repetitions were resampled with replacement. The mean TCAV score was recalculated for each bootstrap sample, thereby incorporating variability related to both the validation folds and repeated CAV construction. Thus, the resulting 95% confidence intervals estimate the stability of the visual TCAV estimates across folds and CAV repetitions, rather than independent-sample uncertainty across facial images.

For tabular TCAV, concept thresholds were obtained globally from the training partition, rather than per emotion class. For each of the seven geometric features from all training samples, regardless of their emotion labels, the 25th (P25) and 75th (P75) percentiles were calculated. Samples with feature values greater than or equal to P75 were assigned to the concept-positive group, and samples with values less than or equal to P25 were assigned to the control group. Observations in the P25-P75 interval were excluded from CAV estimation. So, we did not use emotion labels to select concept-positive or control samples. Once the concept groups were created, we examined the class composition of each concept group to see if there was any potential for confounding between concepts and classes. A strong presence of a single emotion class in a concept-positive or control group was interpreted as a sign that the learned CAV could represent both the intended geometric feature and class-related structure. These cases were retained but interpreted cautiously, rather than as purely class-independent concept effects.

We used an empirical random-concept test to assess the statistical support of each concept–emotion relationship. For each pair, 1000 size-matched random concept directions were generated from the corresponding training partition and evaluated with the same TCAV procedure as the observed concept. The resulting null distribution was used to get two-sided empirical p-values that were corrected for multiple comparisons by the Benjamini–Hochberg (BH) false-discovery-rate procedure. Thus, the values in Table 5, Table 6, Table 7 and Table 8 should be reported as BH-adjusted p-values only. The 10 repeated CAV constructions were only used for evaluating the TCAV stability.

Table 5. Mean TCAV scores and stability of the strongest concept–emotion relationships in the AffectNet image representation.

Concept	Target Emotion	TCAV Mean ± SD	p-Values	95% Hierarchical-Bootstrap CI	Interpretation
open_mouth	anger	0.997 ± 0.012	0.021	[0.991, 1.000]	Very strong, highly stable positive sensitivity
left_eyebrow_raised	contempt	0.989 ± 0.024	0.028	[0.968, 0.999]	Very strong and stable
open_mouth	disgust	0.983 ± 0.042	0.033	[0.961, 1.000]	Very strong and stable
left_eye_open	neutral	0.980 ± 0.131	0.031	[0.924, 1.000]	Strong positive sensitivity
wide_mouth	contempt	0.976 ± 0.049	0.035	[0.942, 0.999]	Strong and stable
wide_mouth	sad	0.974 ± 0.044	0.031	[0.943, 0.993]	Strong and stable
open_mouth	surprise	0.900 ± 0.224	0.042	[0.778, 0.985]	Strong positive association, with some variability
left_eye_open	happy	0.895 ± 0.275	0.040	[0.791, 0.981]	Strong mean effect but less stable
open_mouth	fear	0.851 ± 0.300	0.044	[0.668, 0.986]	Positive but relatively variable

Table 6. Mean TCAV scores and stability of the strongest concept–emotion relationships in the AffectNet tabular data representation.

Concept	Target Emotion	TCAV Mean ± SD	p-Values	95% Hierarchical-Bootstrap CI	Interpretation
wide_mouth	happy	0.769 ± 0.009	0.041	[0.763, 0.777]	Strong, highly stable positive association
left_eyebrow_raised	surprise	0.655 ± 0.015	0.044	[0.651, 0.666]	Clear positive sensitivity
right_eyebrow_raised	surprise	0.653 ± 0.005	0.044	[0.645, 0.657]	Clear and very stable
left_eye_open	surprise	0.633 ± 0.020	0.045	[0.616, 0.643]	Moderate-to-strong positive association
wide_mouth	contempt	0.615 ± 0.009	0.052	[0.609, 0.620]	Stable positive association
right_eye_open	fear	0.608 ± 0.044	0.055	[0.585, 0.645]	Positive association
open_mouth	surprise	0.600 ± 0.008	0.060	[0.595, 0.609]	Stable positive relationship

Table 7. Mean TCAV scores and stability of the strongest concept–emotion relationships in the CK+ image representation.

Concept	Emotion	TCAV Mean ± SD	p-Values	95% Hierarchical-Bootstrap CI	Interpretation
high_asymmetry	disgust	0.945 ± 0.123	0.007	[0.863, 0.997]	Strong positive sensitivity and comparatively good stability
right_eyebrow_raised	disgust	0.919 ± 0.187	0.007	[0.786, 0.998]	Strong positive sensitivity with moderate variability
right_eye_open	disgust	0.878 ± 0.246	0.020	[0.684, 0.985]	Strong positive sensitivity, but variable across runs/folds
left_eye_open	disgust	0.843 ± 0.248	0.022	[0.665, 0.984]	Strong positive sensitivity, with noticeable variability
wide_mouth	contempt	0.833 ± 0.299	0.022	[0.600, 0.993]	Strong mean positive sensitivity but limited stability and small evaluation set
left_eye_open	fear	0.822 ± 0.278	0.029	[0.593, 0.978]	Strong mean positive sensitivity but variable
high_asymmetry	fear	0.726 ± 0.314	0.033	[0.490, 0.952]	Positive directional sensitivity with substantial variability
wide_mouth	anger	0.723 ± 0.390	0.044	[0.348, 0.992]	Positive mean sensitivity but poor stability across folds
right_eye_open	contempt	0.663 ± 0.345	0.045	[0.428, 0.876]	Moderate positive sensitivity with high variability
left_eyebrow_raised	disgust	0.614 ± 0.403	0.048	[0.316, 0.878]	Moderate mean sensitivity but very high variability

Table 8. Mean TCAV scores and stability of the strongest concept–emotion relationships in the CK+ tabular data representation.

Concept	Emotion	TCAV Mean ± SD	p-Values	95% Hierarchical-Bootstrap CI	Interpretation
open_mouth	disgust	0.786 ± 0.019	0.038	[0.779, 0.802]	Strongest positive relationship
open_mouth	contempt	0.636 ± 0.000	0.040	[0.636, 0.636]	Clear and extremely stable
left_eye_open	sadness	0.600 ± 0.025	0.040	[0.588, 0.618]	Moderate positive relationship
right_eye_open	sadness	0.582 ± 0.019	0.044	[0.559, 0.588]	Moderate and stable
left_eyebrow_raised	happy	0.573 ± 0.016	0.048	[0.563, 0.582]	Moderate positive sensitivity
right_eyebrow_raised	happy	0.558 ± 0.014	0.053	[0.552, 0.570]	Moderate positive sensitivity
high_asymmetry	sadness	0.547 ± 0.068	0.055	[0.500, 0.582]	Weak-to-moderate positive effect
left_eyebrow_raised	sadness	0.535 ± 0.033	0.058	[0.512, 0.553]	Slight positive sensitivity
open_mouth	fear	0.523 ± 0.032	0.058	[0.492, 0.538]	Very weak positive tendency

4. Results and Discussion

This section is divided into two main parts: the results obtained from tabular data and images; each is followed by the interpretation of the concepts with TCAV. The results refer to the classification of eight classes from AffectNet and seven from CK+, shown in Table 4. The quality metrics such as accuracy, precision_macro, precision_weighted, recall_macro, recall_weighted, F1_macro, F1_weighted, MCC, AUC_macro, and AUC_weighted were computed for each category of image data.

The tabular data obtained from each image from AffectNet and CK+ are stored in a public repository. The raw data, together with the P25, P50, and P75 percentiles, are shown in Figure 1 for the CK+ dataset and Figure 2 for AffectNet, respectively. We proceeded with this representation because we want to highlight the original structure of the data and the value of each percentile overlay on the violin plots. Class-specific percentile summaries were used only for visualization of the geometric feature distributions. In contrast, the P25/P75 thresholds used to construct tabular TCAV concepts were calculated globally from the training partition without using emotion labels. The percentile markers are also useful for understanding how the concepts used in the tabular TCAV analysis were defined. Each subgraph shows in the title the name of the features and the name of the class. Figure 3 shows the ROC curve graphs for each dataset and deep learning method.

Figure 1. Violin plots of geometric facial features across emotion classes in CK+, with P25, median, and P75 percentile markers.

Figure 2. Violin plots of geometric facial features across emotion classes in AffectNet, with P25, median, and P75 percentile markers.

Figure 3. ROC curves; (a) AffectNet images; (b) AffectNet tabular data; (c) CK+ images; (d) CK tabular data.

The geometric representation and stable concept–emotion relationships for AffectNet and CK+ tabular data in Table 5, Table 6, Table 7 and Table 8 are represented; these are followed by the interpretation of the information.

The comparison between visual and geometric TCAV scores is based on semantic and directional correspondence and not on exact numerical equivalence. Both representations are analyzed using the same human-interpretable facial concepts and target emotion classes. The TCAV score is the fraction of samples tested where the movement in the corresponding concept direction leads to a positive change in the target-class output. This common direction interpretation can be used as a basis for judging whether a concept–emotion association is concordant or discordant across representations. However, these two implementations of TCAV operate in different feature spaces and depend on different procedures for concept construction and sensitivity estimation. Visual CAVs are learned from concept images in the EmoNet latent representation and evaluated via the use of analytical gradients, whereas geometric CAVs are derived from feature groups defined by P25/P75 and evaluated via finite-difference directional sensitivity. Thus, cross-representation comparisons are interpreted mainly in terms of direction and relative consistency of pattern, and absolute TCAV values and ΔTCAV are treated descriptively, not as calibrated measures of effect size.

The violin plots provide a complementary view of the seven geometric facial features, not only showing the class-specific central tendencies but also the shape, spread, and overlap of their distributions. To summarize, the plots show that the geometrical features of facial expressions are significantly different between CK+ and AffectNet. This is expected due to the different nature of the two datasets, CK+ consisting mostly of controlled and posed expressions and AffectNet consisting of more heterogeneous in-the-wild facial images. Thus, CK+ frequently exhibits more class-specific geometric patterns, whereas AffectNet displays broader distributions and more overlaps between emotion classes.

Happiness is especially noticeable in mouth width in both datasets, with the distribution skewed toward larger values. This effect is particularly strong in CK+, where happy expressions are heavily dominated by wide-mouth samples. In AffectNet, the separation is less extreme, although happiness still tends to be in the top part of the distribution. Thus, the increased horizontal extension of the mouth is an important geometric feature of happiness, but its discriminative value decreases in unconstrained conditions where the differences in pose, expression intensity, facial morphology, and image acquisition add additional variability.

The distributions of the mouth openings are different. In CK+, surprise is clearly shifted to larger mouth opening values, representing the strong mouth-opening component of many posed surprise expressions. Happiness also plays a big part in the higher end, but to a lesser degree. In AffectNet, mouth opening is more widely distributed among surprise, fear, disgust, happiness, and other categories. Thus, mouth opening is still relevant but less specific to an emotion in AffectNet. This increased overlap is important as it indicates that geometric characteristics that appear highly discriminative under controlled acquisition conditions may become significantly less class-specific in real-world facial expression data.

The difference is similar for eye openness, as seen from the left and right EAR values. In CK+, surprise has significantly higher eye-openness values than most other classes. This is true in both eyes and reflects the extreme widening of the palpebral aperture in many surprise expressions. In AffectNet, surprise is also mostly in the upper part of the EAR distributions; however, there is a large overlap with neutral, fear, sadness, and other expressions. Again, the broader AffectNet distributions show that the eye aperture alone cannot uniquely characterize an emotion under naturalistic conditions. The left and right EAR distributions also show a close behavior, a sign of a certain bilateral correlation, although the differences between sides are still present and contribute to the asymmetry measure.

One of the clearest emotion-related patterns is in the eyebrow-distance features. For example, in CK+, both left- and right-eyebrow raises are highly skewed to higher values for surprise. In fact, the upper-quartile eyebrow concept groups are largely made up of surprise samples, suggesting that eyebrow raising is highly correlated with this class in CK+. AffectNet shows the same overall trend, with surprise increasing eyebrow measures, but the distributions are much more heterogeneous and overlap much more strongly with neutral, fear, sadness, and other emotions. This cross-dataset difference is methodologically important in that it shows that a geometric concept such as eyebrow_raised can be highly class-dependent in one dataset and a more distributed characteristic in another.

The asymmetry feature is different from other geometric measures. In both datasets, we see that high asymmetry is spread over multiple emotion categories rather than concentrated in a single class. Surprise, anger, disgust, sadness, and happiness significantly populate the upper tail of CK+. The class composition of AffectNet is also heterogeneous. Thus, the violin plots suggest that facial asymmetry is not so much a class-specific geometric marker but rather a complementary property of facial configuration. This could explain why asymmetric concepts show more variable TCAV behavior than concepts associated with the mouth, eye, or eyebrow.

As shown in Table 4, the image-based EmoNet representation outperformed the geometric tabular representation consistently on both AffectNet and CK+. The visual model obtained an accuracy of 0.7111 and a macro F1-score of 0.6408 on AffectNet, whereas the tabular model obtained 0.6640 and 0.6291, respectively. This difference was particularly large for MCC and for ROC-AUC, suggesting that the learned visual representation showed higher class-discrimination performance than the seven geometric measures by themselves. Both models achieved higher performance on CK+, with the image-based model achieving 0.9329 in accuracy and 0.9102 in macro F1, compared to the tabular model, which achieved 0.8265 in accuracy and 0.7694 in macro F1. The almost ideal AUC values for the CK+ image model also suggest good class separability in the feature space of EmoNet. The better performance on CK+ for both representations is explained by the fact that this dataset contains more controlled and pronounced facial expressions, while AffectNet contains more real-world variability and stronger class overlap. This systematic difference between macro and weighted metrics also indicates that class imbalance and class-specific difficulty affect performance, highlighting the necessity of reporting both averaging strategies.

Figure 3 shows the multiclass ROC performance of the image-based and tabular classifiers on AffectNet and CK+, showing distinct differences in discriminative performance across both representation type and dataset.

As shown in Figure 3a, the ROC curves for AffectNet images have strong class-separation capabilities, consistent with the macro- and weighted-average AUC values of 0.9504 and 0.9556, respectively. Conversely, the AffectNet tabular model in Figure 3b achieved considerably lower AUC values, with 0.7682 for macro averaging and 0.7698 for weighted averaging, showing that the seven geometric facial measurements offer useful but more limited discrimination compared to the high-dimensional EmoNet representation. The best results were achieved for the CK+ images, with ROC curves being almost ideal, presenting macro and weighted AUC values of 0.9920 and 0.9928, respectively. The CK+ tabular model Figure 3d also showed good discrimination performance, with 0.9687 macro AUC and 0.9764 weighted AUC, still lower than the image-based model. The ROC analysis confirms two general trends: first, the EmoNet features extracted from images provide better class discrimination than the geometric tabular representation; second, both representations achieved higher performance on CK+ than on AffectNet.

To evaluate the stability of the TCAV estimates, we repeated the CAV construction ten times for each concept and cross-validation fold. In each repetition, a random reference subset of the same size was sampled without replacement from the activations of the respective training fold. The number of reference samples was limited to 100 and was reduced if there were fewer concept or training samples. These repeated CAV constructions were used to characterize variability in TCAV sensitivity and were not treated as independent statistical replicates.

The tabular TCAV analysis had 95% confidence intervals estimated using a non-parametric bias-corrected and accelerated (BCa) bootstrap with 100,000 resamples from the 10 repeated CAV experiments. For each bootstrap iteration, the run-level TCAV scores were resampled with replacement and their mean recalculated. These intervals are therefore indicative of variability due to repeated CAV construction rather than independent variation across facial images.

The TCAV analysis revealed strong associations between concepts and emotions, such as open mouth and anger (TCAV mean of 0.997) and disgust (TCAV mean of 0.983), left-eyebrow raising and contempt (TCAV mean of 0.989), wide mouth and contempt (TCAV mean of 0.976) and sadness (TCAV mean of 0.974). Mouth opening also showed strong positive directional sensitivity for surprise (TCAV mean of 0.900), whereas wide mouth had almost no positive influence on happy or neutral predictions. Larger standard deviations mean less stability across the folds and CAV runs. CAV accuracies greater than 0.996 indicate strong concept separability, although potential dataset-related confounding should be taken into account.

The tabular TCAV analysis based on the P25/P75 exhibited interpretable and mostly stable concept–emotion relationships. The strongest positive association was found between mouth width and happiness (TCAV mean of 0.769). Surprise showed consistent positive sensitivity to raised eyebrows, increased eye opening, and mouth openings. Conversely, mouth-related concepts had negative directional effects for neutral and sadness, meaning that stronger mouth deformation tended to push the classifier away from these classes. Results based on small numbers of correctly classified test cases, especially anger and disgust, should be interpreted with caution. These tabular TCAV results are more directly interpretable at the feature-sensitivity level because the concepts are explicitly linked to measured geometric quantities. But they should still be characterized as model sensitivity to concept directions, not as evidence that a certain facial feature is biologically responsible for an emotion.

Under subject-wise evaluation of the CK+ image representation, the strongest and most stable TCAV relation was found for high facial asymmetry and disgust (TCAV of 0.945 ± 0.123). Disgust also showed high positive directional sensitivity for right-eyebrow raising (0.919 ± 0.187), right-eye openness (0.878 ± 0.246), and left-eye openness (0.843 ± 0.248). The mean sensitivity of mouth width was high for contempt (0.833 ± 0.299), and the left-eye openness was positively associated with the fear-class output (0.822 ± 0.278). Other relationships, including the high asymmetry with fear (0.726 ± 0.314), the mouth width with anger (0.723 ± 0.390), and right-eye openness with contempt (0.663 ± 0.345), exhibited more variability across folds and CAV repetitions and should be interpreted with caution. Compared to the previous image-level analysis, surprise had low positive-direction TCAV fractions for all seven concepts, including mouth opening (0.091 ± 0.210), suggesting that the revised evaluation significantly changes the concept-sensitivity pattern for this class. These results highlight the dependence of TCAV explanations on the evaluation split and concept/reference construction and advocate reporting both mean sensitivity and its variability.

The tabular TCAV analysis with P25/P75 showed different concept-emotion patterns than the visual representation. The strongest positive associations were mouth opening with disgust (TCAV of 0.786) and contempt (TCAV of 0.636), and sadness with eye openness and happiness with eyebrow raising, in moderate associations. For surprise, all TCAV scores were below 0.5, with open mouth the lowest (TCAV of 0.182). Concept-distribution analysis also revealed strong concept-class confounding, especially for eyebrow concepts with surprise and wide mouth with happiness. Thus, the global P25/P75 CAVs may be able to capture the geometric variation and the class-related structure.

The results show that both the image-based and geometric representations hold discriminative information for facial emotion recognition, but their predictive performance and concept-based explanations differ substantially. In both datasets, the EmoNet image representation consistently outperformed the AutoML classifier trained on geometric features. The accuracy of the image-based model on AffectNet was 0.7111 ± 0.0094, while that of the tabular model was 0.6640. On CK+, the accuracies were 0.9329 and 0.8265 for the image-based and tabular models, respectively. The same pattern was seen for macro F1, MCC, and ROC-AUC. Specifically, the CK+ image model achieved an almost perfect macro AUC of 0.9920, while the AffectNet image model achieved 0.9504. The lower performance of both representations on AffectNet is consistent with its more heterogeneous, in-the-wild nature, while CK+ has more controlled and pronounced facial expressions.

The AutoML tabular TCAV results were more modest overall but easier to relate directly to measurable facial geometry. In AffectNet, the strongest geometric association was wide_mouth to happy (TCAV of 0.769) and surprise was consistently positively sensitive to eyebrow raising, eye openness, and mouth opening. In CK+, the strongest tabular effect was open_mouth to disgust (TCAV of 0.786), followed by open_mouth to contempt (TCAV of 0.636). Thus, the visual and geometric representations do not always give the same importance to the same semantic concept.

4.1. Transferability of Facial Concept Directions Across AffectNet and CK+

To evaluate whether visual concepts mined from AffectNet preserve their sensitivity when transferred to CK+, we conducted a cross-dataset TCAV analysis for the same seven facial concepts and the seven emotion classes common to both datasets. The same AffectNet-derived concept definitions were used in the visual TCAV analysis of both datasets, which allows us to examine the consistency and dataset dependence of the resulting concept-emotion sensitivities. In total, we analyzed 49 matched concept–emotion pairs. We summarized their cross-dataset behavior using ${\left|\overline{∆\mathit{TCAV}}\right|}_c$ complementary measures computed from the average absolute difference in TCAV sensitivity between AffectNet and CK+; directional agreement, the extent to which matched TCAV scores have the same majority directional tendency relative to the 0.5 threshold; and the number of pairs with 95% bootstrap confidence intervals that do not include zero, the extent to which the observed cross-dataset differences persist consistently across folds and repeated CAV experiments. Together, these measures provide a descriptive measure of the relative stability or variability of AffectNet concept sensitivities as assessed on CK+. The AffectNet and CK+ TCAV mean is the arithmetic mean of the seven emotion-specific of both datasets TCAV scores. The ${\left|\overline{∆\mathit{TCAV}}\right|}_c$ from absolute difference separately for each matched emotion, database, concept (c) and emotion (e) and is made and them the averaged those seven absolute differences as ${\left|\overline{∆\mathit{TCAV}}\right|}_c=1/7{\sum }_{e=1}^7\left|{\mathit{TCAV}}_{\mathit{AffectNet},C,e}-{\mathit{TCAV}}_{\mathit{CK}+,c,e}\right|$ were computed. The direction agreement column was computed pair by pair for the seven emotion classes common to AffectNet and CK+, not from the two concept-level mean TCAV values.

The Mean |ΔTCAV| in Table 9 is interpreted as a description of cross-dataset variability rather than a calibrated effect size, because the two datasets use different classifiers and have different data distributions.

Table 9. Cross-Dataset Consistency and Variability of TCAV Concept Sensitivity Across AffectNet and CK+.

Concept	Mean|ΔTCAV|	Direction Agreement	Pairs with 95% CI
open_mouth	0.511	2/7 (28.6%)	5/7
left_eye_open	0.368	3/7 (42.9%)	4/7
wide_mouth	0.360	4/7 (57.1%)	4/7
high_asymmetry	0.359	3/7 (42.9%)	6/7
left_eyebrow_raised	0.352	2/7 (28.6%)	3/7
right_eye_open	0.288	4/7 (57.1%)	3/7
right_eyebrow_raised	0.256	5/7 (71.4%)	2/7
Overall	0.356	23/49 (46.9%)	27/49

The Mean |ΔTCAV| values show that cross-dataset concept sensitivity varies considerably across the seven facial concepts. The largest average difference was observed for open_mouth (0.511), indicating the greatest change in TCAV behavior between AffectNet and CK+. Intermediate differences were found for left_eye_open (0.368), wide_mouth (0.360), high_asymmetry (0.359), and left_eyebrow_raised (0.352). Smaller differences were observed for right_eye_open (0.288) and especially right_eyebrow_raised (0.256), suggesting comparatively more consistent cross-dataset sensitivity. Overall, the mean absolute difference across all 49 matched concept–emotion pairs was 0.356, indicating a substantial but concept-dependent variation in TCAV sensitivity between AffectNet and CK+.

Agreement ranged from 28.6% (2/7 emotion classes) for open_mouth and left_eyebrow_raised to 71.4% (5/7) for right_eyebrow_raised. Intermediate agreement was observed for left_eye_open and high_asymmetry at 42.9% (3/7), while wide_mouth and right_eye_open each reached 57.1% (4/7). Overall, only 23 of the 49 matched concept–emotion pairs (46.9%) showed the same directional tendency in AffectNet and CK+. This indicates that fewer than half of the concept–emotion relationships preserved their directional TCAV behavior across the two datasets, supporting the conclusion that the sensitivity of transferred concepts is substantially dataset- and concept-dependent.

The bootstrap analysis showed support for cross-dataset TCAV differences for a substantial proportion of concept–emotion pairs. The strongest evidence was for high_asymmetry: six of seven emotion-specific comparisons had 95% confidence intervals that did not include zero. This was followed by open_mouth (5/7 pairs), and left_eye_open and wide_mouth (4/7 pairs). Differences in support were less for left_eyebrow_raised and right_eye_open (3/7 each) and least for right_eyebrow_raised (2/7). Over half of observed AffectNet–CK+ TCAV differences were robust to the fold- and CAV-run variability captured by the hierarchical bootstrap, with 27 of 49 matched concept–emotion pairs possessing 95% confidence intervals. This finding is in favor of large cross-dataset variation in TCAV sensitivity, but the variation is concept-dependent.

4.2. Comparison with Related State-of-the-Art Studies

For comparisons with state-of-the-art papers, we searched for papers that used different CNN architectures and evaluated the concepts with TCAV. The identified studies did not use the same datasets, as we highlighted the only technologies used.

Li and Mahmoud [34] proposed an intrinsic concept-based FER framework, using concept bottleneck models and concept embedding models. Instead, we employ a post hoc TCAV approach, which allows us to study concept sensitivity without altering the underlying visual or geometric classifiers. Furthermore, the present analysis compares the matched concepts between learned visual representations and explicit geometric features on AffectNet and CK+.

The paper [35] combined preprocessing techniques such as CLAHE, Gaussian filtering, and gamma correction with DeepFace, choosing enhancement strategies based on image quality measures like energy, sharpness, and contrast. The framework compares the emotion prediction results between the raw and enhanced images and demonstrates that enhancement can significantly improve the confidence and accuracy of the classification, particularly for noisy or low-resolution images. The method improved classification in 39–56% of the cases, with accuracy gains up to 11.4% in datasets such as FER-2013 and CK+, showing the value of image-quality enhancement for more robust real-world affective computing systems.

Ng et al. [36] evaluated several CNN-based models for FER on the FER2013 dataset and proposed an optimized CNN using ReLU, Softmax, and Adam/RMSprop optimization. The proposed model achieved the best overall performance while reducing overfitting. The authors emphasize the importance of TCAV for concept analysis.

This paper [13] addresses the issue of limited interpretability of multimodal emotion recognition by interpreting the process of how a neural emotion-recognition model makes predictions using TCAV. The authors use the IEMOCAP dataset to formalize human-understandable emotion-related concepts and study their effects at different layers of a Bi-directional Contextual LSTM (BC-LSTM).

The study [37] proposes a new framework called E-TCAV to improve the efficiency and stability of TCAV. They exhibit that the standard TCAV is computationally expensive, can produce inconsistent scores across the layers of the network and is sensitive to the choice of the latent classifier. They find that layers in the final CNN block generally agree with the penultimate layer, allowing the penultimate layer to be used as a computationally efficient proxy for earlier layers.

Nagashruthi and Hemanth [38] proposed an enhanced 3D CNN with squeeze-and-excitation and residual blocks to classify fMRI data into healthy control, schizophrenia, and ADHD groups, achieving up to 79% accuracy. Its main contribution is an explainability framework combining CAV and TCAV, which identifies and visualizes the brain regions and networks that most influence each class prediction, making the deep learning model more transparent and potentially more useful for clinical interpretation.

Todorov et al. [39] performed an interpretable framework that identifies semantic concepts directly within face-recognition templates and uses them as controllable directions for template editing. It avoids the traditional image-editing and re-encoding pipeline, works across CNN, ViT, and Swin architectures, and enables semantic modifications while maintaining identity-matching performance.

4.3. Limitations and Future Directions

To avoid dataset-specific differences and a lack of generalization, the CK+ and AffectNet datasets were proposed. The CK+ includes controlled and posed expressions; AffectNet includes naturalistic images with much more variation. Thus, some of the concept-emotion associations may reflect properties of the dataset and may not generalize to other datasets of facial expression.

Some of the emotion classes, particularly CK+, are strongly associated with some of the P25/P75 geometric concepts. For instance, surprise samples are very concentrated with eyebrow-raising concepts, so global CAVs may encode both the intended geometric concept and class-related information and can lead to confusion of concept and class.

Differences between concept and reference images in background, acquisition conditions, preprocessing, identity, or source dataset, as well as the intended semantic facial properties, may explain the near-perfect classification accuracy of the CAVs in the image representation; this aspect can influence CAV separability and can include unintended cues.

There are only seven geometric measurements used in the tabular model. These features are interpretable but cannot capture the full facial appearance, such as texture and wrinkles, lip shape, local muscle deformation, and other subtle expression-related information that is available to the CNN.

To verify the generality of the concepts, these were selected from only one dataset. The concepts were selected from the AffectNet dataset and are evaluated on both AffectNet and CK+.

One limitation of the geometric/tabular analysis is that the tabular feature vectors were calculated at the image level, with one set of geometric measurements extracted per facial image. Thus, the reported performance of the CK+ geometric model may be partly subject-specific.

A high TCAV score means that movement in a learned concept direction tends to increase a model output but does not demonstrate that the corresponding facial movement causes an emotion or that the model replicates human psychological mechanisms. Furthermore, results based on small numbers of correctly classified samples should be interpreted with caution.

One interesting path for future work is to combine TCAV with existing attribution approaches such as Grad-CAM, Integrated Gradients, or other saliency-based methods. TCAV gives concept-level insights by quantifying the sensitivity of model predictions to directions in a human-understandable concept space. Attribution methods identify the portions of the image or the input information that are most relevant to a given prediction. Future work could use these views to determine if notions related to strong TCAV sensitivity correspond to spatially consistent attribution patterns in key face regions.

This TCAV attribution analysis could also be an additional layer of validation for concept-based explanations, enabling us to differentiate between semantically meaningful concept effects and spurious correlations or dataset-specific cues, and to make the interpretations robust across visual and geometric representations. In this respect, the integration of concept-level and attribution-level explainability is a critical step toward a more reliable and complete understanding of facial emotion detection models.

Another strategy in future work is related to stratifying eye-related TCAV analyses based on head pose, occlusion, blur, illumination, picture quality parameters, or using face alignment/pose normalization before concept construction.

5. Conclusions

The results indicate that concept-based explainability can offer useful insights in facial emotion recognition for both visual and geometric representations. The image model based on EmoNet outperformed the geometric AutoML model with accuracies of 0.7111 on AffectNet and 0.9329 on CK+ against 0.6640 and 0.8265 in the geometric case, respectively. This finding was also demonstrated by the macro F1-score and ROC-AUC results. In addition, the CK+ image model achieved a macro AUC of 0.9920, which demonstrated good discriminatory power in the visual features learned. In contrast, the geometric representation provided more transparent explanations at the feature level, as its concepts directly corresponded to measurable facial features. The comparison of the two representations also showed consistent and divergent patterns of concept sensitivity. The cross-dataset transfer analysis also demonstrated that visual TCAV sensitivities were only partially preserved between AffectNet and CK+. For the 49 matched concept–emotion pairs, the overall mean |ΔTCAV| was 0.356. 23 of 49 pairs (46.9%) had the same directional tendency relative to the descriptive 0.5 threshold. Moreover, 27 of 49 pairs had 95% bootstrap confidence intervals of the AffectNet–CK+ TCAV difference that did not include zero. The magnitude of the cross-dataset differences was concept-dependent, with open_mouth exhibiting the greatest mean variability and right_eyebrow_raised the least. Our results show that the concept sensitivities learned on AffectNet do not transfer uniformly to CK and that the TCAV explanations are very dependent on the underlying dataset and concept representation. Therefore, Mean |ΔTCAV| should be interpreted as a descriptive measure of cross-dataset variability rather than as a calibrated effect size or evidence of causal differences between datasets.