Interpreting Multi-Branch Anti-Spoofing Architectures: Correlating Internal Strategy with Empirical Performance

Abstract

Multi-branch deep neural networks like AASIST3 achieve state-of-the-art comparable performance in audio anti-spoofing, yet their internal decision dynamics remain opaque compared to traditional input-level saliency methods. While existing interpretability efforts largely focus on visualizing input artifacts, the way individual architectural branches cooperate or compete under different spoofing attacks is not well characterized. This paper develops a framework for interpreting AASIST3 at the component level. Intermediate activations from fourteen branches and global attention modules are modeled with covariance operators whose leading eigenvalues form low-dimensional spectral signatures. These signatures train a CatBoost meta-classifier to generate TreeSHAP-based branch attributions, which we convert into normalized contribution shares and confidence scores ($C_b$) to quantify the model’s operational strategy. By analyzing 13 spoofing attacks from the ASVspoof 2019 benchmark, we identify four operational archetypes—ranging from “Effective Specialization” (e.g., A09, Equal Error Rate (EER) 0.04%, $C=1.56$) to “Ineffective Consensus” (e.g., A08, EER 3.14%, $C=0.33$). Crucially, our analysis exposes a “Flawed Specialization” mode where the model places high confidence in an incorrect branch, leading to severe performance degradation for attacks A17 and A18 (EER 14.26% and 28.63%, respectively). These quantitative findings link internal architectural strategy directly to empirical reliability, highlighting specific structural dependencies that standard performance metrics overlook.

1. Introduction

The rapid advancement of deep learning has led to significant improvements in synthetic speech generation, including Text-to-Speech (TTS) and Voice Conversion (VC) technologies [1,2]. While beneficial for legitimate applications, these technologies pose a severe security threat to Automatic Speaker Verification (ASV) systems, necessitating robust anti-spoofing countermeasures (CMs) [3,4]. State-of-the-art CMs, such as the AASIST3 architecture [5], have evolved into complex, multi-branch neural networks capable of detecting sophisticated spoofing attacks. However, as the complexity of these models increases, their internal decision-making processes become increasingly opaque. Understanding how these models arrive at a decision—specifically, how they utilize their parallel processing branches—is no longer merely a matter of academic interest but an important requirement for ensuring the security and reliability of biometric systems [6].

While multi-branch architectures like AASIST3 achieve low Equal Error Rates (EER) on benchmark datasets like ASVspoof 2019 [3], the functional role of each branch remains largely unexplored. It is unclear whether the four parallel branches (B0–B3) and the global attention modules (GAT) function as a redundant ensemble to improve stability or if they develop specialized roles to detect specific spoofing artifacts. The primary challenge in answering this question lies in the non-linear and high-dimensional nature of deep feature representations. Traditional interpretability methods often focus on input-level saliency maps, which may fail to capture the intermediate coordination strategies employed by the network’s internal components [7], such as the Heterogeneous Stacking Graph Attention Layers (HSGAL) [8].

The field of neural network interpretability has seen the emergence of spectral analysis techniques to probe the internal state of models [9]. Recent studies have demonstrated that the eigenvalue spectrum of activation covariance matrices can serve as a robust signature of a model’s operational state [10]. For instance, Binkowski et al. [11] successfully utilized spectral features of attention maps to detect hallucinations in Large Language Models. Similarly, El Harzli et al. [12] showed that adversarial perturbations systematically shift activations toward specific eigenspaces. Parallel to this, game-theoretic approaches like SHAP (SHapley Additive exPlanations) [13] have become the standard for fair feature attribution. However, few studies have combined these spectral and game-theoretic methods to analyze the architectural efficiency and vulnerability of multi-branch audio anti-spoofing networks [14,15].

Despite the availability of these tools, there is a notable gap in the literature regarding the correlation between a model’s internal processing strategy and its empirical performance on specific attack types [16]. Current research often treats the model as a monolithic entity, ignoring the potential for internal conflicts or misplaced confidence within specific architectural branches [17]. There is a lack of systematic methodologies that can quantify whether a model is succeeding due to a robust consensus among its components or failing due to an over-reliance on a single, incorrect feature extractor [18]. Addressing this gap is essential for diagnosing high-error failure modes and designing more resilient architectures.

In this work, we propose a novel framework for deconstructing the decision-making process of the AASIST3 model. Our contributions are as follows:

• We introduce a robust feature extraction pipeline based on the spectral analysis of layer activation covariance matrices, covering 14 key internal components of the AASIST3 architecture.
• We develop a meta-classification and attribution methodology using CatBoost v1.2.8 and TreeSHAP v0.3.1 [19] to quantify the contribution share of each processing branch (B0–B3) and global module (GAT-S, GAT-T).
• We define four operational archetypes—Effective Specialization, Effective Consensus, Ineffective Consensus, and Flawed Specialization—to classify the model’s behavior.
• We empirically demonstrate that AASIST3 dynamically adapts its strategy for different attacks (A07–A19) and identify a structural misalignment where the model confidently relies on an incorrect branch, leading to high error rates.

To address these challenges, this paper is organized as follows. Section 2 describes the AASIST3 architecture, the proposed interpretation pipeline, and the confidence-based formulation of branch contribution shares. Section 3 presents the experimental results, including ablation studies, the definition and empirical characterization of the four operational archetypes, and detailed per-attack analyses of internal model behavior. Section 4 discusses the architectural implications of these findings, highlighting robustness, vulnerabilities, and the role of global attention modules, while Section 5 concludes the study and outlines directions for future work.

2. Materials and Methods

2.1. AASIST3 Architecture Components

The AASIST3 architecture [5] is a sophisticated multi-branch neural network designed for audio anti-spoofing. Our analysis explicitly focuses on interpreting the roles of its 14 primary internal components. These components process the latent representation generated by the initial RawNet2-based encoder.

The analyzed components are categorized into three functional groups:

(1)

Heterogeneous Stacking Graph Attention Layers (HSGAL): These layers form the computational core of the parallel branches. They employ graph attention mechanisms to capture complex, non-local spectro-temporal patterns within the audio data. We analyze the early-stage (HSGAL1) and late-stage (HSGAL2) layers across all four branches: B0-HSGAL1, B0-HSGAL2, B1-HSGAL1, B1-HSGAL2, B2-HSGAL1, B2-HSGAL2, and B3-HSGAL1, B3-HSGAL2.

(2)

Pooling Layers (Pool): Each branch includes a pooling operation for feature aggregation and dimensionality reduction. We analyze these as: B0-Pool, B1-Pool, B2-Pool, and B3-Pool.

(3)

Global Graph Attention Networks (GAT): Two global modules operate on the multidimensional features to capture holistic dependencies. GAT-S (Spectral) models relationships across frequency bins, while GAT-T (Temporal) models dependencies across time frames.

2.2. Methodology Pipeline

The proposed analysis framework follows a three-phase pipeline designed to extract robust spectral features and quantify the contribution of each AASIST3 [5] component.

Figure 1 provides a high-level visual summary of our three-phase analysis framework. In Phase 1, raw intermediate activations from the AASIST3 encoder are compressed into robust spectral signatures (${\lambda }_1,\cdots ,{\lambda }_{1}0$) that capture the principal variations of the feature space. These signatures serve as the input for Phase 2, where a gradient-boosted decision tree (CatBoost) learns to map spectral patterns to specific spoofing attacks. Finally, in Phase 3, we leverage the trained meta-classifier to extract Shapley values, which are aggregated into confidence scores and operational strategy classifications, enabling a direct link between internal model behavior and empirical performance.

Phase 1: Spectral Feature Extraction. For every audio sample in the evaluation subset of the ASVspoof 2019 Logical Access (LA) dataset [3], we extract intermediate activations from the 14 specified components. Let ${\mathbf{A}}^{\left(l\right)}\in {\mathbb{R}}^{D\times N}$ denote the activation matrix for layer l, where D represents the feature dimension (number of neurons or channels) and N represents the spatial/temporal dimension (or batch size in batch-wise processing). First, we center the data by subtracting the mean vector ${\mathit{\mu }}^{\left(l\right)}$, Equation (1).

$${\overline{\mathbf{A}}}^{\left(l\right)}={\mathbf{A}}^{\left(l\right)}-{\mathit{\mu }}^{\left(l\right)}$$

(1)

Next, we compute the empirical covariance matrix ${\mathbf{C}}^{\left(l\right)}\in {\mathbb{R}}^{D\times D}$, Equation (2).

$${\mathbf{C}}^{\left(l\right)}={\displaystyle \frac{1}{N-1}}{\overline{\mathbf{A}}}^{\left(l\right)}{({\overline{\mathbf{A}}}^{\left(l\right)})}^T$$

(2)

To capture the principal modes of variation within the layer’s representation, we perform an eigenvalue decomposition, Equation (3).

$${\mathbf{C}}^{\left(l\right)}{\mathbf{v}}_k={\lambda }_k{\mathbf{v}}_k$$

(3)

where ${\lambda }_k$ are the eigenvalues and ${\mathbf{v}}_k$ are the eigenvectors. The top-10 eigenvalues $({\lambda }_1^{\left(l\right)},\dots ,{\lambda }_{10}^{\left(l\right)})$ are selected to form the spectral signature for layer l. In Section 3.1, we conduct ablation studies by varying the number of selected eigenvalues used to construct the spectral signature for each layer.

Phase 2: Meta-Classification. The spectral signatures from all 14 layers are concatenated to form a meta-feature vector. This vector is used to train a CatBoost classifier [20]. The model is trained on a multi-class classification task where targets are specific attack types (e.g., A07, A08). Tree-based models like CatBoost are chosen for their compatibility with TreeSHAP [19], allowing for exact and efficient computation of Shapley values.

The CatBoost Classifier was initialized with the default MultiClass objective, which optimizes the Multinomial Cross-Entropy loss using the softmax function to model probability distributions across the 13 attack classes. We set the number of boosting iterations to 1000 to ensure convergence, while maintaining the default learning rate of 0.03 for the MultiClass. To control model complexity and prevent overfitting on the spectral meta-features, we utilized the default symmetric tree structure with a maximum depth of 6, and optimization was performed using 8 parallel threads on a CPU backend.

Phase 3: Strategy Quantification. Using TreeSHAP [19], we compute the SHAP values ${\varphi }_i$ for each of the 14 analyzed architectural components i and each prediction. We employ two aggregation methods to interpret these values:

Branch Attribution Sum. We aggregate the raw SHAP values to determine the total directional contribution of each branch b. Let ${\mathsf{\Phi }}_b$ denote the Branch Attribution Sum, defined as the summation of the mean SHAP values of its constituent components, Equation (4)

$${\mathsf{\Phi }}_b=\sum _{i\in b}\mathrm{mean}\left({\varphi }_i\right)$$

(4)

where positive ${\mathsf{\Phi }}_b$ indicates that the branch generally pushes the prediction toward the “spoof” class, while a negative value indicates a tendency toward “bona fide.”

Normalized Contribution Share. To compare the relative reliance on different branches, we define a confidence score that penalizes internal disagreement within a branch. If components within a branch (e.g., HSGAL1 vs. Pool) have high variance in their SHAP values, the branch’s overall confidence is reduced. Considering the standard deviation $\mathrm{std}(·)$ of the mean SHAP values across components within branch b, the confidence score ${\mathcal{C}}_b$ is defined as in Equation (5).

$${\mathcal{C}}_b={\displaystyle \frac{|{\sum }_{i\in b}\mathrm{mean}\left({\varphi }_i\right)|}{1+{\mathrm{std}}_{i\in b}\left(\mathrm{mean}\left({\varphi }_i\right)\right)}}$$

(5)

Here, the denominator $1+\mathrm{std}(\dots )$ acts as a regularization term: branches with consistent component contributions are favored over those with high internal conflict. Finally, we apply the Softmax function to these scores to obtain the percentage share $S_c$ for each of the six main architectural blocks (B0–B3, GAT-S, GAT-T), Equation (6).

$$S_c={\displaystyle \frac{e^{{\mathcal{C}}_c}}{{\sum }_{j=1}^6{e}^{{\mathcal{C}}_j}}}$$

(6)

To validate the robustness of the confidence score formulation in Equation (5), we perform an ablation study comparing the proposed linear penalty against two alternative formulations: a quadratic penalty $C_b^{quad}$ and an exponential penalty $C_b^{exp}$, as well as a baseline with no penalty $C_b^{none}$. These are defined as follows, Equation (7):

$$C_b^{quad}={\displaystyle \frac{|{\sum }_{i\in b}\mathrm{mean}\left({\varphi }_i\right)|}{1+{\mathrm{std}}_{i\in b}{\left(\mathrm{mean}\left({\varphi }_i\right)\right)}^2}},C_b^{exp}={\displaystyle \frac{|{\sum }_{i\in b}\mathrm{mean}\left({\varphi }_i\right)|}{exp\left({\mathrm{std}}_{i\in b}\left(\mathrm{mean}\left({\varphi }_i\right)\right)\right)}},C_b^{none}=|\sum _{i\in b}\mathrm{mean}\left({\varphi }_i\right)|$$

(7)

We evaluate the stability of the branch ranking across these methods using the Kendall rank correlation coefficient $\tau$. This analysis aims to demonstrate that while the choice of penalty function modulates the absolute confidence scores, the identification of the dominant strategic branch remains consistent across different penalty intensities.

Operational Archetype Definitions: Based on the calculated shares and their correlation with the empirical performance results presented in Section 3, we formally classify the model’s behavior into four distinct archetypes.

Effective Specialization: Defined by a low Equal Error Rate (typically $<1\%$) and a high dominant contribution share. The model successfully identifies a unique artifact and delegates detection to a specialized component.

Effective Consensus: Defined by a low EER (typically $<1\%$) and a balanced distribution of shares. The model achieves high performance through a fault-tolerant, distributed decision-making process.

Ineffective Consensus: Defined by a moderate to high EER (typically $>1\%$) and balanced but weak contribution shares. This indicates model confusion, where no component finds a strong signal.

Flawed Specialization: Defined by a high EER (typically $>10\%$) despite a high dominant share. This represents a high-error mode where the model is “confidently wrong,” relying on a single branch that fails to correctly classify the attack.

Confidence Interval Estimation for Reported Metrics

For the scalar quantities reported in

Table 1. Synthesis of Model Performance (EER) and Internal Strategy (Contribution Share). Dominant Share refers to the highest single-component share after Softmax normalization. The reported values include 95% confidence intervals, expressed in the form value ± CI, to reflect the statistical uncertainty of both EER and SHAP-based contribution estimates.

Attack	EER (%)	Dominant Component(s)	Dominant Share (%)	Confidence Score	Identified Strategy
A09	$0.05\pm 0.01$	B2, B1	$22.85\pm 0.13$, $21.85\pm 0.13$	$2.30\pm 0.01$, $2.26\pm 0.01$	Effective Specialization
A14	$0.27\pm 0.05$	B2, B0	$26.22\pm 0.10$, $18.48\pm 0.08$	$1.87\pm 0.00$, $1.52\pm 0.00$	Effective Specialization
A07	$0.40\pm 0.06$	B2, B1	$22.62\pm 0.11$, $18.84\pm 0.07$	$1.68\pm 0.00$, $1.50\pm 0.00$	Effective Specialization
A11	$0.67\pm 0.07$	B0, B2	$19.50\pm 0.07$, $19.16\pm 0.07$	$1.40\pm 0.00$, $1.38\pm 0.00$	Effective Consensus
A16	$0.74\pm 0.07$	B2, B1	$19.82\pm 0.05$, $19.19\pm 0.05$	$1.32\pm 0.00$, $1.29\pm 0.00$	Effective Consensus
A19	$0.97\pm 0.10$	B1, B2	$20.09\pm 0.07$, $20.02\pm 0.09$	$1.45\pm 0.01$, $1.45\pm 0.01$	Effective Specialization
A13	$1.23\pm 0.10$	B1, B2	$20.45\pm 0.05$, $20.16\pm 0.05$	$1.45\pm 0.01$, $1.43\pm 0.01$	Ineffective Specialization
A15	$2.77\pm 0.15$	B2, B1	$19.55\pm 0.06$, $18.99\pm 0.06$	$1.23\pm 0.00$, $1.20\pm 0.00$	Ineffective Consensus
A08	$3.13\pm 0.17$	B1, B0	$20.19\pm 0.05$, $19.43\pm 0.06$	$1.22\pm 0.01$, $1.18\pm 0.01$	Ineffective Specialization
A12	$7.91\pm 0.20$	B1, B0	$24.00\pm 0.08$, $19.12\pm 0.04$	$1.63\pm 0.01$, $1.40\pm 0.00$	Ineffective Specialization
A17	$14.27\pm 0.40$	B1, B2	$23.91\pm 0.10$, $19.20\pm 0.05$	$1.59\pm 0.01$, $1.37\pm 0.00$	Flawed Specialization (Vulnerability)
A10	$17.28\pm 0.34$	B2, B1	$22.78\pm 0.07$, $20.59\pm 0.08$	$1.66\pm 0.01$, $1.56\pm 0.01$	Ineffective Specialization
A18	$28.61\pm 0.34$	B1, B2	$24.24\pm 0.13$, $20.97\pm 0.08$	$2.08\pm 0.01$, $1.93\pm 0.01$	Flawed Specialization (Vulnerability)

(and summarized in

Table 2. Descriptive statistics of the dominant contribution share (%) stratified by operational archetype.

Identified Strategy	Mean	Std	Var	Count
Effective Consensus	18.76	0.63	0.39	3
Effective Specialization	23.18	2.00	4.02	3
Flawed Specialization	23.81	2.51	6.29	3
Ineffective Consensus	18.84	0.53	0.29	4

),

Table 3. Summed Mean SHAP Values per Branch/Component with 95% confidence intervals reported as $\mathrm{value}\pm \mathrm{CI}$.

Attack	${\mathsf{\Phi }}_{\mathit{B}0}$	${\mathsf{\Phi }}_{\mathit{B}1}$	${\mathsf{\Phi }}_{\mathit{B}2}$	${\mathsf{\Phi }}_{\mathit{B}3}$	${\mathsf{\Phi }}_{\mathit{GAT}-\mathit{S}}$	${\mathsf{\Phi }}_{\mathit{GAT}-\mathit{T}}$
A07	$1.79\pm 0.01$	$1.87\pm 0.01$	$1.95\pm 0.00$	$1.75\pm 0.01$	$1.50\pm 0.01$	$0.94\pm 0.01$
A08	$1.35\pm 0.00$	$1.42\pm 0.00$	$1.37\pm 0.01$	$1.28\pm 0.00$	$0.74\pm 0.01$	$0.86\pm 0.01$
A09	$2.92\pm 0.01$	$2.82\pm 0.01$	$3.35\pm 0.01$	$2.77\pm 0.01$	$1.40\pm 0.01$	$1.23\pm 0.00$
A10	$1.60\pm 0.01$	$1.83\pm 0.01$	$1.96\pm 0.01$	$1.47\pm 0.00$	$1.05\pm 0.01$	$1.59\pm 0.01$
A11	$1.72\pm 0.01$	$1.52\pm 0.00$	$1.74\pm 0.01$	$1.53\pm 0.00$	$1.47\pm 0.01$	$1.00\pm 0.01$
A12	$1.59\pm 0.01$	$1.89\pm 0.01$	$1.62\pm 0.00$	$1.46\pm 0.01$	$1.16\pm 0.01$	$0.84\pm 0.01$
A13	$1.58\pm 0.00$	$1.69\pm 0.01$	$1.70\pm 0.01$	$1.61\pm 0.01$	$1.23\pm 0.01$	$0.72\pm 0.00$
A14	$2.10\pm 0.01$	$1.84\pm 0.01$	$2.28\pm 0.01$	$1.66\pm 0.01$	$1.63\pm 0.01$	$1.15\pm 0.01$
A15	$1.38\pm 0.01$	$1.42\pm 0.01$	$1.49\pm 0.01$	$1.27\pm 0.00$	$1.07\pm 0.01$	$1.10\pm 0.01$
A16	$1.47\pm 0.00$	$1.48\pm 0.01$	$1.50\pm 0.00$	$1.41\pm 0.01$	$1.22\pm 0.01$	$0.76\pm 0.00$
A17	$1.61\pm 0.01$	$1.89\pm 0.01$	$1.65\pm 0.01$	$1.51\pm 0.00$	$1.15\pm 0.01$	$0.89\pm 0.01$
A18	$2.60\pm 0.01$	$2.51\pm 0.01$	$2.39\pm 0.01$	$2.29\pm 0.01$	$1.38\pm 0.01$	$1.36\pm 0.01$
A19	$1.54\pm 0.01$	$1.69\pm 0.01$	$1.76\pm 0.00$	$1.62\pm 0.01$	$1.38\pm 0.01$	$1.16\pm 0.01$

and

Table 4. Contribution Shares (%) per Component after Softmax Normalization. Confidence intervals are provided in the format value ± CI for each value.

Attack	B0 Share	B1 Share	B2 Share	B3 Share	GAT-S Share	GAT-T Share
A07	$17.9\pm 0.05$	$18.8\pm 0.07$	$22.6\pm 0.10$	$18.1\pm 0.07$	$13.1\pm 0.07$	$9.4\pm 0.03$
A08	$19.4\pm 0.05$	$20.2\pm 0.05$	$19.0\pm 0.06$	$18.2\pm 0.04$	$11.0\pm 0.03$	$12.1\pm 0.04$
A09	$20.6\pm 0.09$	$21.9\pm 0.13$	$22.9\pm 0.12$	$20.1\pm 0.11$	$7.8\pm 0.07$	$6.8\pm 0.07$
A10	$17.3\pm 0.06$	$20.6\pm 0.07$	$22.8\pm 0.07$	$15.3\pm 0.05$	$10.2\pm 0.02$	$13.9\pm 0.08$
A11	$19.5\pm 0.07$	$18.0\pm 0.07$	$19.2\pm 0.07$	$18.2\pm 0.05$	$14.2\pm 0.09$	$11.0\pm 0.04$
A12	$19.1\pm 0.04$	$24.0\pm 0.08$	$19.0\pm 0.05$	$17.4\pm 0.05$	$10.8\pm 0.04$	$9.7\pm 0.03$
A13	$19.4\pm 0.05$	$20.4\pm 0.05$	$20.2\pm 0.05$	$18.8\pm 0.06$	$12.1\pm 0.06$	$9.1\pm 0.03$
A14	$18.5\pm 0.07$	$16.4\pm 0.06$	$26.2\pm 0.10$	$16.7\pm 0.07$	$12.0\pm 0.08$	$10.2\pm 0.03$
A15	$18.6\pm 0.05$	$19.0\pm 0.06$	$19.6\pm 0.05$	$17.2\pm 0.04$	$12.6\pm 0.04$	$13.1\pm 0.04$
A16	$19.0\pm 0.05$	$19.2\pm 0.05$	$19.8\pm 0.05$	$18.5\pm 0.04$	$13.3\pm 0.05$	$10.2\pm 0.03$
A17	$17.8\pm 0.05$	$23.9\pm 0.10$	$19.2\pm 0.05$	$17.5\pm 0.06$	$12.0\pm 0.03$	$9.5\pm 0.03$
A18	$19.4\pm 0.15$	$24.2\pm 0.12$	$21.0\pm 0.09$	$18.5\pm 0.08$	$8.3\pm 0.05$	$8.6\pm 0.05$
A19	$17.5\pm 0.05$	$20.1\pm 0.06$	$20.0\pm 0.08$	$17.9\pm 0.08$	$12.2\pm 0.07$	$12.2\pm 0.05$

(Equal Error Rates, summed mean SHAP values, and Softmax-normalized contribution shares), we provide two-sided $(1-\alpha )\times 100\%$ confidence intervals around the sample mean. Let $x_1,\dots ,x_n$ denote the values of a given metric computed over n evaluation samples (e.g., utterances) for a fixed attack and component. The sample mean and sample standard deviation are defined as in Equation (8)

$$\overline{x}={\displaystyle \frac{1}{n}}\sum _{i=1}^n{x}_i,s=\sqrt{{\displaystyle \frac{1}{n-1}}\sum _{i=1}^n{(x_i-\overline{x})}^2}.$$

(8)

The standard error of the mean is defined in Equation (9).

$$\mathrm{SE}={\displaystyle \frac{s}{\sqrt{n}}}.$$

(9)

Assuming approximate normality of the sampling distribution of $\overline{x}$, the two-sided $(1-\alpha )\times 100\%$ confidence interval is given by Equation (10).

$$\overline{x}\pm z_{1-\alpha /2}\mathrm{SE},$$

(10)

where $z_{1-\alpha /2}$ is the $(1-\alpha /2)$ quantile of the standard normal distribution (for a $95\%$ confidence level, $z_{0.975}\approx 1.96$). In all tables, we report each estimate in the compact form $\overline{x}\pm \mathrm{CI}$, where $\mathrm{CI}=z_{1-\alpha /2}\mathrm{SE}$ denotes the half-width of the corresponding confidence interval.

2.3. Justification of the Analysis Method

The proposed methodology integrates spectral theory with game-theoretic attribution to provide a robust interpretation framework. We analyze the eigenvalue spectrum of covariance matrices because features derived from covariance functionals are statistically stable and converge at a rate of $O\left(N^{-1/2}\right)$ [10]. This robustness is crucial when analyzing deep layers where activations can be noisy. Furthermore, recent studies indicate that adversarial perturbations often manifest as shifts toward the null eigenspace of empirical kernels [12], suggesting that spectral features are particularly sensitive to the artifacts introduced by spoofing attacks. By coupling these robust features with SHAP [13], which guarantees the fair distribution of prediction credit among features, we ensure that our attribution of “importance” to specific AASIST3 [5] branches is mathematically principled and not an artifact of the visualization method.

It is important to clarify the epistemological scope and methodological boundaries of the proposed interpretability framework. Our spectral-SHAP approach operates at a meta-level, establishing correlational structures between empirically observable activation patterns and downstream performance outcomes, rather than providing direct causal attributions of AASIST3’s internal computational mechanisms. The SHAP values derived from the CatBoost meta-classifier quantify statistical associations between spectral signatures from intermediate layers and classification success on specific attack types, capturing how the architecture behaves under different input distributions, not why particular feature representations emerged during training. These interpretations reflect emergent operational strategies—consensus, specialization, or internal conflict—as patterns in the attribution landscape, without attributing mechanistic causality to individual neurons, attention heads, or graph convolutions. Consequently, our findings are conditioned on the specific AASIST3 instantiation we analyzed; different training procedures or datasets may yield different operational archetypes even with identical architecture, because learned feature representations would differ. This methodological position is analogous to behavioral modeling in complex systems: we characterize what the model does in its current configuration, not what the architecture is fundamentally capable of doing across all possible parameterizations.

3. Results

In this section, we correlate the internal operational strategies of AASIST3 [5] with its objective performance. Performance is evaluated using the Equal Error Rate (EER), defined as the threshold point where the False Acceptance Rate (FAR) equals the False Rejection Rate (FRR), Equation (11).

$$\mathrm{EER}=\mathrm{FAR}\left(\theta \right)=\mathrm{FRR}\left(\theta \right)$$

(11)

where $\theta$ is the decision threshold. A lower EER indicates better detection performance on a specific attack type from the ASVspoof dataset [3].

3.1. Eigenvalue Count Ablation

We evaluate the sensitivity of the proposed spectral signature to the number of retained eigenvalues $N_{eig}$ by performing an ablation study that jointly considers predictive performance and resource usage (Figure 2, Figure 3 and Figure 4). The results indicate a clear saturation behavior: increasing $N_{eig}$ yields substantial gains in the low-$N_{eig}$ regime, while improvements become marginal beyond $N_{eig}=10$, where the curve enters a diminishing-returns region. In addition, the performance–memory trade-off demonstrates that $N_{eig}=10$ provides an effective operating point, retaining approximately 98% of the maximum F1-Macro score while achieving about 71% memory savings compared with larger configurations. Accordingly, we set $N_{eig}=10$ in all subsequent experiments as a principled compromise between accuracy and computational cost.

Figure 2 illustrates the relationship between the number of eigenvalues $N_{eig}$ selected for the spectral signature and the resulting F1-Macro score. The plot is divided into a high-gain zone, where the performance increases sharply as $N_{eig}$ moves from 2 toward 10, and a subsequent region of diminishing returns for values exceeding 10. A saturation point is explicitly identified at $N_{eig}=10$, marking the transition from rapid performance growth to a plateau where additional computational costs result in marginal quality gains.

Figure 3 presents a scatter plot of F1-Macro score versus memory consumption (MB) for configurations obtained by varying the number of retained eigenvalues $N_{eig}$. Each point corresponds to a specific $N_{eig}$ setting, and the dashed line connects configurations in increasing computational cost, visualizing how model quality changes with memory usage. The figure also annotates a selected operating point at $N_{eig}=10$ (shown together with its reported memory value and F1-Macro score) and includes a reference annotation for a larger baseline configuration (e.g., $N_{eig}=35$).

Figure 4 depicts two curves summarizing the effect of varying the number of retained eigenvalues $N_{eig}$ on both predictive performance and resource usage. The green curve reports the F1 score expressed as a percentage of the maximum observed performance, while the red curve reports the percentage of memory saved relative to the highest-cost configuration, both plotted against $N_{eig}$. The figure highlights a reference point at $N_{eig}=10$ using a vertical guideline and an annotation box that reports the corresponding performance-retention and memory-savings values.

3.2. Penalty Function Ablation

Figure 5 presents the results of the penalty function ablation study. The heatmap visualizes the dominant branch identified for each attack type (A07–A19) under the four different penalty regimes. The consistency of the color coding across each row confirms that the core strategic findings are largely invariant to the specific form of the penalty. For instance, Attacks A09, A14, and A10 consistently identify Branch B2 (red shades) as dominant, while A12, A13, A15, A16, and A18 consistently point to Branch B0 (blue shades).

Quantitatively, the proposed linear penalty achieves a high agreement with the stricter quadratic ($\tau =0.98$) and exponential ($\tau =0.96$) formulations, indicating that it provides a balanced penalization that effectively filters out unstable branches without suppressing valid but slightly noisy signals. The “None” baseline shows a slightly lower correlation ($\tau =0.91$), highlighting that some form of variance penalization is indeed necessary to resolve ambiguities in attacks like A11 and A19, where raw SHAP values alone may lead to inconsistent attributions (e.g., flipping between GAT-S and B0/B1).

3.3. Overview of Model Performance and Internal Strategies

Table 1 synthesizes the key metrics for each attack. The attacks are sorted by their EER to highlight the relationship between difficulty and model strategy. The “Dominant Share” column is derived using the Softmax-normalized confidence metric defined in Equation (6), while the “Confidence score” column reports the raw $C_b$ values from Equation (5). This score explicitly accounts for internal stability: lower values indicate higher internal variance or conflict among the components of a branch, providing quantitative evidence that certain attacks induce disagreement within the model’s decision-making process even when contribution shares appear balanced. The “Identified Strategy” column classifies the model’s overall behavior according to the operational archetypes defined in Section 2.

Figure 6 visualizes the operational strategies of the AASIST3 model by mapping each attack into a 2D space defined by the EER and the contribution share of the dominant branch. This visualization provides the empirical rationale for our categorization thresholds: the 1% EER boundary (green dotted line) cleanly separates reliable performance from failure modes, while the 20% contribution threshold (vertical dashed line) effectively distinguishes distributed consensus from specialized reliance on a single component. By segmenting the performance landscape into these four quadrants—ranging from high-accuracy “Effective Specialization” (bottom right) to the high-error failure mode of “Flawed Specialization” (top right)—the figure confirms that the chosen cutoffs are not arbitrary, but rather reflect natural clusters in the model’s behavior under varying attack conditions.

3.4. Correlation and Variance Analysis of Operational Archetypes

To quantitatively assess the relationship between internal strategy and empirical performance, we conducted a correlation analysis between the dominant contribution share and the Equal Error Rate (EER) across all attacks.

We computed both Pearson’s linear correlation and Spearman’s rank correlation between the dominant Softmax-normalized share and the corresponding EER, Equation (12).

$$r_{\mathrm{Pearson}}=0.537,{\rho }_{\mathrm{Spearman}}=0.077.$$

(12)

The moderate positive Pearson correlation indicates that, on average, higher dominant shares tend to co-occur with higher EER, while the near-zero Spearman coefficient reveals that this effect is not monotonic when attacks are ranked by difficulty. In other words, a large dominant share does not systematically guarantee either low or high error; its impact is modulated by the broader operational archetype (e.g., whether the dominant branch acts as a correct expert or as a confidently wrong failure mode). This aligns with the qualitative findings in Figure 6, where both Effective and Flawed Specialization occupy regions with comparable dominant shares but drastically different EER levels.

We further analyzed how the dominant share behaves within each of the four operational archetypes by grouping attacks according to their identified strategy and computing basic descriptive statistics of the dominant share.

The variance analysis reveals a clear separation between consensus- and specialization-based regimes. Both Effective and Ineffective Consensus exhibit relatively low dominant shares (≈18.8%) with very small variance, reflecting a balanced distribution of responsibility across branches regardless of whether the resulting decision is accurate. In contrast, both specialization regimes show substantially higher dominant shares (≈23–24%) and larger variance, indicating that the model increasingly concentrates its trust in a single branch when it commits to a specialized strategy. Flawed Specialization displays the highest variance ($\mathrm{var}=6.29$), suggesting that failure cases are associated not only with strong reliance on one branch but also with increased instability in how strongly this dominance manifests across attacks. This supports our interpretation that specialization is a double-edged mechanism: it underpins the best-performing attacks but also amplifies vulnerability when the model specializes on the wrong expert.

3.5. Detailed Per-Attack Analysis

The following subsections examine the 13 attacks individually. We analyze the SHAP [13] distribution maps and decision plots to validate the strategies identified in Table 1.

In SHAP distribution plots, each bar corresponds to a model component (for example, one of the branches B0–B3, a pooling layer, or a global attention module), and its length encodes the average magnitude of that component’s SHAP value across many samples, so longer bars indicate components that have a stronger overall influence on the prediction. The sign of the value reflects the direction of the effect relative to the target class: in this study, positive values push the model toward the spoof hypothesis and negative values toward bonafide, so a plot with uniformly large positive bars for one component means the model is consistently relying on that component as an “expert” for detecting artifacts of a particular attack, whereas mixed positive and negative contributions across components reveal internal disagreement or confusion.

Decision plots complement this by showing how these contributions accumulate along the model’s output axis: starting from a baseline score (often near zero), the line traces the step-by-step change in the prediction as each component is added in order of importance, with upward segments indicating components that increase spoof confidence and downward segments indicating components that suppress it. When the trajectory is smooth and dominated by a small set of components, the plot visualizes effective specialization, where the decision is largely driven by one branch or module; when the line oscillates with many small conflicting steps, it reveals ineffective consensus, where no component provides a clear signal and the final prediction is the result of many weak, partially cancelling influences.

3.6. Interpretation Framework for SHAP Analysis

To avoid redundancy, we establish a unified framework for interpreting the visualization results presented in the subsequent figures. The analysis relies on two primary visualization types.

SHAP Distribution Maps (Violin Plots): These plots contrast the model’s response to attack samples (red distributions, positive values pushing toward “spoof”) versus bona fide samples (blue distributions, negative values pushing toward “genuine”). A clear vertical separation implies strong discriminative power, whereas overlapping or diffuse distributions indicate weak feature extraction.

Decision Plots: These visualize the cumulative decision path. A steep, monotonic trajectory driven by one or two dominant components indicates Specialization. Conversely, a chaotic, oscillating trajectory involving many small, conflicting steps signifies Ineffective Consensus, where the model lacks a clear signal and relies on the aggregation of weak features.

3.6.1. Attacks A07 and A08 (Consensus Strategies)

Attack A07 is handled via Effective Consensus, characterized by a moderate Confidence Score of $C_b=0.67$ and balanced contribution shares (Table 1). As shown in Figure 7, virtually all components exhibit strong, positive SHAP contributions with distinct separation between attack and bona fide distributions.

The decision plot (Figure 8, left) confirms this collaborative strategy, showing a smooth accumulation of evidence across both spectral and temporal branches.

In contrast, Attack A08 triggers an Ineffective Consensus. While contribution shares remain balanced, the Confidence Score drops significantly to $C_b=0.33$, the lowest among all attacks (Table 1). The SHAP distributions (Figure 9) are compressed near zero with notable overlap, and the decision plot (Figure 8, right) reveals a jagged trajectory. This visual instability is quantitatively captured by the low $C_b$ score, reflecting the model’s struggle to find a robust artifact in any single domain.

3.6.2. Attacks A09 and A10 (Specialization vs. Complexity)

Attack A09 is a trivial case of Effective Specialization, quantitatively validated by the highest Confidence Score of $C_b=1.56$ and a dominant contribution share of 25.5% from Branch B2 (Table 1). The model delegates detection almost exclusively to B2, which shows an overwhelmingly large positive SHAP signal (Figure 10). Conversely, Attack A10 (Figure 11) represents a failure case of Ineffective Consensus (reclassified from specialization due to high conflict). Although the model attempts to prioritize GAT-T ($25.5\%$) and B2 ($21.9\%$), the decision plot (Figure 12, right) exposes extreme internal conflict, with large opposing contributions canceling each other out. This internal disagreement is measurable: despite high raw contribution shares, the Confidence Score is suppressed to 0.93, confirming that while the model “selects” experts, their predictions are unstable.

3.6.3. Attacks A11 and A12 (Spectral vs. Mixed)

For Attack A11, the model employs Effective Specialization driven by GAT-S ($21.9\%$) and Branch B0 ($20.4\%$), achieving a Confidence Score of 0.76. Notably, Figure 13 demonstrates that the model successfully overcomes a negative (suppressive) contribution from GAT-T to reach a correct decision, highlighting the system’s ability to filter out misleading temporal cues in favor of spectral evidence. Attack A12, however, falls into Ineffective Consensus. The SHAP values (Figure 14) are uniformly low, and the decision plot (Figure 15) is dominated by noise. This lack of signal is reflected in the metrics (Table 1), where no branch exceeds a 20% share, indicating that the artifacts generated by the neural waveform model in A12 do not strongly activate any specific architectural component.

3.6.4. Attacks A13 and A14 (Borderline vs. Distinct)

Attack A13 represents a borderline case of Ineffective Consensus (Figure 16); the model aggregates many small signals to reach a decision, yielding a lower Confidence Score of $0.63$. Attack A14 is efficiently detected via Effective Specialization, with GAT-S ($22.1\%$) and Branch B2 ($20.6\%$) acting as a powerful expert duo (Figure 17). This strong signal results in a high Confidence Score of $1.13$, and the sharp, unidirectional rise in the decision plot (Figure 18) confirms their dominance.

This suggests that while the attack is detected, it lies near the decision boundary of the learned feature space, requiring broad architectural participation rather than expert delegation.

3.6.5. Attacks A15 and A16 (Weak vs. Robust Consensus)

Attack A15 reveals Ineffective Consensus similar to A08, with diffuse SHAP distributions (Figure 19) and a very low Confidence Score of $0.38$ (Table 1). Attack A16, conversely, is a robust example of Effective Consensus. Despite being a low-EER attack ($0.72\%$), the model does not rely on a single branch; instead, Figure 20 shows consistent positive contributions across the entire architecture. This distributed strategy maintains a stable Confidence Score of $0.49$ without high variance, providing a highly stable and fault-tolerant detection mechanism (Figure 21).

3.6.6. Attacks A17 and A18 (Vulnerability and Failure)

These attacks expose the Flawed Specialization vulnerability. For Attack A17 (Figure 22, the model confidently relies on Branch B1 ($20.9\%$ share), yet the high EER ($14.26\%$) indicates this confidence is misplaced. The failure is most high-error for Attack A18, where the model delegates the decision to Branch B0 (25.0% share, Table 1). The decision plot (Figure 23) illustrates this pathology: a strong, confident step driven by the wrong features leads to a systematic misclassification (Figure 24). This behavior is distinct from the low-confidence confusion of A08; here, the Confidence Score is relatively high ($1.33$ for A18), quantitatively proving that the model is confidently wrong rather than uncertain.

3.6.7. Attack A19 (Global Spectral Detection)

Attack A19 is resolved via Effective Specialization led by the global GAT-S module (19.5% share). The decision trajectories presented on Figure 25 confirm that the final decision pivots on the contribution of the global module. Although the Confidence Score is low ($0.45$), Figure 26 shows that the global spectral features provide the decisive signal, effectively overriding minor disagreements from the local pooling layers. This validates the architectural choice of including global attention modules alongside local processing branches.

3.7. Detailed Statistical Data

For completeness, this subsection provides the detailed numerical data used for the analysis. Table 3 shows the summed mean SHAP values for each of the six main architectural components. Table 4 shows the final calculated contribution shares.

3.8. Single-Branch Retention Ablation

To complement the descriptive SHAP-based strategy analysis, a small functional ablation was conducted to test whether the dominant branch identified by the attribution pipeline is sufficient to preserve detection performance on difficult attacks. We focus on the three highest-error cases in our evaluation (A10, A17, A18), where the model either exhibits ineffective consensus/specialization (A10) or flawed specialization (A17, A18) in Table 1. For each selected attack, we retain only the most dominant branch and set the activations of all other branches/modules to zero during inference, after which EER is recomputed on the same evaluation protocol.

The results in

Table 5. Single-Branch Ablation for High-Error Attacks. EER when retaining only the dominant branch identified in Table 1, with all other branches zeroed.

Attack	Dominant Branch Kept	Baseline EER	EER with Only Dominant Branch
A18	B0	28.63	63.34
A17	B1	14.26	66.81
A10	B2	17.31	67.55

show that restricting AASIST3 to a single “best” branch strongly degrades performance, increasing EER to 63–68% for all tested attacks.This behavior is consistent with the interpretation that our SHAP-based dominance scores quantify relative reliance patterns under the full architecture, rather than implying that the dominant branch alone implements a complete and causally sufficient detector. In particular, for A17 and A18 (flawed specialization), the model’s high reliance on B1/B0 already corresponds to a failure mode under the full system (Table 1), and isolating the same branch removes any possibility of corrective evidence from alternative branches, further amplifying the error.

4. Discussion

The results presented in Section 3 allow for a deep interpretation of the AASIST3 [5] model’s internal operating logic. By correlating the model’s chosen operational strategy with its empirical performance (EER), we can move beyond simple descriptions of feature importance to a diagnostic analysis of architectural strengths and weaknesses.

4.1. Analysis of Operational Archetypes

The correlation analysis confirms that the model dynamically switches between the four operational archetypes defined in our methodology. We categorize the behavior of the model on all 13 analyzed attacks (A07–A19) as follows:

Effective Specialization: Our findings demonstrate that the architecture is highly efficient when it can isolate distinct artifacts. This group includes attacks A09, A14, A11, and A19. By successfully delegating detection to a single “expert” component (e.g., Branch B2 for A09 or GAT-S for A11), the model filters out noise from less relevant branches. This suggests that for these “easy” attacks, the multi-branch structure functions primarily as a feature selection mechanism.

Effective Consensus: The low EERs observed for attacks A07 and A16 validate the redundancy hypothesis of multi-branch networks. In these cases, the artifacts appear to be broad enough to be captured by disparate feature extractors (temporal and spectral). The agreement across branches acts as a robust error-correction mechanism, ensuring stability even if one branch provides a noisy signal.

Ineffective Consensus: This behavior reveals a representational limit of the current architecture and is observed for attacks A08, A12, A13, and A15. The balanced but weak contributions indicate that the model is “confused” rather than “collaborative.” The lack of a strong signal from any branch suggests that the specific artifacts of these attacks fall into the blind spots of the learned feature filters in all four branches simultaneously.

Flawed Specialization: This archetype represents the most important discovery of our study. Attacks A18 and A17 fall into this category, as does A10 (which exhibits ineffective specialization). For A18 and A17, high internal confidence does not equate to accuracy. The model’s tendency to “lock on” to a misleading feature in a single branch (e.g., B0 for A18) without cross-verification from other branches points to a lack of inter-branch regularization. Attack A10 similarly shows reliance on specific components (GAT-T) but fails to achieve accuracy, highlighting that specialization is detrimental when the selected expert is incompetent.

4.2. Architectural Implications and Vulnerabilities

The identification of these behaviors has direct implications for understanding the model’s robustness and guiding future improvements.

First, as detailed in Table 4, while effective specialization is efficient, it introduces single points of failure. The model’s heavy reliance on specific branches—for example, Branch B2 contributes 25.5% of the decision for A09, and GAT-S contributes 22.1% for A14—implies that an adversarial attack specifically optimizing against these branches would likely bypass the entire system. Other branches, having low contribution shares (often below 10–15%), would be unable to compensate.

Second, the Flawed Specialization pattern highlights a missing meta-cognitive component in the architecture. The current design aggregates features but does not seemingly evaluate the reliability of a branch’s high activation against the consensus of others. A robust system should be able to penalize a high-confidence branch (like B1 in A17 with 20.9% share) if it contradicts the collective disagreement of the other three branches.

Finally, the roles of the global modules are distinct. The GAT-S (spectral) module acts as a consistent, high-performance anomaly detector, frequently leading effective strategies (e.g., A11, A19). In contrast, the GAT-T (temporal) module appears less reliable as a standalone detector, suggesting that temporal artifacts in the ASVspoof dataset are either more subtle or harder to disentangle from bonafide speech dynamics.

The observed inferiority of GAT-T relative to GAT-S, particularly its failure to specialize under sophisticated neural synthesis attacks like A10 (Tacotron 2 with WaveRNN), can be attributed to the acoustic properties of the ASVspoof 2019 logical access dataset [3]. Neural vocoders typically operate by predicting audio samples or spectral frames sequentially, a process that inherently preserves long-range temporal continuity and prosodic consistency, thereby producing temporal envelopes that closely mimic bona fide speech [21,22]. In contrast, the generation process frequently introduces distinct spectral artifacts—such as overly smoothed formants, phase discontinuities, or checkerboard patterns in the mel-spectrogram—arising from the frame-wise reconstruction loss or upsampling operations [23]. Consequently, the discriminatory signal is far denser in the frequency domain, rendering the spectral graph attention (GAT-S) a more robust expert for artifact detection, whereas the temporal graph attention (GAT-T) struggles to find reliable distinguishing features in the comparatively well-modeled time domain [24].

4.3. Connection to Broader Research

Our findings empirically support theories from the wider field of deep learning. The dynamic adoption of specialized roles for specific tasks confirms observations that multi-branch networks naturally diversify to cover different feature subspaces [25]. However, our identification of “Flawed Specialization” adds nuance to the work on uncertainty estimation [26]: we show that model failure is not always preceded by high uncertainty (entropy); often, the most high-error failures occur when the model is irrationally certain about a wrong feature.

Although the empirical analysis in this work is restricted to the AASIST3 architecture [5], the proposed spectral–SHAP framework is, by construction, model-agnostic and can in principle be applied to other mainstream multi-branch anti-spoofing systems, including RawNet-style encoders [27], graph-based countermeasures [24], and recent convolutional–transformer hybrids [28] used in ASVspoof challenges [2]. The only requirements are access to intermediate activations of a finite set of branches or experts and the ability to compute covariance operators over these activations, so that spectral signatures and SHAP-based contribution shares can be defined in the same manner as for AASIST3. This makes the framework suitable not only for classical parallel-branch countermeasures but also for modern mixture-of-experts [29] and gating-based anti-spoofing models [30], where each expert or specialist sub-network can be treated as a branch and analyzed in terms of its operational strategy, redundancy, and potential single points of failure.

5. Conclusions

This study presented a rigorous methodology for interpreting the internal decision-making strategies of the multi-branch AASIST3 architecture. By correlating spectral feature attributions with the Equal Error Rate (EER) across 13 distinct spoofing attacks, we moved beyond black-box performance metrics to a granular understanding of architectural behavior.

Our analysis revealed that AASIST3 is not a static feature extractor but a dynamic system that adopts different operational strategies based on the input. We identified that for “easy” attacks (e.g., A09, A14), the model effectively utilizes a Specialization strategy, delegating detection to a specific branch (often B2 or GAT-S) that filters out noise. Conversely, for broadly detectable attacks (e.g., A07, A16), it employs an Effective Consensus strategy, leveraging redundancy to ensure stability.

Crucially, our framework exposed a significant architectural vulnerability: Flawed Specialization. In the most difficult attack scenarios (A17, A18), the model exhibited high internal confidence in a single branch (e.g., B0 for A18), while other branches provided weak or conflicting signals. This misplaced confidence led to high-error performance (EER > 28%), identifying a notable lack of inter-branch regularization and cross-verification mechanisms. Furthermore, the identification of “Single Points of Failure” in successful specialization scenarios suggests that while efficient, the current architecture may be susceptible to adversarial attacks targeting specific “expert” branches.