Sparse Attention-Based Residual Joint Network for Aspect-Category-Based Sentiment Analysis

Abstract

Aspect-based sentiment analysis (ABSA) aims at identifying the sentiment polarity for a particular aspect in a review. ABSA studies based on deep learning models have exploited the attention mechanism to detect aspect-related parts. Conventional softmax-based attention mechanisms generate dense distributions, which may limit performance in tasks that inherently require sparsity. Recent studies on sparse attention transformation functions have demonstrated their effectiveness over the conventional softmax function. However, these studies primarily focus on highly sparse tasks based on self-attention architectures, leaving their applicability to the ABSA domain unexplored. In addition, most ABSA research has focused on leveraging aspect terms despite the usefulness of aspect categories. To address these issues, we propose a sparse-attention-based residual joint network (SPA-RJ Net) for the aspect-category-based sentiment analysis (ACSA) task. SPA-RJ Net incorporates two aspect-guided sparse attentions—sparse aspect-category attention and sparse aspect-sentiment attention—that introduce sparsity in attention via a sparse distribution transformation function, enabling the model to selectively focus on aspect-related information. In addition, it employs a residual joint learning framework that connects the aspect category detection (ACD) task module and the ACSA task module via residual connections, enabling the ACSA module to receive explicit guidance on relevant aspect categories from the ACD module. Our experiment validates that SPA-RJ Net consistently outperforms existing models, demonstrating the effectiveness of sparse attention and residual joint learning for aspect category-based sentiment classification.

1. Introduction

Aspect-based sentiment analysis (ABSA) is fine-grained sentiment analysis (SA) [1]. While traditional SA treats a review as a whole and assigns it a single sentiment label, ABSA aims at detecting the sentiment polarity corresponding to a particular aspect. Consider the sentence “pizza is good but sometimes service is so bad” as one of the common reviews about restaurants. It focuses on ‘pizza’ and ‘service’ as aspects of interest, each of which may have a different sentiment, e.g., positive for pizza and negative for service. ABSA, i.e., sentiment analysis pinpointing at the aspect level, will be useful to businesses and consumers who want valuable insights for decision-making from reviews. However, as reviews are generally written in an unstructured manner by individuals, the review texts vary in length, sentence structure, expression of aspect/sentiments, and other factors. Moreover, ABSA must handle multiple aspects, each potentially associated with different sentiment labels within a single review, which presents a significant challenge.

Recently, attention-based deep learning models have led to significant improvement in performance in ABSA by effectively identifying aspect-relevant words within an input review [2]. The conventional attention mechanism employs the softmax function to convert attention scores into a dense probability distribution, assigning positive weights to all input elements [3]. However, this property results in non-zero weights even for irrelevant elements, which can be a limitation in applications that benefit from sparsity [4,5,6]. A recent study by Google DeepMind [6] highlights the significance of exploring alternatives to softmax, pointing out its fundamental limitations in enabling robust reasoning over diverse inputs. Various methods have been introduced to address the sparsity issue in attention, such as regularization-based techniques [7,8,9] and temperature-scaled softmax variants [6,10,11]. However, these approaches still rely on the inherently dense nature of softmax, which restricts the level of sparsity achievable and hinders the model’s ability to distinguish sharply and filter out irrelevant input tokens. They often fail to fully capture the selective focus intended by attention mechanisms.

As an alternative to softmax, several studies have proposed the functions that directly transform logits into sparse probability distributions. Functions such as sparsemax [12] and entmax [13] explicitly assign zero weights to irrelevant tokens, thereby introducing sparsity into the attention mechanism. These methods have shown potential for improving interpretability, as their attention weights more clearly reflect which elements are considered important. Although sparse attention distributions have proven their effectiveness successfully in high-sparsity tasks, such as translation and summarization tasks, which are usually based on self-attention models such as Transformer [14] and BERT [15], their integration into the ABSA domain remains underexplored. In many ABSA studies, we observe that attention tends to be overly diffused across the entire review, which can negatively affect prediction accuracy. This issue becomes more pronounced when multiple aspects are present within a single sentence, where a sparse distribution could help the model focus more precisely on relevant information. Therefore, we believe that incorporating sparse attention into general ABSA frameworks presents a promising research direction.

On the other hand, most ABSA research has focused on aspect terms despite the practical utility of aspect categories. Aspects can be classified into aspect terms and aspect categories [1]. Aspect terms are explicitly mentioned in the text, providing clearer guidance to models, while aspect categories refer to high-level features that are often implicit, making aspect-category-based sentiment analysis (ACSA) more challenging. Nevertheless, the aspect-category-based approach offers advantages such as broader applicability and easier data collection [16]. In this paper, we focus on aspect-category-based sentiment analysis (ACSA), and hereafter, we use the term “aspect” to refer to aspect category.

To address these challenges, we propose SPA-RJ Net, a sparse-attention-based residual joint network. SPA-RJ Net provides two aspect-guided sparse attentions, i.e., sparse aspect-category attention and sparse aspect-sentiment attention. They incorporate sparse probability distribution into the attention mechanism by using the α-entmax function, which transforms attention scores into sparse distributions via Tsallis entropy. To evaluate the effect of sparse attention in ABSA, we adopt an aspect-guided attention model, a commonly used approach in this task, and apply varying degrees of sparsity by adjusting the sparsity parameter α. These sparse attentions enable the model to focus more effectively on aspect-related information. To the best of our knowledge, this is the first work to directly apply the sparse attention distribution to the ABSA task with benchmark datasets such as SemEval [17,18,19] and Multi-Aspect Multi-Sentiment (MAMS) [20].

Next, we adopt a joint learning framework that integrates ACD and ACSA, which enhances sentiment classification by providing explicit guidance on relevant aspect categories. In our joint learning setting [21], the ACD task module, as an auxiliary task, learns aspect-category-related features via sparse aspect-category attention and transfers this sparse attention information to the main ACSA task module. ACSA predicts sentimental polarity for each aspect through CNN-biLSTM layers and sparse aspect-sentiment attention. However, learning multiple tasks simultaneously can lead to information interference between tasks, which may result in unstable or suboptimal performance. To mitigate the problem, we propose to employ a residual connection [22] between ACD and ACSA and adjust a balancing hyperparameter to regulate the learning pace discrepancy between tasks. They help to promote more stable optimization and improve overall performance. Our experiments show that SPA-RJ Net outperforms existing models, particularly dense attention-based ones, demonstrating the effectiveness of sparse attention for ABSA.

In summary, the main contributions of this paper are as follows:

• We propose SPA-RJ Net to explore the effectiveness of sparse attention in ACSA. It employs two aspect-guided sparse attentions—aspect-category and aspect-sentiment attention—which introduce sparsity into attention via the α-entmax function. This enables the model to focus more effectively on a smaller number of aspect-specific elements, improving accuracy and interpretability.
• SPA-RJ Net adopts a residual joint learning framework with adjusting a balancing hyperparameter. The ACD module extracts aspect features by sparse aspect-category attention and transfers them to the main ACSA module via the residual pathway, which enhances final sentiment classification by providing explicit guidance on relevant aspect categories.
• Our experiments with ABSA benchmark datasets (SemEval and MAMS) validate the efficiency and the interpretability of SPA-RJ Net, which highlights the benefits of sparse attention and residual joint learning in the ABSA task domain.

The rest of the paper is designed as follows: The related work and background are described in Section 2 and Section 3. Then, our proposed model architecture is presented in Section 4. Section 5 presents our experimental setup and results, model analysis, and case study demonstrating the effectiveness of the proposed SPARJ-Net model. Lastly, this paper draws a conclusion and addresses research directions in Section 6.

2. Related Work

2.1. Attention-Based Deep Learning Models for ABSA

Deep learning techniques have demonstrated promising results in ABSA [2,23]. Deep learning models can automatically learn semantic representation from raw data without manual feature engineering [24,25]. Most models are based on a framework composed of task-independent word embedding and task-centric deep neural networks. Word embeddings become fundamental components in modern NLP tasks. Word2Vec [26] and GloVe [27] are popular pre-training models but focus on static word embeddings. Recently, BERT [15] introduced context-sensitive word representations by considering both left and right contexts of a word simultaneously. Its self-attention mechanism is effective at capturing the full context of a sentence.

For task-centric deep neural network models, RNN-based models and CNNs have been extensively utilized [28,29]. RNN-based models, such as long short-term memory (LSTM) [30] and gated recurrent units (GRUs) [31], are particularly effective at capturing sequential text information. BiLSTM enhanced performance of LSTM by allowing the model to access both prior and future context in a sequence, making it more effective in tasks requiring nuanced language understanding [32]. On the other hand, CNNs can efficiently detect patterns in shorter texts and contiguous words. The 2D-CNN can capture broader contextual information by transforming the text into a 2D image [33]. The BERT model also plays a role as the flexible and powerful post-training model when fine-tuned for specific tasks, including ABSA, by utilizing a multi-layer self-attention mechanism [34].

With the success of attention mechanisms in various tasks, such as machine translation [14,35] and emotion analysis [36], numerous attention-based deep learning models have been proposed for the ABSA task. Wang et al. [37] first introduced an LSTM model with a self-attention layer, known as ATAE-LSTM, which incorporates aspect embeddings into the attention mechanism. Tang et al. [38] proposed a deep memory network with multiple stacked computational layers, each containing an attention module. Building upon the HEAT network [39], Cheng et al. [40] developed an LSTM-based model that jointly learns to represent aspects and aspect-level sentiment information using two distinct attention mechanisms: aspect attention and sentiment attention. Ma et al. [41] presented the interactive attention network (IAN), which models the interactive attention between context word vectors and multi-word aspect term vectors. Huang et al. [42] proposed an attention-over-attention (AOA) neural network, which captures the interaction between aspects and context sentences by generating mutual attention representations. Xu et al. [43] introduced the multi-attentional network (MAN), which leverages both intra-level and inter-level attention to enhance the representation of aspect-specific sentiment.

Compared to previous models, hybrid learning models with attention, such as the LSTM-CNN with attention model, have shown improved performance because the hybrid model aims to compensate for the weaknesses of one model by leveraging the strengths of other models [44]. In ABSA where local context and sequential information are important, various attention-based hybrid models show their usefulness [45,46,47].

Currently, as another approach to effectively capture the dependency relation between aspect and context, graph convolution network (GCN) models are proposed [48,49,50]. However, since we aim to focus on the effectiveness of hierarchical sparse attention over the ABSA reviews in this work, we employ the model consisting of a BERT and a CNN-biLSTM for text processing.

2.2. Attention with Sparsity Constraints

The attention mechanism is known to improve both the accuracy and interpretability of models by assigning different weights to input elements based on their relevance. During the computation of attention, raw attention scores are transformed into a probability distribution that reflects the relative importance of each element [51]. Various functions have been proposed for this transformation, including sum-normalization [52], softmax [53], and spherical softmax [54], each of which maps vectors from Euclidean space to a probability simplex. Among these, softmax has become the standard choice for most attention implementations.

However, several studies have revealed the limitations of softmax in scenarios that benefit from sparsity. Luong et al. [55] reported that the dense attention distribution produced by softmax can degrade performance in long-sequence settings, particularly in machine translation tasks. Veličković et al. [6] further argued that softmax fundamentally fails to support robust reasoning across all input configurations. They validated this claim through empirical studies on tasks such as max-retrieval and CLRS-text, using Transformer-based architectures.

To address this issue, various methods have been proposed. Regularization-based approaches encourage sparsity by introducing constraints into the loss function, making them both flexible and easy to integrate into existing models [7,8]. However, these methods rely on indirect control over attention distributions, which can result in suboptimal sparsity levels and require careful tuning of the regularization strength. Instead of indirectly influencing the attention distribution, some researchers have proposed methods that directly induce sparsity. One such approach involves modifying the softmax function to suppress elements with lower attention scores [6,10,11]. Veličković et al. [6] introduced an adaptive temperature mechanism that adjusts based on the dataset, while Feng et al. [10] and Sun et al. [11] proposed techniques that retain only attention scores above a predefined threshold or select the top-k most relevant tokens using softmax variants. These methods were applied to tasks such as document classification, summarization, and question-answering; however, they have not yet been explored in the context of ABSA.

As an alternative line of research, several functions have been proposed to replace softmax in attention mechanisms. Since the introduction of sparsemax, a range of sparse transformation functions, such as sparsegen and entmax, have been developed to enable more selective and interpretable attention. The sparsemax function performs Euclidean projection onto the probability simplex, resulting in sparse attention distributions that improve interpretability [12]. It has shown promising empirical performance in tasks such as natural language inference and multi-label classification. Sparsegen, a generalization of sparsemax, offers greater flexibility in controlling the level of sparsity while retaining favorable optimization properties [56]. Entmax further generalizes both softmax and sparsemax by introducing a tunable parameter, α, which allows the dynamic adjustment of sparsity levels within a continuous range [13]. In this study, we adopt the entmax function to explore the effectiveness of sparse attention in the ABSA task, specifically investigating how varying degrees of sparsity influence model performance.

In the domain of ABSA, research on sparse attention remains relatively limited and has primarily focused on regularization-based methods [16,57,58]. Dhanith et al. [57] proposed a sparse-self-attention-based gated recurrent unit (SSA-GRU-AE), which applies L1 regularization to enforce sparsity in self-attention. Hu et al. [58] introduced two aspect-level regularization strategies: sparse regularization for individual aspects and orthogonal regularization across multiple aspects. In their subsequent work [16], they extended this approach by incorporating task-level regularization for ACD, forming a hybrid regularization framework. Although their model addressed the ACSA task using a multi-task learning setup similar to ours, it did not incorporate residual connections between tasks, and it required each input review to be accompanied by a predefined set of aspect categories. Furthermore, most existing studies have relied on self-attention architectures that naturally exhibit a high level of sparsity, which may limit the generality and adaptability of the learned representations. To the best of our knowledge, this study is the first to incorporate a sparse attention transformation function into the ABSA task within query-key attention architectures. We evaluate the effectiveness of sparse attention using widely adopted ABSA benchmark datasets, including SemEval and MAMS.

2.3. Joint Learning for ABSA

Joint learning has been increasingly adopted in ABSA to simultaneously optimize multiple related tasks, such as aspect term extraction (ATE), aspect category detection (ACD), and sentiment classification for target aspects. This multi-task paradigm, which is often used interchangeably with multi-task learning [21], leverages inter-task dependencies, thereby improving both generalization and overall model performance.

Previous ABSA studies exploring joint frameworks have primarily focused on the aspect-term-based sentiment analysis (ATSA) task, where the aspect term extraction (ATE) and sentiment classification tasks address the limitations of existing models that focus on a single task [46,59,60,61]. However, these approaches heavily rely on the accurate detection of aspect terms, which can significantly degrade sentiment classification performance.

There have also been studies in ACSA that utilize joint learning [62,63,64]. Schmitt et al. [62] proposed an end-to-end joint learning network, while Wang et al. [63] introduced the BERT-SAN model. Both employed ACD as an auxiliary task to guide sentiment prediction with explicit aspect category supervision. Pei et al. [64] proposed a multi-task joint learning framework to simultaneously solve the ACSA and ATSA tasks using BERT. However, prior works did not explicitly integrate the attention learned from ACD into the sentiment prediction module and did not incorporate residual learning to connect tasks. Our work introduces a residual joint learning architecture that enhances the information flow between ACD and ACSA through residual connections, while incorporating sparse attention to improve focus and interpretability.

3. Background

3.1. Attention Mechanism

To enhance the understanding of sparse attention, we first describe the general attention mechanism. It consists of four components: input features, query, attention computation, and output. The inputs of the attention model are the matrices of feature vectors $F=\{f_1,\dots ,f_{n_f}\}\in {\mathbb{R}}^{d_f\times n_f}$ and the query $Q\in {\mathbb{R}}^{d_q}$. The query tells the attention model which and how many input feature vectors to attend to the query.

To generate the attention, two different matrices are extracted from $F$: the keys $K=\{k_1,\dots ,k_{n_f}\}\in {\mathbb{R}}^{d_k\times n_f}$, and the values $V=\{v_1,\dots ,v_{n_f}\}\in {\mathbb{R}}^{d_v\times n_f}$, where $d_k$ and $d_v$ are the dimensions of the key vectors and value vectors. The general way of obtaining $K$ and $V$ is through a linear transformation of $F$ using the weight matrices, $W_K$ and $W_V$, respectively. $W_K$ and $W_V$ can be learned during training or previously predefined:

$$K=W_K\times F,\mathrm{V}=W_V\times F$$

(1)

Attention computation is composed of three steps: attention score calculation, probability distribution transformation, and context vector construction, as presented in Figure 1.

Its goal is to produce some weighted vector(s) with vectors in $V$. Each weight means a relative importance of each input feature vector according to a given query. These relative weights are obtained via attention score calculation and probability transformation.

At the step of attention score calculation, an attention score vector $e=\{e_1,\dots ,e_{n_f}\}\in {\mathbb{R}}^{n_f}$is produced with the query $q$ and the key matrix $K$ via some score function, $f_{score}$:

$${\mathrm{e}}_i=f_{score\left(k_i, q\right)}$$

(2)

The $f_{score}$ function can be defined as any function that measures the relatedness or similarity of the key and query, for example, dot product.

Then, in the probability transformation step, the attention scores are transformed into a probability distribution of [0, 1] by a probability transformation function. The probability transformation function allows the model to assign relative importance to different input elements while keeping the overall distribution balanced. There are various probability transformation functions, but current attention models typically use softmax. The softmax function converts a vector of weights to a vector that represents a posterior label probability, $a=\{a_1,\dots ,a_{n_f}\}\in {\mathbb{R}}^{n_f}$, where the sum of $a_i$ is 1, as follows:

$$a_i=softmax\left({\mathrm{e}}_i\right)={\displaystyle \frac{\mathit{exp}\left({\mathrm{e}}_i\right)}{{\sum }_{j=1}^{n_f}\mathit{exp}\left({\mathrm{e}}_j\right)}}$$

(3)

Each element of the input vector $e$ is executed by the exponential function, and then the output values are normalized by dividing by the sum of the exponentials. The sum of the output vector $a_i$ is 1, satisfying the conditions of a probability distribution. The attention weights provide a rather intuitive interpretation for the attention module.

In the final weighted vector construction, a context vector $c\in {\mathbb{R}}^{d_v}$ is calculated as:

$$c= \sum _{i=1}^{n_f}{a}_i{v}_i$$

(4)

We observe that in the probability transformation step, softmax produces a dense probability distribution, assigning positive values to all elements due to its exponential operation. As a result, unrelated elements with positive weights can introduce noise that can degrade the model’s accuracy. It becomes particularly critical in multi-aspect presence.

3.2. Sparse Attention Distribution Transformation

Softmax is one of the possible transform functions that aim to map a given input $c \in R_d$ onto the probability simplex ${\Delta }^{d-1}$ in a differentiable manner. A simplex is a geometric structure in which vectors are constrained to be positive and sum to 1, as follows:

$${\Delta }^{d-1}=\left\{p\in R^d∣p\ge 0,{\sum }_{i=1}^d{p}_i=1\right\}$$

(5)

This forces the function to output a probability distribution that will lie on a corner of the probability simplex. Therefore, $p$ will be a binary vector with a 1 at the component of the highest value in $c$, which leverages softmax (Equation (3)) to map $c$ onto the simplex, written in a differentiable approximation of the argmax operation:

$$softmax\left(c\right)=argma{x}_{p\in {\mathsf{\Delta }}^{\mathrm{d}-1}}{c}^{T}p+H^S\left(p\right),$$

(6)

where $H^S\left(\mathrm{p}\right)=-{\sum }_j{p}_j\mathit{ln}{p}_j$ is the Gibz–Boltmann–Shannon entropy [65].

By changing the entropy, we can consistently describe other transformation functions. The sparsemax function can be defined with the Gini entropy $H^G\left(\mathrm{p}\right)={\displaystyle \frac{1}{2}}{\sum }_j{p}_j\left(1-p_j\right)$:

$$\begin{gathered} SparseMax\left(c\right)=argma{x}_{p\in {\mathsf{\Delta }}^{\mathrm{d}-1}}{c}^{T}p+H^G\left(p\right) \\ = argmi{n}_{p\in {\mathsf{\Delta }}^{\mathrm{d}-1}}|p-c{|}^2 \end{gathered}$$

(7)

This transformation can assign a zero value to low-attention-scoring elements, thereby generating a sparse attention distribution where only relevant elements have weights.

As a follow-up study, the α-entmax function was proposed [13]. Unlike sparsemax, α-entmax utilizes the Tsallis entropy [66] and controls the propensity of sparsity in probability distribution through the α parameter, defined as:

$$\begin{gathered} \alpha -entmax\left(c\right)=argma{x}_{p\in {\mathsf{\Delta }}^{\mathrm{d}-1}}{c}^{T}p+H_{\alpha }^T\left(p\right) \\ ={\left[\left(\alpha -1\right)c-\tau \left(c\right)\mathbf{1}\right]}_{+}^{{\displaystyle \frac{1}{\alpha -1}}} \end{gathered}$$

(8)

$$H_{\alpha }^T\left(p\right)=\left\{\begin{array}{c}{\displaystyle \frac{1}{\alpha \left(\alpha -1\right)}} {\displaystyle \sum _j}({\pi }_j-{\pi }_j^{\alpha }{\displaystyle )}, \alpha \ne 1 \\ -{\displaystyle \sum _j}{\pi }_j\left(\mathit{log}{\pi }_j\right), \alpha =1 \end{array}\right.$$

(9)

In Equation (8), ${\left[· \right]}_{+}$ is the positive part (ReLU) function, and τ: R_n → R is a normalizing function satisfying the constraint $\sum _j{\left[\left(\alpha -1\right)c_j-\tau \left(c\right)\mathbf{1}\right]}_{+}^{{\displaystyle \frac{1}{\alpha -1}}}=1,$for any $c$. As a result, entries with score $c_j\le {\displaystyle \frac{\tau \left(c\right)}{\alpha -1}}$ get exactly zero probability and can be disregarded during computation.

As the value of α approaches 1, i.e., when $H_1^T$ becomes the Shannon entropy, the α-entmax function recovers the softmax function. For all α > 1, it produces continuously sparser output distributions. In particular, for α = 2, it recovers the sparsemax function.

Both sparsemax and α-entmax offer efficient and differentiable algorithms to generate sparse attention distributions, making them seamless alternatives to the softmax function. In this work, we use α-entmax to control sparsity in attention distributions based on the value of α.

3.3. Residual Learning

Residual learning, originally introduced in ResNet [67], is a widely adopted technique in deep neural networks, allowing layers to learn modifications (residuals) to the identity mapping, rather than learning the entire transformation. Instead of directly mapping the input feature $x$ to an output $\widehat{x}$, the residual learning module reformulates the desired mapping as:

$$\widehat{x}=\mathrm{F}\left(\mathrm{x}\right)+\mathrm{x}, \mathrm{where} \mathrm{F}\left(\mathrm{x}\right) \mathrm{the} \mathrm{residual} \mathrm{function} \mathrm{to} \mathrm{be} \mathrm{learned}$$

(10)

This skip connection to directly pass input information to subsequent layers can prevent vanishing gradients and accelerate network training.

In the context of multi-task learning and NLP, residual connections can be used to combine information from sub-tasks or earlier layers with downstream modules. This approach enables the model to support more stable learning and effective information transfer. By selectively adding refined signals rather than replacing input representations entirely, residual connections promote better generalization and interpretability in complex architectures.

4. Sparse-Attention-Based Residual Joint Network (SPA-RJ Net)

This section presents the SPA-RJNet architecture, as illustrated in Figure 2.

4.1. Input Layer for Embedding

Given a review $s$ consisting of $n$ words, represented as $s=\{w_1,w_2,\dots ,w_n\}$, and a set of aspect categories $a=\{a_1,a_2,\dots ,a_k\},$ where each aspect $a_i$ is a sequence of $m$ word tokens, SPA-RJ Net M predicts the sentiment polarity $p$ of $s$ with respect to $a$:

$$M\left(s,a\right)=p, \mathrm{where} p=\left\{p_j|p_j\in \left\{positive, negative, neutral\right\}, j=\mathrm{1,2},\dots ,i \right\}$$

Each word $w_i$ in the input sequence $s$ is initially mapped to a fixed d-dimensional (d = 768) vector, corresponding to the default hidden size of the pre-trained BERT base model. Thus, the input review $s$ is embedded into $E_s$ as follows:

$$E_s={\mathit{BERT}}_{review}\left(s\right)\in {\mathbb{R}}^{n\times 768}$$

Each aspect $a_i=\{t_1^{\left(i\right)},t_2^{\left(i\right)},\dots ,t_m^{\left(i\right)}\}$ is first encoded by BERT at the token level. We apply mean pooling to obtain a fixed-length vector, which is then projected into a 1024-dimensional space via a linear transformation followed by a $\mathrm{t}\mathrm{a}\mathrm{n} \mathrm{h}$ activation, as follows:

$$m_i={\displaystyle \frac{1}{M}}{\displaystyle \sum _{k=1}^M}{{BERT}_{aspect}(t}_k)$$

(11)

$$e_{a_i}=\tan \mathrm{h}\left(W_a\ast m_i+b_a\right)\in {\mathbb{R}}^{1024}$$

(12)

4.2. Sparse Aspect-Category Attention

The ACD module identifies which predefined aspect categories are present in a given review. To enhance precision and interpretability, it employs sparse aspect-category attention, which enables the model to selectively focus on the most informative words for each aspect. For aspect-category attention, the transformed aspect vector $e_{a_i}$ is used as the query $q_i^{acd}$, while $E_s$ serves as both the key $K_{s,i}^{acd}$ and the value $V_{s,i}^{acd}$ for the aspect $a_i$, as follows:

$$\begin{gathered} q_i^{acd}=W_{q,i}^{acd}\ast e_{a_i} \\ {K}_{s,i}^{acd}=W_{k,i}^{acd}\ast E_s \\ {V}_{s,i}^{acd}=W_{v,i}^{acd}\ast E_s \end{gathered}$$

(13)

To compute attention weights for ACD, we employ entmax in Equation (7), and then the attention score for the $i$-th aspect is computed as follows:

$${\alpha }_i^{acd}= \alpha -entmax\left({\displaystyle \frac{q_i^{acd}\ast {\left(K_{s,i}^{acd}\right)}^{\top }}{\sqrt{d}}}\right),$$

(14)

where $q_i^{acd}\in {\mathbb{R}}^d$ is the aspect-specific query, and $K_{s,i}^{acd}\in {\mathbb{R}}^{n\times d}$ is the key matrix projected from the input sequence $E_s$.

Using ${\alpha }_i^{acd}$, we obtain a context vector by a weighted sum over the value matrix:

$$c_i^{acd}= {\alpha }_i^{acd}\ast V_{s,i}^{acd}$$

(15)

To incorporate explicit aspect category information, we residually enhance the context vector by adding a linear transform of $e_{a_i}$. Also, the resulting representation ${\widehat{c}}_i^{acd}$ is then passed through a task-specific classifier to produce the ACD logit:

$${\widehat{c}}_i^{acd}=c_i^{acd}+W_{r,i}\ast e_{a_i}$$

(16)

$${\mathcal{l}}_i^{ACD}=W_{c,i}^{acd}\ast {\widehat{c}}_i^{acd}+b_{c,i}^{acd}$$

(17)

Since aspect category detection is formulated as a binary classification task for each aspect, we apply the binary cross-entropy loss with logits to supervise each logit ${\mathcal{l}}_i^{ACD}$ against a binary ground-truth label $y_i^{ACD}\in \left\{\mathrm{0,1}\right\}$. The overall ACD loss is defined as the average over all $l$ aspects:

$${\mathcal{L}}_{ACD}= {\displaystyle \frac{1}{l}}{\displaystyle \sum _{i=1}^l}BCEWithLogits\left({\mathcal{l}}_i^{ACD},y_i^{ACD}\right)$$

(18)

4.3. Residual Connection from ACD to ACSA

To incorporate the attention signal generated during the ACD stage into the downstream ABSA task, we refine the original review representation by applying a residual connection. For an aspect $a_i$, the attention distribution ${\alpha }_i^{acd}$ is used to reweigh the word-level input embeddings $E_s$. This reweighting is performed via element-wise multiplication across the sequence length, allowing the model to amplify aspect-relevant features while attenuating irrelevant ones, as follows:

$${\widehat{E}}_s^{\left(i\right)}=\left({{\alpha }_i^{acd}}^{\top }⨀E_s\right)+E_s,$$

(19)

where $⨀$ denotes element-wise multiplication applied across the sequence length.

The residual connection ensures that the original semantic content captured by the pre-trained BERT encoder is preserved, while the attention weights ${\alpha }_i^{acd}$ selectively emphasize tokens that are informative for the aspect $a_i$.

This fused representation ${\widehat{E}}_s^{\left(i\right)}$ serves as an enriched input for the subsequent ACSA-specific processing layers. By integrating attention-based reweighting with residual learning, the model benefits from both a targeted focus on aspect-relevant regions and retention of broader contextual meaning, which is critical for accurate sentiment classification.

4.4. Feature Extraction

At this stage, we encode the residual-enhanced review representation using a CNN-BiLSTM hybrid encoder, which captures both rich local and sequential features.

The aspect-enhanced review representation ${\widehat{E}}_s^{\left(i\right)}$, obtained through residual integration, is then processed by a hybrid encoder consisting of a convolutional neural network (CNN) followed by a bidirectional long short-term memory (BiLSTM) network. The CNN layer captures local n-gram features by applying convolutions over the word sequence:

$$C_i=\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v}\left({\widehat{E}}_s^{\left(i\right)}\right)\in {\mathbb{R}}^{\mathrm{n}\times {\mathrm{d}}_{\mathrm{c}}},$$

(20)

where $d_c$ is the number of output channels (feature maps).

This operation emphasizes localized patterns around each word, which are important for identifying sentiment-bearing phrases.

To model global dependencies beyond fixed local windows, the output $C_i$ is passed through a BiLSTM layer, which consists of two LSTM networks running in opposite directions. The forward LSTM processes the token sequence from the beginning to the end, capturing dependencies based on preceding context, while the backward LSTM traverses the sequence in reverse, modeling dependencies from future context. The outputs from both directions are concatenated at each timestep:

$$H^{\left(i\right)}=\mathrm{B}\mathrm{i}\mathrm{L}\mathrm{S}\mathrm{T}\mathrm{M}\left(C_i\right)\in {\mathbb{R}}^{n\times d_b},$$

(21)

where $d_b$ = 2 × hidden_size denotes the dimensionality of the concatenated forward and backward hidden states.

The resulting representation $H^{\left(i\right)}$ provides rich, context-aware features for each token, which are used in the subsequent aspect-specific attention mechanism.

4.5. Sparse Aspect-Sentiment Attention

After obtaining the contextualized token representations $H^{\left(i\right)}$ from the BiLSTM encoder, we apply a second attention mechanism to aggregate aspect-relevant information for sentiment classification. This attention is guided by same transformed aspect vector $u_i$, which is now projected into a new query vector $q_i^{acsa}$, suitable for this stage. Like the first attention mechanism in ACD, the output sequence from the preceding BiLSTM layer serves as the source of both key and value representations for aspect-specific sentiment attention:

$$\begin{gathered} q_i^{acsa}=W_{q,i}^{acsa}\ast e_{a_i}, \\ {K}_i^{acsa}=W_{k,i}^{acsa}\ast H^{\left(i\right)}, \\ {V}_i^{acsa}=W_{v,i}^{acsa}\ast H^{\left(i\right)} \end{gathered}$$

(22)

We then compute attention weights using the scaled dot product formulation, with sparsemax again employed to obtain a sparse, interpretable distribution over tokens:

$${\alpha }_i^{acsa}= \mathrm{sparsemax}({\displaystyle \frac{q_i^{acsa}\ast {\left(K_i^{acsa}\right)}^{\top }}{\sqrt{d_b}}})$$

(23)

Using these attention weights, we compute the final aspect-aware summary vector for the $i$-th aspect, which serves as the input to the sentiment classification layer. The resulting context vector aggregates sentiment-relevant information over the entire sequence, weighted by the aspect-specific attention distribution, and serves as a compact yet informative representation for downstream classification:

$$c_i^{acsa}= {\alpha }_i^{acsa}\ast V_i^{acsa}$$

(24)

The context vector $c_i^{absa}$, computed from the aspect-specific attention, is passed through a task-specific linear classifier to produce the sentiment logit for the $i$-th aspect:

$${\mathcal{l}}_i^{ACSA}=W_{c,i}^{acsa}\ast c_i^{acsa}+b_{c,i}^{acsa}$$

(25)

Here, $W_{c,i}^{acsa}\in {\mathbb{R}}^{C\times d_b}$, and $C$ denotes the number of sentiment classes (e.g., positive, neutral, and negative). The predicted sentiment is then supervised via the categorical cross-entropy loss with respect to the ground-truth label $y_i^{ACSA}\in \left\{\mathrm{0,1}, \dots ,C-1\right\}$. The overall ABSA loss is computed as the mean over all aspects with valid labels:

$${\mathcal{L}}_{ACSA}= {\displaystyle \frac{1}{k}}{\displaystyle \sum _{i\in V}}CrossEntropy\left({\mathcal{l}}_i^{ACSA},y_i^{ACSA}\right)$$

(26)

This formulation enables the model to flexibly handle partial labels while still optimizing for fine-grained, aspect-specific sentiment prediction.

4.6. Loss Function for Joint Learning

To jointly optimize the ACD module and the ACSA module, we adopt a multi-task learning framework where the total loss is a weighted sum of the task-specific losses. The overall loss for a single instance is defined as:

$${\mathcal{L}}_{joint}= \gamma \ast {\mathcal{L}}_{ACD}+\left(1-\gamma \right)\ast {\mathcal{L}}_{ACSA},$$

(27)

where $\gamma \in \left[\mathrm{0,1}\right]$ is a balancing hyperparameter that regulates the learning pace discrepancy between tasks.

This joint formulation allows the model to share and reinforce relevant representations across tasks. By jointly training both tasks within a unified architecture, the model benefits from mutual inductive bias: ACD helps guide attention toward sentiment-bearing phrases relevant to the identified aspect, while sentiment signals from ACSA serve to regularize and refine the identification of aspect categories.

Tasks in a joint learning framework often progress at different learning rates. In our experiments, the ACD task consistently converged more quickly than the ACSA task. When the loss functions are combined without adjusting a balancing parameter, the faster-learning task can dominate the optimization process, potentially hindering effective learning of the slower task and resulting in suboptimal performance. To mitigate this issue, we initialize the weighting coefficient γ to assign equal importance to both tasks, but as the ACD task stabilizes earlier on the validation set, γ is dynamically adjusted according to the following rule:

$$\gamma = 1-ACS{A}_{Accurac{y}_{val}}$$

(28)

This adaptive scheduling strategy reflects the intuition that, once the ACD task reaches a reliable level of performance, the model should gradually shift its focus toward more effective sentiment polarity learning in the ACSA task. The threshold for this transition is determined empirically.

5. Experiment

5.1. Dataset Description

To validate the performance of the SPA-RJ Net, we ran experiments on four benchmark datasets in the experiments, including the restaurant domain from datasets from SemEval-2014 task4 [17], SemEval-2015 task 12 [18], SemEval-2016 task 5 [19], and the MAMS dataset [20]. Particularly, as most reviews in the MAMS dataset contain at least two different aspects with different sentiment polarities per review, our experiments effectively present the ability to handle multiple aspects.

We first removed any review instances that were not assigned to any aspect category and then categorized each aspect category into three sentiment labels: positive, negative, and neutral. To mitigate the effect of outliers, we excluded the top and bottom 5% of instances based on review length. The dataset was split into training and testing sets using an 80:20 ratio. Detailed statistics for the datasets are summarized in

Table 1. Sentiment-label distribution (train/test) for ABSA benchmark datasets.

Data Set	Positive		Negative		Neutral
	Train	Test	Train	Test	Train	Test
REST 14	1631	530	640	181	144	43
REST 15	754	252	280	245	39	25
REST 16	1348	321	647	131	134	31
MAMS	1332	334	1485	372	1508	377

.

Table 2. Aspect category distribution (train/test) for ABSA benchmark datasets.

Data Set	Food		Service		Ambience		Price
	Train	Test	Train	Train	Test	Test	Train	Test
REST 14	1166	402	562	167	385	105	302	80
REST 15	530	254	256	159	177	69	110	40
REST 16	692	257	933	145	234	57	270	24
MAMS	1158	267	1134	269	183	50	118	19

summarizes the per-aspect counts (food, service, ambience, and price) that fell into the training and test partitions of each dataset. We kept the split close to the customary 80:20 rule so that no category was under-represented in the held-out set.

5.2. Evaluation Metrics

To evaluate the performance of our model, we selected accuracy and Macro-f1, both of which are widely used metrics. Accuracy offers an intuitive measure of the proportion of correctly predicted instances across the dataset. Macro-F1 is computed as the unweighted average of F1 scores across all classes, where F1 score is the harmonic mean of precision and recall. Precision measures the correctness of positive predictions, while recall reflects the model’s ability to capture all relevant instances. These metrics are defined as follows:

$$Precision= {\displaystyle \frac{TP}{TP+FP}}, Recall= {\displaystyle \frac{TP}{TP+FN}},$$

where TP, FP, and FN denote true positives, false positives, and false negatives, respectively:

$$F1= 2·{\displaystyle \frac{Precision·Recall}{Precision+Recall}}$$

$$Macro-F1={\displaystyle \frac{1}{C}}{\displaystyle \sum _{i=1}^C}{F1}_i$$

By aggregating precision and recall per class and then averaging equally, Macro-F1 ensures that the model’s performance is not dominated by majority classes, offering a more balanced assessment. Therefore, we used accuracy and Macro-F1 as evaluation metrics in our experiments.

5.3. Comparison Models

We compared our model to existing attention-based deep learning models in the ABSA domain. They are built on LSTM, RNN, MemNet, and GNN models, each incorporating attention mechanisms in different ways:

• ATAE-LSTM [37]. The concatenations of an aspect embedding vector and each word embedding vector of the review are served as inputs to the LSTM model and an attention layer, allowing to selectively focus on aspect-relevant information.
• TD-LSTM [68]. TD-LSTM employs two target-dependent LSTMs that process the preceding part with the target term in both directions. Attention is applied to the concatenated hidden states, emphasizing words around the target.
• MemNet [38]. MemNet is a deep memory network with multiple computational layers, each of which includes an attention mechanism and explicit memory. Multi-hop attention enhances the ability to capture nuanced patterns by focusing on key context.
• IAN [41]. The interactive attention network (IAN) learns the interaction between aspect terms and words, generating separate representations for both, which effectively reflects the mutual influence between targets and contexts through two attention layers.
• AOA [42]. The attention-over-attention (AOA) jointly models the interaction between aspects and sentences, explicitly capturing the relationship between aspects and context, which enhances accuracy, particularly in complex multi-aspect contexts.
• ASGCN [48]. This model first uses GCN over the dependency tree to exploit syntactical information and word dependencies and to learn the aspect-specific representations.
• SAGCN [49]. SAGCN utilizes GCN to find correlation between the aspect and sentence using the dependency tree.
• DSMN [69]. DSMN uses the multi-hop attention guided by dynamically selected context memory. Then, it integrates the aspect information with the memory networks.
• CMA-MemNet [70]. The rich semantic information between the aspect and the sentence is extracted by this memory network. It uses multi-head self-attention to make up for the problem where it ignores the semantic information of the sequence itself.
• DualGCN [50]. This model uses two GCNs, SynGCN and SemGCN, that jointly consider the syntax structures and semantic correlations, with regularizers constraining attention scores in the SemGCN.
• MAT+2R_O [16]. In a multi-task learning setting with ACD and ACSA, this LSTM-based model applies hybrid regularizations to the attention mechanism, combining aspect-level and task-level constraints to enhance performance.

5.4. Experiment Environment

All experiments were executed on a 64-bit workstation equipped with an Intel Core i9-10900X CPU (10 cores), 128 GB DDR4 RAM, and two NVIDIA GeForce RTX 3090 GPUs (24 GB GDDR6X VRAM each). The SPA-RJ Net was fine-tuned from BERT-base (12 layers, 768 hidden units) using the AdamW optimizer. We applied a learning rate of 2 × 10⁻⁵ to the encoder parameters and 1 × 10⁻⁴ to the newly added CNN-BiLSTM layers in the ACSA task. Training used a batch size of 32 and a maximum input length of 128 tokens. Dropout 0.1 was applied to all layers. Sparse attention for both ACD and ACSA tasks was implemented with entmax 2.0 (α = 2.0), and the threshold for the transition of balancing parameter was 0.92.

5.5. Experiment Result

We propose several variants of the SPA-RJ Net, distinguished along two dimensions: the sparsity of the attention mechanism—dense attention (DA) versus sparse attention (SPA)—and the learning framework—single-task learning (ST) versus residual joint learning (RJ). DA adopts the softmax function, while SPA employs the entmax function with a sparsity-controlling hyperparameter α. ST represents a single-task learning framework for the ACSA task, whereas RJ refers to a joint learning architecture that integrates ACD and ACSA through a residual connection. We evaluated SPA-RJ Net and its three variant models—DA-ST Net, SPA-ST Net, and DA-RJ Net—on four widely used ABSA benchmarks with three labels (positive, neutral, and negative): REST 14, REST 15, REST 16, and MAMS. We note that the sparse-attention-based learning models, i.e., SPA-ST Net and SPA-RJ Net, had the entmax sparsity parameter α set to 2.

Table 3. Performance comparison on four datasets. The best performance is bold typed and unreported results are represented with a hyphen mark.

Model	REST 14		REST 15		REST 16		MAMS
	Acc.	Macro-F1	Acc.	Macro-F1	Acc.	Macro-F1	Acc.	Macro-F1
ATAE-LSTM	77.20	67.02	78.48	60.53	83.77	61.71	70.63	67.02
TD-LSTM	78.00	66.73	76.39	58.70	82.16	54.21	74.59	66.73
MemNet	78.20	65.80	77.31	58.28	85.44	65.99	63.29	69.64
IAN	79.26	70.09	78.50	52.65	84.74	55.21	76.60	70.09
AOA	79.97	70.42	78.17	57.02	87.50	66.21	77.26	70.42
ASGCN	80.77	72.02	79.89	61.89	88.99	67.48	-	-
SAGCN	77.14	67.16	76.19	59.73	88.30	67.40	-	-
DSMN	78.17	70.13	79.16	61.70	86.70	67.40	-	-
CMA-MemNet	79.14	71.23	80.90	62.90	87.13	68.76	-	-
DualGCN	84.27	78.08	81.73	65.05	89.29	68.08	-	-
MAT-2RO	84.28	74.45	77.51	52.78	-	-	-	-
DA-ST Net	85.69	67.51	80.95	55.19	92.95	83.65	74.47	74.10
SPA-ST Net (α = 2)	89.64	76.30	83.53	65.17	93.38	85.78	78.00	77.99
DA-RJ Net	84.73	66.35	81.15	55.45	89.74	82.25	73.06	72.32
SPA-RJ Net (α = 2)	89.41	74.56	84.33	67.34	94.44	88.26	80.78	79.56

summarizes the performance comparison with a wide range of baseline models, including LSTM-based methods (e.g., ATAE-LSTM, IAN, and AOA), memory networks (e.g., MemNet, DSMN, and CMA-MemNet), and graph-based approaches (e.g., ASGCN and DualGCN). It is noteworthy in that DualGCN and MAT+2R_O apply sparsity into attention using regularization techniques and MAT+2R_O also adopts multi-task learning setting with ACD and ACSA. The baseline results are taken from the original and relevant papers, which used the same dataset under comparable settings. A summary table is provided in Appendix A.

Based on the experimental results, our SPA-RJ Net and its variant models demonstrated superior performance across most of the ABSA datasets. Notably, SPA-RJ Net, which integrates sparse attention with a residual joint learning strategy, yielded the highest scores in all evaluation metrics except for the result on the REST 14 dataset. On the REST 14 dataset, SPA-ST Net—a variant of SPA-RJ Net that still incorporates sparse attention—achieved the highest accuracy, while DualGCN, which employs a regularization-based approach, obtained the highest F1 score. These results suggest that introducing sparsity into the attention mechanism can be effective for ABSA tasks.

Focusing on the impact of sparse attention, we analyzed its effectiveness by separately examining performance under single-task learning and multi-task joint learning settings. In the single-task learning setting, SPA-ST Net outperformed its counterpart, DA-ST Net. SPA-ST Net achieved higher Macro-F1 scores by 8.8% on REST 14, 10.0% on REST 15, 2.1% on REST 16, and 3.9% on MAMS. In terms of accuracy, it yielded improvements of 3–4% on REST 14, REST 15, and MAMS, and 1.4% on REST 16. In the residual joint learning setting, SPA-RJ Net using sparse attention outperformed its dense-attention counterpart DA-RJ Net. These gains confirm that a sparse probability simplex helps the model intensify aspect-relevant parts. We observed that DA-based models showed slightly lower accuracy but significantly worse F1 scores compared to SA-based models. These results suggest that the inherent tendency of dense attention to diffuse focus across all tokens may lead to poor recall for minority classes. In contrast, sparse attention selectively emphasizes important tokens, enhancing class-wise balance and improving overall F1 performance.

From the perspective of the learning framework (single-task learning vs. multi-task learning), SPA-ST Net and SPA-RJ Net—both of which utilize sparse attention but differ in their frameworks—demonstrated that residual joint learning led to improved performance, in addition to the benefits of sparse attention. SPA-RJ Net produced an additional 1.8% Macro-F1 on average and push accuracy above 94% on REST 16 and 79% on MAMS over SPA-ST Net.

Through the experiments, we observed that sparse attention played an important role in improving performance on the ABSA task. Using sparse attention instead of dense attention led to bigger improvements than switching the setting from single-task learning to residual joint learning. We also note that combining sparse attention with residual joint learning demonstrated a kind of synergistic effect, consistently improving both accuracy and Macro-F1 across various ABSA datasets. Therefore, we believe that this combination model, like SPA-RJ Net, is applicable to various kinds of applications and domains.

To complement overall accuracy and F1 results, we report class-wise precision and recall to better understand the model’s behavior across sentiment classes.

Table 4. Class-wise precision and recall of SPA-RJ Net (sparsemax, α = 2).

Data Set	Positive		Neutral		Negative
	Precision	Recall	Precision	Recall	Precision	Recall
REST 16	90.50	81.02	72.21	80.11	95.02	97.10
MAMS	83.50	82.01	73.78	78.14	79.00	81.21

shows that SPA-RJ Net with sparsemax (α = 2) achieved strong and balanced performance. Positive and negative classes on both REST 16 and MAMS exhibited high precision and recall (e.g., ≥81% recall and ≥90% precision on REST 16), indicating effective detection of clear polarities. While the neutral class was more ambiguous than the others, the model still performed solidly (precision ≥ 72% and recall ≥ 78%) and avoided collapsing toward extremes. The small precision–recall gaps further suggested well-calibrated decision boundaries. These results support the model’s overall robustness across datasets and sentiment categories.

Table 5. The effect of sparsity levels in the attention distribution. The best performance is bold typed.

Model	Sparsity Parameter $\mathit{\alpha }$	REST 14		REST 15		REST 16		MAMS
		Acc.	Macro-F1	Acc.	Macro-F1	Acc.	Macro-F1	Acc.	Macro-F1
SPA-RJ Net	1 (=softmax)	84.73	56.35	81.15	55.45	89.74	62.25	73.06	72.32
	1.5	86.97	62.78	78.18	53.25	93.59	86.51	81.00	80.67
	2	90.23	74.63	84.33	67.34	94.44	88.26	80.78	79.56
	4	88.31	59.03	83.73	57.22	90.08	70.90	78.24	77.91

illustrates how the sparsity level of the attention distribution affected the performance of ACSA. We controlled the sparsity of the attention distribution by adjusting the value of $\alpha$, a parameter of the entmax function. As the value of $\alpha$ increased from 1 (i.e., equivalent to softmax when α = 1 [13]) to 4, the attention distribution became sparser.

In our experiments, every configuration that introduced sparsity (α > 1) outperformed dense attention across all four benchmark datasets. On all SemEval datasets, the optimal performance was observed at α = 2, whereas the best results on the MAMS dataset were achieved with α = 1.5. For instance, with α = 2.0, the model reached up to 94.44% accuracy and 88.26% Macro-F1 on REST 16, and with α = 1.5, the model achieved 81.00% accuracy and 80.67% Macro-F1 on MAMS. Then, beyond these values of α, performance gradually declined as α increased. This may result from dataset-specific characteristics in inherent sparsity, which can influence the optimal degree of attention sparsity for each dataset. Nevertheless, this result strongly highlights the limitations of relying on the conventional softmax function in the ABSA task and learning mode without a self-attention mechanism and suggests that sparse attention be actively considered as a viable alternative for achieving optimal performance.

5.6. Ablation Study

To investigate how individual components and task-specific sparsity settings influence performance, we conducted a series of ablation studies, as summarized in Table 5 and

Table 6. Ablation experiments on SPA-RJ Net. The best performance is bold typed.

Ablation Model	REST 14		REST 15		REST 16		MAMS
	Acc.	Macro-F1	Acc.	Macro-F1	Acc.	Macro-F1	Acc.	Macro-F1
SPA-RJ Net (with GloVe)	72.90	52.10	66.07	50.13	88.89	77.92	62.69	62.13
SPA-RJ Net (w/o ACD)	89.41	73.73	82.14	58.54	90.6	81.2	75.65	74.02
SPA-RJ Net (w/o residual connection)	85.14	66.18	84.33	57.61	92.31	83.02	75.47	72.48
SPA-RJ Net (w/o balancing hyperparameter)	86.80	68.94	82.33	56.96	90.44	87.02	76.79	75.88
SPA-RJ Net (complete)	90.23	74.63	84.33	67.34	94.44	88.26	80.78	79.56

.

In Table 6, we list several ablation models. SPA-RJ Net (with GloVe) denotes the model with pre-trained GloVe embedding vectors (*d = 300) instead of BERT to evaluate the impact of different input representations. The model using GloVe embeddings showed inferior performance compared to the original SPA-RJ Net (complete) using BERT. As GloVe provides static embeddings that assign a single representation to each word regardless of context, this limitation hinders the model’s ability to capture context-dependent meanings. This result underscores the effectiveness of contextualized embeddings like BERT for nuanced sentiment analysis in ABSA.

SPA-RJ Net (w/o ACD) ablates the ACD task, passing inputs only through a single sparse aspect-sentiment attention block for ACSA. Compared to the complete version (ACSA with ACD), SPA-RJ Net (w/o ACD) demonstrated overall lower performance. The performance gap was small on the SemEval 2014 dataset (less than 1%) but more significant on other datasets, with accuracy dropping by up to 5.13% and Macro-F1 by 9.8%. This underscores the critical role of the ACD module in providing explicit aspect-level guidance. To further explore the interaction between the ACD and ABSA attention mechanisms, we provide their respective attention maps in a case study.

Both SPA-RJ Net (w/o residual connection) and SPA-RJ Net (w/o balancing parameter) retain the two-task joint-learning framework. To examine the effect of residual connection from ACD to ACSA, we implemented SPA-RJ Net (w/o residual connection) by removing the residual learning submodule from SPA-RJ Net (complete). In this setting, only the ACD logits—without the refined, aspect-aware representations—are available to ACSA. This design restricts the transfer of aspect-aware context to the ACSA module, causing notable performance degradation—about 5% lower accuracy on REST 14 and MAMS, and nearly 10% lower Macro-F1 on REST 15. These findings suggest that the residual connection plays an essential role in capturing deeper semantic dependencies between ACD and ACSA by transferring semantically enriched information to the main task.

The performance difference between SPA-RJ Net (w/o balancing hyperparameter) and SPA-RJ Net (complete) reflects the effectiveness of the balancing hyperparameter. In our framework, which consists of separate ACD and ACSA tasks, the ACD task—being simpler than ACSA—generally learned significantly faster. Without adjusting the balancing hyperparameter, ACD may dominate the shared representation space, which can lead to suboptimal performance for the slower-learning ACSA task and hinder the convergence of both tasks. On average, SPA-RJ Net (complete) achieved 3.43% higher accuracy and a 5.69% gain in Macro-F1, highlighting the importance of controlling task imbalance during joint learning.

Table 7. The effect of attention configuration on SPA-RJ Net. The best performance is bold typed.

Attention Config.		REST 14		REST 15		REST 16		MAMS
Aspect Category Att.	Aspect Sentiment Att.	Acc.	Macro-F1	Acc.	Macro-F1	Acc.	Macro-F1	Acc.	Macro-F1
DA	DA	84.73	66.35	81.15	55.45	89.74	62.25	73.06	72.32
SPA	DA	87.90	68.60	82.74	56.53	89.96	76.06	80.66	79.49
DA	SPA	87.90	71.42	82.14	56.03	92.09	83.37	79.62	79.03
SPA	SPA	90.23	74.63	84.33	67.34	94.44	88.26	80.78	79.56

explores the impact of different attention configurations in performance. In our model, there are two attention modules: the aspect-category attention in the ACD task and the aspect-sentiment attention in the ACSA task, applied sequentially in that order. We investigated how the type of attention distribution—sparse (SPA) or dense (DA)—in each attention affected the overall performance. For example, DA-SPA refers to a configuration where the aspect-category attention generates a dense attention distribution by using softmax, and the aspect-sentiment attention yields a sparse attention distribution by using entmax (α = 2). SPA-RJ Net adopts the SPA-SPA configuration, in which sparse attention distributions are applied in both attentions.

Among the four configurations, SPA–SPA consistently achieved the best results across all datasets. For instance, it achieved an accuracy of 94.44% and a Macro-F1 score of 88.26 on REST 16, and 80.78% accuracy and 79.56 Macro-F1 on MAMS. In contrast, the DA–DA configuration, which applies dense attention in both modules, showed the lowest performance across most datasets, particularly in terms of Macro-F1 score, which demonstrates the limitations of dense attention in capturing fine-grained sentiment signals and the effectiveness of applying sparse attention in the domain of ACSA. The DA–SPA and SPA–DA configurations, each of which used a mix of sparse and dense attention, achieved intermediate results, suggesting that partial sparsity improved performance over fully dense configurations but did not outperform the fully sparse attention configuration.

The superior performance of the SPA–SPA setting suggests that sparse attention, by focusing on a limited number of informative tokens, is effective for both ACD and ACSA. In ACD, it highlights aspect-related cues while suppressing irrelevant information, leading to more accurate detection. In ACSA, it helps the model focus on sentiment-bearing expressions, reducing contextual noise. These results indicate that sparse attention offers a beneficial inductive bias for the task.

5.7. Case Study

In this section, we present a case study to examine how the proposed sparse attention in SPA-RJ Net operates during inference. We analyze several representative examples drawn from the MAMS dataset, which include review samples annotated with either a single aspect category (from case 1 to 4) or multiple aspect categories (from case 5 to 7).

As summarized in

Table 8. Case study on MAMS.

Sample Cases	Aspect/Label	Results
1. This happened while many dinners were enjoying their meal and it was great.	Food/Positive	Positive (✓)
2. The noise here is so bad that people entered and left without ordering and they lowered lights during the dinner.	Ambience/Negative	Negative (✓)
3. Stat by enjoying the atmosphere while having a glass of wine at the bar and we are ready to having fun.	Ambience/Positive	Positive (✓)
4. The service was rather slow but waiter was very pleasant and obliging	Service/Positive	Negative (✗)
5. The food was barely decent and out server was nowhere to be found.	Service/Negative Food/Positive	Negative (✓) Positive (✓)
6. The atmosphere was amazing for such a reasonable price.	Ambience/Positive Price/Positive	Positive (✓) Positive (✓)
7. The food was good. But not at all worth the price	Food/Positive Price/Negative	Positive (✓) Positive (✗)

, these cases are selected to illustrate the model’s ability to focus on aspect-relevant words and to demonstrate how sparse attention enhances interpretability. For each review, the table presents the ground-truth labels provided by the dataset alongside the model’s predictions. Figure 3 visualizes the attention distributions under different settings (dense att. vs. sparse att.; ACD att. vs. ACSA att.) to provide an intuitive understanding of how the model allocates attention, with darker colors representing higher attention scores. Each distribution illustrates how various attention mechanisms or task settings influence the model’s focus on aspect-relevant words.

For cases with a single aspect category (cases 1 to 4), the model correctly predicts the sentiment polarity in cases 1 to 3. In particular, cases 1 and 2 offer direct visual comparisons between sparse and dense attention for the same aspect category (food), allowing us to examine their interpretability. As shown in Figure 3a, sparse attention assigns significantly higher weights to sentiment-relevant words, such as ‘dinner’, ‘meal’, and ‘great’ in case 1, resulting in a correct prediction. While case 1 involves a relatively simple sentence structure with clear aspect–opinion pairs, case 2 is more challenging due to the dispersed nature of aspect-related words. Nonetheless, sparse attention highlights key tokens like ‘noise’, ‘bad’, ‘people’, and ‘dinner’ more distinctly than dense attention, which tends to distribute weights more uniformly across tokens. These examples demonstrate that sparse attention offers better focus and interpretability, particularly in identifying opinion-bearing expressions critical for accurate sentiment analysis.

On the other hand, case 3 illustrates the distinct roles of aspect-category attention (ACD att.) and aspect-sentiment attention (ACSA att.) in SPA-RJ Net, where both are configured with sparse attention distributions. While the sparse aspect-category attention assigns high scores primarily to aspect targets, such as ‘atmosphere’, the sparse aspect-sentiment attention focuses not only on the same aspect targets but also on sentiment-bearing expressions like ‘enjoying’ and ‘having fun’.

However, our model makes a misprediction in case 4 where multiple aspect terms under the same aspect category express conflicting sentiment polarities. An analysis of two sparse attention distributions in SPA-JR Net reveals that the sparse aspect-category attention over-concentrates on the aspect target word ‘service’, while overlooking related terms such as ‘waiter’. This suggests that the strong focus of sparse attention on a few key words can lead to the omission of other important information. Therefore, it is important to consider how to apply sparsity appropriately to achieve a better balance between focus and contextual coverage.

In cases 5 to 7, which involve multiple aspect categories, we observe that different words within the same sentence receive distinct attention scores depending on the given aspect category. This provides an opportunity to examine the interpretability of sparse attention in more complex settings. In case 5 (Figure 3b), for instance, sparse attention distinctly highlights ‘food’ and ‘decent’ for the food category, while assigning higher attention to ‘server’, ‘no-where’, and ‘found’ for the service category. In contrast, the dense attention yields a near-uniform distribution, with much of the review text shaded in light red, making it difficult to distinguish sentiment-relevant tokens. This comparison visually demonstrates that sparse attention not only filters out irrelevant tokens but also adapts its focus based on the aspect category, offering more interpretable and fine-grained attention behavior.

Case 6 demonstrates that the two distinct attentions, as observed in case 3, also effectively fulfill their respective roles in multi-aspect scenarios. For the ambience aspect category, the sparse aspect-category attention (ACD att.) strongly focuses on the word ‘atmosphere’, which is closely related to the aspect, while the sparse aspect-sentiment attention (ACSA att.) additionally attends to the associated opinion expression ‘was amazing’. Although the review sentence remains the same, given the price aspect category, the ACD and ACSA attentions focus on different words compared to the ambience aspect category—namely, ‘price’ and ‘reasonable’, respectively. It indicates that even in multi-aspect scenarios, the ACD and ACSA attentions are able to correctly identify the aspect-relevant content based on the given aspect category.

On the other hand, case 7 produces an incorrect result for one of the two aspects. As shown in the sparse attention distribution for the price aspect category in case 7 of Figure 3b, the model fails to attend to the word ‘not’ in the opinion expression ‘not at all worth’ for the price aspect. As a result, it predicts the opposite sentiment label of the original negative. This outcome, like case 4, appears to result from excessively high attention sparsity, which causes the model to focus on too few words and miss a single critical term—ultimately leading to a reversal of sentiment polarity. This observation suggests the need to calibrate sparsity appropriately for the given dataset. Nevertheless, as demonstrated in the experiments from the previous section, the selective focus of sparse attention proved effective in many cases.

6. Discussion and Conclusions

In this paper, we proposed SPA-RJ Net to address the limitations of conventional dense attention generated by the softmax function and to explore the effectiveness of sparse attention in ACSA. SPA-RJ Net leverages two sparse attentions induced by the $\alpha$-entmax function and it also incorporates a residual joint learning framework between aspect category detection (ACD) and aspect category sentiment prediction (ACSA) tasks. This design enables our SPA-RJ Net to selectively focus on important words/context for the corresponding aspect category.

Our experiments empirically demonstrated the effectiveness of SPA-RJ Net, which largely outperformed a range of attention-based models, including those employing attention regularization or multi-task learning. In particular, the use of sparse attention led to meaningful performance gains, highlighting its value for ABSA tasks. The ablation study validated the contribution of each technical component, while case analysis illustrated the difference in sparse and dense attention distributions and clarified the roles of aspect-category and aspect-sentiment attentions. Overall, SPA-RJ Net improved both performance and interpretability in single- and multi-aspect ACSA scenarios.

We employed the entmax transformation function [13] to induce sparse attention, adjusting the α parameter in the range of 1.0 to 4.0. This allowed us to generate attention distributions with varying degrees of sparsity, including softmax (α = 1.0) and sparsemax (α = 2.0). Notably, we observed that the optimal α value varied across datasets, depending on their sparsity characteristics. For datasets with more concentrated opinion expressions, higher sparsity (larger α) led to better performance, whereas denser expressions favored lower α values. These results indicate that entmax provides a flexible and adaptive attention mechanism, capable of adjusting to the structure of different datasets rather than being limited to a specific domain or task.

While our experiments were conducted solely on the restaurant domain to align with prior works and ensure comparability, our model is designed to be domain-agnostic and can be easily applied to other domains. In future work, we plan to explore cross-domain and cross-lingual ABSA settings to validate the model’s effectiveness in transferring knowledge across diverse domains and languages, thereby providing broader insights into its generalizability and robustness.

Furthermore, we expect that SPA-RJ Net can be effectively applied to various downstream tasks, such as targeted summarization, question-answering, and the construction of explainable recommendation systems grounded in fine-grained opinion mining.

We plan to pursue future research in the following directions. First, we aim to dynamically learn sparsity-related parameters to adapt the level of attention sparsity to each dataset. By employing dataset-specific sparsity levels, we expect to achieve improved and more robust performance. In addition, we plan to extend our model by incorporating GNNs. Specifically, we aim to integrate sparse attention into GNNs, enabling each node to attend to its most relevant neighbors. This approach leverages syntactic dependencies while reducing noise, computational cost, and memory usage. We believe sparse attention inherently improves efficiency and scalability, and we plan to evaluate runtime and memory usage under varying dataset scales as part of future work, aiming to provide a more efficient solution for ABSA.