CogSent: Cognition-Driven Multimodal Sentiment Analysis Through Fast–Slow Thinking

Abstract

Sentiment analysis of multimodal social media data is of great importance, not only for recognizing objective information but also for capturing subjective emotional states. While single-modal sentiment analysis has achieved notable progress, existing multimodal approaches still face two key challenges: (1) inadequate modeling of subjective emotional characteristics and (2) insufficient handling of cross-modal inconsistencies. To address these limitations, we propose an image-text multimodal sentiment analysis framework (CogSent) grounded in the psychological Dual-System Theory. First, inspired by human fast and slow thinking, we develop a Hierarchical Dual-Channel Cognition (HDCC) architecture to extract intuitive and rational affective features, respectively. Second, we introduce an Intuition-Guided Cognition Refinement (IGCR) module that uses System-I-inspired intuitive representations as affective priors to retrieve and refine System-II-inspired contextual representations via cross-attention. Third, we propose a Dynamic Cross-Modal Cognition Fusion (${\mathrm{DC}}^2$F) network that predicts sample-adaptive thresholds from modality discrepancy, agreement, and attention statistics, dynamically regulating directional image–text interaction to mitigate interference from conflicting cross-modal sentiment signals. Extensive experiments on public benchmarks demonstrate that CogSent achieves improved performance compared to existing methods and yields competitive results on multiple evaluation settings.

1. Introduction

With the rapid proliferation of multimodal content in online platforms, human affective expression has increasingly expanded beyond text to include images, audio, and videos. Such heterogeneous data not only enriches communication but also shapes social interactions, public opinion, and decision-making in the real world. As a result, understanding and modeling sentiment from multimodal signals has become a critical challenge in affective computing and multimedia analysis [1,2].

Current multimodal sentiment analysis methods [2,3,4] can be broadly categorized into three paradigms. Early studies rely on feature concatenation or tensor fusion [3] to model inter-modal correlations, but these approaches incur high computational cost and produce redundant representations, limiting their scalability and expressiveness. To overcome these issues, subsequent works leverage attention mechanisms and graph-based models [4] to selectively highlight sentiment-relevant signals across modalities, which enhances interpretability and alleviates noise interference. However, these methods still struggle with modality imbalance and fail to fully exploit the complementary nature of multimodal data. More recently, pretraining based frameworks and contrastive learning techniques have been introduced to learn robust cross-modal representations from large scale unlabeled or weakly labeled data, extracting global-local features [2]. These approaches improve generalization and alignment between image and text modalities, yet their effectiveness heavily depends on the availability of high-quality labeled or well-aligned datasets, which remain scarce in many real world scenarios. Despite these advances, existing methods face two critical limitations. (1) inadequate modeling of subjective emotional characteristics. Current approaches primarily extract objective semantic features from images and text using deep networks, implicitly treating them as proxies for human emotional responses. Such representations resemble the slow thinking process in psychology and fail to account for immediate perceptual cues (e.g., all-caps text indicating intense emotion, or image color/brightness influencing affective judgment, as the first and second examples in Figure 1). (2) insufficient handling of cross-modal inconsistency. Due to conflicting emotional polarities across modalities and the absence of strict image-text consistency constraints, existing models often propagate contradictory signals during cross-modal interaction, resulting in suboptimal semantic alignment. This misalignment hinders reliable affective judgments under inconsistent multimodal expressions (e.g., as the third example in Figure 1). Although text-dominant strategies have been widely adopted to mitigate this issue, the validity of the text dominance hypothesis has yet to be rigorously established.

Figure 1. Qualitative comparison of CogSent with single-modal baselines. For each case, the figure presents the image-text input together with the sentiment predictions of the three models, illustrating that CogSent produces more reliable affective judgments under cross-modal sentiment inconsistency.

To tackle the above-mentioned challenges, inspired by the functional differences between fast-thinking and slow-thinking modes from the psychological Dual-System Theory, we propose CogSent, a multimodal sentiment analysis framework. Rather than faithfully simulating the temporal, psychological, or neural mechanisms of human cognition, CogSent translates the functional propositions of this theory regarding cue-driven intuitive processing, context-dependent analytical processing, and their interactions into computable modules. Specifically, first, inspired by human fast thinking and slow thinking, we design the Hierarchical Dual-Channel Cognition (HDCC) architecture to extract System-I-inspired intuitive affective features and System-II-inspired rational affective features, respectively. The former captures directly perceivable, salient-cue-driven affective information, while the latter models context-dependent semantic and visual relationships. Second, we build the Intuition-Guided Cognition Refinement (IGCR) module. Taking System-I-inspired intuitive representations as affective priors, it retrieves and refines sentiment-relevant information from System-II-inspired contextual representations via cross-attention. Dynamic gating is further adopted to adaptively balance intuitive information and refined rational information, yielding robust unimodal affective representations. Third, we present the Dynamic Cross-Modal Cognition Fusion (${\mathrm{DC}}^2$F) network to address cross-modal sentiment inconsistency. ${\mathrm{DC}}^2$F predicts sample-adaptive thresholds based on modal discrepancies, consistency measures, and attention statistics. Asymmetric masks are employed to dynamically regulate directional information exchange between text and images, mitigating the interference of conflicting affective signals on fusion outputs.

The main contributions of this work are summarized as follows:

• We propose CogSent, a multimodal sentiment analysis framework inspired by the Dual-System Theory, which organizes cue-driven intuitive information and context-dependent rational information into complementary affective representations. This framework offers a functional computational operationalization of the Dual-System Theory instead of a direct simulation of human dual-system cognitive processes.
• We develop HDCC to separately model System-I-inspired intuitive cues and System-II- inspired contextual representations. Built upon HDCC, IGCR uses intuitive affective representations as priors to retrieve and refine rational-side evidence through cross-attention, followed by dynamic gating for adaptive balancing of the two types of representations.
• We put forward ${\mathrm{DC}}^2$F, which dynamically modulates directional cross-modal interaction by predicting sample-adaptive modal thresholds so as to alleviate the impact of conflicting image-text affective signals.
• Experiments conducted on three public benchmarks, including five-seed evaluation, controlled interaction comparisons, feature-group analysis, and consistency-stratified tests, verify the effectiveness and stability of CogSent and provide quantitative evidence for its key design choices.

The remainder of this paper is organized as follows. Section 2 reviews related studies on single-modal sentiment analysis, multimodal sentiment analysis, and sentiment analysis combined with psychological theories. Section 3 presents the proposed CogSent framework, including the Hierarchical Dual-Channel Cognition (HDCC), Intuition-Guided Cognition Refinement (IGCR), and Dynamic Cross-Modal Cognition Fusion (${\mathrm{DC}}^2$F) modules. Section 4 reports the experimental settings and results, including comparisons with baseline methods, ablation studies, and further analyses. Finally, Section 5 concludes the paper and discusses potential directions for future research.

2. Related Works

Research on sentiment analysis has progressively evolved from single-modal modeling toward multimodal and theory-informed approaches. To provide the background for CogSent, this section reviews three lines of related research.

2.1. Single-Modal Sentiment Analysis

Early studies on image sentiment analysis mainly relied on handcrafted visual features. For example, ref. [5] extracted color and texture cues motivated by art theory and psychological studies, while ref. [6] incorporated artistic principles into visual sentiment analysis.Although these methods demonstrated the relevance of low-level visual properties to affective perception, their reliance on manually designed features limited their ability to capture more complex patterns. Deep learning methods subsequently enabled more flexible visual representation learning. Ref. [7] focused on local facial regions, ref. [8] developed a dual-stream super-resolution CNN, and ref. [2] combined global self-attention with local multi-scale features to capture sentiment-related visual information at different levels. For text sentiment analysis, early approaches commonly relied on sentiment lexicons to assign polarity scores to words or phrases [9]. Such methods are relatively simple but can be sensitive to domain-specific vocabulary and linguistic variation. The development of pretrained language models has since led to stronger contextual representations, with models such as XL-Net [10] and BERT-based classifiers [11] being applied to sentiment-related tasks. However, stronger semantic representations do not necessarily provide a clearer account of why a particular sentiment is predicted. To incorporate more explicit affective information, ref. [12] integrated sentiment labels into the pretraining process, while ref. [13] introduced a two-level LSTM with enriched embeddings and polarity inversion to model linguistic and sentiment information jointly.

Despite the progress of single-modal approaches, sentiment is rarely expressed through a single source of information in real-world settings. Visual and textual signals can provide complementary emotional cues, and in some cases may even convey different sentiments. This motivates the development of multimodal sentiment analysis methods that can jointly model information from multiple modalities.

2.2. Multimodal Sentiment Analysis

Multimodal sentiment analysis combines information from different modalities, typically text and visual content, to obtain a more complete representation of affective information [14,15]. A central challenge in this setting is how to capture interactions between modalities without introducing unnecessary or conflicting information. Early work such as the Tensor Fusion Network (TFN) [3] explicitly modeled higher-order interactions between modalities, but its fusion strategy can introduce considerable computational cost and does not explicitly address long-range dependencies. Later approaches explored more selective forms of cross-modal interaction. Ref. [16] used gated cross-modal attention with pairwise learning to filter inconsistent information, while ref. [4] dynamically adjusted unaligned multimodal features to improve word representations. These methods improve the flexibility of cross-modal interaction, although misalignment between modalities remains a challenge.

Recent studies have further explored contrastive learning and pretrained multimodal models. Ref. [17] employed multilevel contrastive learning to connect global representations with fine-grained intermodal cues. Ref. [18] used CLIP together with a missing-modality inference module to generate virtual modalities when one modality is unavailable. Ref. [19] combined hierarchical disentanglement, representation enhancement, and dual-channel interaction to improve multimodal representation learning. Other studies have focused more directly on the problem of conflicting information. Ref. [20] introduced hybrid curriculum learning to reduce interference from latent conflicting signals, while ref. [21] proposed the Cross-Modal Joint Representation Transformer (CMJRT) to model both inter-modal consistency and complementarity. Another line of work places greater emphasis on textual information during multimodal fusion. Ref. [22] introduced the Cross-modal Attention-based Text-Centric Fusion Network (TeFNA), and ref. [23] used text-guided adaptive fusion to integrate multimodal information. Ref. [24] further considered the problem of textual over-reliance through the TFPN model. Although these approaches can be effective when textual information provides reliable sentiment cues, treating text as the dominant modality may not be appropriate for every multimodal sample. In particular, visual content may contain affective information that is absent from or inconsistent with the accompanying text. This motivates a fusion strategy that allows the relative contribution of each modality to vary according to the available evidence.

2.3. Sentiment Analysis Combined with Psychology

Psychological factors provide useful insights into how humans perceive and express emotions. Visual properties such as color, brightness, and saturation can influence affective responses, while linguistic patterns can also reflect emotional states. For example, ref. [25] examined the relationship between visual properties and emotional responses. Motivated by such findings, several studies have incorporated psychological theories into sentiment analysis. Ref. [26] adopted the Stimulus–Organism–Response (S-O-R) framework for emotion modeling, while ref. [27] incorporated personality traits into sentiment analysis. These studies demonstrate the value of psychological knowledge in sentiment modeling, but primarily focus on specific psychological factors or individual modalities. Related cognitive computing studies further explore how cognitive information can be represented and modeled from multimodal or neuroimaging data. Ref. [28] investigated multimodal feature interaction for cognitive state detection and introduced self-training for high-quality pseudo-label generation. Ref. [29] modeled brain functional organization using fMRI-based graphs, while 3D convolutional neural networks have also been applied to cognitive state decoding and neuroimaging analysis [30]. Although these studies target cognitive-state or brain-related tasks rather than sentiment analysis, they highlight the importance of explicitly modeling cognitive information and preserving structured representations. Recent multimodal affective computing research has also explored psychology-inspired and reasoning-oriented approaches. DEAN [31] incorporates an emotional arousal model into cross-modal interaction, while SSL-MEPR [32] explores semi-supervised multi-task learning for emotion and personality recognition. Emotion-LLaMA [33] further integrates visual, audio, and textual information within an instruction-tuned multimodal language model for emotion recognition and reasoning. These studies provide complementary perspectives on psychological modeling and multimodal affective understanding, but they do not explicitly model the interaction between intuitive and rational affective processes. Inspired by the Dual-System Theory [34], CogSent takes a different perspective by explicitly modeling fast, intuitive and slow, rational representations from both visual and textual modalities. It further allows intuitive information to guide rational feature refinement and dynamically regulates conflicting cross-modal sentiment cues through HDCC, IGCR, ${\mathrm{and DC}}^2$F.

3. CogSent

Psychological Dual-System Theory posits that human cognition operates through two interacting processes: a fast, intuitive system and a slow, rational system. Motivated by this, we propose a cognition-driven multimodal sentiment analysis framework (Figure 2), which models emotion generation as the interaction of fast thinking and slow thinking. In particular, we leverage the intuitive process to guide the rational process during feature integration, thereby achieving more authentic and reliable affective inference.

Figure 2. The overview of CogSent. After preprocessing, the image-text data is processed by the HDCC module (Section 3.1) to extract the Textual Intuitive Feature (TIF), Textual Rational Feature (TRF), Visual Intuitive Feature (VIF), and Visual Rational Feature (VRF). The IGCR module (Section 3.2) then enables intuitive cues to guide and refine rational analysis. Finally, the ${\mathrm{DC}}^2$F module (Section 3.3) generates fused multimodal cognitive representations for sentiment analysis. Note that the red font denotes personal pronoun in uppercase letters, and the underline marks words with relatively clear emotional attributes that appear in the NRC dictionary.

3.1. Hierarchical Dual-Channel Cognition (HDCC)

The design of HDCC is inspired by the psychological Dual-System Theory, which describes two broadly different modes of cognitive processing: relatively rapid and cue-driven perception, and more context-dependent and analytical processing [34]. This distinction provides a useful organizational perspective for affective computing. In social-media posts, emotionally salient words, emojis, emphatic punctuation, facial expressions, colors, and illumination can offer directly observable affective cues, whereas negation, discourse context, semantic dependencies, and global visual composition require deeper contextual modeling. Motivated by these complementary information-processing roles, we construct two parallel computational branches: a System-I-inspired fast-thinking branch that captures intuitive affective cues and a System-II-inspired slow-thinking branch that models deeper contextual information.

For each image-text pair, HDCC produces four representations: Textual Intuitive Feature (TIF), Textual Rational Feature (TRF), Visual Intuitive Feature (VIF), and Visual Rational Feature (VRF). TIF and VIF are derived from predefined affect-related indicators, whereas TRF and VRF are learned from contextual encoders. Table 1 lists the indicators used for TIF and VIF and their functions. These indicators are intended to capture specific affective cues associated with intuitive responses; they are not meant to represent intuition in its entirety.

Table 1. Functional taxonomy of the explicit affective cues used by HDCC.

Modality	Feature Group	Symbol	Components and Intended Role
Text	Learned explicit affect	${\mathbf{T}}_{\mathrm{aff}}$	Seven-category emotion probabilities from BERT-GoEmotions, providing an explicitly decoded affective distribution.
Text	Lexical and rule-based affect	${\mathbf{T}}_{\mathrm{lex}}$	NRC EmoLex category scores, NRC-EIL intensity scores, VADER polarity scores, and polarity/modifier statistics, representing lexical emotion, valence, and intensity.
Text	Expressive style	${\mathbf{T}}_{\mathrm{sty}}$	Emoji frequency, expressive punctuation, uppercase-character ratio, and personal-pronoun statistics; describing non-semantic emphasis, writing style, and discourse perspective of affective expression.
Image	Social-affective signal	${\mathbf{V}}_{\mathrm{face}}$	Seven-category facial-emotion probabilities from DeepFace, representing affective information conveyed by facial expressions.
Image	Chromatic and illumination stimulus	${\mathbf{V}}_{\mathrm{chr}}$	HSV statistics, dominant-color ratios, hue distribution, brightness, highlight/shadow ratios, illumination variation, saturation, warm–cold ratio, and color balance.
Image	Structural and texture stimulus	${\mathbf{V}}_{\mathrm{str}}$	Edge density, grayscale entropy, Laplacian variance, fractal characteristics, Gabor response, residual noise, and sharpness, describing local complexity and image structure.

3.1.1. System-I-Inspired Intuitive Representation

Textual affect is frequently signaled not only by semantic content but also by explicit emotion markers and expression style [35]. Following the order in Table 1, we extract three textual feature groups and three visual feature groups, yielding six groups in total.

The learned explicit-affect feature ${\mathbf{T}}_{\mathrm{aff}}$ is obtained from BERT-GoEmotions and contains the predicted probabilities of seven emotion categories. This feature converts the text into an explicit emotion distribution rather than using its contextual hidden states directly.

Lexical and rule-based affect, denoted by ${\mathbf{T}}_{\mathrm{lex}}$, combines NRC [36,37] and VADER information. Given a token sequence $\mathcal{T}=\{t_1,\dots ,t_L\}$ and an emotion category e, the contribution of token $t_i$ is adjusted by its lexicon association, negation, degree modification, and, for the intensity representation, its emotion-intensity value. The category and intensity scores are aggregated as

$$\begin{array}{cc}\hfill s_e^{\mathrm{cat}}\left(\mathcal{T}\right)& ={\displaystyle \sum _{i=1}^L}\mathbb{I}(t_i\in {\mathcal{L}}_e)r_i^{\mathrm{neg}}{r}_i^{\mathrm{deg}},\hfill \\ \hfill s_e^{\mathrm{int}}\left(\mathcal{T}\right)& ={\displaystyle \sum _{i=1}^L}\mathbb{I}(t_i\in {\mathcal{L}}_e)r_i^{\mathrm{neg}}{r}_i^{\mathrm{deg}}v(t_i,e),\hfill \end{array}$$

(1)

where ${\mathcal{L}}_e$ is the lexicon associated with emotion e, $v(t_i,e)$ is its emotion-intensity value, and $r_i^{\mathrm{neg}}$ and $r_i^{\mathrm{deg}}$ denote the effects of nearby negation and degree modifiers. The resulting NRC category and intensity vectors are concatenated with the positive, negative, neutral, and compound scores returned by VADER.

The textual style feature ${\mathbf{T}}_{\mathrm{sty}}$ captures non-semantic expressive cues. It integrates emoji frequency, the occurrence of exclamation and question marks, the proportion of uppercase characters, as well as personal-pronoun usage that characterizes the perspective from which emotion is expressed. These indicators describe emphasis and narrative perspective that may not be fully represented by lexical semantics.

The three textual groups are concatenated and projected to obtain TIF:

$$\mathbf{TIF}=\mathrm{ReLU}\left({\mathbf{W}}_t^I[{\mathbf{T}}_{\mathrm{aff}};{\mathbf{T}}_{\mathrm{lex}};{\mathbf{T}}_{\mathrm{sty}}]+{\mathbf{b}}_t^I\right).$$

(2)

Visual affect can be associated with facial expression, color and illumination, and structural appearance [5,6]. Accordingly, the visual explicit cues are organized into three groups.

The social-affective feature ${\mathbf{V}}_{\mathrm{face}}\in {\mathbb{R}}^7$ contains the facial-emotion probabilities predicted by DeepFace. The chromatic and illumination feature ${\mathbf{V}}_{\mathrm{chr}}$ combines the original color, brightness, and related auxiliary descriptors. Its color descriptors include HSV channel means; the ratios of red, orange, and blue pixels; channel standard deviations; hue skewness; and hue-histogram entropy. Given a hue histogram $h_H$, the entropy is

$$E_H=-{\displaystyle \sum _{k=1}^{n_H}}{p}_k{\log }_2{p}_k,p_k={\displaystyle \frac{h_H\left(k\right)}{{\sum }_{j=1}^{n_H}{h}_H\left(j\right)}}.$$

(3)

The illumination descriptors contain mean brightness, brightness deviation, highlight and shadow ratios, and the mean Sobel gradient magnitude

$$G={\displaystyle \frac{1}{N}}{\displaystyle \sum _{p=1}^N}\sqrt{{\left({\partial }_x{V}_p\right)}^2+{\left({\partial }_y{V}_p\right)}^2},$$

(4)

where V denotes the HSV value channel and N is the number of pixels. This group also includes mean saturation, warm–cold ratio, and red-green color balance.

The structural and texture feature ${\mathbf{V}}_{\mathrm{str}}$ contains Canny edge density, grayscale-histogram entropy, Laplacian variance, contour-based fractal characteristics, Gabor-response deviation, non-local-means residual noise, and sharpness. The fractal and sharpness descriptors are computed as

$$D_f={\displaystyle \frac{\log P}{\log A}},I_{\mathrm{sharp}}=I-G_{\sigma }\left(I\right),S_{\mathrm{sharp}}={\displaystyle \frac{1}{N}}{\displaystyle \sum _{p=1}^N}{\parallel \nabla I_{\mathrm{sharp},p}\parallel }_2,$$

(5)

where P and A are the perimeter and area of the foreground structure, respectively, and $G_{\sigma }$ denotes Gaussian smoothing. Canny thresholds are set to 50 and 150; the Gabor filter uses wavelength 10 and direction 0°; non-local means filtering uses strength 7; and sharpness is calculated with $\sigma =3$. These indicators describe local complexity, texture regularity, blur, and structural variation. The equations retain the nontrivial operations required for reproduction, whereas elementary counts and summary statistics are specified directly in Table 1. The three visual groups are concatenated and projected to obtain VIF:

$$\mathbf{VIF}=\mathrm{ReLU}\left({\mathbf{W}}_v^I[{\mathbf{V}}_{\mathrm{face}};{\mathbf{V}}_{\mathrm{chr}};{\mathbf{V}}_{\mathrm{str}}]+{\mathbf{b}}_v^I\right).$$

(6)

To preserve distinct properties of affective clues, each feature group is individually projected into the shared hidden space and stacked as independent tokens:

$$\begin{array}{cc}\hfill {\tilde{\mathbf{T}}}_g^I& =\mathrm{ReLU}\left({\mathbf{W}}_{t,g}^I{\mathbf{T}}_g+{\mathbf{b}}_{t,g}^I\right),g\in \{\mathrm{aff},\mathrm{lex},\mathrm{sty}\},\hfill \\ \hfill {\tilde{\mathbf{V}}}_g^I& =\mathrm{ReLU}\left({\mathbf{W}}_{i,g}^I{\mathbf{V}}_g+{\mathbf{b}}_{i,g}^I\right),g\in \{\mathrm{face},\mathrm{chr},\mathrm{str}\},\hfill \\ \hfill {\mathbf{F}}_t^I& ={\mathrm{Stack}}_{g\in \{\mathrm{aff},\mathrm{lex},\mathrm{sty}\}}\left({\tilde{\mathbf{T}}}_g^I\right),{\mathbf{F}}_i^I={\mathrm{Stack}}_{g\in \{\mathrm{face},\mathrm{chr},\mathrm{str}\}}\left({\tilde{\mathbf{V}}}_g^I\right),\hfill \end{array}$$

(7)

where ${\mathbf{W}}_{t,g}^I$,${\mathbf{W}}_{i,g}^I$,${\mathbf{b}}_{t,g}^I$,${\mathbf{b}}_{i,g}^I$ denote learnable projection weights and biases, mapping each raw feature group into shared dimension D.

3.1.2. System-II-Inspired Rational Representation

The System-II-inspired branch focuses on deep, context-dependent features. The overall semantic structure and contextual information of text play a decisive role in emotional expression. To obtain the Textual Rational Feature (TRF), we construct a RoBERTa-BiLSTM-Attention model (RoBERTa-BA). RoBERTa [38], as a pre-trained language model, provides rich contextual semantic representations. Building upon this, BiLSTM is used to model sequential dependencies, and the attention mechanism highlights emotionally relevant parts of the text. We adopt RoBERTa as the base encoder, leveraging its powerful semantic representation capabilities. Given an input text T, the encoding process is defined as

$$H_{\mathrm{roberta}}=\mathrm{RoBERTa}\left(T\right)\in {\mathbb{R}}^{m\times d_{roberta}},$$

(8)

where $H_{\mathrm{roberta}}$ is the RoBERTa encoding of T with m tokens and dimension $d_{roberta}$. To model sequential dependencies, we use a bidirectional LSTM (BiLSTM) on top of the RoBERTa encoder:

$$H=\mathrm{BiLSTM}\left(H_{\mathrm{roberta}}\right)\in {\mathbb{R}}^{n\times 2d_h},$$

(9)

where $d_h$ is the LSTM hidden size. To further emphasize emotionally salient components, we apply an Emotion Attention Mechanism (EAM) to the BiLSTM outputs, assigning a dynamic importance score to each hidden state, thereby enhancing the model’s sensitivity to emotionally informative regions in the text. Two representations are derived from EAM outputs. For the vector-form TRF, EAM produces weighted summation over token states to yield a single vector. Alternatively, EAM can re-weight token hidden states without pooling; a subsequent learnable linear projection maps the full token sequence to shared hidden dimension D to obtain ${\mathbf{F}}_t^{II}$.

In multimodal sentiment analysis, images contain rich visual emotional cues [39]. This study employs Vision Transformer (ViT) [40] to extract Visual Rational Feature (VRF). Its global modeling capability aligns with human deep visual cognition, enabling direct modeling of pixel relationships at arbitrary spatial distances via self-attention, capturing complex emotional cues. The ViT architecture treats an image as a sequence of visual tokens. Given an image $\mathbf{I}\in {\mathbb{R}}^{H\times W\times C}$ (where H, W, and C denote height, width, and channels, respectively), the preprocessed image is divided into fixed-size patches. Each patch is flattened and linearly projected into a D-dimensional embedding space. A special [CLS] token is prepended to the sequence, and learnable positional embeddings are added. The resulting sequence is passed through L layers of Transformer encoders, each comprising a multi-head self-attention module and feed-forward networks. After encoding, the [CLS] token aggregates contextual interactions with all image patches and is used as the global visual representation. Two outputs are obtained from ViT encoding. The [CLS] token serves as the vector-form VRF. Meanwhile, the complete ViT output token sequence (including patch tokens and [CLS]) is projected to dimension D to obtain ${\mathbf{F}}_i^{II}$.

3.2. Intuition-Guided Cognition Refinement (IGCR)

Simply using or concatenating intuitive and rational features cannot fully represent the dynamic evolution of emotions during human thought processes. This limitation significantly impacts the comprehensive understanding of complex emotional expressions.

Given this, we propose IGCR, whose core mechanism leverages intuitive affective cues as query priors to drive the refinement of rational cognitive representations. By leveraging attention mechanisms and dynamic gating mechanisms, it achieves complementary fusion and deep interaction of these dual representations, thereby constructing emotional features that are more aligned with human cognitive laws.

By treating the intuitive feature sequence ${\mathbf{F}}^I$ as query priors and the rational feature sequence ${\mathbf{F}}^{II}$ as keys and values, multi-head cross-attention retrieves rational-side evidence that is relevant to intuitive affective cues. We aggregate the cross-attention output via mean-pooling to obtain a global guidance vector $\mathit{g}$, which is broadcast-added to every token of the rational sequence:

$$\begin{array}{cc}\hfill {\mathbf{A}}_1& =\mathrm{MultiHeadAttn}(Q={\mathbf{F}}^I,K={\mathbf{F}}^{II},V={\mathbf{F}}^{II}),\hfill \\ \hfill \mathit{g}& =\mathrm{MeanPool}\left({\mathbf{A}}_1\right),\hfill \\ \hfill {\mathbf{C}}^{\prime }& ={\mathbf{F}}^{II}+\mathit{g}.\hfill \end{array}$$

(10)

this mechanism calculates semantic relevance between each intuitive token and all rational tokens. The derived global guidance signal injects intuitive perceptual priors into each position of the rational sequence, preserving rational contextual depth while strengthening semantic consistency with intuitive perception.

Based on the enhanced rational feature sequence ${\mathbf{C}}^{\prime }$, we further apply multi-head self-attention over the full sequence for progressive refinement, capturing deep intra-sequence dependencies:

$$\begin{array}{cc}\hfill {\mathbf{A}}_2& =\mathrm{MultiHeadAttn}(Q={\mathbf{C}}^{\prime },K={\mathbf{C}}^{\prime },V={\mathbf{C}}^{\prime }),\hfill \\ \hfill {\mathbf{C}}^{″}& =\mathrm{LayerNorm}({\mathbf{C}}^{\prime }+{\mathbf{A}}_2).\hfill \end{array}$$

(11)

this progressive refinement step captures deeper internal dependencies and enhances the robustness of rational emotional representations.

To balance and fuse the refined rational emotional features $C^{″}$ and the original representation, we introduce a dynamic gating mechanism. Before gating, we perform mean-pooling over both the refined rational sequence ${\mathbf{C}}^{″}$ and the original intuitive sequence ${\mathbf{F}}^I$. This condenses sequence-wise features into two global unimodal vectors ${\mathit{c}}_{\mathrm{global}}$ and ${\mathit{f}}_{\mathrm{global}}$ for subsequent vector-based gating computation:

$$\begin{array}{cc}\hfill G& =\sigma \left(W_g[{\mathit{c}}_{\mathrm{global}};{\mathit{f}}_{\mathrm{global}}]+{\mathit{b}}_g\right),\hfill \\ \hfill O& =G\odot {\mathit{c}}_{\mathrm{global}}+(1-G)\odot {\mathit{f}}_{\mathrm{global}},\hfill \end{array}$$

(12)

where $\sigma$ denotes the Sigmoid function, $W_g\in {\mathbb{R}}^{D\times 2D}$ and ${\mathit{b}}_g\in {\mathbb{R}}^D$ are learnable parameters, $[·;·]$ represents vector concatenation, and ⊙ denotes element-wise multiplication. The output $O\in {\mathbb{R}}^D$ serves as the aggregated unimodal representation ($T_E$ for text branch, $I_E$ for visual branch), which is fed into the subsequent $D{C}^{2}F$ module.

Finally, layer normalization is applied to the fused feature O to standardize the feature distribution. Through IGCR, we align sequence-level intuitive perceptual cues and contextual rational representations within each modality. It yields robust unimodal aggregated embeddings that synthesize perceptual prior and contextual reasoning, laying a solid foundation for subsequent cross-modal interaction.

3.3. Dynamic Cross-Modal Cognition Fusion (${\mathit{DC}}^2$F)

Different modalities may convey conflicting emotions. Although text is generally believed to capture sentiment more effectively, real-world social-media text can employ sarcasm and metaphor to conceal genuine emotions. Given that the dominant modality varies across samples, assuming universal text dominance a priori is not tenable. To address this issue, we present the sample-adaptive Dynamic Cross-Modal Cognition Fusion module. It predicts per-sample modality-specific thresholds, adaptively selects text-dominant, image-dominant, or balanced interaction modes for each attention head, and modulates cross-modal information exchange via differentiable soft masks.

Given IGCR output textual and visual affective representations ${\mathbf{T}}_E,{\mathbf{I}}_E\in {\mathbb{R}}^D$, we construct the bimodal sequence $\mathbf{X}=[{\mathbf{T}}_E;{\mathbf{I}}_E]\in {\mathbb{R}}^{2\times D}$. For the h-th attention head, unmasked attention is computed as:

$${\mathbf{S}}^h={\displaystyle \frac{{\mathbf{Q}}^h{\left({\mathbf{K}}^h\right)}^{\top }}{\sqrt{d_h}}},{\overline{\mathbf{A}}}^h=\mathrm{Softmax}\left({\mathbf{S}}^h\right),$$

(13)

where ${\mathbf{Q}}^h=\mathbf{X}{\mathbf{W}}_Q^h$, ${\mathbf{K}}^h=\mathbf{X}{\mathbf{W}}_K^h$, ${\mathbf{V}}^h=\mathbf{X}{\mathbf{W}}_V^h$. We define $a_h^{t\to i}={\overline{\mathbf{A}}}_{1,2}^h$ and $a_h^{i\to t}={\overline{\mathbf{A}}}_{2,1}^h$, denoting text-to-image and image-to-text directional attention weights, respectively.

Sample-wise thresholds are derived from modal features and initial attention statistics:

$$\begin{array}{cc}\hfill \mathbf{z}& =\mathrm{LayerNorm}\left([{\mathbf{T}}_E;{\mathbf{I}}_E;|{\mathbf{T}}_E-{\mathbf{I}}_E|;{\mathbf{T}}_E\odot {\mathbf{I}}_E;\mathbf{r}]\right),\hfill \\ \hfill {\delta }_m& =ϵ+(1-2ϵ)\sigma \left({\mathrm{MLP}}_m\left(\mathbf{z}\right)\right),m\in \{t,i\}.\hfill \end{array}$$

(14)

here, $|{\mathbf{T}}_E-{\mathbf{I}}_E|$ and ${\mathbf{T}}_E\odot {\mathbf{I}}_E$ encode cross-modal discrepancy and consistency. $\mathbf{r}$ aggregates per-head statistics: mean, standard deviation of directional scores, average directional difference, and mean attention entropy. Thresholds ${\delta }_t,{\delta }_i$ are shared across heads for one sample but differ between samples; $ϵ$ avoids threshold values of exactly 0 or 1.

For each attention head, three-state probabilities are calculated:

$$\begin{array}{cccc}\hfill g_{t,h}& ={\displaystyle \frac{a_h^{i\to t}-{\delta }_t}{\tau }},\hfill & \hfill g_{i,h}& ={\displaystyle \frac{a_h^{t\to i}-{\delta }_i}{\tau }},\hfill \\ \hfill g_{b,h}& =-\max (g_{t,h},g_{i,h}),\hfill & \hfill \left[\begin{array}{c}{p}_{t,h}\\ p_{i,h}\\ p_{b,h}\end{array}\right]& =\mathrm{Softmax}\left(\left[\begin{array}{c}{g}_{t,h}\\ g_{i,h}\\ g_{b,h}\end{array}\right]\right).\hfill \end{array}$$

(15)

where $\tau$ is temperature. $p_{t,h},p_{i,h},p_{b,h}$ correspond to probabilities of text-dominant, image-dominant, balanced states. A large $a_h^{i\to t}$ implies the image query depends heavily on text evidence; symmetric logic holds for image-dominance. When both directional scores are below thresholds, $g_{b,h}$ becomes positive and the model favors bidirectional interaction.

The asymmetric soft mask is applied to attention logits:

$${\tilde{\mathbf{M}}}^h=\left[\begin{array}{cc}0& p_{t,h}\\ p_{i,h}& 0\end{array}\right],{\mathbf{A}}^h=\mathrm{Softmax}\left({\mathbf{S}}^h+\log \left[\mathrm{clamp}\left(\mathbf{1}-{\tilde{\mathbf{M}}}^h,ϵ,1\right)\right]\right).$$

(16)

larger entries in ${\tilde{\mathbf{M}}}^h$ produce stronger suppression for corresponding query-to-key connections. Diagonal entries remain zero such that intra-modal self-attention is never suppressed.

During training, continuous soft-state probabilities yield fully differentiable computation for end-to-end optimization of threshold predictors via classification loss. To narrow the train-inference gap between soft training masks and discrete inference masks, we adopt exponential temperature annealing:

$${\tau }_e=\max \left({\tau }_{\min },{\tau }_0·{\gamma }^e\right),$$

with hyper-parameters ${\tau }_0=1.0$, ${\tau }_{\min }=0.3$, $\gamma =0.8$, and e denoting training epoch.

At inference, we apply $\arg \max$ per head to select the highest-probability state and generate binary hard masks $\{0,1\}$: text-dominant/image-dominant suppress the respective cross-modal links, while the balanced state uses an all zero mask.

Each head yields ${\mathbf{O}}^h={\mathbf{A}}^h{\mathbf{V}}^h$. After head concatenation, output projection, residual connection, layer normalization and feed-forward layers, mean-pooling over two modality tokens produces fused representation $\mathbf{F}$. Finally, $\mathbf{F}$ is concatenated with TRF and VRF for sentiment prediction.

4. Experiments

4.1. Implementation Details

Datasets. For evaluation, we used three common image-text multimodal sentiment analysis datasets: MVSA-Multiple, MVSA-Single [41], and TumEMO [42]. The TumEMO dataset covers seven emotions: anger, boredom, calm, fear, happiness, love, and sadness. MVSA-Single and MVSA-Multiple datasets contain positive, neutral, and negative sentiments. All datasets are from social media platforms and contain rich emotional information. The datasets are randomly split into training, validation, and test sets in an 8:1:1 ratio. Detailed statistics of the datasets are summarized in Table 2.

Table 2. Statistics for dataset. Details of the datasets used for evaluation, including their label distributions and data splits.

Dataset	Label	Sample	Train	Valid	Test
MVSA-Single	positive	2683	2147	268	268
	neutral	470	376	47	47
	negative	1358	1088	135	135
MVSA-Multiple	positive	11,318	9056	1131	1131
	neutral	4408	3528	440	440
	negative	1298	1040	129	129
TumEMO	Angry	14,553	11,643	1455	1455
	Bored	32,283	25,827	3228	3228
	Calm	18,109	14,487	1811	1811
	Fear	20,264	16,212	2026	2026
	Happy	50,267	40,213	5027	5027
	Love	34,511	27,609	3451	3451
	Sad	25,277	20,221	2528	2528

Experimental Settings. The configuration of the simulation environment used in this study is summarized in Table 3. All experiments are conducted on a server with Ubuntu 20.04 operating system, equipped with an Intel Xeon Platinum 8352V CPU (2.10 GHz), 64 GB RAM, and an NVIDIA RTX 4090 GPU (24 GB VRAM) using CUDA 11.8. All experiments are implemented based on the PyTorch 2.0 framework. In the data preprocessing phase, all text inputs are padded or truncated to a maximum length of 128 tokens, and all image samples are resized to a resolution of $224\times 224$ pixels. All model selections and hyperparameter tuning strategies are strictly conducted on the validation set to avoid overfitting. To quantitatively evaluate the stability and reduce the randomness of model training, we train CogSent and all baseline models with five fixed random seeds: [42, 123, 3407, 766, 2026]. The detailed parameter settings used in this study are presented in Table 3.

Table 3. Hyperparameter configuration. Settings used for MVSA and TumEMO.

Hyperparameter	MVSA	TumEMO
Number of classes	3	7
Epochs	20	15
Optimizer	AdamW	AdamW
Learning rate	$1\times {10}^{-5}$	$1\times {10}^{-5}$
Batch size	64	64
Random seeds	42, 123, 3407, 766, 2026
Initial temperature ${\tau }_0$	1.0	1.0
Minimum temperature ${\tau }_{\min }$	0.3	0.3
Annealing factor $\gamma$	0.8	0.8

4.2. Baselines

To validate the effectiveness of our approach, we conduct comparisons with unimodal sentiment models as well as state-of-the-art multimodal baselines.

4.2.1. Unimodal Sentiment Baselines

Text Modality: For text feature extraction, we employ two widely adopted pre-trained language models: BERT [43] and RoBERTa [38]. These models are renowned for their powerful natural language understanding capabilities and have demonstrated superior performance across a variety of downstream text classification tasks.

Image Modality: For the image modality, we integrate two mainstream deep learning architectures. First, we adopt classical convolutional neural networks (CNNs), namely VGG [44] and ResNet [45], which are capable of effectively capturing visual features after being pre-trained on large-scale image recognition tasks. In addition, we incorporate the Vision Transformer (ViT) [40], a Transformer-based architecture that has attracted considerable attention due to its superior performance in image classification and its ability to model long-range dependencies.

4.2.2. Multimodal Sentiment Baselines

MLFC [46]: Proposes a multimodal sentiment analysis framework based on supervised contrastive learning. It integrates CNN and Transformer architectures to reduce redundant features and employs contrastive learning to enhance prototypical emotion feature extraction.

MDSE [47]: Improves robustness by fusing unimodal private and shared features, and designs a modality-agnostic contrastive loss to unify cross-modal representations and bridge modality gaps.

CIGNN [48]: Constructs image-text co-occurrence graphs to capture global associations, employs Graph Neural Networks for representation learning, and utilizes multi-head attention for deep multimodal fusion.

SentiCLIP [49]: Leverages vision language pretraining to extract unified semantic features, with a core interaction module that explicitly and implicitly captures image-text correlations to significantly enhance sentiment analysis performance.

S-MA [50]: Incorporates a Shor-Term Memory Attention mechanism to capture intra and inter modal interactions, combined with self-distillation strategy to optimize representation learning, demonstrating superior performance across multiple benchmarks.

4.3. Main Results

This section evaluates the experimental performance on selected datasets and includes both classical unimodal methods and advanced multimodal methods. All baselines are run using the same data partitioning, preprocessing process, and five random seeds as CogSent.

Comparison with single-modal baselines. As reported in Table 4, multimodal approaches consistently outperform their single-modal counterparts. Image-only models yield the lowest performance due to sparse and indirect emotional cues, whereas text-only models perform better given their richer semantic information. Notably, our RoBERTa-BA module achieves the best results among text-based models, providing a strong backbone for the overall CogSent framework.

Table 4. Main results. We compare our CogSent with previous state-of-the-art methods on MVSA-Single, MVSA-Multiple, and TumEMO datasets.

Model	MVSA-Single		MVSA-Multiple		TumEMO
	Acc	F1	Acc	F1	Acc	F1
Image Sentiment Analysis
ResNet [45]	$64.61\pm 0.13$	$61.52\pm 0.31$	$62.13\pm 0.29$	$61.38\pm 0.51$	$45.54\pm 0.25$	$32.82\pm 0.57$
VGG [44]	$65.12\pm 0.94$	$64.02\pm 0.44$	$63.70\pm 0.67$	$61.91\pm 0.27$	$46.41\pm 0.29$	$34.14\pm 0.42$
ViT [40]	$67.53\pm 0.26$	$65.70\pm 0.19$	$66.25\pm 0.32$	$63.57\pm 0.40$	$48.34\pm 0.19$	$35.92\pm 0.46$
Text Sentiment Analysis
BERT [43]	$69.49\pm 0.13$	$67.43\pm 0.17$	$68.39\pm 0.36$	$68.98\pm 0.19$	$59.20\pm 0.75$	$48.71\pm 0.73$
RoBERTa [38]	$69.59\pm 0.39$	$68.34\pm 0.25$	$69.93\pm 0.32$	$69.29\pm 0.34$	$59.87\pm 0.84$	$51.29\pm 1.03$
RoBERTa-BA	$70.23\pm 0.16$	$69.63\pm 0.61$	$70.02\pm 0.60$	$69.47\pm 0.37$	$60.55\pm 1.18$	$52.22\pm 0.68$
Image-Text Sentiment Analysis
MLFC [46]	$75.42\pm 0.17$	$72.31\pm 0.37$	$70.74\pm 0.36$	$68.23\pm 0.45$	$65.75\pm 0.62$	$63.17\pm 0.31$
MDSE [47]	$76.27\pm 0.19$	$75.45\pm 0.34$	$72.56\pm 0.34$	$70.19\pm 0.11$	$67.15\pm 0.16$	$66.47\pm 0.40$
CIGNN [48]	$74.99\pm 0.10$	$73.76\pm 0.51$	$73.40\pm 0.16$	$70.91\pm 0.05$	$67.27\pm 0.24$	$66.73\pm 0.52$
SentiCLIP [49]	$75.87\pm 0.20$	$75.40\pm 0.19$	$73.31\pm 0.12$	$70.50\pm 0.18$	$67.44\pm 0.31$	$67.15\pm 0.18$
S-MA [50]	$77.66\pm 0.14$	$76.67\pm 0.31$	$72.55\pm 0.09$	$70.52\pm 0.23$	$67.77\pm 0.34$	$67.57\pm 0.08$
CogSent	$\mathbf{77}.\mathbf{95}\pm \mathbf{0}.\mathbf{13}$	$\mathbf{77}.\mathbf{38}\pm \mathbf{0}.\mathbf{28}$	$\mathbf{75}.\mathbf{40}\pm \mathbf{0}.\mathbf{17}$	$\mathbf{72}.\mathbf{47}\pm \mathbf{0}.\mathbf{17}$	$\mathbf{68}.\mathbf{49}\pm \mathbf{0}.\mathbf{26}$	$\mathbf{68}.\mathbf{04}\pm \mathbf{0}.\mathbf{21}$

Note: Bold values indicate the best performance.

Comparison with multimodal baselines. Among the comparative models, MLFC [46] optimizes multimodal analysis through contrastive learning but focuses only on high-level fusion. MDSE [47] fuses single-modal features and reduces the modality gap but neglects cross-modal interaction. CIGNN [48] leverages co-occurrence graphs and graph neural networks to fuse features but insufficiently mines potential information. SentiCLIP [49] captures associations through visual-language pre-training but does not achieve deep fusion. S-MA [50] optimizes representation learning but lacks cross-modal integration. Inspired by the Dual-System Theory, our model extracts intuitive and rational emotional features from modalities. The IGCR module models the interaction between fast and slow thinking, while ${\mathrm{DC}}^2$F adaptively adjusts the contribution of each modality for individual samples. CogSent achieves the highest mean Accuracy and macro-F1 across all three datasets. On MVSA-Single, CogSent improves Accuracy and macro-F1 by 0.29 and 0.71 percentage points, respectively, over the strongest competitor, S-MA. On MVSA-Multiple, CogSent outperforms CIGNN, the best-performing baseline on both metrics, by 2.00 percentage points in Accuracy and 1.56 percentage points in macro-F1. On TumEMO, CogSent achieves gains of 0.72 and 0.47 percentage points in Accuracy and macro-F1, respectively, over S-MA. The standard deviations of the six metrics range from 0.13 to 0.28 across five runs, indicating relatively small variation between runs. These results show that CogSent maintains consistent performance across the three datasets. The results of the t-test are presented in Appendix A.1.

4.4. Ablation Study

As shown in Table 5, different modules are ablated to evaluate their individual impacts, considering the following ablated variants: (1) w/o ${\mathrm{DC}}^2$Fablates the ${\mathrm{DC}}^2$F module and replaces it with simple feature concatenation for multi-modal fusion; (2) w/o IGCR_T removes the IGCR in the text modality, replacing the interaction of dual-system emotional features with concatenation and projection; (3) w/o HDCC_TI and w/o HDCC_TII, respectively, eliminate the acquisition of text’s intuitive (System I) and rational (System II) emotional features, thus disabling the intra-modal dual-system feature interaction; (4) w/o IGCR_V ablates the IGCR in the visual modality, with dual-system feature interaction replaced by concatenation and projection; (5) w/o HDCC_VI and w/o HDCC_VII remove the acquisition of visual intuitive and rational emotional features, disabling intra-modal dual-system interaction; (6) w/o Vision retains the text modality’s HDCC and IGCR while excluding visual input, and w/o Text retains the visual modality’s components while excluding text.

Table 5. Ablation results. We validate the impact of each component of CogSent on the MVSA-Single, MVSA-Multiple, and TumEMO datasets. Values are reported as mean ± standard deviation.

Model	MVSA-Single		MVSA-Multiple		TumEMO
	Acc	F1	Acc	F1	Acc	F1
CogSent	$\mathbf{77}.\mathbf{95}\pm \mathbf{0}.\mathbf{13}$	$\mathbf{77}.\mathbf{38}\pm \mathbf{0}.\mathbf{28}$	$\mathbf{75}.\mathbf{40}\pm \mathbf{0}.\mathbf{17}$	$\mathbf{72}.\mathbf{47}\pm \mathbf{0}.\mathbf{17}$	$\mathbf{68}.\mathbf{49}\pm \mathbf{0}.\mathbf{26}$	$\mathbf{68}.\mathbf{04}\pm \mathbf{0}.\mathbf{21}$
w/o DC2F	$77.23\pm 0.21$	$76.32\pm 0.19$	$74.87\pm 0.23$	$71.69\pm 0.26$	$67.58\pm 0.27$	$66.97\pm 0.32$
w/o IGCR_T	$76.58\pm 0.16$	$75.98\pm 0.27$	$74.64\pm 0.16$	$71.25\pm 0.18$	$66.72\pm 0.15$	$65.98\pm 0.28$
w/o HDCC_VI	$75.33\pm 0.09$	$75.12\pm 0.12$	$74.25\pm 0.24$	$70.87\pm 0.22$	$65.35\pm 0.23$	$65.13\pm 0.23$
w/o HDCC_TI	$74.64\pm 0.18$	$74.27\pm 0.17$	$73.47\pm 0.11$	$70.15\pm 0.19$	$64.51\pm 0.22$	$64.25\pm 0.19$
w/o HDCC_VII	$74.61\pm 0.15$	$74.15\pm 0.11$	$73.43\pm 0.18$	$69.36\pm 0.24$	$64.14\pm 0.22$	$63.49\pm 0.24$
w/o IGCR_V	$73.08\pm 0.23$	$72.92\pm 0.14$	$73.25\pm 0.16$	$69.28\pm 0.15$	$62.95\pm 0.21$	$62.98\pm 0.20$
w/o HDCC_TII	$69.86\pm 0.17$	$68.19\pm 0.16$	$69.29\pm 0.12$	$66.57\pm 0.18$	$58.78\pm 0.19$	$52.94\pm 0.23$
w/o Vision	$72.08\pm 0.13$	$71.69\pm 0.23$	$71.42\pm 0.17$	$69.11\pm 0.09$	$62.07\pm 0.16$	$61.74\pm 0.22$
w/o Text	$68.93\pm 0.24$	$66.58\pm 0.18$	$68.22\pm 0.14$	$64.28\pm 0.21$	$51.92\pm 0.24$	$46.59\pm 0.26$

Note: Bold values indicate the best performance.

Effect of HDCC. This module extracts intuitive (System I) and rational (System II) cognitive features from each modality. As shown in Table 5, removing the visual cognitive features causes a ∼2% performance drop, smaller than that from removing textual features, indicating the stronger discriminative power of the textual cognition. Furthermore, at the intuitive and rational feature levels, the performance loss caused by removing rational features (System II) in both modalities is greater than that from removing intuitive (System I) features. This further confirms that rational features possess superior analytical capabilities in complex emotion analysis, and also suggests that the text modality’s intuitive features are the most discriminative. However, in the overall HDCC ablation, we cannot determine whether each explicit feature set contributes. Therefore, we deleted the six sets of displayed emotional representations described in module 3.1 group by group, and the results are shown in the Table 6.

Table 6. Contribution of the six HDCC explicit feature groups. Each variant removes one complete functional group. $\Delta$Acc and $\Delta$F1 denote the changes in mean Accuracy and macro-F1 relative to the complete model. Negative values indicate performance degradation after removing the corresponding group.

Variant	Dim	MVSA-Single		MVSA-Multiple		TumEMO
		ΔAcc	ΔF1	ΔAcc	ΔF1	ΔAcc	ΔF1
w/o learned affect	7	$-0.91$	$-0.93$	$-0.58$	$-0.77$	$-0.95$	$-0.89$
w/o lexical affect	20	$-0.96$	$-0.95$	$-0.62$	$-0.76$	$-0.97$	$-0.94$
w/o textual style	5	$-0.81$	$-0.82$	$-0.48$	$-0.64$	$-0.84$	$-0.88$
w/o facial affect	7	$-0.89$	$-0.88$	$-0.50$	$-0.78$	$-0.95$	$-0.98$
w/o chromatic/illumination	16	$-0.86$	$-0.86$	$-0.45$	$-0.69$	$-0.96$	$-0.94$
w/o structural/texture	10	$-0.85$	$-0.72$	$-0.58$	$-0.76$	$-0.92$	$-0.97$

Ablation results demonstrate that dropping any of the six System-I feature groups reduces both Accuracy and macro-F1 across MVSA-Single, MVSA-Multiple and TumEMO, confirming that every group supplies complementary affective evidence.To assess redundant information among explicit descriptors, we compute group-wise Spearman correlations solely on the training set.

To evaluate potential redundancy across HDCC explicit descriptors, we compute group-wise Spearman correlations using only training-set samples. Figure 3 visualizes mean-absolute correlation coefficients between dimensions of different feature groups. Diagonal cells report average within-group associations, where larger values reflect stronger statistical overlap.

Figure 3. Spearman correlations between HDCC explicit feature groups. Heatmap of mean-absolute group-level coefficients computed on the training split. Diagonal entries denote within-group feature association.

The highest cross-group correlation appears between learned affect and lexical affect, as both capture textual emotional signals via different paradigms. Lexical affect and textual-style features, as well as learned affect and facial affect, also show notable association. Nevertheless, these magnitudes indicate moderate overlap rather than full redundancy. Most other cross-group coefficients remain low, with textual-style and structural/texture yielding the weakest linkage due to their disparate modalities and functions.

These correlation results should be considered together with the group-ablation results. Although the learned-affect and lexical-affect groups are relatively highly correlated, removing either group still reduces performance across all datasets. This suggests that they capture some overlapping but also distinct affective information. The same observation applies to the two modalities. The textual and visual groups have relatively low correlations, while removing the visual features consistently leads to lower performance. This indicates that visual features provide additional affective information that is not fully captured by the textual features. Overall, the feature groups contain some redundancy, but each group still contributes to the final prediction.

Effect of IGCR. IGCR integrates intuitive and rational features, with intuition-derived affective priors driving the overall cognition-refinement process. Compared to the full model, the w/o IGCR_T and w/o IGCR_V variants show performance drops in both modalities, with a larger loss when removing visual fusion (IGCR_V), reaching up to 4.09%. Without IGCR, simply concatenating dual-system features introduces noise and weakens cross-system synergy. For example, in the image modality, removing IGCR disrupts the guidance from intuitive to rational features, preventing System I from effectively supporting complex reasoning in System II, leading to even worse performance than removing HDCC_VI alone.

To further examine the directional design and refinement mechanism of IGCR, we construct three variants while keeping the other components unchanged. Bidirectional IGCR adds a rational-prior-driven reverse path to the original intuition-prior-driven process. Reverse IGCR retains only this reverse path. Concat-Projection removes attention-based refinement and dynamic gating, directly concatenating and projecting the two feature types.

As shown in Table 7, CogSent achieves the best performance across all six metrics. Bidirectional IGCR reduces performance by 0.37–0.76 percentage points, while Reverse IGCR results in larger drops of 1.13–3.70 points. These results suggest that the refinement process benefits from using intuitive affective priors as the guiding signal rather than introducing a reverse path or using rational priors for guidance. Concat-Projection performs worst, with decreases of 1.84–5.61 points, indicating that simple feature aggregation is less effective than the attention-based refinement and adaptive fusion used in IGCR. Overall, the results support the asymmetric design of IGCR, where intuitive affective priors guide the refinement of rational features while the two systems retain complementary roles.

Table 7. Effect of different interaction directions and fusion strategies in IGCR. Values are reported as mean ± standard deviation.

Variant	MVSA-Single		MVSA-Multiple		TumEMO
	Acc	Macro-F1	Acc	Macro-F1	Acc	Macro-F1
CogSent	77.95 ± 0.13	77.38 ± 0.28	75.40 ± 0.17	72.47 ± 0.17	68.49 ± 0.26	68.04 ± 0.21
Bidirectional IGCR	77.19 ± 0.35	76.68 ± 0.34	75.02 ± 0.31	72.10 ± 0.26	68.05 ± 0.29	67.37 ± 0.26
Reverse IGCR	75.21 ± 0.35	73.68 ± 0.36	74.12 ± 0.27	71.34 ± 0.29	66.87 ± 0.22	66.07 ± 0.36
Concat-Projection	72.34 ± 0.41	72.12 ± 0.33	73.56 ± 0.38	69.21 ± 0.37	63.45 ± 0.32	63.31 ± 0.39

Note: Bold values indicate the best performance.

Effect of ${\mathbf{DC}}^2$F. Prior studies often assumed absolute text dominance to resolve modal polarity conflict, but we argue this dominance is not absolute. To validate this, we removed the original masking in multi-head attention and visualized weight matrices of different heads using the MVSA-Multiple dataset. As shown in Figure 4, most heads exhibit asymmetry with text attending to itself, but some heads show text focusing on images while images mainly self-attend.

Figure 4. Visualization of attention weight matrices. We present attention maps from the multi-head attention module heads. Results show most heads are text-dominant, but some still prioritize images. Based on this, we propose ${\mathrm{D}}^2$CF to dynamically shift text–image focus.

We compared three settings while keeping other module parameters constant: (i) text only dominance (using only fixed masks), (ii) strict modal fairness (no mask mechanism), (iii) asymmetric cross modal interaction under fixed thresholds, and (iv) asymmetric cross modal interaction under sample adaptive thresholds.

As shown in Figure 5, the strict modality-fair baseline obtains the lowest results, since it enforces balanced cross-modal interaction without considering intrinsic text-bias characteristics. The text-only dominance setting brings modest gains by amplifying text priority. Further, the fixed-threshold asymmetric cross-modal interaction continues to boost performance, demonstrating that asymmetric text-image dependency is beneficial for multimodal sentiment understanding. Our sample-adaptive threshold method achieves the highest Accuracy and Macro-F1 on both datasets. This verifies that sample-wise adaptive constraints can better calibrate cross-modal information flow than static fixed masks or rigid modality-fair constraints, alleviating excessive text dominance and yielding consistent performance improvements.

Figure 5. Performance comparison of modality fusion strategies. We present bar charts comparing the Accuracy and $F1$-Score of various modality fusion strategies across MVSA-Multiple and TumEMO datasets. The results visually demonstrate that our proposed “Asymmetric modality consideration under sample adaptive thresholds” consistently achieves superior performance in sentiment analysis.

To verify whether the sample-adaptive ${\mathrm{DC}}^2$F yields greater performance gains under cross-modal sentiment conflicts, we split the MVSA-Single dataset into two subsets according to whether the text and image sentiment polarities are consistent or inconsistent. This partitioning is only applied to the held-out test set, while keeping the original training-validation-test splits used in the main experiments unchanged. The sample statistics and evaluation metrics for the resulting subsets are presented in Table 8.

Table 8. Consistency-stratified results on cleaned MVSA-Single. N denotes the number of test samples in each subset; the remaining columns report Accuracy and F1 of CogSent and its w/o ${\mathrm{DC}}^2$F variant. Results are mean ± standard deviation over five matched seeds.

Subset	N	CogSent		w/o ${\mathrm{DC}}^2$F
		Acc	Macro-F1	Acc	Macro-F1
inconsistent	190	$71.73\pm 0.36$	$76.67\pm 0.32$	$70.20\pm 0.46$	$74.66\pm 0.38$
consistent	260	$82.31\pm 0.25$	$78.49\pm 0.24$	$81.92\pm 0.31$	$78.12\pm 0.36$

As shown in Table 8, ${\mathrm{DC}}^2$F improves Accuracy/Macro-F1 by $1.53/2.01$ points on label-discrepant pairs, compared with $0.39/0.37$ points on label-consistent pairs. Thus, its larger observed gain occurs when the two modalities carry neutral–polar discrepant labels, supporting the intended sample-adaptive fusion role within the scope of the cleaned benchmark.

4.5. Case Study

To further validate the model’s effectiveness, we conducted case studies comparing the performance of the complete CogSent model against variants without HDCC_I/II or DC²F. As shown in Table 9, when image-text pairs involve complex emotional cues or exhibit sentiment polarity inconsistency across modalities, accurately identifying users’ sentiment tendencies becomes particularly challenging.

Table 9. Case study. Evaluate the effectiveness of Cogsent using specific cases on the MVSA dataset, and conduct a comparative analysis with models that remove fast thinking and ${\mathrm{DC}}^2$F model.

Image	Text	Ground Truth	w/o HDCC_I/II	w/o ${\mathrm{DC}}^2$F	Ours
	#maharashtra is the worst state to be a #farmer. grim details of farmers’ suicides in india	Negative	Negative	Negative	Negative
	peter recovered the mayan lenses that colonel soyer had snatched during the ruined wedding ceremony.	Positive	Neutral	Positive	Positive
	#empty looks good enough #p#ew #wrrr	Positive	Neutral	Negative	Positive
	#sunsetlover #moneytrees vegan #ebooks -weight- loss program, and recipe books: vibrant	Neutral	Positive	Positive	Positive

Note: Bold values indicate the ground truth and the prediction results of CogSent.

Effectiveness of System-I-Inspired Intuitive Representation in Emotion Perception. Taking the second data sample in Table 9 as an example, the image presents an overall bright tone, strong vertical contrast, and a substantial proportion of red regions at the perceptual level, which may activate fast thinking and facilitate the extraction of salient emotional cues. In contrast, the corresponding text does not convey an explicit emotional tendency. Without incorporating the fast thinking mechanism, this sample is misclassified as neutral sentiment. It proves that the interaction between System-I-Inspired Representation and System-II-Inspired Representation contributes to a more comprehensive integration of emotional information from image-text pairs.

Handling Cross-Modal Sentiment Polarity Inconsistency. The third sample in Table 9 illustrates the model’s ability to address sentiment polarity inconsistencies between modalities. Specifically, when the image conveys negative emotion while the text expresses positive sentiment, removing the ${\mathrm{DC}}^2$F module results in misclassification, whereas the complete model produces the correct prediction. This comparison highlights the role of ${\mathrm{DC}}^2$F in reconciling cross-modal emotional conflicts.

Limitations in Ambiguous Emotional Content. In the fourth sample of Table 9, the model fails to correctly classify the emotion, indicating that challenges remain when handling ambiguous or weakly expressed emotional content. This case reveals potential limitations in capturing subtle or context-dependent affective signals.

Overall Case Study Analysis. Overall, the case study suggests that the proposed model more effectively captures complementary emotional cues across modalities and produces more reliable sentiment predictions, particularly in scenarios involving cross-modal polarity conflicts. These findings provide qualitative evidence supporting the design of the proposed cognitive-inspired mechanisms.

5. Conclusions

We present CogSent, a cognition-inspired multimodal sentiment analysis framework that integrates System-I-Inspired Representation and System-II-Inspired Representation. Visual cues frequently feed into System-I-Inspired Representations for perceptual affective judgment, whereas textual semantics provide contextual evidence for System-II-Inspired Representations. Through HDCC and IGCR, System-I-Inspired Representations guide contextual reasoning to refine affective features, and System-II-Inspired contextual analysis in turn calibrates System-I-Inspired outputs to mitigate perceptual bias. The ${\mathrm{DC}}^2$F module further harmonizes information exchange across modalities, alleviating inconsistencies. This synergistic interaction among modalities, fast–slow cognition, and intuitive–rational processing yields more authentic and reliable sentiment understanding, advancing cognitively inspired affective computing. For future work, we plan to extend CogSent to broader modalities such as audio and video and to explore its applicability in real-world, large-scale affective interaction scenarios.

Appendix A.1. Statistical Significance Tests

To examine whether the improvements of CogSent are consistent across different training runs, we conduct two-sided paired t-tests using the results from five matched random seeds: 42, 123, 3407, 766, and 2026. For each dataset and metric, CogSent is compared with the multimodal baseline that achieves the highest mean score. S-MA is used for MVSA-Single and TumEMO, while CIGNN is used for MVSA-Multiple. The null hypothesis is that the mean seed-wise performance difference between CogSent and the corresponding baseline is zero. We use $p<0.05$ as the significance criterion.

As shown in Table A1, all six comparisons yield positive mean differences and $p<0.05$. The results indicate that CogSent consistently outperforms the strongest multimodal baseline across the five matched runs on Accuracy and macro-F1 for all three datasets.

Table A1. Paired t-test results for CogSent and the strongest multimodal baseline. $\Delta$ denotes the mean performance difference, calculated as CogSent minus the comparator in percentage points.

Dataset	Metric	Comparator	Δ	$\mathit{t}\left(4\right)$	p
MVSA-Single	Accuracy	S-MA	0.296	3.770	0.0196
MVSA-Single	Macro-F1	S-MA	0.708	4.000	0.0161
MVSA-Multiple	Accuracy	CIGNN	1.998	22.749	<0.0001
MVSA-Multiple	Macro-F1	CIGNN	1.558	18.459	<0.0001
TumEMO	Accuracy	S-MA	0.716	4.031	0.0157
TumEMO	Macro-F1	S-MA	0.470	4.273	0.0129

Table A1. Paired t-test results for CogSent and the strongest multimodal baseline. $\Delta$ denotes the mean performance difference, calculated as CogSent minus the comparator in percentage points.

Dataset	Metric	Comparator	Δ	$\mathit{t}\left(4\right)$	p
MVSA-Single	Accuracy	S-MA	0.296	3.770	0.0196
MVSA-Single	Macro-F1	S-MA	0.708	4.000	0.0161
MVSA-Multiple	Accuracy	CIGNN	1.998	22.749	<0.0001
MVSA-Multiple	Macro-F1	CIGNN	1.558	18.459	<0.0001
TumEMO	Accuracy	S-MA	0.716	4.031	0.0157
TumEMO	Macro-F1	S-MA	0.470	4.273	0.0129

Appendix A.2. Supplementary Attention Analysis for IGCR

To intuitively reveal the retrieval-and-refinement mechanism inside IGCR, we visualize the cross-attention weights of IGCR. The attention weight matrix has a dimension of $3\times L_R$, where rows correspond to the three intuitive tokens and columns correspond to individual tokens of the rational sequence, with each row normalized by softmax. We plot the cross-attention heatmaps for text and visual modalities separately, as shown in Figure A1 and Figure A2. As illustrated in Figure A1 and Figure A2, different intuitive tokens assign disparate attention weights over rational tokens and concentrate on emotionally salient positions rather than distributing attention uniformly. This demonstrates that intuitive perceptual priors can retrieve emotion-relevant contextual evidence from rational representations.

Beyond cross-attention, IGCR adopts a progressive self-attention module upon the enhanced rational sequence ${\mathbf{C}}^{\prime }$, which produces a square $L_R\times L_R$ attention matrix modeling pairwise intra-dependencies among rational-side tokens. For the visual branch, the matrix reaches $197\times 197$ with massive elements, bringing heavy visualization overhead. We omit its heatmap here and only provide the above cross-attention qualitative analysis.

Figure A1. Heatmap of IGCR cross-attention weights for text branch. Rows denote three intuitive tokens, columns represent rational text tokens. Brighter color indicates higher attention weight.

Figure A1. Heatmap of IGCR cross-attention weights for text branch. Rows denote three intuitive tokens, columns represent rational text tokens. Brighter color indicates higher attention weight.

Figure A2. Heatmap of IGCR cross-attention weights for visual branch. Rows denote three intuitive tokens, columns represent rational image tokens. Brighter color indicates higher attention weight.

Figure A2. Heatmap of IGCR cross-attention weights for visual branch. Rows denote three intuitive tokens, columns represent rational image tokens. Brighter color indicates higher attention weight.