A Review of Multimodal Sentiment Analysis in Online Public Opinion Monitoring

Abstract

With the rapid development of the Internet, online public opinion monitoring has emerged as a crucial task in the information era. Multimodal sentiment analysis, through the integration of multiple modalities such as text, images, and audio, combined with technologies including natural language processing and computer vision, offers novel technical means for online public opinion monitoring. Nevertheless, current research still faces many challenges, such as the scarcity of high-quality datasets, limited model generalization ability, and difficulties with cross-modal feature fusion. This paper reviews the current research progress of multimodal sentiment analysis in online public opinion monitoring, including its development history, key technologies, and application scenarios. Existing problems are analyzed and future research directions are discussed. In particular, we emphasize a fusion-architecture-centric comparison under online public opinion monitoring, and discuss cross-lingual differences that affect multimodal alignment and evaluation.

1. Introduction

In the era of ubiquitous internet connectivity, online platforms have become the primary channels for public opinion expression and information dissemination. The rise of social media platforms such as X (Twitter), Instagram, and TikTok has amplified the scale, speed, and influence of public discourse. Online public opinion monitoring has consequently become essential for government agencies, corporations, and researchers, enabling real-time detection of emerging trends, crises, and sentiment shifts [1].

Multimodal sentiment analysis, a subfield of multimodal learning, integrates text, images, audio, and sometimes video, combining natural language processing (NLP), computer vision (CV), and speech processing to achieve more accurate and nuanced sentiment detection [2]. Its capacity to capture cross-modal cues allows better sentiment inference than unimodal approaches, thereby improving the timeliness and precision of public opinion monitoring.

2. Review Methodology

2.1. Scope

With multimodal sentiment analysis assuming escalating importance in online public opinion monitoring, systematic surveys are essential for consolidating emergent methodologies and forecasting research trajectories. While recent years have witnessed several influential surveys, each exhibits distinct limitations that leave critical gaps unfilled.

Xu et al. [3] recently surveyed Transformer-based multimodal learning through a geometric–topological perspective, analyzing modality-agnostic self-attention mechanisms across diverse applications. However, this breadth spanning vision, language, and beyond yields limited domain-specific insights for sentiment analysis within public opinion contexts. He et al. [4] systematically categorized multimodal fusion architectures into joint, collaborative, and encoder–decoder frameworks, meticulously detailing early, late, and hybrid fusion paradigms. Yet this foundational work predates the large-scale pretrained model era and advanced attention mechanisms that now dominate the field.

Zhao et al. [5] introduced the pioneering survey on Multimodal Aspect-Based Sentiment Analysis, focusing specifically on text–image fusion methods for aspect-level classification. While seminal, their analytical scope remains confined to aspect-based scenarios, neglecting the broader sentiment analysis challenges inherent to public opinion monitoring. Zhang et al. [6] examined large language model applications in cybersecurity, highlighting critical challenges in safety, interpretability, and resource dependency that parallel public opinion system concerns. Nevertheless, their cybersecurity-centric framework does not address multimodal sentiment analysis methodologies themselves.

These surveys collectively reveal a conspicuous void: none provide a comprehensive examination of multimodal sentiment analysis techniques specifically tailored for online public opinion monitoring, particularly within the contemporary landscape shaped by large language models and evolving social media platforms. Notably, despite conducting extensive performance evaluations across state-of-the-art models, existing surveys fail to systematically contextualize these advances within public opinion monitoring frameworks.

The present review establishes its scope across four interconnected dimensions:

• the evolutionary trajectory of multimodal fusion techniques, from traditional feature-level approaches to contemporary Transformer-based architectures;
• task-specific methodological deployments across social media monitoring, product/service feedback analysis, and public safety/crisis management;
• comparative performance assessment across English and Chinese benchmark datasets, including CMU-MOSI, CMU-MOSEI, CH-SIMS, and CH-SIMSv2;
• emerging challenges and future research directions arising from LLM integration and domain-specific requirements.

2.2. Strategy

The purpose of this systematic review is to chart the evolution and application of multimodal sentiment analysis methodologies within the specific context of online public opinion monitoring. We followed a structured and reproducible search-and-screening procedure, while using a targeted query design to focus on the most relevant and impactful studies in this fast-evolving domain. Our initial search focused on publications that concurrently addressed “multimodal sentiment analysis” and “online public opinion” between 2018 and 2025, scanning six major academic databases—Google Scholar, Web of Science, IEEE, Elsevier, ACM, and CNKI. This precise query yielded 116 promising studies that directly bridged computational methods with public opinion challenges.

Recognizing the interdisciplinary nature of this field, we subsequently broadened our exploration to include individual searches for “multimodal sentiment analysis”, “online public opinion monitoring”, and related fusion methodologies, enabling us to capture foundational techniques and parallel innovations that inform current practice. From this curated collection of literature, we prioritized works demonstrating clear methodological contributions, empirical validation, or practical deployment in real-world monitoring scenarios. Publications lacking substantive AI/ML frameworks, direct relevance to sentiment analysis applications, or sufficient experimental detail were excluded from further consideration. The resulting synthesis draws upon 97 carefully selected references spanning dataset development, feature extraction advances, fusion architecture innovations, and domain-specific implementations across social media, product feedback, and crisis management applications.

2.3. Contributions

To make the lens explicit, we organize and evaluate prior work primarily from a public opinion monitoring perspective, with a focus on how fusion architectures behave across application scenarios and English/Chinese settings. This work aims to

• Synthesize existing methodologies to provide researchers with a detailed understanding of available methods and resources;
• Systematically analyze the evolution of fusion strategies from conventional paradigms to modern transformer-based approaches;
• Evaluate practical applications across key public opinion monitoring scenarios;
• Identify pressing challenges and prospective research avenues in the current technological landscape.

3. Multimodal Sentiment Analysis

Sentiment analysis is currently a research hotspot in the interdisciplinary field of computer science, covering domains such as computer science, psychology, and social sciences. Similar to affective computing and emotion recognition research methods, it utilizes natural language processing, machine learning, and other techniques to mine and analyze opinions and topics contained in different modalities of data, and to identify sentiment polarity and orientation [5].

The technical development history of sentiment analysis, as shown in Figure 1, can be roughly divided into three stages:

Initial Exploration (1960–1990): During this foundational period, research predominantly focused on computer-assisted sentiment analysis techniques that relied heavily on manually constructed sentiment lexicons. Scholars developed rule-based approaches using predefined dictionaries of sentiment-laden words and phrases, with computational methods serving primarily as auxiliary tools for linguistic analysis rather than autonomous prediction systems.

Technological Development (1990–2010): This era witnessed the formal conceptualization of sentiment analysis as a distinct research field, accompanied by the emergence of systematic sentiment polarity analysis methodologies. A pivotal advancement was the introduction of the Latent Dirichlet Allocation (LDA) topic model around 2003, which enabled probabilistic modeling of semantic structures in textual data. Statistical machine learning approaches began supplementing purely lexicon-based methods, establishing the technical groundwork for subsequent data-driven paradigms.

Application Maturity (2010–present): The integration of sophisticated machine learning and deep learning architectures has propelled sentiment analysis into a phase of widespread practical deployment. Representative breakthroughs include the Word2Vec model (circa 2013) for dense vector representations, the Global Vectors for Word Representation (GloVe) framework (circa 2014), and the transformative Bidirectional Encoder Representations from Transformers (BERT) model (2018). These innovations have facilitated large-scale implementation across diverse domains, particularly in comprehensive public opinion monitoring, granular product and service feedback analysis, and critical public safety management systems.

Compared with traditional sentiment analysis methods, multimodal sentiment analysis can leverage data from multiple modalities such as text, audio, and images, enabling more comprehensive extraction and judgment of implicit emotional information [7].

3.1. Unimodal Sentiment Analysis

Over its development, unimodal sentiment analysis has achieved significant results in multiple aspects, including multi-dimensional data processing, big data computation, and complementary information across different data types.

3.1.1. Text Modality

As shown in Figure 2, the general process of text sentiment analysis consists of

Data Collection: This initial phase involves gathering textual data from diverse sources, including manual acquisition through curated datasets and automated web crawling conducted under strict legal and ethical conditions, ensuring compliance with platform policies and data protection regulations.

• Data Preprocessing: The raw data undergoes systematic cleaning and normalization, which includes removing irrelevant characters such as stop words and punctuation marks, performing dictionary matching to identify sentiment-bearing lexicons, and conducting lexical recognition to parse words and phrases for further analysis.
• Feature Extraction: At this stage, the processed text is transformed into machine-readable numerical representations using techniques such as Bag-of-Words for frequency-based encoding or word embeddings for semantic vectorization. Simultaneously, sentiment scores and polarity values are computed to quantify the emotional orientation embedded in the textual content.
• Model Training: The extracted features are fed into appropriate learning algorithms for training predictive models. This includes traditional machine learning methods such as Support Vector Machines (SVMs) for classification, as well as advanced deep learning architectures like BERT for contextual language understanding and ResNet for handling complex feature mappings.
• Result Visualization: Finally, the analysis outcomes are presented through intuitive visual formats—such as charts, graphs, or interactive dashboards—to effectively convey sentiment patterns, trends, and insights derived from the model predictions.

Early text sentiment analysis methods primarily relied on lexicon-based and rule-based approaches for sentiment recognition and classification [8]. On this basis, Hu et al. [9] used adjectives as prior knowledge to determine sentence sentiment polarity, but this approach was limited in handling non-adjective sentiment expressions and complex contexts. To capture implicit semantic and emotional associations between words, Maas et al. [10] proposed learning word vectors containing both semantic and sentiment information via joint optimization of semantic and sentiment objectives. However, their method lacked adaptability to dynamic semantic changes, limiting performance in complex sentiment analysis tasks.

For simpler text sentiment analysis tasks, the Bag-of-Words (BoW) model is often used, representing source text as a vector of word occurrence counts:

$$\mathrm{BoW}\left(D\right)=(w_1,w_2,\dots ,w_n),$$

(1)

Another common method is Term Frequency–Inverse Document Frequency (TF-IDF), which evaluates the importance of a word in a document relative to a corpus:

$$\mathrm{TF}-\mathrm{IDF}(t,d)=\mathrm{TF}(t,d)\times \mathrm{IDF}\left(t\right),$$

(2)

where $\mathrm{TF}(\mathrm{t},\mathrm{d})$ is the term frequency, and $\mathrm{IDF}\left(\mathrm{t}\right)$ is the inverse document frequency.

Mikolov et al. [11] proposed using Word2Vec to compute continuous word vectors from large-scale datasets, enabling quantitative semantic analysis of words. In Word2Vec, the Continuous Bag-of-Words (CBOW) model minimizes the cross-entropy loss of predicting a target word, while the Skip-gram model predicts context words from a given target word. The CBOW objective function is

$$J={\displaystyle \frac{1}{T}}\sum _{t=1}^{T}logP\left(w_t\right|w_{t-k},\dots ,w_{t+k}),$$

(3)

where T is the total number of words, k is the context window size, $w_t$ is the target word, and $w_{t-k}$, $...$, $w_{t+k}$ are its context words.

He et al. [12] proposed a deep learning model enhanced with emotion semantics for microblog sentiment analysis, mapping emojis into an emotional space and combining them with deep models—which are effective when emojis are present but limited when they are absent. Jin et al. [13] treated sentiment data as an auxiliary task within a multi-task learning framework for offensive language detection, reducing reliance on explicit sentiment cues. Li et al. [14] employed Convolutional Neural Networks (CNNs) and Bidirectional Long Short-Term Memory (BiLSTM) networks for feature extraction and fusion, improving classification accuracy but limiting performance on long-text tasks due to CNN constraints. Han et al. [15] applied multi-dimensional attention mechanisms to capture inter-word dependencies and high-level semantic–emotional information, achieving effective feature extraction but facing high computational complexity, making it unsuitable for large-scale datasets.

3.1.2. Visual Modality

In the visual modality, sentiment analysis was initially applied to image emotion prediction. Tamura et al. [16] computed texture features of images for sentiment analysis, but their method failed to fully exploit multi-scale features in similarity measurement and lacked precision in describing texture elements. Compared to simple surface-feature computation, Colombo et al. [17] applied the Hough transform to compute contour features of image regions, generating slope histograms and establishing a correspondence with emotional features, thus capturing the influence of line orientation on emotion. Machajdik et al. [18] extracted low-level visual features to predict image sentiment, but both methods suffered from dataset sensitivity and limited generalization.

With the advancement of machine learning, many researchers have adopted machine learning-based methods for visual sentiment analysis. Borth et al. [19] proposed a large-scale visual sentiment ontology based on Plutchik’s emotion wheel, building a detector library by detecting adjective–noun pairs related to emotions in images. However, because their sentiment classification was simplified to a three-polarity system, the method struggled with complex emotional information. Yang et al. [20] introduced binary label encoding and label noise augmentation to address the ambiguity in image sentiment, enabling multi-label sentiment recognition.

To address model dependency on high-quality datasets, Zhu et al. [21] leveraged adversarial and cycle-consistency losses to map between unpaired image domains, but their approach was limited in handling geometric transformations. Chen et al. [22] trained models using noisy emoji labels easily obtained from microblogs, alleviating dataset labeling scarcity; however, recognition performance for complex emotions declined compared to specialized visual sentiment models. He et al. [23] improved CNN classification performance on small datasets by reducing marginal and joint distribution discrepancies, though large domain shifts between source and target datasets could introduce noise. Zhao et al. [24] used a convolutional spatial Transformer and a temporal Transformer to learn spatial and temporal features, addressing challenges like occlusion, non-frontal poses, and head movements, but struggled with emotion categories having sparse samples.

The development of visual sentiment analysis parallels that of text sentiment analysis: both aim to achieve efficient and accurate data recognition while increasingly focusing on extracting deeper-level emotional information.

3.1.3. Speech Modality

For basic speech modality sentiment analysis, Lin et al. [25] captured the temporal dynamics of speech using a Mel-frequency sub-band energy difference feature extraction method, achieving high classification accuracy and robustness in gender-independent scenarios.

Benefiting from advancements in machine learning and neural networks, research on acoustic and prosodic features has become a hotspot in speech sentiment analysis. Wu et al. [26] improved sentiment recognition accuracy through acoustic modeling with multiple classifiers combined via a meta-decision tree; however, the method was constrained by pre-defined emotional rules and knowledge bases, making it less effective for ambiguous or personalized emotional expressions. Sunberg et al. [27] applied multiple discriminant analysis and canonical correlation analysis to acoustic parameters, revealing correlations between vocal physiological signals and emotions, but their method was limited by small dataset size and weak acoustic features. Jin et al. [28] extracted low-level acoustic features and emotional vectors for feature representation, but the method relied on acted emotional data, making it less representative of genuine emotional expression and inconsistent in recognizing different emotion categories. Mencattini et al. [29] developed a dynamic cooperative speaker model for continuous emotion estimation in natural speech, but due to subjective labeling and scarce related data, the method—like that of Sunberg et al. [27]—had limited generalizability and stability.

To address dataset scarcity, Eskimez et al. [30] employed variational autoencoders, adversarial autoencoders, and adversarial variational Bayes to learn features from unlabeled speech data, improving sentiment recognition. However, due to emotional diversity, these methods showed inconsistent performance across different emotion categories. Pourebrahim et al. [31] reduced label distribution discrepancies between samples by using parallel shared encoders with a maximum mean discrepancy loss, but their model had high complexity due to the combined use of autoencoders and classification tasks.

3.2. Multimodal Feature Fusion

Multimodal sentiment analysis refers to techniques that use multiple modalities (such as text, images, and audio) to perform sentiment analysis. Compared with unimodal sentiment analysis, it leverages richer data sources to obtain more comprehensive emotional information, thereby improving the accuracy and reliability of sentiment recognition [32].

3.2.1. Fusion Strategy

In multimodal learning, the timing of fusion strategy implementation has a significant impact on the effectiveness of multimodal integration. With the continuous development of machine learning, deep learning models have been introduced into the fusion process to narrow the gap between modalities and enhance feature representation. Since each fusion method has its own advantages and disadvantages, it is often necessary to experiment within training tasks to achieve optimal results [33]. The main fusion strategies are as follows:

• Early Fusion (Feature-Level Fusion)

As shown in Figure 3, early fusion refers to combining the features from each modality before decision-making. This approach merges features at the feature level, which can help reduce subsequent processing costs, but requires handling a large volume of heterogeneous feature formats [34].

ii.

Late Fusion (Decision-Level Fusion)

As shown in Figure 4, late fusion integrates the outputs of different modalities after independent decision-making. This allows each modality to use its own optimal classifier, but may incur additional training costs [4].

iii.

Hybrid Fusion (Mid-Level Fusion)

As shown in Figure 5, hybrid fusion is performed after feature extraction but before the final decision, allowing the model to capture complementary information between modalities while also leveraging individual modality-specific features [35]. For example, Zhang Xinyou et al. [36] addressed the problem of uncertain information propagation direction in fake news detection by fusing multi-view features from content and news context to generate more comprehensive representations.

iv.

Tensor Fusion

As shown in Figure 6, tensor fusion represents data from different modalities as tensors and fuses them through specific mathematical operations. This can capture intrinsic correlations between modalities but may face challenges in handling high-dimensional data [2].

Zhao Xinhe et al. [37] applied tensor fusion in gambling website detection, aligning textual and visual features to unified dimensions and employing focal loss to enhance classification performance on imbalanced datasets. The tensor fusion operation can be expressed as follows:

$$z_{\mathrm{fusion}}=\mathrm{concat}(z_t,z_v,z_a,z_t\otimes z_v,z_v\otimes z_a,z_a\otimes z_t,z_t\otimes z_v\otimes z_a),$$

(4)

where $z_t$, $z_v$, and $z_a$ denote the feature vectors from text, visual, and audio modalities, respectively, and ⊗ represents the outer product operation.

v.

Model-Level Fusion

As shown in Figure 7, Model-level fusion integrates multimodal data at various stages of model learning, jointly optimizing feature extraction and fusion strategies during training. This requires addressing issues such as balancing the contribution of each modality and handling modality-specific characteristics. Lueangwitchajaroen et al. [35] proposed a multi-layer feature fusion approach based on EfficientNet-B7, integrating spatial and temporal information from RGB video frames at early, middle, and late stages to improve action recognition accuracy. Zheng et al. [38] designed a reinforcement learning strategy leveraging category priors to perform category-wise feature fusion and address data imbalance, thereby reducing reliance on large-scale training data.

vi.

Transformer-Based Fusion

Since Vaswani et al. [39] proposed the Transformer architecture, Transformer-based multimodal fusion has become a research hotspot [3]. For instance, Shvetsova et al. [40] developed a Transformer-based fusion mechanism for zero-shot video retrieval in modality-agnostic environments. Xu et al. [41] built a unified Transformer framework to combine object detection and captioning into pre-training, jointly learning visual representation and cross-modal semantic alignment—though this increases training complexity and imposes high requirements on input image quality.

Researchers have also introduced attention mechanisms into Transformer-based fusion to enhance the learning of key content across modalities. Girdhar et al. [42] combined modalities into spatio-temporal blocks and applied a self-attention-based Transformer for multimodal classification tasks. The attention mechanism can be expressed as follows:

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left({\displaystyle \frac{Q{K}^{\mathrm{T}}}{\sqrt{d_k}}}\right)V,$$

(5)

where Q, K, and V denote the query, key, and value matrices, and $d_k$ is the dimensionality of the key vectors.

Other examples include Tschannen et al. [43], who adapted the CLIP model [44] to process both images and text using ViT (Vision Transformer) and contrastive learning; Huan et al. [45], who used attention-based fusion to complete missing modalities; and Yi et al. [46], who proposed a two-stage stacked Transformer capturing intra-modality communication and inter-fusion representation interactions with an adaptive weight accumulation mechanism. A schematic diagram of the Transformer-Based Fusion Strategy is shown in Figure 8:

vii.

Hierarchical Fusion

As shown in Figure 9, hierarchical fusion integrates multimodal features at multiple abstraction levels, such as low-level perceptual features and high-level semantic features [47]. This approach can better preserve contextual and semantic information, making it effective for handling complex multimodal data.

A generic hierarchical fusion function can be expressed as

$$F=g(f_1,f_2,...,f_m),$$

(6)

where $f_i$ are the features extracted from different sources and g is the fusion function.

Given the distinct characteristics of each fusion strategy discussed above,

Table 1. Comparison of multimodal fusion methods.

Fusion Method	Advantages	Disadvantages
Early Fusion	Preserves fine-grained cross-modal interactions Enables end-to-end joint optimization Learns low-level feature correlations	High computational complexity Strict temporal/spatial alignment required Vulnerable to noise and missing modalities
Late Fusion	Modular design, independent training High computational efficiency Robust to misalignment	Cannot capture modal interactions Loses cross-modal complementarity Struggles to optimize ensemble weights
Hybrid Fusion	Balances expression and efficiency Captures mid-level interactions Tolerant to partially missing modality	Requires careful fusion layer design Increases model complexity Fusion timing relies on heuristics
Tensor Fusion	Models high-order interactions Preserves complete correlations Strong theoretical capacity	Suffers from dimensionality explosion Requires large datasets Poor interpretability
Model-level Fusion	Deep integration, parameter-efficient Enables cross-modal sharing Facilitates transfer learning	Complex architecture design High coupling reduces flexibility Difficult training convergence
Transformer-based Fusion	Attention learns adaptive weights Captures long-range dependencies Highly scalable and generalizable	Quadratic computational complexity Requires large-scale pretraining Limited interpretability
Hierarchical Fusion	Multi-scale interaction capture Combines complementary advantages Strong robustness and adaptability	Complex structure, hard to train High computation and memory cost Tedious hyperparameter tuning

provides a systematic comparison of their respective strengths and limitations.

3.2.2. Current Research Status in Multimodal Sentiment Analysis

Recent work includes multimodal aspect-based sentiment analysis (MABSA), multi-task learning frameworks, and the integration of large language models for cross-modal reasoning. MABSA aims to analyze sentiment evaluations of specific aspects within multimodal data. Challenges include handling the complexity of multimodal data, aligning different modalities’ temporal sequences, and improving model generalization and interpretability. For example, Zhang et al. [48] used a gating mechanism to reduce noise interference and enhance image semantic representation via adjective–noun pairs. Wang et al. [49] combined orthogonally constrained self-attention with a gated local cross-modal interaction mechanism to improve MABSA accuracy, but their method suffered from low training efficiency and high sensitivity to hyperparameters. Zhang et al. [50] integrated cross-attention and graph attention networks to improve performance, though the results depended on graph construction parameters. Li et al. [51] used a text-guided fusion approach to reduce redundancy and employed adaptive context enhancement to improve polarity recognition.

In multi-task multimodal sentiment recognition, research focuses on using multi-task learning frameworks to improve performance. Lin et al. [52] proposed a model with shared layers for visual and speech modalities that could jointly learn emotional information.

Multimodal emotion recognition based on deep learning also demonstrates great potential [53]. These models can learn and adapt to specific emotion analysis tasks with a small amount of samples, to a certain extent solving the problem of scarce data. Moreover, multimodal emotion recognition technology is increasingly important in the detection and intervention of emotional disorders such as depression, and relevant scholars are constantly exploring how to use multimodal data for more accurate assessment and intervention [54].

Meanwhile, with the development of large language models (LLMs), the amount of research on the processing of multimodal data has been increasing. By leveraging large language models, in-depth data mining and analysis of modal data such as text, images, and audio can be achieved, such as the multimodal data understanding and text generation tasks based on large language models like ChatGPT-4, Qwen-14B series [55], and DeepSeek-R1-Zero [56]. Pang et al. [57] utilized the auxiliary knowledge of multimodal large language models to improve the accuracy of sentiment analysis and reasoning ability. This method requires computational resources to generate auxiliary knowledge, which increases the complexity of the model and the training cost.

4. Online Public Opinion Monitoring

Public opinion refers to the collective emotional tendencies and viewpoints widely held by the public regarding a particular issue within a certain social space. With the rapid development of the Internet, the concept of online public opinion has emerged, characterized by its wide reach and high transmission speed. Therefore, online public opinion monitoring is of great significance for the decision-making of governments, enterprises, and various organizations.

4.1. Theoretical Foundation

As one of the main forms of public opinion, online public opinion retains the core characteristics of general public opinion, preserving its essential attributes while manifesting in digital environments. As systematically illustrated in Figure 10, the dissemination process of online public opinion unfolds through three sequential stages: information generation, where original content is initially created and introduced into the digital ecosystem; information diffusion, which involves the propagation and spread of that content across various platforms and networks; and formation of influence, where the cumulative effect of disseminated information shapes public perceptions and generates tangible impacts. This dynamic process fundamentally involves key elements such as information sources that serve as the originators of content, transmission channels that act as conduits for distribution and amplification, and audience responses that reflect the reception, interpretation, and reactive behaviors of end-users, together constituting the interactive framework of online public opinion dissemination.

The formation and evolution of online public opinion are influenced by multiple factors, including social events, media reports, and online user interactions. Thus, evaluating online public opinion requires a comprehensive assessment of its dissemination dynamics, scope of influence, and potential social impact, so as to predict its development trends and possible consequences [58].

4.2. Manual Monitoring Methods

Manual content analysis performed by professionals has always been an indispensable part of public opinion monitoring. From the early 20th century to the mid-20th century, information monitoring mainly relied on reading traditional media and collecting materials via clipping services. With the popularity of radio and television, monitoring expanded to include program listening and random telephone surveys.

In the 1950s, many government agencies and enterprises began using focus groups to conduct qualitative research on public opinion. In the 1980s, the rapid development of computer and mobile communication technologies further improved the informatization of databases and archive management, greatly enhancing the efficiency of data storage, retrieval, and analysis. Questionnaires also evolved from paper-based to electronic formats [59].

4.3. Machine Learning–Based Methods

As depicted in Figure 11, the machine learning-based approach to public opinion monitoring harnesses natural language processing (NLP) and data mining techniques to scrutinize and oversee public sentiment and opinions circulating on the Internet. This method involves several key steps: initially, it gathers data from various online sources; following this, it employs NLP to process and interpret textual content; next, it applies data mining to identify trends and patterns; subsequently, it conducts real-time monitoring to track the evolution of public sentiment; and finally, it predicts trends to inform practical applications and decision-making processes. This comprehensive strategy effectively translates raw data into valuable insights, facilitating a proactive stance on public opinion management.

Based on the CLIP framework, Wang et al. [60] introduced a linear feature fusion layer to significantly improve multimodal representation. However, this method required finding optimal fusion ratios during training and was less adaptable to variations in language and data quality across contexts.

Chen Jie [61] combined the DR-Transformer multimodal fusion mechanism with hierarchical multimodal features for sentiment polarity recognition, mapping relationships between graded features and high-level emotional information while narrowing semantic gaps.

Yang et al. [62] introduced a multi-channel graph neural network to learn global multimodal representations, combined with a multi-head attention mechanism for predicting sentiment from image–text pairs. While effective, their model was relatively complex and had limited training efficiency and generalization.

The general process includes model training, integration and deployment, real-time monitoring, data source processing, trend prediction, and application feedback.

4.4. Multimodal Sentiment Analysis in Public Opinion Monitoring

This emerging field combines multiple data modalities to detect and analyze public sentiment, with applications including the following.

4.4.1. Social Media Monitoring

Multimodal sentiment analysis can be applied to social platforms to analyze user-generated text, images, and videos for sentiment trends. For instance, analyzing tweets with images can yield more accurate sentiment assessments, as text and image content may convey different emotions.

Zadeh et al. developed two benchmark multimodal sentiment corpora:

CMU-MOSI [63]—2199 opinionated video segments with audio;

CMU-MOSEI [64]—over 23,500 sentences from YouTube speakers with audio.

Leveraging MABSA, Zhou et al. [65] used aspect extraction, polarity prediction, and adversarial training to enhance text–image–aspect learning, but their model’s complex interaction mechanisms were sensitive to hyperparameters. Xiang et al. [66] applied feature smoothing and multi-channel attention to bridge semantic gaps across modalities, improving MABSA performance but also requiring complex computations. Yang et al. [67] introduced an image-assisted module with multimodal prompt fusion to improve text–image feature integration, though the method required substantial training data.

To address unbalanced modality proportions in real-world data, Hu et al. [68] enhanced linguistic information while reducing non-linguistic redundancy, slightly lowering performance on non-linguistic features. Xie et al. [69] introduced uncertainty estimation and ordinal regression for dynamic modality quality weighting, improving prediction stability but at the cost of higher computational complexity. Wang et al. [70] incorporated fuzzy deep neural networks for multi-scale emotion uncertainty modeling but faced difficulties in real-time applications.

In Chinese social networks, Du Peipei [71] used topic matching and emoji-masking tasks with gating mechanisms to filter redundant information in Weibo sentiment analysis. Ni Ningning [72] addressed heterogeneous cross-media data using a graph-based cross-media fusion framework with a background-topic model, though performance depended on high-quality graph construction.

4.4.2. Product and Service Feedback

Enterprises can combine text, speech, and image feedback to better assess customer satisfaction. Xu et al. [73] designed a multi-interaction memory network exploiting cross-modal dependencies and built the Multi-ZOL dataset (5288 phone reviews from the ZOL forum). Xue et al. [74] applied co-attention fusion after filtering noise and capturing multi-granular feature correlations, though their model was complex and untested in dynamic scenarios.

In specialized industries, Huawei’s Pangu Model uses an encoder–decoder architecture integrating language and vision for predictive tasks in meteorology, medicine, and water resource management [75].

4.4.3. Public Safety and Crisis Management

Multimodal sentiment analysis can detect emergencies and shifts in collective emotions via video surveillance and social media. Xu Yang et al. [76] combined ensemble empirical mode decomposition (EEMD) with Transformer attention to analyze COVID-19 public opinion heat trends, though performance depended heavily on preprocessing. Liu et al. [8] improved PP-OCR text detection/recognition with a global–local attention mechanism for multimodal sentiment tasks, but the method was dataset- and language-specific.

4.5. Performance Comparison of Multimodal Sentiment Analysis Methods

Based on the MMSA [77] integrated framework, the following models were tested in both Chinese and English settings using the CMU-MOSI [63], CMU-MOSEI [64], CH-SIMS [78], and CH-SIMSv2 [79] datasets: LMF [80], MFN [81], MISA [82], EF-LSTM [83], LF-DNN [84], Self-MM [85], MMIM [86], MFM [87], Graph-MFN [64].

Table 2. Information of multimodal sentiment computing datasets.

No.	Dataset Name	Application	Modality Types	Data Volume	Language	Year	Institution
1	CMU-MOSI [63]	Sentiment Computing, Public Opinion Analysis	Text, Visual, Audio	2199 video clips	English	2018	Carnegie Mellon University
2	CMU-MOSEI [64]	Sentiment Computing, Public Opinion Analysis, Human-Computer Interaction	Text, Visual, Audio	23,500 video clips	English	2018	Carnegie Mellon University
3	CH-SIMS [78]	Sentiment Computing, User Behavior Analysis, Public Opinion Analysis	Text, Visual, Audio	2281 video clips	Chinese	2020	Tsinghua University
4	CH-SIMSv2 [79]	Sentiment Computing, User Behavior Analysis, Public Opinion Analysis	Text, Visual, Audio	14,563 video clips	Chinese	2022	Tsinghua University

outlines key information about four datasets used in multimodal sentiment computing, detailing their applications, modality types, data volume, language, year of creation, and the institutions responsible for their development.

• CMU-MOSI: Developed by Carnegie Mellon University in 2018, this dataset contains 2199 video clips and focuses on sentiment computing and public opinion analysis. It includes text, visual, and audio modalities and is available in English.
• CMU-MOSEI: Also from Carnegie Mellon University, this dataset was created in 2018 and comprises 23,500 video clips. It is used for sentiment computing, public opinion analysis, and human–computer interaction, incorporating text, visual, and audio data in English.
• CH-SIMS: This dataset, developed by Tsinghua University in 2020, contains 2281 video clips. It is utilized for sentiment computing, user behavior analysis, and public opinion analysis, covering text, visual, and audio modalities in Chinese.
• CH-SIMSv2: An extension of CH-SIMS, this dataset was also developed by Tsinghua University and released in 2022. It includes a larger volume of 14,563 video clips and is used for similar applications as CH-SIMS, focusing on text, visual, and audio data in Chinese.

Each dataset offers a rich resource for researchers and practitioners in the field of sentiment analysis, enabling the development and evaluation of models that can interpret and analyze multimodal data effectively.

Table 3. Summary of mainstream multi-modal model information.

Model Name	Core Idea	Applicable Scenarios	Trainable Parameters
LMF [80]	Dynamic fusion of modalities to capture inter-modal interactions	Multimodal sentiment computing, intent recognition	≈0.5 M
MFN [81]	Multi-perspective sequence learning to fully utilize cross-perspective interaction information	Multi-perspective video analysis, dialogue sentiment recognition	≈2.2 M
MISA [82]	Decompose modalities into invariant and specific features to reduce modal differences	Cross-modal sentiment transfer, low-resource scenarios	≈104 M
EF-LSTM [83]	Use early fusion and model phrase/sentence-level semantic composition	Text-speech sentiment computing, real-time interaction systems	≈0.89 M
LF-DNN [84]	Joint prediction based on BLSTM-based late fusion	Multimodal emotion recognition, human–computer interaction	≈0.6 M
Self-MM [85]	Self-supervised generation of single-modal labels and joint training	Label-scarce scenarios, cross-modal alignment	≈103 M
MMIM [86]	Maximize mutual information between input and fusion layer	Noisy environments, information-missing scenarios	≈103 M
MFM [87]	Decompose and represent cross-modal discriminative factors and modality-specific generative factors	Modality-missing scenarios, data completion	≈1.41 M
Graph-MFN [64]	Use graph structure to dynamically control modality weights and explicitly model modal interactions	Complex multimodal dialogue, sentiment computing	≈2.11 M

synthesizes specifications for nine multimodal architectures utilized in sentiment computing, capturing their fundamental principles, deployment contexts, and parameter magnitudes—metrics that determine both intricacy and data-driven adaptation potential.

• The LMF [80] model can dynamically and selectively fuse information from language, visual, and audio modalities to capture the interaction relationships among different modalities, achieving multimodal emotion computation and intent recognition.
• The MFN [81] independently models the interactions within each perspective and captures the cross-interactions between different perspectives, while storing and updating this interaction information through a multi-perspective gated memory module to achieve multi-modal and multi-perspective sequence learning.
• MISA [82] decomposes each modality into modality-invariant and modality-specific features, fuses them, and predicts emotional states, reducing the modality gap while lowering model complexity.
• EF-LSTM [83] uses recurrent neural networks and tensor operations to obtain semantic combination relationships at the phrase and sentence levels.
• LF-DNN [84] is a multi-modal, multi-perspective sequence learning method based on early fusion of input-level multi-modal DNN features, using a BLSTM network to jointly process audio, video, and text features, achieving simultaneous prediction of six types of emotions and their intensities.
• Self-MM [85] automatically generates single-modal labels to jointly train multi-modal and single-modal tasks, effectively capturing the consistency and differences between modalities, and achieving self-supervised multi-task learning without additional manual annotations.
• MMIM [86] maximizes mutual information at the input and fusion levels to reduce the loss of task-related information, using both parametric and non-parametric methods to estimate the lower bound of maximizing mutual information, thereby improving the quality of multi-modal data fusion.
• MFM [87] decomposes multi-modal representations into “cross-modal discriminative factors” and “modality-specific generative factors”, with the former used for task prediction and the latter for data reconstruction and missing modality completion, achieving joint optimization of generation and target discrimination.
• Graph-MFN [64] uses a graph structure to dynamically control the weights of language, visual, and acoustic modalities in real time, explicitly modeling single-, dual-, and triple-modal interactions, achieving more efficient modality fusion.

The experiments were divided into two parts:

(1)

Those using a BERT model trained in English as the text modality encoder;

(2)

Those using a BERT model trained in Chinese as the text modality encoder.

Evaluation metrics included top five classification accuracy, Mean Absolute Error (MAE), and correlation coefficient. Each result was the average of three runs with random seeds (1111, 1112, 1113).

• Top-5 Classification Accuracy: This metric evaluates the performance of a classification model by considering whether the correct label is within the top five predictions for each sample. The formula is given by

  $${\mathrm{Acc}}_5={\displaystyle \frac{1}{N}}\sum _{i=1}^{N}I\left(\mathrm{Top-}5{\mathrm{labels}}_i\cap \mathrm{True}{\mathrm{label}}_i\ne ⌀\right),$$

  (7)

  where N is the total number of samples and I is an indicator function returning 1 if at least one of the predicted top five sentiment labels matches the ground-truth label.
• Mean Absolute Error (MAE): This metric measures the average magnitude of errors between predicted and actual values without considering their direction. It is calculated using the following formula:

  $$\mathrm{MAE}={\displaystyle \frac{1}{N}}\sum _{i=1}^N|y_i-{\widehat{y}}_i|,$$

  (8)

  where $y_i$ is the true sentiment intensity for the i-th sample, and ${\widehat{y}}_i$ is the predicted value.
• Correlation coefficient:This metric quantifies the strength and direction of the linear relationship between predicted and actual sentiment intensities. It is defined by the following formula:

  $$\mathrm{Corr}={\displaystyle \frac{{\sum }_{i=1}^N(y_i-\overline{y})({\widehat{y}}_i-\overline{\widehat{y}})}{\sqrt{{\sum }_{i=1}^N{(y_i-\overline{y})}^2}\sqrt{{\sum }_{i=1}^N{({\widehat{y}}_i-\overline{\widehat{y}})}^2}}},$$

  (9)

  where $\overline{y}$ and $\overline{\widehat{y}}$ are the mean true and predicted sentiment intensities, respectively.

As shown in

Table 4. Comparison of performance of multimodal sentiment analysis methods in English environments (based on the MOSI dataset). The best-performing results are highlighted in bold.

Model Name	Bert_en+MOSI
	Mult_acc_5%	MAE%	Corr%
LMF [80]	39.65	96.81	65.02
MFN [81]	39.21	96.69	66.14
MISA [82]	46.99	80.91	76.60
EF-LSTM [83]	30.66	113.38
LF-DNN [84]	38.39	96.13	65.53
Self-MM [85]	51.50	72.62	79.62
MMIM [86]	51.26	74.89	77.68
MFM [87]	39.94	93.29	65.53
Graph-MFN [64]	41.45	93.50	65.89

and

Table 5. Comparison of performance of multimodal sentiment analysis methods in English environments (based on the MOSEI dataset). The best-performing results are highlighted in bold.

Model Name	Bert_en+MOSEI
	Mult_acc_5%	MAE%	Corr%
LMF [80]	53.55	56.51	73.25
MFN [81]	52.46	57.32	71.56
MISA [82]	53.92	54.79	76.04
EF-LSTM [83]	51.34	59.49	68.94
LF-DNN [84]	53.65	55.88	73.32
Self-MM [85]	55.41	53.57	75.95
MMIM [86]	51.07	58.49	71.38
Graph-MFN [64]	53.18	56.74	72.60

, the performance comparison results of the above-mentioned partial models tested in an English environment are as follows:

Based on the MOSI dataset (Table 4), the Self-MM model performs best in multimodal sentiment analysis tasks. Self-MM utilizes a self-attention mechanism and multimodal fusion strategy to effectively capture the correlations and complementary information between different modalities. Its five-class accuracy is 51.5%, its mean absolute error is 72.62%, and its correlation coefficient is 79.62%. The next best model is MISA. MISA improves the model’s understanding and the fusion capability of multimodal data by learning modality-invariant and modality-specific representations. Its five-class accuracy is 46.99%, its mean absolute error is 80.91%, and its correlation coefficient is 76.6%. Although MISA has a higher mean absolute error compared to Self-MM, its five-class accuracy and correlation coefficient are lower than those of Self-MM.

Based on the MOSEI dataset (Table 5), the Self-MM model achieves the highest performance in multimodal sentiment analysis tasks. It employs a self-supervised learning approach and multimodal fusion to effectively capture the relationships and complementary information across different modalities. The model attains a five-class accuracy of 55.41%, a mean absolute error (MAE) of 53.57%, and a correlation coefficient (Corr%) of 75.95%. Following closely behind is the MISA model. MISA enhances the model’s comprehension and integration of multimodal data by learning invariant and specific features for each modality. It achieves a five-class accuracy of 53.92%, an MAE of 54.79%, and a Corr% of 76.04%. Despite MISA’s slightly higher MAE compared to Self-MM, its five-class accuracy and Corr% are marginally lower than those of Self-MM.

As shown in

Table 6. Comparison of performance of multimodal sentiment analysis methods in Chinese environments (based on the CH-SIMS dataset). The best-performing results are highlighted in bold.

Model Name	Bert_cn+CH-SIMS
	Mult_acc_5%	MAE%	Corr%
LMF [80]	36.69	44.57	56.98
MFN [81]	38.73	44.62	56.12
MISA [82]	37.49	44.16	57.14
EF-LSTM [83]	36.40	44.94	59.20
LF-DNN [84]	64.62	45.25	54.58
Self-MM [85]	42.16	41.47	59.28

and

Table 7. Comparison of performance of multimodal sentiment analysis methods in Chinese environments (based on the CH-SIMSv2 dataset). The best-performing results are highlighted in bold.

Model Name	Bert_cn+CH-SIMSv2
	Mult_acc_5%	MAE%	Corr%
LMF [80]	48.87	35.66	58.32
MFN [81]	54.52	29.79	71.99
MISA [82]	41.52	38.70	55.33
EF-LSTM [83]	51.22	31.57	69.42
LF-DNN [84]	53.35	30.29	71.19
Self-MM [85]	52.35	31.63	70.76
Graph-MFN [64]	43.84	40.38	52.54

, the above-mentioned models were tested in the Chinese environment, and the performance comparison results are as follows:

Based on the CH-SIMS dataset (Table 6), the Self-MM model demonstrates remarkable performance in multimodal sentiment analysis tasks, achieving a five-class accuracy of 42.16%, a mean absolute error (MAE) of 41.47%, and a correlation coefficient of 59.28%. The second-best performing model is still MISA.

Based on the CH-SIMSv2 dataset (Table 7), both MFN and LF-DNN models exhibit excellent performance. MFN achieves a five-class accuracy of 54.52%, an MAE as low as 29.79%, and a correlation coefficient as high as 71.99%. LF-DNN attains a five-class accuracy of 53.35%, an MAE of 30.29%, and a correlation coefficient of 71.19%. MFN effectively integrates multimodal information through its modal fusion network and attention mechanisms, while LF-DNN leverages its deep architecture to achieve highly efficient feature extraction.

As shown in

Table 8. Comparison of trainable parameter quantities for each model.

Model Name	Trainable Parameters
LMF [80]	≈0.5 M
MFN [81]	≈2.2 M
MISA [82]	≈104 M
EF-LSTM [83]	≈0.89 M
LF-DNN [84]	≈0.6 M
Self-MM [85]	≈103 M
MMIM [86]	≈103 M
MFM [87]	≈1.41 M
Graph-MFN [64]	≈2.11 M

, models with fewer parameters (such as LMF, EF-LSTM, LF-DNN) perform well on specific datasets. For example, LF-DNN achieves high classification accuracy on both the CH-SIMS and CH-SIMSv2 datasets, indicating that its lightweight structure has good adaptability and efficiency in Chinese contexts. Models with a medium number of parameters (such as MISA, EF-LSTM, MMIM) perform well on certain datasets. For instance, MISA achieved the best performance on the MOSI dataset, demonstrating some advantages in handling complex multimodal data. Models with larger numbers of parameters (such as Self-MM, Graph-MFN) generally perform better on large-scale datasets. For example, Self-MM achieved optimal performance on the MOSEI dataset, showing that its complex structure is better equipped to handle large-scale, complex multimodal data.

As shown in Table 3, Table 4, Table 5 and Table 6, model performance exhibits a significant cross-lingual gap, with architectures that excel in Chinese contexts often underperforming in English environments. In Chinese datasets, MFN and LF-DNN demonstrate superior adaptability, where MFN achieves 54.52% accuracy on CH-SIMSv2 (Table 6) and LF-DNN reaches 64.62% on CH-SIMS (Table 5), indicating that attention-heavy fusion mechanisms effectively capture nuanced interplay between text and culturally specific visual cues. In contrast, Self-MM leads in English datasets, attaining 55.41% accuracy on MOSEI (Table 4) and 51.50% on MOSI (Table 3), revealing that self-supervised multimodal alignment excels when visual and textual modalities maintain direct semantic correspondence. This discrepancy stems from the semantic ambiguity of visual modalities in Chinese communication—particularly the polysemy of indigenous emojis and sticker derivatives whose meanings shift across subcultural contexts—versus English datasets where visual cues exhibit clearer alignment with textual sentiment. Future research must develop cross-lingual alignment frameworks that incorporate culture-aware visual disambiguation modules and meta-learning techniques to dynamically adapt fusion weights across linguistic landscapes.

5. Conclusions

This review does not introduce new models. Instead, it consolidates how multimodal sentiment analysis methods have been adapted for online public opinion monitoring, where data are noisy, modality availability is uneven, and evaluation settings vary across platforms and languages. By using fusion architectures as the main comparative axis and by contrasting representative English and Chinese benchmarks, this work summarizes both established findings and unresolved issues.

Multimodal sentiment analysis has gradually integrated technologies and theories from multiple disciplines. With the rise of large language models, large-scale datasets, and high-performance computing, new challenges have emerged:

(1)

Collaborative Representation

Emotional information embedded in different modalities varies in nature, and in practical applications, the proportion of each modality can differ significantly. Effectively integrating multimodal data while eliminating inter-modal discrepancies is key to improving sentiment polarity recognition accuracy.

(2)

Fine-Grained Sentiment Recognition

In psychology, there is no universally accepted definition of human emotions, and emotional expression varies widely across contexts [88]. Current multimodal sentiment analysis often focuses on broad categories such as joy, anger, sadness, and annoyance. There is an urgent need for more fine-grained sentiment analysis frameworks. While large language models such as GPT, LLaMA, Qwen, and DeepSeek have partially addressed this challenge in general domains, issues such as long-term dependency and “hallucinations” [6] mean that performance in domain-specific scenarios is still inadequate.

(3)

Datasets

As shown in

Table 9. Multimodal datasets for sentiment analysis.

No.	Dataset Name	Applications	Modalities	Year	Institution
1	VQA 2.0 [89]	Emotion Classification, Product Recommendation, Visual Question Answering	Text, Visual	2017	Virginia Tech, Georgia Institute of Technology
2	Twitter 2017 [90]	Sentiment Analysis, User Behavior Analysis, Cross-lingual Sentiment Analysis	Text, Visual	2018	Fudan University
3	CMU-MOSI [63]	Sentiment Analysis, Public Opinion Monitoring	Text, Visual, Audio	2018	Carnegie Mellon University
4	CMU-MOSEI [64]	Sentiment Analysis, Public Opinion Monitoring, Cross-modal Representation Learning	Text, Visual, Audio	2018	Carnegie Mellon University, University of Rochester
5	UR-FUNNY [91]	Humor Detection, Multi-modal Sentiment Analysis, Human-computer Interaction	Text, Visual, Audio	2019	Carnegie Mellon University
6	CH-SIMS [78]	Sentiment Analysis, User Behavior Analysis, Chinese Public Opinion Monitoring	Text, Visual, Audio	2020	Tsinghua University
7	MUGE [92]	Emotion Classification, Image Captioning, Text-to-image Retrieval, Text-based Image Generation	Text, Visual	2022	Alibaba DAMO Academy, Tsinghua University, Alibaba Cloud TI Platform
8	Wukong [93]	Image-text Retrieval, Zero-shot Image Classification, Chinese Public Opinion Monitoring	Text, Visual	2022	Huawei Noah’s Ark Lab
9	CH-SIMSV2 [79]	Sentiment Analysis, User Behavior Analysis, Chinese Public Opinion Monitoring	Text, Visual, Audio	2022	Tsinghua University
10	Touch100k [94]	Haptic Perception, Imitation Learning, Sentiment Analysis	Haptic, Visual	2024	Beijing Jiaotong University, Beijing University of Posts and Telecommunications, Tencent WeChat AI Team
11	PanoSent [95]	Sentiment Analysis, User Behavior Analysis, Public Opinion Monitoring	Text, Visual, Audio	2024	National University of Singapore
12	SEED-VII [96]	Cross-modal Analysis, Sentiment Analysis, Brain-computer Interface, Psychology Research	EEG, Eye-tracking	2024	Shanghai Jiao Tong University

, most mainstream multimodal sentiment analysis datasets are non-Chinese in origin; Chinese datasets emerged later. In the era of short videos, the number of topics is exploding, but certain new forms of emotional expression in Chinese social media—such as novel gestures, sticker packs, and emoji derivatives—cannot yet be effectively recognized. Additionally, text often dominates in dataset composition.

This table catalogues essential specifications for twelve datasets employed in multimodal sentiment analysis investigations, specifying their use cases, modality configurations, publication years, and the research entities that compiled them.

• VQA 2.0: Released by Virginia Tech and Georgia Institute of Technology in 2017, this dataset targets emotion classification, product recommendation, and visual question answering, integrating text and visual modalities.
• Twitter 2017: Curated by Fudan University in 2018, this resource facilitates sentiment analysis, user behavior analysis, and cross-lingual sentiment analysis, comprising text and visual data.
• CMU-MOSI: Produced by Carnegie Mellon University in 2018, this dataset serves sentiment analysis and public opinion monitoring, encompassing text, visual, and audio modalities.
• CMU-MOSEI: Also compiled in 2018 by Carnegie Mellon University and the University of Rochester, this collection supports sentiment analysis, public opinion monitoring, and cross-modal representation learning, featuring text, visual, and audio inputs.
• UR-FUNNY: Issued by Carnegie Mellon University in 2019, this dataset is designed for humor detection, multi-modal sentiment analysis, and human-computer interaction, combining text, visual, and acoustic information.
• CH-SIMS: Developed by Tsinghua University in 2020, this resource addresses sentiment analysis, user behavior analysis, and Chinese public opinion monitoring, integrating textual, visual, and auditory channels.
• MUGE: Published in 2022 by Alibaba DAMO Academy, Tsinghua University, and Alibaba Cloud TI Platform, this dataset enables emotion classification, image captioning, text-to-image retrieval, and image generation from textual descriptions, utilizing text and visual modalities.
• Wukong: Released by Huawei Noah’s Ark Lab in 2022, this collection supports image-text retrieval, zero-shot image classification, and Chinese public opinion monitoring, incorporating text and visual data.
• CH-SIMSV2: An expanded version from Tsinghua University published in 2022, this dataset continues to serve sentiment analysis, user behavior analysis, and Chinese public opinion monitoring, featuring text, visual, and audio modalities.
• Touch100k: Introduced in 2024 by Beijing Jiaotong University, Beijing University of Posts and Telecommunications, and Tencent WeChat AI Team, this pioneering dataset focuses on haptic perception, imitation learning, and sentiment analysis, uniquely merging haptic and visual sensory data.
• PanoSent: Developed by the National University of Singapore in 2024, this resource is applied to sentiment analysis, user behavior analysis, and public opinion monitoring, integrating text, visual, and audio modalities.
• SEED-VII: Released in 2024 by Shanghai Jiao Tong University, this specialized dataset facilitates cross-modal analysis, sentiment analysis, brain-computer interface research, and psychological studies, employing EEG and eye-tracking modalities.

These datasets jointly constitute a diverse repository for the research community, facilitating the design and assessment of models spanning varied languages, cultural contexts, and sensory modalities.

(4)

Uncertain data processing

Uncertainty characteristics embedded in real-world data manifest in heterogeneous forms, and in practical applications of multimodal sentiment analysis and public opinion monitoring, the interplay between randomness, fuzziness, and inconsistency can differ significantly across modalities. Effectively modeling uncertain data while eliminating the interference of noisy information is key to improving sentiment recognition robustness and accuracy [97]. Moreover, there is no universally accepted taxonomy for data uncertainty in opinion analysis, and its manifestation varies widely across contexts.

In the era of social media, the volume of multimodal data is exploding, but certain new forms of uncertainty in online public opinion—such as acquisition interference, transmission distortions, and storage inconsistencies—cannot yet be effectively recognized.