Multihead Average Pseudo-Margin Learning for Disaster Tweet Classification

Abstract

During natural disasters, social media platforms, such as X (formerly Twitter), become a valuable source of real-time information, with eyewitnesses and affected individuals posting messages about the produced damage and the victims. Although this information can be used to streamline the intervention process of local authorities and to achieve a better distribution of available resources, manually annotating these messages is often infeasible due to time and cost constraints. To address this challenge, we explore the use of semi-supervised learning, a technique that leverages both labeled and unlabeled data, to enhance neural models for disaster tweet classification. Specifically, we investigate state-of-the-art semi-supervised learning models and focus on co-training, a less-explored approach in recent years. Moreover, we propose a novel hybrid co-training architecture, Multihead Average Pseudo-Margin, which obtains state-of-the-art results on several classification tasks. Our approach extends the advantages of the voting mechanism from Multihead Co-Training by using the Average Pseudo-Margin (APM) score to improve the quality of the pseudo-labels and self-adaptive confidence thresholds for improving imbalanced classification. Our method achieves up to 7.98% accuracy improvement in low-data scenarios and 2.84% improvement when using the entire labeled dataset, reaching 89.55% accuracy on the Humanitarian task and 91.23% on the Informative task. These results demonstrate the potential of our approach in addressing the critical need for automated disaster tweet classification. We made our code publicly available for future research.

1. Introduction

According to the Centre for Research on the Epidemiology of Disasters (CRED), 399 natural hazards were recorded worldwide in 2023, resulting in 86473 fatalities, affecting $93.1$ million people, and causing a total economic loss of $202.7$ billion USD. In 2023, the United States alone was hit by 28 disasters with a billion-dollar damage magnitude, exceeding the previous record of 22 events set in 2020 [1]. In such situations, the lack of information represents a serious obstacle that local authorities face when distributing resources and managing intervention processes. Furthermore, the time-critical constraints associated with the nature of disasters amplify these problems and place additional pressure on the intervention teams.

The most straightforward way to assess damage levels and streamline resource distribution is to gather information from eyewitnesses and victims. However, entering into contact and centralizing large volumes of data are both difficult and time-consuming. An alternative is to use the extensive messages uploaded on X (formerly Twitter) and other social media platforms available for analysis during natural disasters. This method can provide valuable situational awareness while reducing the overhead of data collection and eliminating the need for centralization. In this idea, recent studies collected disaster data from social media and manually annotated them to identify situational awareness categories. Then, they trained machine learning models for these supervised classification tasks [2,3]. However, the manual annotation required for training machine learning models can be expensive and slow. To address this problem, recent studies have explored semi-supervised learning (SSL) approaches to harness the abundance of unlabeled user-generated data while minimizing the reliance on labeled data. While most of the research has focused on single-modality data [4,5], Sirbu et al. [6] introduced the first multimodal unlabeled corpus and an SSL method specifically designed for multimodal disaster tweet classification. Although the work of Sirbu et al. [6] represents a significant milestone, it does not compare multiple SSL methods, potentially overlooking algorithms better suited for this task. Such a comparison is essential not only to identify the most effective methods but also to deepen the understanding of what approaches are best suited for specific scenarios and why.

To respond to this need, we first adapt and explore a variety of SSL methods for the task of multimodal disaster tweet classification. Apart from the two methods from Sirbu et al. [6] that rely on the techniques of pseudo-labeling and consistency regularization (FixMatch [7] and FixMatch LS), we also experiment with two enhancements that solve the global thresholding problem of FixMatch (FlexMatch [8] and FreeMatch [9]), one method that uses a different paradigm for creating pseudo-labels, based on the history of predictions instead of the current logits (MarginMatch [10]), and an approach relying on a different SSL technique, co-training (Multihead Co-Training [11]).

As each SSL paradigm analyzed has its own strengths and weaknesses, we recognize the need for a unified framework that leverages their advantages while addressing their limitations.To this end, we introduce a semi-supervised method that combines the strengths of the other approaches to create a hybrid framework that we call Multihead Average Pseudo-Margin (Multihead APM). Specifically, we extend the Multihead Co-Training algorithm [11]. While the original algorithm uses the agreement between two classification heads to generate pseudo-labels for the third, we argue that hard examples—instances where the two heads fail to agree—carry critical information. To incorporate these cases into the training process, we introduce a method inspired by MarginMatch [10], employing the APM score to effectively handle such examples, as it is less sensitive to noisy predictions. Additionally, building on the insights from FlexMatch [8] and FreeMatch [9], which highlighted the limitations of global thresholding, we propose a self-adaptive thresholding mechanism tailored to our algorithm. The technical background and a detailed explanation of how Multihead APM fits into this context are provided in Section 3.

We analyze the performance of our approach in comparison to the other semi-supervised learning methods on the CrisisMMD [2] tasks, the largest publicly available disaster-related multimodal dataset. Since SSL requires domain-specific unlabeled data, we use the unlabeled corpus introduced by Sirbu et al. [6], containing 122k tweets with text and image in the same domain. The experimental results confirm that our proposed algorithm, leveraging a co-training architecture, achieves state-of-the-art performance on both the Humanitarian and Informative classification tasks in the CrisisMMD dataset [2]. Notably, compared to the supervised baseline, Multihead APM delivers up to 7.98% improvement with only 50 labeled examples per class and a 2.84% gain when utilizing the entire dataset. At the same time, it delivers important improvements compared to all the other SSL methods we have explored.

Our contributions are summarized as follows:

1.

We are the first to adapt and compare multiple semi-supervised learning methods on multimodal disaster tweet classification tasks.

2.

We introduce a new SSL method by extending the Multihead Co-Training framework with self-adaptive thresholds based on the Average Pseudo-Margins score.

3.

We achieve state-of-the-art results on both the Humanitarian as well as the Informative CrisisMMD [2] classification tasks, and we prove the feasibility of our approach in low-resource settings.

4.

We provide an in-depth analysis of the training process and failure cases for the baseline SSL methods in the context of highly imbalanced classes.

This paper continues with a section on related work, establishing the context and foundation for this study. This is followed by a detailed description of the methods employed in the proposed approach. The fourth section outlines the experimental setup, providing the necessary details for reproducibility. The results are then presented and discussed in the fifth section, highlighting the key findings. The sixth section delves deeper into the analysis of confidence thresholds, offering insights into their unexpected behavior. Finally, this paper concludes with a summary of the findings.

2. Related Work

2.1. Semi-Supervised Learning

Semi-supervised learning is a machine learning technique that utilizes labeled data together with large amounts of unlabeled data to train models. This approach is particularly useful when the labeled data are either scarce or difficult to obtain (i.e., high annotation costs, lack of time or skilled annotators, etc.).

According to Yang et al. [12], SSL algorithms can be grouped into several categories, based on loss function and model design. Their taxonomy includes categories for model types such as generative methods and graph-based approaches, as well as categories reflecting the SSL techniques employed, such as consistency regularization [13,14] or pseudo-labeling [15]. The core principle of consistency regularization is that realistic perturbations applied to data points should not alter the predictions of the model. Meanwhile, pseudo-labeling methods utilize high-confidence predictions as pseudo-labels, which are incorporated into the training dataset as labeled examples. Finally, hybrid methods form a category that integrates multiple SSL techniques to leverage their combined strengths.

A key representative of this category is FixMatch [7], one of the most successful SSL algorithms, which combines consistency regularization and pseudo-labeling. In this approach, the model first generates pseudo-labels for weakly augmented unlabeled images. Then, consistency regularization minimizes the cross-entropy loss between the predicted pseudo-labels and the output of the model on strongly augmented images. Given its effectiveness across a wide range of tasks [16,17,18], recent years have seen a surge in methods building upon FixMatch, with the research community increasingly focused on improving the two key concepts—consistency regularization and efficient pseudo-labeling.

For example, FlexMatch [8] uses the same concepts as FixMatch but additionally tries to solve the problem of imbalanced class distributions in the dataset. While FixMatch uses a static confidence threshold to filter out the unlabeled samples that should not be included in the unlabeled loss, FlexMatch proposes the use of an individual threshold for each class and adjusts the thresholds at each time step.

FreeMatch [9] further improves the adaptive threshold used in FlexMatch by splitting it into two components: a local and a global threshold. The local threshold is computed for each class independently, whereas the global threshold is common for all classes. Both are adaptive, and the final class threshold is computed by combining the local and global class thresholds.

MarginMatch [10] extends FlexMatch by introducing a new indicator to filter out the “untrustworthy” unlabeled samples: Average Pseudo-Margin (APM). Inspired by the Area Under the Margin [19], the APM score is a value calculated per sample, which keeps track of the model’s historical predictions on that particular sample. As opposed to the class threshold filters used in FlexMatch and FreeMatch that rely solely on the model’s prediction for the current iteration, the APM score incorporates information from all past epochs.

Another classical SSL technique is co-training [20], a disagreement-based method where multiple models are trained in parallel using different views of the data. Deep Co-Training [21] adapts the method to deep learning by using only one data source and relies on adversarial examples to ensure that models use different views of the data, which is necessary to avoid individual classifiers from collapsing into each other. JointMatch [22] extends the classical co-training framework by incorporating additional techniques: the consistency regularization introduced by FixMatch is employed using cross-labels instead of pseudo-labels and adaptive local thresholds are used similarly to FreeMatch. Additionally, the algorithm introduces a weighting mechanism for disagreement examples in the loss function, as these examples are shown to be more informative and keep the two networks diverged.

Multihead Co-Training [11] solves the drawback of having to train multiple classifiers by proposing a multihead architecture where each head uses the remaining ones to predict the pseudo-labels for a given data sample. If the heads reach an agreement, the sample is included in the unlabeled loss; otherwise, it is excluded.

Building on the strengths of these methods, we propose Multihead Average Pseudo-Margin (APM), a novel approach that combines the multihead architecture of Multihead Co-Training with the APM-based filtering mechanism introduced by MarginMatch and a self-adaptive thresholding strategy inspired by FreeMatch. This combination enables Multihead APM to address challenges such as class imbalance and unreliable pseudo-labels more effectively.

2.2. Disaster Tweet Classification

The relationship between deep learning and disaster tweets is not new. In fact, several studies have researched the ability to use the vast amounts of data generated by victims and eyewitnesses on social media platforms to streamline the intervention process of local authorities. Some of these studies focused on the information within text messages [23,24,25,26], whereas others tried to make use of the uploaded images [27,28,29,30,31]. More recently, with the observation that unimodal approaches lose sight of valuable information—since many disaster tweets contain both text and images—the attention has shifted toward multimodal models that aim to integrate information across both modalities [32,33,34,35,36,37,38]. These models typically utilize parallel architectures, such as combining a CNN-based image encoder with an RNN or transformer-based text encoder, followed by a fusion mechanism to integrate the two modalities. For example, Agarwal et al. [34] introduced a gated multimodal model that utilizes a Recurrent Convolutional Neural Network (RCNN) to process textual data [39] and an Inception-v3 network [40] for image features. These were integrated using an attention mechanism [41] to enhance the fusion of features and applied to multiple damage-related tasks defined by Alam et al. [2]. Abavisani et al. [35] proposed a framework that pairs DenseNet [42] for image encoding with BERT [43] for textual representation, incorporating a cross-attention module to suppress irrelevant information from either modality. Khattar and Quadri [44] proposed the Cross-Attention Multimodal (CAMM) framework, designed to leverage complementary information from the tweet text and images. The cross-attention mechanism is applied to text features extracted using a Bi-LSTM [45] and image features extracted using a VGG-16 network [46]. Koshy and Elango [47] introduced a transformer-based multimodal deep learning framework aimed at classifying informative tweets during disasters. Their approach employs a Vision Transformer [48] to extract visual features and a fine-tuned RoBERTa model [49] to extract word embeddings, followed by a Bi-directional LSTM [45] to capture contextual dependencies. A multimodal fusion module is then used to obtain the joint representation of the extracted features.

However, all of these methods rely on a relatively small number of manually labeled examples rather than taking advantage of the huge amounts of unlabeled data generated by social media users during natural disasters. Some studies [5,50] have applied semi-supervised learning to disaster tweets, but their focus was limited to textual data. Sirbu et al. [6] were the first to demonstrate the significant improvements that SSL algorithms can achieve over the supervised counterparts for this task when integrated with multimodal architectures for disaster tweet classification. They adapted FixMatch to a multimodal setup and introduced FixMatch LS, an enhancement that uses soft pseudo-labels and a linear schedule for the unlabeled loss coefficient. Although their work showed the feasibility of FixMatch and its enhancement for disaster tweet classification, it lacked a comparison to other semi-supervised algorithms and techniques. As a result, we are the first to adapt and compare several semi-supervised algorithms for this multimodal task, as well as the first to introduce a co-training approach in the context of disaster tweet classification.

3. Methods

In this section, we first describe the baselines we have employed—both supervised and semi-supervised. Then, we introduce Multihead APM, our proposed method built upon the Multihead Co-Training framework, which leverages the APM score to generate high-confidence pseudo-labels and introduces self-adaptive thresholds to address the class imbalance. The semi-supervised baselines also provide the theoretical background and justification for creating Multihead APM.

3.1. Baseline Supervised Models

Following Sirbu et al. [6], we adopt the Multimodal Bitransformer (MMBT) [51] as our base architecture, given its ability to simultaneously leverage text and image modalities for disaster tweet classification. This model has demonstrated strong performance on this task in both fully supervised [37] and semi-supervised setups [6].

To establish a stronger supervised baseline, we integrate weak text augmentation using EDA [52] and refer to the resulting model as MMBT Supervised Aug. While other multimodal models could have been investigated as potential baselines, doing so would have incurred additional GPU costs and falls beyond the scope of this work. Our primary focus is on analyzing semi-supervised approaches rather than benchmarking various multimodal architectures.

3.2. Baseline Semi-Supervised Methods

3.2.1. FixMatch for Multimodal Data

In order to make use of a large number of unlabeled data samples, FixMatch [7] heavily relies on two important principles in SSL: pseudo-labeling and consistency regularization [13,14]. Similar to Sirbu et al. [6], we adapt FixMatch to the multimodal setup by introducing two new augmentation functions: $\beta (·)$ (used for weak text augmentations) and $\mathcal{B}(·)$ (used for strong text augmentations). As a result, the multimodal supervised loss is defined as follows:

$$l_s={\displaystyle \frac{1}{B}}\sum _{b=1}^{B}H(p_b,p_m\left(y\right|\alpha \left(x_b^{img}\right),\beta \left(x_b^{txt}\right)))$$

(1)

while the multimodal unsupervised loss is the following:

$$l_u={\displaystyle \frac{1}{\mu B}}\sum _{b=1}^{\mu B}{\mathbb{1}}_{\tau }\left(q_b\right)H(\widehat{q_b},Q_b)$$

(2)

where B is the batch size; $\mu$ is the labeled-to-unlabeled data ratio; H is the cross-entropy loss; ($x_b^{img}$, $x_b^{txt}$) is an (image, text) pair from the labeled dataset; ($u_b^{img}$, $u_b^{txt}$) is an (image, text) pair from the unlabeled dataset; $p_m\left(y\right|·)$ is the model’s prediction (probability distribution over all classes) for a given input; $q_b=p_m\left(y\right|\alpha \left(x_b^{img}\right),\beta \left(x_b^{txt}\right))$; $Q_b=p_m\left(y\right|\mathcal{A}\left(u_b^{img}\right),\mathcal{B}\left(u_b^{txt}\right))$; $\alpha (·)$ is the weak image augmentation function; $\beta (·)$ is the weak text augmentation function; $\mathcal{A}(·)$ is the strong image augmentation function; and $\mathcal{B}(·)$ is the strong text augmentation function. $p_b$ is the one-hot encoding of the target vector, while ${\displaystyle \widehat{q_b}=arg\underset{y}{max}\left(q_b\right)}$ is the hard pseudo-label of the weakly augmented input sample and ${\mathbb{1}}_{\tau }\left(q_b\right)=max\left(q_b\right)>\tau$ is the filtering function using the static threshold $\tau$.

The final loss is $l=l_s+{\lambda }_u·l_u$, where ${\lambda }_u$ is a fixed scalar denoting the weight of the unlabeled loss.

3.2.2. FixMatch LS

Sirbu et al. [6] provides two enhancements that improved the classification performance of FixMatch on the CrisisMMD tasks. The first change replaces hard pseudo-labels with soft ones to incorporate information about the entire probability distribution in the pseudo-labels. This enhancement proves to be useful for tweets with a strong semantic overlap between multiple classes. The second change uses a linear unlabeled loss weight, instead of a fixed one, which should decrease the unlabeled loss during the first iterations of the algorithm, when the model’s predictions are still not qualitative. The updated unlabeled and total loss formulas are as follows:

$$l_u={\displaystyle \frac{1}{\mu B}}\sum _{b=1}^{\mu B}H(q_b,Q_b)$$

(3)

$$l=l_s+{\lambda }_u\left(t\right)·l_u$$

(4)

where ${\lambda }_u\left(t\right)=2t$, and t is the epoch number.

3.2.3. FlexMatch

FlexMatch [8] addresses the drawbacks encountered by FixMatch due to using a static threshold $\tau$ by replacing it with an adaptive one.

In order to achieve this goal, FlexMatch analyzes the correlation between the learning difficulty of a class and the count of its pseudo-labels that exceed the static threshold. A class with a large number of samples that exceed the threshold is considered to have a small learning difficulty, hence the need to use a high threshold for it. On the other hand, a class having a reduced number of samples above the threshold is difficult to learn and requires lowering the filtering threshold.

To this end, FlexMatch calculates a threshold for each individual class and dynamically changes it after each training iteration. To define the adaptive class thresholds, it first computes the number of pseudo-labels for each class that exceeds the static threshold:

$${\sigma }_t\left(c\right)=\sum _{b=1}^{\mu B}{\mathbb{1}}_{\tau }\left(q_b\right)·\mathbb{1}(\widehat{q_b}=c)$$

(5)

Therefore, the adaptive class threshold can be expressed as follows:

$${\mathcal{T}}_t\left(c\right)={\beta }_t\left(c\right)·\tau ={\displaystyle \frac{{\sigma }_t\left(c\right)}{{\displaystyle \underset{c}{max}\{{\sigma }_t\left(c\right)\}}}}·\tau$$

(6)

where $\tau$ is a hyperparameter denoting the static confidence threshold. ${\beta }_t\left(c\right)$ is used to normalize the pseudo-label count distribution and ensures that each adaptive class threshold is in the range $[0,\tau ]$.

During the first iterations, FlexMatch introduces the concept of threshold warm-up to decrease the confirmation bias. Thus, the samples that have not exceeded the confidence threshold for any class are introduced into a new virtual class, and the new normalization formula becomes the following:

$${\beta }_t\left(c\right)={\displaystyle \frac{{\sigma }_t\left(c\right)}{max{\displaystyle \{\underset{c}{max}\left\{{\sigma }_t\left(c\right)\right\},\mu B-\sum _c{\sigma }_t\left(c\right)\}}}}$$

(7)

Finally, FlexMatch employs a non-linear mapping function $\mathcal{M}(·)$, which has a higher sensitivity for large ${\beta }_t\left(c\right)$ values and vice versa. This is used to reduce threshold fluctuations during early training iterations when the model is still unstable. The final threshold formula is ${\mathcal{T}}_t\left(c\right)=\mathcal{M}\left({\beta }_t\left(c\right)\right)·\tau$. While several convex functions can be used for $\mathcal{M}(·)$, we employ the one proposed in the FlexMatch paper: $\mathcal{M}\left(x\right)={\displaystyle \frac{x}{2-x}}$.

By adapting FlexMatch to the multimodal setup, we obtain the multimodal loss formulas used in this work:

$$l_s={\displaystyle \frac{1}{B}}\sum _{b=1}^{B}H(p_b,p_m\left(y\right|\alpha \left(x_b^{img}\right),\beta \left(x_b^{txt}\right)))$$

(8)

$$l_{u,t}={\displaystyle \frac{1}{\mu B}}\sum _{b=1}^{\mu B}\mathbb{1}(max\left(q_b\right)>{\mathcal{T}}_t\left(\widehat{q_b}\right))H(\widehat{q_b},Q_b)$$

(9)

$$l=l_s+{\lambda }_u·l_{u,t}$$

(10)

3.2.4. FreeMatch

FreeMatch [9] aims to improve the adaptive threshold used in FlexMatch by introducing a new technique called self-adaptive thresholding. According to Wang et al. [9], the thresholding scheme used in FlexMatch heavily relies on a fixed global threshold to compute the adaptive class thresholds and may struggle to adjust to the model’s learning progress. This happens especially when the labeled data are scarce and provides limited supervision for the model. To solve this problem, FreeMatch splits the threshold into two components: a global part and a local one.

The self-adaptive global threshold reflects the confidence of the model on the whole unlabeled dataset. Ideally, it should increase over time to exclude low-quality pseudo-labels in late training iterations:

$${\tau }_t=\left\{\begin{array}{cc}{\displaystyle \frac{1}{C}},\hfill & \mathrm{if}t=0\hfill \\ \lambda {\tau }_{t-1}+(1-\lambda ){\displaystyle \frac{1}{\mu B}}\sum _{b=1}^{\mu B}max\left(q_b\right),\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(11)

where C is the number of classes and $\lambda \in (0,1)$ is the momentum decay of the exponential moving average (EMA).

The self-adaptive local threshold reflects the confidence of the model on the unlabeled dataset for a specific class:

$${\tilde{p}}_t\left(c\right)=\left\{\begin{array}{cc}{\displaystyle \frac{1}{C}},\hfill & \mathrm{if}t=0\hfill \\ \lambda {\tilde{p}}_{t-1}\left(c\right)+(1-\lambda ){\displaystyle \frac{1}{\mu B}}\sum _{b=1}^{\mu B}{q}_b\left(c\right),\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(12)

By combining the global and local thresholds, the final self-adaptive class thresholds are obtained:

$${\tau }_t\left(c\right)={\displaystyle \frac{\widetilde{p_t}\left(c\right)}{{\displaystyle \underset{c^{\prime }}{max}\left(\widetilde{p_t}\left(c^{\prime }\right)\right)}}}·{\tau }_t$$

(13)

Using the new class thresholds, the unlabeled loss formula adapted for the multimodal setup becomes the following:

$$l_{u,t}={\displaystyle \frac{1}{\mu B}}\sum _{b=1}^{\mu B}\mathbb{1}(max\left(q_b\right)>{\tau }_t\left(\widehat{q_b}\right))H(\widehat{q_b},Q_b)$$

(14)

Additionally, FreeMatch introduces a new loss member called Self-Adaptive Fairness (SAF), which helps the model converge faster and make more diverse predictions:

$$\widetilde{h_t}=\lambda {\tilde{h}}_{t-1}+(1-\lambda ){\mathrm{Hist}}_{\mu B}\left(\widehat{q_b}\right)$$

(15)

$$\overline{p}={\displaystyle \frac{1}{\mu B}}\sum _{b=1}^{\mu B}\mathbb{1}\left(max\left(q_b\right)\ge {\tau }_t\left(\widehat{q_b}\right)\right)Q_b,$$

(16)

$$\overline{h}={\mathrm{Hist}}_{\mu B}\left(\mathbb{1}\left(max\left(q_b\right)\ge {\tau }_t(\widehat{q_b}\right)){\widehat{Q}}_b\right).$$

(17)

$$l_{f,t}=-H(\mathrm{SumNorm}\left({\displaystyle \frac{\widetilde{p_t}}{\widetilde{h_t}}}\right),\mathrm{SumNorm}\left({\displaystyle \frac{\overline{p}}{\overline{h}}}\right))$$

(18)

where $Hist$ is the histogram distribution function, ${\displaystyle \widehat{Q_b}=arg\underset{y}{max}\left(Q_b\right)}$, and $\mathrm{SumNorm}(·)={\displaystyle \frac{(·)}{\sum (·)}}$. Adding SAF, the final loss formula is defined as follows:

$$l=l_s+{\lambda }_u{l}_{u,t}+{\lambda }_f{l}_{f,t}$$

(19)

3.2.5. MarginMatch

MarginMatch [10] uses the same adaptive threshold presented in FlexMatch but additionally claims that the confidence checks performed on the pseudo-labels offer a limited amount of information, as they rely solely on the current model’s predictions. In order to gain full insight into the predictions from previous iterations, MarginMatch introduces an additional filter based on the Area Under the Margin (AUM) [19].

MarginMatch adapts the AUM score to unlabeled data, calling this new score the Average Pseudo-Margin (APM). Similar to the margin, the pseudo-margin (PM) proposed by MarginMatch is calculated per data sample after each epoch t. For a particular class c, the pseudo-margin is defined as the difference between the logit of class c and the greatest logit of a different class:

$${\displaystyle P{M}_c^{\left(t\right)}=z_c^{\left(t\right)}-\underset{i\ne c}{max}{z}_i^{\left(t\right)}}$$

(20)

The APM for a particular sample is defined as the average of the previously computed pseudo-margins across all epochs up to the current one:

$$AP{M}_c^{\left(t\right)}\left(x\right)={\displaystyle \frac{1}{t}}\sum _{i=1}^{t}P{M}_c^{\left(i\right)}\left(x\right)$$

(21)

Judging by the above formula, it is straightforward that the APM score incorporates information from all past epochs. Furthermore, some important properties of the APM need to be mentioned. A negative APM score for a class c indicates that the predictions for this sample have fluctuated between different classes in the past and the historical confidence for class c is low. On the other hand, a positive score showcases a higher level of confidence and guarantees that class c has been selected as a pseudo-label in previous epochs.

MarginMatch further adapts the APM formula to an exponential moving average (EMA) setting similar to FreeMatch:

$$AP{M}_c^{\left(t\right)}\left(x\right)=P{M}_c^{\left(t\right)}\left(x\right)·{\displaystyle \frac{\delta }{1+t}}+AP{M}_c^{(t-1)}·(1-{\displaystyle \frac{\delta }{1+t}})$$

(22)

where $\delta$ is the smoothing factor.

MarginMatch uses the APM score as an additional filter to the one already used by FlexMatch before introducing a data sample in the unlabeled loss. The unlabeled loss formula, after adapting the model to a multimodal setting, becomes the following:

$$l_{u,t}={\displaystyle \frac{1}{\mu B}}\sum _{b=1}^{\mu B}{\mathbb{1}}_{APM}^{\left(t\right)}·{\mathbb{1}}_{FLEX}^{\left(t\right)}·H(\widehat{q_b},Q_b)$$

(23)

where ${\mathbb{1}}_{APM}^{\left(t\right)}=\mathbb{1}(AP{M}_{\widehat{q_b}}^{\left(t\right)}\left(q_b\right)>{\gamma }^{\left(t\right)})$, and ${\gamma }^{\left(t\right)}$ is the APM threshold for the current iteration computed by introducing a virtual class of erroneous examples that mimics the training dynamics of incorrect pseudo-labels and ${\mathbb{1}}_{FLEX}^{\left(t\right)}=\mathbb{1}(max\left(q_b\right)>{\mathcal{T}}_t\left(\widehat{q_b}\right))$.

3.2.6. Multihead Co-Training

Multihead Co-Training [11] adapts the Deep Co-Training [21] framework to use a single model with multiple classification heads on top. Pseudo-labels are calculated independently for each classification head using the weakly augmented version of the data (similar to FixMatch). Afterwards, each individual classification head uses the remaining ones to predict its pseudo-labels, based on head agreement. If a majority element is found among the pseudo-labels predicted by the remaining heads, an agreement is reached and this pseudo-label is included in the unsupervised loss for the current head. In particular, for three heads, an agreement is reached if and only if the remaining two heads will both predict the same pseudo-label. The decision process employed for selecting a pseudo-label is shown in Figure 1, when the two skip paths are active. The unsupervised loss is calculated for each head independently as the cross-entropy loss between the head’s pseudolabel—determined using the head agreement strategy—and the strongly augmented version of the sample. The final unsupervised loss is computed as the average of the individual unsupervised losses from all heads.

We adapt Multihead Co-Training to a multimodal setup by defining the multimodal losses for a three-headed model:

$$l_s={\displaystyle \frac{1}{B}}\sum _{b=1}^B\sum _{m\in \{1,2,3\}}H(p_b,p_m\left(y\right|\alpha \left(x_b^{img}\right),\beta \left(x_b^{txt}\right)))$$

(24)

$$l_u={\displaystyle \frac{1}{3}}\sum _{m\in \{1,2,3\}}{\displaystyle \frac{1}{{\sum }_{b=1}^{\mu B}{\mathbb{1}}_{b,m}}}\sum _{b=1}^{\mu B}{\mathbb{1}}_{b,m}H({\overline{q}}_{b,m},Q_b)$$

(25)

where $p_m$ is the head classifier with index m; ${\mathbb{1}}_{b,m}=[{\widehat{q}}_{b,i}={\widehat{q}}_{b,j}],\{i,j\}=\{1,2,3\}\setminus \left\{m\right\}$ marks if the remaining two heads have reached an agreement; and ${\overline{q}}_{b,m}={\widehat{q}}_{b,i},i\ne m$ is the pseudo-label generated by one of the other heads. The supervised loss is computed as the sum of cross-entropy losses for each individual classification head, and the unsupervised loss is very similar to the one used in FixMatch, the main differences being that the confidence threshold was replaced with the head agreement technique and the cross-entropy loss is applied only on the samples that have reached an agreement.

3.3. Multihead Average Pseudo-Margin

Our main contribution is that we introduce a new semi-supervised learning method called Multihead Average Pseudo-Margin (Multihead APM) that combines co-training with MarginMatch [10]. For this, we draw inspiration and the best takeaways from the other approaches while using the Multihead Co-Training [11] framework as our backbone. This allows us to combine benefits from all three SSL techniques analyzed: co-training, pseudo-labeling, and consistency regularization. At the same time, we identify and fix a couple of shortcomings of the original approach. In addition, we perform a rigorous comparison of Multihead APM to several state-of-the-art semi-supervised learning algorithms for disaster tweet classification tasks, showing the benefits of our proposed method.

First, we notice that in the late-stage training process of Multihead Co-Training, the heads reach a very high degree of agreement, resulting in ineffective filtering of the unlabeled examples, with almost all of them being used in the loss computation. As this is a result of relying only on the agreement of the heads for filtering the samples, this can be solved by introducing adaptive thresholding techniques, similar to FlexMatch [8] or FreeMatch [9]. However, MarginMatch [10] has demonstrated the importance of historical predictions in filtering out untrustworthy data, so we decide to use the APM indicator. To exclude some of the samples where the heads have reached an agreement, we propose the following heuristic. If at least one head has the APM score for the selected pseudo-label above the current threshold, we keep the sample inside the unsupervised loss; otherwise, we drop it.

Second, Jointmatch [22] has shown that head disagreements should not be ignored, since they provide challenging examples that could improve the model’s performance, and they prevent the heads from converging or becoming indistinguishable. To incorporate the above idea into our architecture, we also use the APM score. If exactly one of the two remaining heads has an APM score above the threshold, the sample is included in the unsupervised loss using the pseudo-label of that head. Despite the disagreement, we use the APM score as a tiebreaker for selecting the pseudo-label. If both heads exceed the threshold or neither of them does, we cannot choose between the two pseudo-labels, and the sample is ignored.

Finally, we define a new method for selecting the APM thresholds. While we could use a virtual class of erroneous examples similar to MarginMatch, we prefer to take advantage of our three-headed co-training architecture and simply define the samples for which agreement is reached between the heads as the correctly classified examples that can be used for computing the self-adaptive thresholds. Moreover, as there can be large differences between the APM values belonging to different classes, we also draw inspiration from FreeMatch [9] and define our self-adaptive thresholds for each class and update them after each training epoch:

$${\displaystyle {\gamma }_m^{\left(t\right)}\left(c\right)={\mathrm{percentile}}_f\left({\cup }_{i\ne m}\{AP{M}_c^{\left(t\right)}\left(q_b\right)\mid {\mathbb{1}}_{b,m}\wedge {\widehat{q}}_{b,i}=c\}\right)}$$

(26)

where f is a hyperparameter denoting the percentile and m and c are the current classification head and class. The APM thresholds are calculated individually for each $(head,class)$ pair as the bottom $f^{th}$ percentile of the reunion of APM scores for all remaining classification heads, where an agreement is reached and the selected pseudo-label is equal to the current class.

In the next section, we present the experiments conducted with different versions of the currently proposed approach:

• APM is used only for disagreements, and we set an additional threshold lower bound of 0 (Multihead APM No Agree).
• APM is used for both agreements and disagreements, and we set an additional threshold lower bound of 0 (Multihead APM Agree).
• APM is used only for disagreements without a lower bound (Multihead APM No Agree No Low).
• APM is used for both agreements and disagreements with no lower bound (Multihead APM Agree No Low).

The decision process employed for selecting a pseudo-label is shown in Figure 1. For Multihead APM Agree, no skip path is active, while for Multihead APM No Agree, the green skip path is followed. All the variants described above yield strong results. However, as expected, the versions that use the APM for both agreements and disagreements outperform the other ones, since they reduce the number of unlabeled samples that are used in the final iterations of the training process. In the next section, we will perform a detailed analysis of our setup and present our experimental results.

4. Experiments

4.1. Datasets

4.1.1. Labeled Data

CrisisMMD [2] is a multimodal dataset consisting of approximately 18,000 manually annotated tweets containing both text and images, collected from 7 natural disasters that occurred during the year 2017: Hurricane Irma, Hurricane Harvey, Hurricane Maria, the California wildfires, the Mexico earthquake, the Iraq–Iran earthquake, and the Sri Lanka floods. The dataset was designed for three distinct classification tasks, Informativeness, Humanitarian Categories, and Damage Severity, each offering a different perspective on the current state of the disaster.

The first task is to divide the tweets into two different categories, Informative and Not Informative, which distinguish useful tweets that should be analyzed by the intervention teams from those that do not convey important information. The second task aims to categorize the relevant tweets into eight distinct humanitarian categories. However, for the multimodal setup, the tweets were grouped by Ofli et al. [53] into only five categories that we also use: Affected individuals; Infrastructure and utility damage; Rescue, volunteering, or donation effort; Other relevant information; and Not humanitarian. The third task is to assess the severity of the damage as Little or no damage, Mild damage, or Severe damage. However, this task is annotated only based on the images, so it is not available for the multimodal setup. Therefore, we exclude it from our experiments.

The official train–validation–test splits for the multimodal setup were later released by Ofli et al. [53], so we use this version of the dataset in our study.

4.1.2. Unlabeled Data

To maintain consistency in our choices and simplify the comparison of our results with previous work, we use the same unlabeled dataset introduced by Sirbu et al. [6], which consists of 122K text and image pairs sampled using the Twitter streaming API during disasters that occurred in 2017. The tweets have been filtered in advance to avoid duplicates, retweets, and posts with multiple attached images. This dataset is further denoted as the unlabeled corpus and has been used in all our semi-supervised algorithm benchmarks.

4.2. Augmentation

As consistency regularization is the technique of training the model to predict similar outputs for augmented versions of the same input, it is clear that augmentation is an important part of the studied semi-supervised approaches. Moreover, most of them use pseudo-labeling in a manner similar to FixMatch [7], where the pseudo-label is computed on a weakly augmented version of the input and then used as a target for the strongly augmented version of the input. This underlines the importance of using not just one but two augmentation functions of different strengths.

As the baseline semi-supervised approaches have been introduced for images, it is easy for us to follow them and use RandAugment [54] for strong image augmentation and as weak image augmentation a standard flip-and-shift strategy with $50\%$ horizontal flip probability and up to $12.5\%$ horizontal or vertical translation.

For text augmentation, previous work [6] explored two different approaches: Easy Data Augmentation [52] (EDA) and back-translation [55]. EDA is a simple augmentation technique that randomly applies 4 possible operators to a text: synonym replacement, random insertion of a word, random swap of two words, and random deletion of a word. On the other hand, back-translation follows a more complex strategy: it translates the sentence to an intermediate language and then back to the original one. Thus, a different version of the sentence is obtained while preserving the underlying message. While both augmentation strategies showed comparable performance in Sirbu et al. [6], EDA offers the critical advantage of being run on a CPU without the need for a pre-trained generative model for machine translation that additionally requires GPU time. As a result, we employ EDA as our text augmentation strategy. For weak augmentation, we apply EDA to $10\%$ of the words in the sentence, whereas for strong augmentation, we apply it to $20\%$ of the words.

4.3. Low-Data Regimes

Low-data regimes refer to settings where training data are scarce or hard to obtain. For example, during natural disasters, only a small number of user-generated messages can be annotated due to time-critical constraints. This setup is compatible with a low-data setting, where labeled data are very limited, even though unlabeled data are widely available.

Previous work [6] proposed to mimic a low-data setting by considering that only a smaller portion of the labeled dataset is available at the time of training a model. Similar to them, we evaluate our algorithms on the Humanitarian and Informative tasks using only 50 labeled samples per class, which would better mimic the resources available during an active crisis event.

This final step in our experiments would help us understand the performance improvements that can be achieved in settings close to the real world. As the experiments show that the Multihead APM Agree No Low configuration performs the best among the Multihead APM variations presented in Section 3.3, we additionally test this version of the Multihead APM in low-data regimes to quantify the performance boost generated by unlabeled data in such settings.

4.4. Experimental Setup

To ensure that our experimental results are not influenced by non-deterministic factors, we run each experiment three times and report the average results.

We evaluate all the algorithms described in Section 3 on both the Humanitarian and the Informative tasks (as the Damage Severity task does not provide a multimodal setup). Each training process, using the entire labeled dataset, has an average execution time of 4h30m for 20 training epochs. Considering the total training time for the 11 proposed algorithms, we obtain 297 GPU hours or approximately 12 days. Including all the experimental settings—used for hyperparameter tuning, statistics accumulation, or other purposes—that have been omitted from this paper, the total execution time would be about double. All of our experiments were conducted in PyTorch, on a Linux system with an NVIDIA A100 GPU with 80 GB VRAM.

4.5. Hyperparameters

USB [56] is a benchmark for evaluating SSL algorithms across vision, language, and audio tasks. Building on this, we extend the USB codebase by implementing our Multihead APM approach, incorporating the multimodal tasks from CrisisMMD, and adapting the evaluated methods to the multimodal setup.

Each of the simulated algorithms has a distinct set of hyperparameters and configuration properties. We experiment with several values and choose the ones that generate the best F1 scores on the validation set while also keeping in mind the required computational power and the hardware limitations. The hyperparameters used are presented in

Table 1. Hyperparameters used in all experiments.

Hyperparameter	Used Value	Details
Ratio of unlabeled to labeled data, $\mu$	3	Computational limitations, as 7 should work better [7].
Unlabeled loss coefficient, ${\lambda }_u$	1	Compatibility to previous work [7].
Image size	224 × 224	Required size for the MMBT network.
Labeled batch size, B	8	Computational limitations.
Number of epochs, $N_e$	20	Sufficient for all the models to converge.
Learning rate, $lr$	${10}^{-5}$
Optimizer	AdamW
Layer decay rate	$0.5$
Weight decay rate	$5\times {10}^{-4}$
Confidence threshold, $\tau$	$0.95$	For FixMatch and FlexMatch.
EMA momentum, $\lambda$	0.999	For FreeMatch. Experimented with $\{0.9,0.95,0.99,0.999\}$.
Smoothing factor $\delta$	0.997	For MarginMatch.
Fixed APM cutoff	0	For MarginMatch.
Number of heads	3	For Multihead Co-Training and Multihead APM.
Percentile threshold, f	$5\%$	For Multihead APM. Experimented with $\{5\%,10\%,95\%\}$.

Note: The upper section contains the hyperparameters shared by all the methods. The lower section contains the algorithm-specific parameters. Missing details represent that the hyperparameter was chosen for compatibility with previous work—[6] for the shared ones and the introducing paper for the others.

. For the shared ones, we use the same values to ensure a fair comparison.

4.6. Efficiency Considerations

While SSL algorithms may bring a substantial performance boost to a supervised model by utilizing additional unlabeled data, this usually results in slower training, as more data need to be passed through the model. Specifically, for a supervised model, the total training time $\mathcal{T}$ is directly proportional to the number of epochs $N_e$ and the number of labeled examples $N_L$ (i.e., $\mathcal{T}\approx N_L·N_e$). However, for an SSL approach, as the total data passed through the model each epoch are $(\mu +1)·N_L$, the total training time becomes $\mathcal{T}\approx (\mu +1)·N_L·N_e$. This is valid for all the SSL algorithms used in this work.

Furthermore, some of the design choices may bring additional performance overhead. For example, a traditional co-training method requires double the time and memory, as it trains two models in parallel. While Multihead Co-Training addresses this by using a shared backbone, the use of three heads still implies passing data through two additional classification layers. However, the classification heads are usually small, and the impact on performance varies relative to the backbone used. In our experiments, with MMBT as a shared backbone, the performance overhead of Multihead Co-Training and Multihead APM is consistently less than $1\%$ in terms of additional training time compared to FixMatch. Similarly, computing the APM score or the adaptive thresholds introduces a slight overhead, but based on our experiments, it is negligible relative to the forward and backward passes through the model. Multihead APM, while incorporating these techniques, has a similar training time.

During inference, a model trained with SSL behaves similarly to the same model trained in a supervised manner, as the same amount of data are passed through the model, and the adaptive thresholds are no longer updated. However, the overhead associated with using a multihead architecture still remains. In our experiments, the prediction during inference was computed by averaging the logits from all three heads. This results in preserving a slight overhead (∼1%) compared to single-headed architectures, which may be relevant for real-world applications. However, the overhead may be eliminated post-training by using a single head during inference. This comes at a performance cost of ∼0.1% on our tasks, as the three heads reach a very high agreement during the final training stages.

5. Results

In this section, we present the results for the Humanitarian and Informative classification tasks, analyzing the key factors that contributed to the success or limitations of the various algorithms. We focus on the performance of our proposed method, Multihead APM, comparing the four variations and providing an explanation for the best-performing results. The comprehensive results for all the methods, using the full labeled dataset, are summarized in

Table 2. Classification results for the Humanitarian and Informative CrisisMMD tasks.

Model	Humanitarian				Informative
	Acc	P	R	F1	Acc	P	R	F1
MMBT Supervised Aug  [51]	$86.71$	$87.20$	$86.75$	$86.74$	$89.44$	$90.07$	$90.06$	$89.87$
FixMatch [7]	$88.55$	$88.87$	$88.59$	$88.51$	$89.96$	$89.91$	$90.00$	$89.91$
FixMatch LS [6]	$88.66$	$89.04$	$88.70$	$88.74$	$90.38$	$90.35$	$90.42$	$90.36$
FlexMatch [8]	$85.19$	$87.88$	$85.24$	$86.11$	$90.05$	$90.09$	$90.09$	$90.08$
FreeMatch [9]	$84.88$	$86.93$	$84.93$	$85.54$	$90.35$	$90.52$	$90.39$	$90.43$
MarginMatch  [10]	$87.39$	$88.48$	$87.43$	$87.54$	$89.86$	$90.12$	$89.86$	$89.94$
Multihead Co-Training [11]	$88.34$	$88.68$	$88.38$	$88.28$	$90.68$	$90.66$	$90.71$	$90.67$
Multihead APM Agree	$89.34$	89.74	$89.37$	$89.34$	$90.81$	$90.82$	$90.84$	$90.70$
Multihead APM Agree No Low	89.55	$89.73$	89.58	89.55	91.23	91.21	91.27	91.22
Multihead APM No Agree	$88.61$	$88.99$	$88.64$	$88.65$	$90.71$	$90.69$	$90.74$	$90.63$
Multihead APM No Agree No Low	$88.91$	$89.10$	$88.96$	$88.89$	$90.51$	$90.60$	$90.55$	$90.39$

Note: Acc represents the accuracy of the model, whereas P, R, and F1 denote the weighted precision, recall, and F1 score, respectively. The best results for each task are highlighted in bold. All the results are averaged over three runs.

.

Our model achieves an accuracy improvement of $2.84\%$ and $1.79\%$ on the Humanitarian and Informative tasks, respectively, compared to the supervised baseline. Furthermore, it surpasses the previous state of the art (SOTA) established by the semi-supervised approach of Sirbu et al. [6], delivering at least a $0.85\%$ improvement for both tasks. In low-data scenarios, our approach demonstrates even greater effectiveness, achieving up to a $7.98\%$ accuracy increase over the supervised baseline with only 50 labeled examples per class, as shown in

Table 3. Classification results for the Humanitarian and Informative CrisisMMD tasks using 50 labeled samples per class.

Model	Humanitarian 50 Labels/Class				Informative 50 Labels/Class
	Acc	P	R	F1	Acc	P	R	F1
MMBT Supervised Aug	$68.70$	$79.35$	$68.70$	$71.61$	$75.79$	$78.98$	$75.79$	$76.46$
FixMatch	$70.59$	$79.58$	$70.59$	$71.47$	$78.27$	$80.31$	$78.27$	$78.78$
FixMatch LS	$71.11$	$78.78$	$71.11$	$71.95$	$81.61$	$81.24$	$81.61$	$81.29$
Multihead APM Agree No Low	74.05	82.32	74.05	76.74	83.77	83.59	83.77	83.22

Note: Acc represents the accuracy of the model, whereas P, R, and F1 denote the weighted precision, recall, and F1 score, respectively. The best results for each task are highlighted in bold. All the results are averaged over three runs.

.

Given the distinct characteristics of the Humanitarian and Informative tasks, we next analyze the results for each task in detail.

5.1. Humanitarian Task

As expected, the fully supervised baseline (MMBT Supervised Aug) is ranked among the lowest places in our list, since it does not use any unlabeled data. The obtained results are similar to Sirbu et al. [6]; however, our algorithm seems to have a slightly better precision, recall, and F1 score. Most likely, these changes arise from the fact that we use a weak augmentation strategy when computing the labeled loss.

The results obtained by FixMatch and FixMatch LS are again very similar to those reported by Sirbu et al. [6]. This was expected as we followed their experimental setup as closely as possible. The increase in performance of FixMatch LS over FixMatch demonstrates again the effectiveness of the enhancements proposed by Sirbu et al. [6] for the original FixMatch algorithm: the usage of soft pseudo-labels and the linear unlabeled loss coefficient.

For FlexMatch and FreeMatch the presented results look surprisingly low. In fact, these two algorithms rank in the lowest places in our benchmarks, both in terms of accuracy and F1 score, being outperformed not only by FixMatch and FixMatch LS but also by MMBT Supervised Aug. This outcome is quite the opposite of our initial expectations, especially as we decided to use these algorithms because the Humanitarian class distribution is highly unbalanced and the fixed cutoff threshold proposed by FixMatch was unable to account for the different learning difficulties of the classes. However, by analyzing the evolution of the pseudo-labels and confidence thresholds during the training process, as well as comparing the confusion matrices of different algorithms, we gain insight into the causes of this poor performance. FlexMatch and FreeMatch, which use adaptive confidence thresholds, end up using very low thresholds for the underrepresented classes—especially for Affected individuals, which accounts for less than $1\%$ of the labeled data. This makes the model consider any pseudo-label of an underrepresented class as accurate and includes it in the loss computation, thus affecting performance, as the model learns to predict the infrequent classes way too often. A more in-depth analysis is provided in Section 6.

Although MarginMatch shows an improvement over the supervised baseline, it still obtains worse results than FixMatch LS and FixMatch. A similar in-depth analysis is provided in Section 6.

The Multihead Co-Training approach obtains better results than MarginMatch but still underperforms compared to FixMatch. The main problem with this approach is that during training, as the heads make more confident predictions, they collapse into each other and more than $99.8\%$ of the unlabeled data are included in the unsupervised loss.

Analyzing the results of our proposed method, we see that all four Multihead APM variants stand at the top of the ranking. In particular, Multihead APM Agree and Multihead APM Agree No Low obtain the most promising results, with the latter performing slightly better. The dominance of the methods using the APM for both agreements and disagreements over those that use it just for disagreements can be easily explained. The problem with the classical Multihead Co-Training approach was the excessive usage of unlabeled data (i.e., the failure to filter out irrelevant samples). Adding an additional check on the APM threshold for the agreements limits the number of pseudo-labeled samples that the model will use. In particular, our proposed quantile value of $5\%$ uses approximately $95\%$ of the unlabeled dataset in the late-stage training process.

For comparison, Multihead APM Agree No Low achieves some impressive results, improving the accuracy of MMBT Supervised Aug by $2.84\%$ and the weighted F1 score by $2.81\%$. Compared to FixMatch, we managed to increase the accuracy by $1\%$ and the weighted F1 score by $1.01\%$. If we compare our results with those of Sirbu et al. [6], we can see that our algorithm has improved the accuracy of the best proposed model by $1.05\%$ and the F1 score by $1.45\%$. This is particularly impressive since all of our experiments have used a labeled-to-unlabeled ratio of $\mu =3$, whereas Sirbu et al. [6] used $\mu =7$. A lower $\mu$ means that fewer unlabeled data are passed through the model during each iteration, resulting in faster training and more efficient memory usage. However, this may hinder the model from achieving its full potential, as demonstrated by Sirbu et al. [6].

5.2. Informative Task

For the Informative task, the relative rankings of MMBT Supervised Aug, FixMatch, and FixMatch LS are the same as for the Humanitarian task, with FixMatch LS outperforming FixMatch and both surpassing MMBT Supervised Aug.

For FlexMatch and FreeMatch, we can see that the results relative to other approaches are much better compared to the Humanitarian task. The more balanced class distribution of the Informative dataset played an important role in obtaining these results. While FlexMatch ranks between FixMatch and FixMatch LS, FreeMatch manages to slightly outrank FixMatch LS judging by the F1 score. Meanwhile, MarginMatch provides a relatively low improvement over the supervised baseline, as it performs comparably to FixMatch.

The Multihead Co-Training approach ranks above all previously analyzed algorithms in both accuracy and F1 score. We believe that the usage of a two-class dataset with a more uniform distribution compared to the Humanitarian task helps the model achieve better performance. Even though the head collapsing problem is still present toward the end of the training process, the model can distinguish more clearly between unlabeled samples belonging to different classes and predict the pseudo-labels with higher confidence.

As expected, the Multihead APM algorithms achieve some of the best results in our benchmarks. In particular, Multihead APM Agree No Low obtains the best accuracy, precision, recall, and F1 score from all the algorithms analyzed. Compared to MMBT Supervised Aug, Multihead APM Agree No Low improves the accuracy by $1.79\%$ and the F1 score by $1.35\%$. If we use FixMatch LS as a baseline, our algorithm enhances the accuracy by $0.85\%$ and the F1 score by $0.86\%$.

Comparing Multihead APM Agree No Low with the results from Sirbu et al. [6], we observe that our algorithm outperforms their best model in both accuracy and F1 score. This result is particularly impressive considering that Sirbu et al. [6] uses twice as much unlabeled data per batch, which helps improve performance at the cost of higher computational resources and slower training. Furthermore, their model employs back-translation for strong text augmentation, which has shown better results than EDA for this task.

5.3. Low-Data Regimes

We employ four methods for our low-regime benchmark: MMBT Supervised Aug, FixMatch, FixMatch LS, and our proposed model that achieved the best results on the CrisisMMD classification tasks—Multihead APM Agree No Low.

The results presented in Table 3 demonstrate the dominance of semi-supervised algorithms. This is expected as only a small number of labeled samples are used during the training process. Both FixMatch and FixMatch LS outperform MMBT Supervised Aug on both the Humanitarian and Informative classification tasks. However, our custom model ranks first by a significant margin in both accuracy and F1 score. This shows that Multihead APM is even more robust in the low regime compared to the currently best solution for these tasks (FixMatch LS).

Compared to the fully supervised algorithm, Multihead APM Agree No Low improves the accuracy and F1 score by $5.35\%$ and $5.13\%$ on the Humanitarian task and by $7.98\%$ and $6.76\%$ on the Informative task. Using FixMatch LS as the baseline, considered the best model introduced by Sirbu et al. [6], our method achieves an increase in accuracy and F1 score of $2.94\%$ and $4.79\%$ on the Humanitarian task and of $2.16\%$ and $1.93\%$ on the Informative one.

6. Confidence Thresholds Analysis

In Section 5, we noticed a few discrepancies between the obtained results and our initial expectations. More precisely, for the Humanitarian task consisting of highly imbalanced classes, the SSL methods FlexMatch and FreeMatch performed even worse than the supervised model MMBT Supervised Aug. This is surprising especially because they use adaptive confidence thresholds that aim to independently filter pseudo-labels for each class, favoring the underrepresented classes. To understand this phenomenon, we analyze the training process of SSL algorithms, focusing on confidence thresholds and their impact on the model’s performance. As the findings are similar for both the FlexMatch and FreeMatch algorithms, we present details only for one of them (FlexMatch). Then, we also analyze the thresholds based on the APM scores and explain why our method performs better than MarginMatch and why the variant Multihead APM Agree No Low outperforms the others.

The Affected Individuals class is particularly underpopulated in the Humanitarian dataset, having around $1\%$ of the labeled samples. On the other hand, the Not Humanitarian class consists of $53\%$ of the labeled dataset. In

Table 4. FlexMatch adaptive class thresholds at the end of the first training epoch.

Class	Labeled Percent	Pseudo-Label Count	Threshold
Not Humanitarian	$53\%$	1422 $(7.8\%)$	$0.043$
Other relevant information	$22\%$	780 $(4.2\%)$	$0.023$
Rescue, volunteering, or donation effort	$15\%$	0 $(0.0\%)$	0
Infrastructure and utility damage	$9\%$	0 $(0.0\%)$	0
Affected individuals	$1\%$	0 $(0.0\%)$	0
No class	−	16,176 $(88.0\%)$	−

, we present the pseudo-label distribution and adaptive class thresholds for FlexMatch at the beginning of the training process, while

Table 5. FlexMatch adaptive class thresholds at the end of the last training epoch.

Class	Labeled Percent	Pseudo-Label Count	Threshold
Not Humanitarian	$53\%$	4806 $(26.1\%)$	$0.950$
Other relevant information	$22\%$	4766 $(25.9\%)$	$0.934$
Rescue, volunteering, or donation effort	$15\%$	4231 $(23.1\%)$	$0.747$
Infrastructure and utility damage	$9\%$	3892 $(21.2\%)$	$0.646$
Affected individuals	$1\%$	601 $(3.3\%)$	$0.063$
No class	−	82 $(0.4\%)$	−

shows the same information at the end of the training process. In the beginning, the confidence threshold is less than $5\%$ for the most frequent class and $0\%$ for the bottom three classes. In the end, we observe some major changes. The confidence thresholds for the two most frequent classes are above $90\%$ and higher than $60\%$ for the top four. However, the last class still has a confidence threshold of $6.3\%$, which is 10 times smaller than the threshold of the next smallest class.

The use of 0 thresholds during the first epochs of the training process will automatically incline the model to include more samples in these classes. This is especially true for the last class, which has a very small threshold even at the end of the training. The above idea can be observed in Figure 2, which showcases the confusion matrices from FixMatch and FlexMatch. We can clearly see that FlexMatch has a higher accuracy for the Affected Individuals class, but this comes at a great cost. The accuracy for the first three classes has decreased and the model started to include a larger number of mislabeled samples in the last two classes.

Looking at the confusion matrices, the values started to shift from the first three columns to the right. The number of samples from the last two classes that were incorrectly labeled by FixMatch as belonging to the first three classes (the first three columns under the main diagonal) has decreased. Meanwhile, the number of samples from the first three classes incorrectly labeled as the last two classes (the last two columns above the main diagonal) has increased. This shows the bias that FlexMatch introduces when making predictions, which affects the overall performance of the algorithm. Since the first three classes have a cumulative weight of $90\%$ of the entire labeled dataset, the performance drop for them will quickly affect the overall accuracy and weighted F1 score of the model. The same is also true for FreeMatch since both methods rely on adaptive thresholding techniques based on the current confidence of the model, which allow for the use of low-precision pseudo-labels. Multihead APM addresses this by incorporating APM-based thresholding combined with a voting mechanism, which increases the quality of the pseudo-labels.

In

Table 6. Average APM scores for all the samples grouped by the same pseudo-label.

Class	Not Hum.	Other Rel.	Rescue Vol.	Infrastruct.	Affected Ind.
Not Humanitarian	$0.929$	$-5.914$	$-6.208$	$-6.821$	$-8.183$
Other relevant information	$-5.008$	$0.879$	$-7.985$	$-6.963$	$-8.293$
Rescue, volunteering, or donation effort	$-4.672$	$-6.685$	$0.809$	$-6.478$	$-6.918$
Infrastructure and utility damage	$-5.098$	$-5.323$	$-5.824$	$0.779$	$-6.872$
Affected individuals	$-3.678$	$-4.669$	$-3.086$	$-2.549$	$-0.190$

Note: The row represents the pseudo-label of the sample after the last training epoch, while the column indicates the class for which the APM score was computed.

, we can see the average APM scores of MarginMatch for all the samples belonging to a certain pseudo-label at the end of the training process. The important values are located on the main diagonal, representing the average APM thresholds used for each class during the filtering process. It is easy to observe that all these values are quite different; hence, the use of a common threshold for all the classes, as proposed by MarginMatch, would not be suitable. However, the introduction of class-wise APM-based thresholds proposed by Multihead APM addresses this drawback by allowing a proportional number of pseudo-labels to be selected for each class. Furthermore, we can see that the Affected Individuals class has a negative average APM score. Since all our experiments used 0 as a fixed APM cutoff, we expect that our model will have difficulties predicting this class. The above fact is supported by Figure 3 in which we see the confusion matrix from MarginMatch. The last column is almost completely filled with zeros, and the accuracy of the Affected Individuals class is just $33\%$.

Additionally, in Table 6, we can see that the average APM score for the Affected Individuals class was negative. This is the main reason why Multihead APM Agree No Low performs better than Multihead APM Agree since the latter uses 0 as a lower bound for the APM threshold. Hence, it struggles to predict this class. This fact can be observed in Figure 4, where the last column of Multihead APM Agree is almost completely filled with zeros. In contrast, the confusion matrix for Multihead APM Agree No Low has a better accuracy for the Affected Individuals class and multiple non-zero values on the last column.

7. Conclusions

As part of our research, we analyzed several state-of-the-art semi-supervised algorithms and adapted them to a multimodal setting. We created a centralized benchmark for the Humanitarian and Informative classification tasks from the CrisisMMD dataset [2] and performed a detailed analysis of the key factors that influenced the performance of each solution. Furthermore, we introduced a new semi-supervised learning method called Multihead APM by integrating the APM scores [19] proposed by MarginMatch [10] into our custom Multihead Co-Training [11] approach. We then demonstrated that it outperforms all the other analyzed algorithms and achieves state-of-the-art results on the CrisisMMD dataset with additional unlabeled data. As a final step in our research, we investigated the performance boost that semi-supervised methods can achieve compared to fully supervised ones by running our models in low-data regimes.

Our work has demonstrated that semi-supervised algorithms significantly improve the accuracy for both the Humanitarian and Informative classification tasks. Notably, our proposed Multihead APM method sets a new state of the art over strong SSL baselines, achieving substantial improvements for both tasks and showcasing its effectiveness in leveraging the extensive user-generated messages posted on X (Twitter) and other social media platforms. This progress directly supports our primary goal of enhancing the intervention process of local authorities during natural disasters.

For future work, we aim to evaluate the robustness and generalizability of Multihead APM on a broader range of datasets and modalities to explore its potential applications in other disaster-related contexts and beyond this domain. Experimenting with Multihead APM on other disaster-related datasets involves collecting relevant unlabeled data or researching domain-adaptation techniques that would allow for the use of an unlabeled corpus from a different domain. For example, the DisasterMM [57] dataset would require relevant tweets in the Italian language, which need to be collected or translated from an existing English corpus. This is beyond the scope of this work, but it is definitely a promising direction to further assess the generalizability of our approach. Additionally, we plan to integrate Multihead APM as part of the USB benchmark [56], making it easily accessible for comparison with more SSL approaches and diverse datasets.