Learning from Crowds Using a Focal Loss Function: Dealing with Imbalanced Annotations

Abstract

Obtaining high-quality labeled data for supervised learning is costly, motivating the use of crowdsourcing, which distributes the annotation process across multiple workers with varying levels of expertise. A key challenge in crowdsourced data is annotation sparsity, as each worker labels only a limited subset of instances. This sparsity can amplify class imbalance, reduce supervision for minority classes, and bias standard cross-entropy-based models toward the majority classes. To address this problem, we propose a correlated chained Gaussian process framework trained on a focal-loss-based variational objective (CCGPFL). This probabilistic framework jointly models latent ground-truth and instance-dependent annotator reliability while accounting for correlations among annotators. In addition, the focal-weighted objective mitigates the imbalance induced by sparse annotations by assigning greater importance to harder examples during training. Experiments on synthetic, semi-synthetic, and fully real multi-annotator datasets show that CCGPFL achieves competitive and often superior performance relative to state-of-the-art learning-from-crowds baselines in terms of Overall Accuracy (OA) and Area Under the ROC Curve (AUC).

1. Introduction

The recent success of supervised learning depends largely on the availability of large amounts of well-labeled data for model training [1]. Domain experts commonly provide such labels and are often treated as ground truth. However, in many real-world scenarios, obtaining high-quality labels is expensive and time-consuming, especially in the era of large-scale data [2]. A practical alternative is crowdsourcing, which distributes the labeling process across multiple annotators with unknown levels of expertise [3]. Each annotator contributes a partial and potentially biased view of the latent ground truth to a limited number of training instances. Crowdsourced labels have enabled applications in medicine [4], biology [5], and natural language processing [6]. However, because crowdsourced labels are noisy and incomplete, standard supervised learning algorithms cannot be directly applied.

These challenges have motivated the field of learning from crowds (LFC), which seeks to build supervised models from multiple annotators [7]. Two perspectives dominate. The first, label inference, comprises a two-stage algorithm. In the first stage, noisy annotations are aggregated into a soft estimate of the latent ground truth; in the second, a standard supervised learner is trained on these inferred labels [2,7]. The most common baseline is majority voting (MV), which assigns each instance the label supported by the majority of annotators. Despite its simplicity, MV assumes homogeneous annotator reliability and equal contribution across instances, ignoring input-dependent expertise, systematic biases, and adversarial behavior. To address these limitations within the label-inference paradigm, probabilistic aggregators (e.g., confusion-matrix/EM approaches) weight annotators by estimated reliability and can produce calibrated soft labels [8,9].

The second approach involves modifying standard supervised learning models to perform learning directly from multiple noisy labels. Hence, in the training stage, the model jointly provides a soft estimate of the actual hidden labels, computes the workers’ performance, and trains the supervised learning model [10]. Prior work has shown that, unlike the label inference approach, training directly on multi-annotator data often yields better performance because input features inform both the latent labels and annotator behavior [11]. The seminal contribution of [10] introduced a logistic-regression formulation in which each annotator’s performance is captured via a confusion matrix. This idea has inspired numerous extensions, including crowd-layers [12,13] for deep neural networks and Gaussian-process models as natural ways to represent uncertainty present in the information from multiple annotators [11,14].

While these models have progressively improved the modeling of annotator behavior, their primary focus has been on refining the estimation of workers’ performance (WP). For instance, WP has been parameterized through quantities such as annotator accuracy [15] or confusion matrices [16]. However, these parameters are often assumed to be homogeneous across the input space and estimated under the assumption that annotators act independently. More sophisticated approaches based on Gaussian processes (GPs) and deep neural networks (DNNs) have attempted to relax these assumptions by modeling WP as a function of the input features and allowing dependencies among annotators [17,18].

Another fundamental challenge in learning from crowds is annotation sparsity [19], where each annotator labels only a small subset of the dataset, and most instance–annotator pairs remain unobserved. This issue is not merely a missing-data inconvenience. Even when the underlying class distribution is relatively balanced, sparse annotation patterns can distort the empirical distribution of observed labels, thereby biasing the learning process toward majority classes [20]. Moreover, the limited number of labels per annotator can hinder reliable estimation of annotator-specific parameters [21]. Existing strategies for mitigating annotation sparsity can be broadly grouped into three categories: methods that reconstruct missing entries in the annotation matrix [22,23], methods that share information across annotators to improve the estimation of annotator-specific parameters [13,21,24], and methods that augment sparse annotations with pseudo-labels generated by predictive models [25]. Although these strategies improve the use of incomplete annotations, they generally introduce additional mechanisms for label completion, parameter transfer, or iterative retraining. In contrast, comparatively less attention has been paid to directly mitigating the dominance of frequent annotation patterns within the training objective itself.

To address this limitation, we propose a focal-weighted learning objective for multi-annotator classification under sparse and noisy supervision. Our key idea is to replace the standard cross-entropy term used in variational learning with focal reweighting, so that the training procedure reduces the dominance of easy, frequently observed examples and places greater emphasis on harder, potentially underrepresented cases. Rather than applying focal loss to a conventional classifier, we incorporate it into a probabilistic learning-from-crowds framework that jointly estimates latent ground-truth labels and annotator reliabilities. To instantiate this idea, we build on the Correlated Chained Gaussian Processes (CCGP) framework [26], where likelihood parameters are represented by latent functions drawn from correlated Gaussian process priors. Through a semi-parametric latent factor model [27], annotator reliabilities become input-dependent latent functions that can also exhibit explicit inter-annotator correlations [18]. This leads to our proposed method, termed Correlated Chained Gaussian Processes with Focal Loss (CCGPFL), which combines flexible annotator modeling with a focal-weighted variational objective tailored to sparse, noisy, and potentially imbalanced crowdsourced annotations. Unlike prior CCGP-based learning-from-crowds models [17,28], the proposed approach explicitly modifies the training objective to address the imbalance effects induced by sparse annotation patterns.

The main contributions of this work are as follows:

• We identify annotation sparsity as a source of imbalance in learning-from-crowds settings, showing that sparse annotation patterns can distort the empirical distribution of observed labels and hinder standard CE-based training.
• We propose Correlated Chained Gaussian Processes with Focal Loss (CCGPFL), a probabilistic framework for multi-annotator classification that jointly learns latent ground-truth labels and annotator reliabilities under sparse and noisy supervision.
• We derive a focal-weighted variational objective within a evidence lower bound (ELBO) of a CCGP-based model, enabling the incorporation of focal reweighting into learning from crowds while preserving flexible modeling of heterogeneous, potentially correlated annotators via input-dependent latent functions.
• We evaluate the proposed method on synthetic, semi-synthetic, and real multi-annotator datasets, demonstrating competitive, and often superior, performance relative to existing LFC baselines, with clearer gains at moderate-to-high annotation sparsity.

The remainder of this paper is structured as follows: Section 2 presents the related work and the main contributions. Section 3 provides a comprehensive overview of the foundational concepts in probabilistic modeling for learning from crowds using our CCGPFL approach. Section 4 outlines the experimental setup in detail. Section 5 and Section 6 then present a thorough discussion of the experimental findings and offer concluding insights.

2. Related Work and Main Contributions

2.1. Learning from Crowds

Learning from crowds (LFC) addresses a core challenge in modern machine learning: training models from multiple noisy annotators with varying expertise, bias, and label coverage. Unlike conventional supervised learning—which assumes a single, accurate label per instance—LFC must explicitly model heterogeneous annotator quality, instance-dependent performance, sparse and non-overlapping coverage across instances, and often severe class imbalance. As a result, practical LFC pipelines prioritize robustness to noise and subjectivity, enabling scalable learning without relying exclusively on scarce expert labels.

The LFC paradigm spans several core application families. In computer vision, crowds provide annotations for image classification [4,29], object detection (e.g., bounding boxes and points) [30], and semantic segmentation [31]. In natural language processing, crowd labels support topic modeling [6], text classification [32,33], and sentiment analysis. For regression, representative examples remain comparatively scarce: ref. [34] considers multiple domain experts providing continuous angle annotations for unmanned aerial vehicle (UAV) images of sugarcane fields, which are fused via a static Kalman filter to obtain a single ground-truth row orientation per image. In contrast, ref. [35] proposes MAR-CCGP, a multi-annotator regression framework based on correlated chained Gaussian processes that integrates cacao physicochemical descriptors with noisy sensory scores from multiple panelists, jointly inferring latent ground-truth perceptions and annotator trustworthiness that depend on the inputs.

Methodologically, crowdsourcing pipelines can be grouped into two main categories: label inference and end-to-end models. The first category estimates the latent ground-truth labels, which are subsequently used to train a conventional supervised model. Conversely, end-to-end models directly learn the downstream task from the crowdsourced data [29].

2.1.1. Label Inference

The simplest baseline is majority voting (MV), which selects the class supported by the most annotators for each instance. Despite its practicality, MV assumes equal annotator reliability, which can affect model performance when labels are sparse, when annotators are highly heterogeneous, or when classes are imbalanced [36,37]. To partially address unequal expertise, weighted majority voting (WMV) assigns a nonnegative weight to each annotator, typically proportional to an estimate of their accuracy or trustworthiness [38].

A significant milestone in LFC was the introduction of probabilistic label noise models, which treat the ground-truth label as a latent variable and model annotator responses conditionally on it. The seminal Dawid–Skene (DS) model [8] assumes that, given the true class, annotators respond independently according to individual confusion matrices. Under DS, the joint likelihood of the observed labels factorizes over annotators, and the latent ground truth and annotator confusion matrices can be jointly estimated via maximum likelihood or maximum a posteriori inference, typically using the expectation–maximization algorithm. Building on this idea, the Generative model of Labels, Abilities, and Difficulties (GLAD) [39] explicitly models item difficulty and worker expertise, reflecting the intuition that some annotators perform better on particular tasks and that certain instances are inherently more challenging to label. In a related direction, ref. [40] uses information from multiple annotators to induce clusters of instances and then associates each cluster with a class label to address multi-class classification problems. More recently, ref. [41] proposed Dual-View Learning from Crowds (DVLFC), a two-stage method that represents each instance through two complementary views: its original attributes and a vector of noisy worker annotations treated as additional features. After augmenting the sparse annotations and estimating an integrated label for each instance, DVLFC trains a supervised model for each view and combines their class-membership probabilities to obtain the final prediction.

2.1.2. End-to-End Learning from Crowds

Conversely, end-to-end (E2E) approaches have been proposed to jointly infer ground-truth labels and train the downstream predictor within a single optimization problem. In these models, the label-integration mechanism is embedded as a learnable component within a discriminative architecture, and the parameters governing annotator performance are updated simultaneously with those of the predictor. By directly optimizing a task-specific loss over the raw crowdsourced labels, E2E methods can exploit richer interactions between instance features and annotator behavior than two-stage pipelines, while avoiding the need to commit to a fixed aggregated label before training the classifier [19].

A seminal contribution in this line of research is the work of [10], who formulated learning from crowds as a joint probabilistic model in which the latent ground-truth labels, the classifier parameters, and the annotator-specific parameters are estimated jointly. In their formulation, a linear classifier is coupled with annotator-specific confusion matrices, allowing the learning algorithm to account for heterogeneous worker reliability while leveraging input features to guide the estimation of the hidden labels. This framework established a foundational paradigm for end-to-end learning from crowds and has inspired several subsequent research directions. One important direction is to explore models in which annotator performance depends on the input features, enabling instance-dependent representations of worker expertise. Another direction investigates dependencies among annotators, moving beyond the common assumption of independent labeling behavior.

One important line of research models annotator performance as a function of the input features. In many real-world scenarios, annotator expertise is not uniform across the input space: workers may perform well on certain types of instances while making systematic errors on others. To capture this phenomenon, several approaches introduce instance-dependent formulations in which annotator parameters vary with data characteristics. For example, ref. [15] proposes a per-annotator, instance-dependent binary variable that models each worker’s reliability as a function of the input features. Similarly, recent works, such as [31,42], extend the CrowdLayer architecture by modeling each transformation matrix ${\mathit{M}}^{\left(w\right)}\left(\mathit{x}\right)\in {\mathbb{R}}^{K\times K}$ as a function of the input features, with K being the number of classes. $w\in \{1,\dots ,W\}$ indicates the labeler, and W denotes the number of available annotators. Similarly, ref. [43] assumes that only a small subset of instances is difficult to annotate. Under this assumption, annotator-specific transition matrices remain largely static across the input space, with instance-dependent deviations occurring only occasionally. To capture this behavior, they model noisy labels as additive outliers subject to a sparsity constraint.

Another common assumption in many LFC models is that annotators act independently. Most of the aforementioned approaches assume conditional independence among annotators given the true label; however, this assumption may not hold in practice. Annotators who share similar training, background, or domain knowledge may exhibit correlated labeling behaviors, leading to systematic agreement or disagreement patterns [29]. To address this limitation, several approaches introduce models that explicitly capture dependencies among annotators. For instance, a recent family of methods [17,28] builds upon Correlated Chained Gaussian Processes (CCGP) [26], where annotator parameters are modeled as latent functions governed by Gaussian process priors. Since CCGP is formulated within the multi-output Gaussian process framework, the latent functions associated with different annotators can exhibit statistical correlations, allowing the model to capture dependencies among annotators.

While a substantial body of LFC research has focused on improving the modeling of annotator behavior, much less attention has been paid to the effects of sparse annotation patterns on training. In many practical settings, each annotator labels only a small fraction of the dataset, resulting in a highly sparse output matrix [19]. In this scenario, the empirical class distribution induced by the observed annotations may deviate from the underlying class distribution, leading to an optimization process increasingly dominated by frequent annotation patterns [25]. Moreover, the limited number of labels per annotator can hinder reliable estimation of annotator-specific parameters [21]. To alleviate these limitations, several approaches have mainly focused on recovering missing annotations or sharing statistical strength across annotators. One line of research addresses annotation sparsity by completing missing entries in the annotation matrix before applying a label-integration algorithm. For instance, ref. [22] uses cosine similarity to identify annotators with comparable labeling patterns and to complete missing labels based on their responses. Similarly, in [23], a three-stage method is proposed that first identifies and flips low-confidence annotations, then applies probabilistic matrix factorization to generate soft labels for missing entries, and finally filters unreliable predictions before converting the remaining labels into hard annotations. Another line of research mitigates sparsity by sharing information across annotators [21]. For instance, ref. [13] first trains a global transition network on all available noisy labels to capture shared noise patterns. They then fine-tune this model using the annotations provided by each worker to obtain annotator-specific, instance-dependent transition matrices. Finally, the individual networks are refined using information from neighboring annotators with similar noise patterns, identified through a graph-convolutional network. Similarly, ref. [24] proposes Coupled Confusion Correction, which jointly trains two classifiers and uses each one to distill a class-balanced meta set of likely clean pseudo-labeled instances for correcting the annotator-specific confusion parameters learned by the other. Although the distilled meta sets do not retain annotator identities, their information is transferred to the annotator-specific parameters through a bi-level meta-learning procedure. Moreover, annotators with similar expertise are clustered according to their learned confusion matrices, allowing their parameters to be corrected jointly. Finally, a third line of research leverages predictive models to impute missing annotations. In particular, ref. [25] proposes training a base LFC model on the available crowdsourced labels and using its predictions to generate pseudo-annotations for the missing entries. Only high-confidence pseudo-annotations are retained to augment the annotation matrix, while a distribution-aware strategy balances their contribution across classes. The resulting annotation matrix is then used to retrain the base model, progressively improving the quality of the reconstructed annotations.

2.2. Main Contributions

The approaches discussed above mitigate annotation sparsity through label completion, parameter sharing across annotators, or iterative pseudo-annotation procedures. Although these strategies improve the use of incomplete annotations, they introduce additional mechanisms for reconstructing missing labels, transferring information among annotators, or retraining auxiliary models. Moreover, comparatively less attention has been paid to the effect of sparsity-induced class imbalance on the training objective itself. As discussed previously, sparse annotation patterns may distort the empirical class distribution and increase the dominance of frequent annotation patterns during optimization [25].

In contrast, CCGPFL integrates flexible annotator modeling and imbalance-aware learning within a single probabilistic framework. On the one hand, it builds upon correlated chained Gaussian processes [18,26] to jointly model the latent ground-truth process and annotator-specific parameters via input-dependent latent functions that explicitly capture statistical dependencies across annotators. On the other hand, it introduces a focal-loss-based variational objective that replaces the conventional categorical cross-entropy term. By downweighting well-classified annotations, this objective reduces the dominance of frequent annotation patterns and places greater emphasis on difficult observations.

Therefore, the main distinction of the proposed approach lies not only in its flexible modeling of heterogeneous and potentially correlated annotators [17,28] but also in its direct treatment of sparsity-induced imbalance within end-to-end probabilistic learning. Unlike previous sparsity-oriented methods, CCGPFL does not require an explicit label-completion stage, an additional parameter-transfer mechanism, or an iterative pseudo-annotation and retraining procedure. In this way, CCGPFL extends previous GP-based LFC models [17,28] by incorporating a training objective tailored to sparse, noisy, and potentially imbalanced multi-annotator data.

3. Probabilistic Model for Learning from Crowds

3.1. Problem Setup and Notation

Consider a K-class classification problem with inputs in ${\mathbb{R}}^P$. In the standard supervised setting, one seeks to learn a function f:${\mathbb{R}}^P\to {[0,1]}^K$ from a dataset ${\left\{({\mathit{x}}_n,{\mathit{y}}_n)\right\}}_{n=1}^N$, where each ${\mathit{x}}_n\in {\mathbb{R}}^P$ is an input instance and each ${\mathit{y}}_n\in {\{0,1\}}^K$ is a one-hot vector encoding the ground-truth label of sample n. In the crowdsourced setting, however, the true labels are unobserved. Instead, each instance ${\mathit{x}}_n$ is labeled by a subset of annotators drawn from a pool of W workers. Let $W_n\subseteq \{1,\dots ,W\}$ denote the subset of annotators who provide a label for instance n. The resulting multi-annotator dataset is written as

$$\mathcal{D}=\{\mathit{X},\mathit{Y}\}={\left\{{\mathit{x}}_n,{\left\{{\mathit{y}}_n^{\left(w\right)}\right\}}_{w\in W_n}\right\}}_{n=1}^N,$$

where each ${\mathit{y}}_n^{\left(w\right)}\in {\{0,1\}}^K$ is a possibly noisy one-hot annotation provided by annotator w for instance n.

Because each annotator labels only a limited subset of instances, the corresponding instance–annotator supervision pattern is typically sparse. Let ${\overline{N}}_k$ denote the total number of observed annotations assigned to class k across all annotators, and let

$$N_{\mathrm{obs}}={\displaystyle \sum _{k=1}^K}{\overline{N}}_k$$

be the total number of observed annotations. We define the annotation sparsity level $S\in [0,1]$ as

$$\begin{array}{c}\hfill S=1-{\displaystyle \frac{N_{\mathrm{obs}}}{NW}},\end{array}$$

(1)

where $NW$ is the maximum possible number of instance–annotator labels, and values of S closer to one indicate a higher proportion of unobserved pairs.

To distinguish between imbalance in the observed annotations and imbalance in the underlying dataset, we further define

$$\begin{array}{c}\hfill \overline{R}={\displaystyle \frac{{max}_k{\overline{N}}_k}{{min}_k{\overline{N}}_k}},R={\displaystyle \frac{{max}_k{N}_k}{{min}_k{N}_k}},\end{array}$$

(2)

where $\overline{R}$ measures the imbalance in the observed annotations, and R measures the imbalance in the latent true class frequencies [25]. In general, sparse annotation patterns may distort the empirical distribution of observed labels relative to the underlying class distribution, so that $\overline{R}$ can differ substantially from R.

Given this sparse multi-annotator setting, the goal is to learn a classifier $f(·)$ that generalizes well while accounting for the mechanism through which annotators transform latent labels into noisy observations. This requires modeling annotator-specific reliability and, at the same time, handling the imbalance that may arise in the observed annotation distribution under sparse supervision.

3.2. Modeling Crowdsourced Data

Following [15], we assume a latent true label ${\mathit{y}}_n$ for each training instance. The labels provided by multiple annotators are then modeled as conditionally dependent on this latent ground truth and on each annotator’s reliability. To quantify the latter, we introduce a latent binary variable $z_n^{\left(w\right)}$ representing the reliability of annotator w for instance n. When $z_n^{\left(w\right)}=1$, the annotator provides the correct label, modeled through a categorical distribution; conversely, when $z_n^{\left(w\right)}=0$, the annotation is assumed to follow a uniform distribution over classes, representing an incorrect or random response.

Given the latent variables and under the assumption of conditional independence across annotators, the likelihood of the observed multi-annotator data can be expressed as

$$\begin{array}{cc}\hfill p\left(\mathit{Y}|\widehat{\mathit{Y}},\mathit{Z}\right)& ={\displaystyle \prod _{n=1}^N}\prod _{w\in W_n}\left[{\left({\displaystyle \prod _{k=1}^K}{\widehat{y}}_{n,k}^{y_{n,k}^{\left(w\right)}}\right)}^{z_n^{\left(w\right)}}{\left({\displaystyle \frac{1}{K}}\right)}^{1-z_n^{\left(w\right)}}\right],\hfill \end{array}$$

(3)

where ${\widehat{y}}_{n,k}$ denotes the k-th component of the latent ground-truth probability vector for instance n, and $y_{n,k}^{\left(w\right)}$ is the k-th component of the one-hot label provided by annotator w. Consequently, the likelihood depends on two sets of latent parameters: $\mathit{Z}\in {[0,1]}^{N\times W}$, which collects the annotator reliabilities, and  $\widehat{\mathit{Y}}\in {[0,1]}^{N\times K}$, which represents the estimated ground-truth class probabilities.

3.3. Modeling the Likelihood Parameters as Latent Functions: Correlated Chained Gaussian Processes (CCGP)

The estimation of the hidden true labels $\widehat{\mathit{Y}}$ and the annotators’ reliabilities $\mathit{Z}$ are modeled from the input features $\mathit{X}$ using the Correlated Chained Gaussian Processes (CCGP) framework [26]. The key idea is to associate each likelihood parameter with a latent function (LF) $g_j(·):{\mathbb{R}}^P\to \mathbb{R}$, whose output is transformed through a deterministic mapping $h_j(·):\mathbb{R}\to {\mathcal{B}}_j$ into the valid parameter space ${\mathcal{B}}_j$ of the j-th likelihood term.

In the multi-annotator model, this requires $J=K+W$ latent functions: the first K represent the hidden ground-truth probabilities, while the remaining W encode the annotators’ reliabilities. The hidden labels are obtained via a softmax transformation $\iota (·)$ applied to the corresponding latent functions,

$$\begin{array}{c}\hfill {\widehat{y}}_{n,k}=\iota \left(g_k\left({\mathit{x}}_n\right)\right)={\displaystyle \frac{exp\left(g_k\left({\mathit{x}}_n\right)\right)}{{\sum }_{l=1}^{K}exp\left(g_l\left({\mathit{x}}_n\right)\right)}}.\end{array}$$

(4)

The remaining W LFs $g_{l_w}(·)$, with $l_w=K+w\in \{K+1,\dots J\}$, are employed to codify the labelers’ reliabilities. We highlight that, despite $z_n^{\left(w\right)}$ being a binary variable, we use the sigmoid transformation $\varsigma (·)$ as the link function to avoid discontinuities and favor the model’s stability.

To capture dependencies among annotators and classes, each LF $g_j(·)$ is modeled as one of the outputs of a multi-output Gaussian process. In the multi-annotator setting, this formulation enables the model to represent inter-annotator correlations and the dependence of annotator performance on the input features. Specifically, each latent function is expressed as a linear combination of D independent Gaussian process priors, following the Semi-Parametric Latent Factor Model (SLFM) [44,45]:

$$g_j\left({\mathit{x}}_n\right)={\displaystyle \sum _{d=1}^D}{a}_{j,d}{u}_d\left({\mathit{x}}_n\right),$$

(5)

where each $u_d\left({\mathit{x}}_n\right)$ is an IID sample from a Gaussian process, $u_d(·)\sim \mathcal{GP}(0,{\kappa }_d(·,·))$, which is represented by a mean function (assumed to be zero) and the covariance function given by the kernel function ${\kappa }_d(·,·):{\mathbb{R}}^P\times {\mathbb{R}}^P\to \mathbb{R}$; in addition, $a_{j,d}\in \mathbb{R}$ are the combination parameters. This shared latent factor structure enables each output function to be expressed as a mixture of common latent processes, thereby capturing correlations among annotators, classes, and input-dependent patterns in a unified manner.

However, vanilla GPs have a computational complexity of $\mathcal{O}\left(N^3\right)$, making them prohibitive for large datasets. Accordingly, we resort to well-known sparse GPs to reduce this computational complexity. Thus, the goal is to augment the GP priors by defining a vector of inducing variables $\mathit{u}={[{\mathit{u}}_1^{\top },\dots ,{\mathit{u}}_D^{\top }]}^{\top }\in {\mathbb{R}}^{MD\times 1}$. The inducing variables vector is built from vectors ${\mathit{u}}_d=[u_d\left({\mathit{\lambda }}_1^{\left(d\right)}\right),\dots ,u_d\left({\mathit{\lambda }}_M^{\left(d\right)}\right)]$, which correspond to additional evaluations of $u_d(·)$ at M unknown inducing inputs ${\mathit{\lambda }}_m^{\left(d\right)}\in {\mathbb{R}}^P$ summarizing the information in the N training points. Note that we define a set of inducing points ${\mathbf{\Lambda }}_d={[{\mathit{\lambda }}_1^{\left(d\right)},\dots ,{\mathit{\lambda }}_M^{\left(d\right)}]}^{\top }\in {\mathbb{R}}^{M\times P}$ for each latent function $u_d(·)$. In addition, let $\mathit{\lambda }={[vec\left({\mathbf{\Lambda }}_d\right),\dots ,vec\left({\mathbf{\Lambda }}_D\right)]}^{\top }\in {\mathbb{R}}^{MPD\times 1}$ be the vector of all inducing points [46]. According to the above, our CCGP models the latent true labels and the annotators’ reliabilities based on the following joint distribution (Figure 1 shows the probabilistic graphical model),

$$p(\mathit{Y},\widehat{\mathit{Y}},\mathit{Z},\widehat{\mathit{g}},\mathit{u})=p\left(\mathit{Y}\right|\widehat{\mathit{Y}},\mathit{Z}){\displaystyle \prod _{j=1}^J}p\left({\mathit{g}}_j\right|\mathit{u})p\left(\mathit{u}\right),$$

(6)

where ${\mathit{g}}_j={\left[g_j\left({\mathit{x}}_1\right),\dots ,g_j\left({\mathit{x}}_N\right)\right]}^{\top }\in {\mathbb{R}}^{N\times 1}$, and $\widehat{\mathit{g}}={\left[{\mathit{g}}_1^{\top },\dots ,{\mathit{g}}_J^{\top }\right]}^{\top }\in {\mathbb{R}}^{NJ\times 1}$. Alike,

$$\begin{array}{cc}\hfill p\left({\mathit{g}}_j\right|\mathit{u})=& \mathcal{N}\left({\mathit{g}}_j|{\mathit{K}}_{{\mathit{g}}_j,\mathit{u}}{\mathit{K}}_{\mathit{u},\mathit{u}}^{-1}\mathit{u},{\mathit{K}}_{{\mathit{g}}_j,{\mathit{g}}_j}-{\mathit{K}}_{{\mathit{g}}_j,\mathit{u}}{\mathit{K}}_{\mathit{u},\mathit{u}}^{-1}{\mathit{K}}_{\mathit{u},{\mathit{g}}_j}\right),\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill p\left(\mathit{u}\right)=& \mathcal{N}\left(\mathit{u}|\mathbf{0},{\mathit{K}}_{\mathit{u}\mathit{u}}\right)={\displaystyle \prod _{d=1}^D}\mathcal{N}\left({\mathit{u}}_d\right|\mathbf{0},{\mathit{K}}_{{\mathit{u}}_d{\mathit{u}}_d}),\hfill \end{array}$$

(8)

where ${\mathit{K}}_{\mathit{u}\mathit{u}}\in {\mathbb{R}}^{DM\times DM}$ is a block-diagonal containing blocks ${\mathit{K}}_{{\mathit{u}}_d{\mathit{u}}_d}\in {\mathbb{R}}^{M\times M}$, based on the kernel function ${\kappa }_d(·,·)$. Similarly, ${\mathit{K}}_{{\mathit{g}}_j{\mathit{g}}_j}\in {\mathbb{R}}^{N\times N}$ contains elements ${\sum }_{d=1}^D{a}_{j,d}^2{\kappa }_d({\mathit{x}}_n,{\mathit{x}}_{n^{\prime }})$, where ${\mathit{x}}_n,{\mathit{x}}_{n^{\prime }}\in \mathit{X}$. Likewise, ${\mathit{K}}_{{\mathit{g}}_j\mathit{u}}=[{\mathit{K}}_{{\mathit{g}}_j{\mathit{u}}_1},\dots ,{\mathit{K}}_{{\mathit{g}}_j{\mathit{u}}_D}]\in {\mathbb{R}}^{N\times DM},$ where ${\mathit{K}}_{{\mathit{g}}_j{\mathit{u}}_d}\in {\mathbb{R}}^{N\times M}$ gathers elements $a_{j,d}{\kappa }_d({\mathit{x}}_n,{\mathit{\lambda }}_m^{\left(d\right)}),$$m\in \{1,\dots ,M\}$. To obtain the posterior distributions over the latent parameters, the true labels, and the workers’ reliabilities, we require integrating out the expression in Equation (6) with respect to $\widehat{\mathit{g}}$ and $\mathit{u}$. Nevertheless, because we are not dealing with Gaussian likelihoods, such integration is analytically intractable. Thus, we employ the so-called Variational Inference, which approximates the actual posterior as follows:

$$p(\widehat{\mathit{g}},\mathit{u}|\mathit{Y})\approx q(\widehat{\mathit{g}},\mathit{u})=q\left(\widehat{\mathit{g}}\right|\mathit{u})q\left(\mathit{u}\right)={\displaystyle \prod _{j=1}^J}q\left({\mathit{g}}_j\right|\mathit{u}){\displaystyle \prod _{d=1}^D}q\left({\mathit{u}}_d\right),$$

(9)

where the distributions $q\left({\mathit{g}}_j\right|\mathit{u}),\forall j\in 1,\dots ,J$ are considered to be equal to the prior (see Equation (7)), allowing for efficient mini-batch training as established. In addition, each $q\left({\mathit{u}}_d\right)$ is computed as a multivariate Gaussian distribution with unknown mean ${\mathit{m}}_d\in {\mathbb{R}}^{M\times 1}$ and covariance matrix ${\mathit{V}}_d\in {\mathbb{R}}^{M\times M}$. Consequently, the approximation of the posterior comprises the computation of the variational parameters ${\{{\mathit{m}}_d,{\mathit{V}}_d\}}_{d=1}^D$ by maximizing an evidence lower bound (ELBO). For classification from crowdsourced data, the ELBO is derived from the likelihood in Equation (3)

$$\begin{array}{cc}\hfill \mathcal{L}& ={\displaystyle \sum _{n=1}^N}\sum _{w\in W_n}{\mathbb{E}}_{q\left({\mathit{g}}_1\right)\dots q\left({\mathit{g}}_J\right)}\left[logp\left({\mathit{y}}_n^{\left(w\right)}|{\widehat{\mathit{y}}}_n,z_n^{\left(w\right)}\right)\right]-{\displaystyle \sum _{d=1}^D}{\mathbb{D}}_{KL}(q\left({\mathit{u}}_d\right)\left|\right|p\left({\mathit{u}}_d\right)),\hfill \end{array}$$

(10)

where

$$\begin{array}{c}\hfill q\left({\mathit{g}}_j\right)=\int q\left({\mathit{g}}_j\right|\mathit{u})q\left(\mathit{u}\right)d\mathit{u}\end{array}$$

(11)

and ${\mathbb{D}}_{KL}(p(·),q(·))$ stand for the Kullback–Leibler divergence between distributions $p(·)$ and $q(·)$; similarly,

$$\begin{array}{cc}\hfill logp\left({\mathit{y}}_n^{\left(w\right)}\right|{\widehat{\mathit{y}}}_n,z_n^{\left(w\right)})& =z_n^{\left(w\right)}\underset{CCE}{\underbrace{\left(\sum _{k=1}^K{y}_{n,k}^{\left(w\right)}log\left({\widehat{y}}_{n,k}\right)\right)}}+\left(1-z_n^{\left(w\right)}\right)log\left({\displaystyle \frac{1}{K}}\right).\hfill \end{array}$$

(12)

Figure 1. Probabilistic graphical model of our CCGP. Shadow (light) circles represent observed (latent) variables and arrows define conditional dependences. Each ${\mathit{y}}_w\in {[0,1]}^{N_r\times K}$ contains the one-hot representation of the $N_r$ labels given by w-th worker; similarly, ${\mathit{z}}_w\in {\mathbb{R}}^{N\times 1}$ represents the reliability for labeler w.

We highlight that, from the ELBO in Equation (10), we also compute the model hyperparameters, including the combination parameters in the SLFM ${\left\{a_{j,d}\right\}}_{j,d=1}^{J,D}$, the latent variables $\mathit{\lambda }$, and the kernel. From Equation (12), we note that the first term on the right side is based on the Categorical Cross-Entropy (CCE), which is prone to overfitting the majority class in an imbalanced scenario.

3.4. CCGPFL: From Categorical Cross-Entropy to Focal Loss in the ELBO Formulation

In the variational framework described in Equation (10), the data-fitting term of the Evidence Lower Bound (ELBO) is defined through the log-likelihood $logp\left({\mathit{y}}_n^{\left(w\right)}\right|{\widehat{\mathit{y}}}_n,z_n^{\left(w\right)})$, which is based on the Categorical Cross-Entropy (CCE) loss, as expressed in Equation (12). Although effective in balanced settings, the CCE objective tends to overfit majority classes and underrepresent minority ones, particularly when annotation sparsity amplifies class imbalance. Consequently, the resulting gradient updates are dominated by abundant, well-classified samples, leading to biased learning.

To mitigate this problem, we incorporate the focal loss (FL) [47] into the variational objective. The focal loss extends the standard cross-entropy by adding a modulating factor ${(1-{\widehat{y}}_{n,k})}^{\gamma }$ that dynamically downweights easy or well-classified examples, focusing learning on minority and hard samples. Formally, for a K-class problem, the focal loss for a single instance ${\mathit{x}}_n$ with one-hot label ${\mathit{y}}_n$ is defined as

$$\begin{array}{c}\hfill {\mathcal{L}}_{FL}({\mathit{y}}_n,{\widehat{\mathit{y}}}_n)=-{\displaystyle \sum _{k=1}^K}{y}_{n,k}{(1-{\widehat{y}}_{n,k})}^{\gamma }log\left({\widehat{y}}_{n,k}\right),\end{array}$$

(13)

where $\gamma \ge 0$ controls the focusing strength, and ${\widehat{\mathit{y}}}_n=f\left({\mathit{x}}_n\right)$ denotes the predicted class probabilities. When $\gamma =0$, the focal loss reduces to the CCE, ensuring backward compatibility.

Accordingly, to train a model aware of class imbalance, we replace the CCE-based data-fit term with the following focal-loss-based function:

$$\begin{array}{cc}\hfill {\mathcal{L}}_{FL}({\mathit{y}}_n^{\left(w\right)},{\widehat{\mathit{y}}}_n,z_n^{\left(w\right)})& =z_n^{\left(w\right)}\left({\displaystyle \sum _{k=1}^K}{y}_{n,k}^{\left(w\right)}{(1-{\widehat{y}}_{n,k})}^{\gamma }log\left({\widehat{y}}_{n,k}\right)\right)+(1-z_n^{\left(w\right)})log\left(\frac{1}{K}\right).\hfill \end{array}$$

(14)

By substituting Equation (14) into the ELBO in Equation (10), the resulting focal-loss-based variational objective becomes

$$\begin{array}{c}\hfill {\mathcal{L}}_{FL}={\displaystyle \sum _{n=1}^N}\sum _{w\in W_n}{\mathbb{E}}_{q\left({\mathit{g}}_1\right)\dots q\left({\mathit{g}}_J\right)}\left[{\mathcal{L}}_{FL}({\mathit{y}}_n^{\left(w\right)},{\widehat{\mathit{y}}}_n,z_n^{\left(w\right)})\right]-{\displaystyle \sum _{d=1}^D}{\mathbb{D}}_{\mathrm{KL}}(q\left({\mathit{u}}_d\right)\parallel p\left({\mathit{u}}_d\right)).\end{array}$$

(15)

The focal modulation reshapes the gradient flow within the variational objective in Equation (15): the expected log-likelihood term now weights each observation by its classification difficulty, effectively balancing the contributions of sparse and minority-class annotations. As a result, the proposed focal-loss-based variational objective encourages the model to learn from informative, underrepresented, and hard-to-classify samples while retaining the probabilistic regularization induced by the Kullback–Leibler divergence. This modification directly addresses the imbalance issues inherent in sparse crowdsourced data.

On the other hand, the expectation ${\mathbb{E}}_{q\left({\mathit{g}}_1\right)\dots q\left({\mathit{g}}_J\right)}\left[{\mathcal{L}}_{FL}({\mathit{y}}_n^{\left(w\right)},{\widehat{\mathit{y}}}_n,z_n^{\left(w\right)})\right]$ term in Equation (15) does not admit an analytical solution due to the softmax and sigmoid transformations employed to, respectively, estimate the ground truth ${\left\{{\widehat{\mathit{y}}}_n\right\}}_{n=1}^N$ and the annotators’ reliabilities ${\left\{z_n^{\left(w\right)}\right\}}_{n,w=1}^{N,W}$. Therefore, it is approximated numerically using Monte Carlo estimation and the reparameterization trick [48]. In particular, let $q\left({\mathit{g}}_j\right)=\mathcal{N}({\mathit{g}}_j\mid {\mathit{m}}_j,{\mathit{C}}_j),j\in \{1,\dots ,J\},$ denote the variational distribution associated with the j-th latent function (computed as in Equation (11)). For each Monte Carlo sample $t\in \{1,\dots ,T\}$, we draw ${\mathit{ϵ}}_j^{\left(t\right)}\sim \mathcal{N}(\mathbf{0},\mathit{I}),$ and construct a sample from the corresponding variational distribution as

$$\begin{array}{c}\hfill {\mathit{g}}_j^{\left(s\right)}={\mathit{m}}_j+{\mathit{L}}_j\odot {\mathit{ϵ}}_j^{\left(s\right)},\end{array}$$

(16)

where ${\mathit{L}}_j$ is the Cholesky factor of ${\mathit{C}}_j$, and  ⊙ represents the Hadamard product. Consequently, the expected focal-loss term for the annotation provided by worker w, for sample n, is approximated as

$$\begin{array}{c}\hfill {\mathbb{E}}_{q\left({\mathit{g}}_1\right)\dots q\left({\mathit{g}}_J\right)}\left[{\mathcal{L}}_{FL}\left({\mathit{y}}_n^{\left(w\right)},{\widehat{\mathit{y}}}_n,z_n^{\left(w\right)}\right)\right]\approx {\displaystyle \frac{1}{T}}{\displaystyle \sum _{t=1}^T}{\mathcal{L}}_{FL}\left({\mathit{y}}_n^{\left(w\right)},{\widehat{\mathit{y}}}_n^{\left(t\right)},z_n^{(w,t)}\right),\end{array}$$

(17)

where ${\widehat{\mathit{y}}}_n^{\left(t\right)}$ and $z_n^{(w,t)}$ are obtained by propagating the sampled latent functions ${\left\{{\mathit{g}}_j^{\left(t\right)}\right\}}_{j=1}^J$ through the corresponding transformations.

Then, to reduce computational cost and enable stochastic gradient updates, we approximate the focal-loss-based ELBO using mini-batching. At each optimization step, we uniformly sample a mini-batch $\mathcal{B}$ of size $m_b$ from the dataset, where each element contains an input sample ${\mathit{x}}_n$ and all its available annotations ${\left\{y_n^w\right\}}_{w\in W_n}$. The index set $\mathcal{I}\subset \{1,\dots ,N\}$ is drawn uniformly, so $\left|\mathcal{I}\right|=m_b$ and $\mathcal{B}={\left\{({\mathit{x}}_n,{\left\{y_n^{\left(w\right)}\right\}}_{w\in W_n})\right\}}_{n\in \mathcal{I}}$. Accordingly, the mini-batch estimator of the focal-loss-based ELBO is given by

$$\begin{array}{c}\hfill {\widehat{\mathcal{L}}}_{FL}={\displaystyle \frac{N}{T{m}_b}}\sum _{n\in \mathcal{I}}\sum _{w\in W_n}{\displaystyle \sum _{t=1}^T}{\mathcal{L}}_{FL}\left({\mathit{y}}_n^{\left(w\right)},{\widehat{\mathit{y}}}_n^{\left(t\right)}{z}_n^{(w,t)}\right)-{\displaystyle \sum _{d=1}^D}{\mathbb{D}}_{\mathrm{KL}}\left(q\left({\mathit{u}}_d\right)\parallel p\left({\mathit{u}}_d\right)\right),\end{array}$$

(18)

Note that, since the instances are sampled uniformly, the data-fit term is rescaled by the factor $N/m_b$. Finally, the computational complexity of our model is dominated by the matrix inversion of ${\mathit{K}}_{\mathit{u}\mathit{u}}$, with $\mathcal{O}\left(D{M}^3\right)$ operations, and by the matrix products ${\left\{{\mathit{K}}_{{\mathit{g}}_j\mathit{u}}{\mathit{K}}_{\mathit{u}\mathit{u}}^{-1}\right\}}_{j=1}^J$, with $\mathcal{O}\left(J{m}_{b}D{M}^2\right)$ operations. Here, ${\mathit{g}}_j$ represents the $m_b$ values of the j-th latent function evaluated on a mini-batch [49]. Figure 2 summarizes the proposed CCGPFL framework.

Figure 2. Overview of the proposed CCGPFL framework. First, a set of latent Gaussian processes $u_d\left(\mathit{x}\right)$ is combined through a semiparametric latent factor model (SLFM) to generate the correlated latent functions $g_j\left(\mathit{x}\right)$. Then, the link function $\iota (·)$ produces the ground-truth class estimates $\widehat{\mathbf{y}}$, while the functions ${\varsigma }_w(·)$ estimate the instance-dependent reliability $z^{\left(w\right)}$ of each annotator. Finally, both outputs are incorporated into the focal-loss-based variational objective ${\widehat{\mathcal{L}}}_{FL}$ (see Equation (18)).

3.5. Predictive Distributions

Once the CCGPFL model has been trained, the final step is to perform a prediction on unseen data. Given a new input ${\mathit{x}}_{*}$, the objective is to estimate both its class probabilities and the reliability associated with each annotator. Formally, the predictive class probability for category k is defined as

$${\widehat{y}}_{*,k}=p(y_{*}=k\mid {\mathit{x}}_{*},\widehat{\mathit{f}},\mathit{u}),$$

while the predictive reliability for annotator w is given by $z_{*}^{\left(w\right)}=p(z^{\left(w\right)}=1\mid {\mathit{x}}_{*},\widehat{\mathit{f}},\mathit{u})$.

Since both quantities depend on latent Gaussian process outputs, their analytical computation is intractable. Instead, we approximate their predictive moments under the variational posterior $q\left({\widehat{\mathit{g}}}_{*}\right)$, defined as

$$q\left({\widehat{\mathit{g}}}_{*}\right)=\int p({\widehat{\mathit{g}}}_{*}\mid \mathit{u})q\left(\mathit{u}\right)d\mathit{u}.$$

Accordingly, the first and second predictive moments for class probabilities and annotator reliabilities are computed as

$$\begin{array}{cc}\hfill \mathbb{E}\left[{\widehat{y}}_{*,k}\right]& \approx \int \iota \left(g_k\left({\mathit{x}}_{*}\right)\right)q\left({\widehat{\mathit{g}}}_{*}\right)d{\widehat{\mathit{g}}}_{*},\hfill \end{array}$$

(19)

$$\begin{array}{cc}\hfill \mathbb{E}\left[{\widehat{y}}_{*,k}^2\right]& \approx \int \iota {\left(g_k\left({\mathit{x}}_{*}\right)\right)}^{2}q\left({\widehat{\mathit{g}}}_{*}\right)d{\widehat{\mathit{g}}}_{*},\hfill \end{array}$$

(20)

$$\begin{array}{cc}\hfill \mathbb{E}\left[z_{*}^{\left(w\right)}\right]& \approx \int \varsigma \left(g_{l_w,*}\right)q\left({\widehat{\mathit{g}}}_{*}\right)d{\widehat{\mathit{g}}}_{*},\hfill \end{array}$$

(21)

$$\begin{array}{cc}\hfill \mathbb{E}\left[{\left(z_{*}^{\left(w\right)}\right)}^2\right]& \approx \int \varsigma {\left(g_{l_w,*}\right)}^{2}q\left({\widehat{\mathit{g}}}_{*}\right)d{\widehat{\mathit{g}}}_{*},\hfill \end{array}$$

(22)

where $\iota (·)$ and $\varsigma (·)$ denote the softmax and sigmoid transformations, respectively. The predictive variances can then be obtained from the standard moment identity $Var\left[A\right]=\mathbb{E}\left[A^2\right]-\mathbb{E}{\left[A\right]}^2$.

Since the integrals in Equations Equations (19)–(22) do not admit closed-form solutions, all expectations are estimated via Monte Carlo sampling from the variational posterior $q\left({\widehat{\mathit{g}}}_{*}\right)$. This procedure provides both the mean predictive class probabilities and interpretable measures of annotator reliability while naturally incorporating uncertainty arising from the latent Gaussian processes. The complete training and prediction procedure of the proposed CCGPFL model is summarized in Algorithm 1.

Algorithm 1: CCGPFL for learning from sparse and imbalanced crowdsourced labels

4. Experimental Set-Up

We validate the performance of our CCGPFL in different datasets with simulated and real workers. Below, we present the details of the considered experiments.

4.1. Sensitivity Analysis of the Focal-Loss Parameter $\gamma$ Under Varying Annotator Sparsity Levels

We consider a fully synthetic one-dimensional dataset for a three-class classification problem, which allows access to the ground-truth labels in both training and test sets. The latent label of each instance is determined by the maximum among $sin\left(2\pi x_n\right)$, $-sin\left(2\pi x_n\right)$, and $0.5-sin\left(2\pi (x_n+0.25)\right)$. We generate 100 training samples and 200 test samples over the interval $[0,1]$. To emulate the learning-from-crowds setting, we follow [18] and simulate five correlated annotators with input-dependent reliabilities. Figure 3 illustrates the resulting classification problem together with the annotators’ reliability functions, whose average accuracies are $[0.88,0.61,0.59,0.58,0.24]$.

Figure 3. One-dimensional synthetic dataset used for multi-class classification. The left panel depicts the ground-truth partition of the input space into three classes, with each color representing a different class. The five panels on the right show the reliability of each simulated annotator as a function of the input, highlighting heterogeneous labeling behaviors. Annotator 1 is reliable almost everywhere except in a narrow central interval; Annotator 2 is reliable in the left and middle parts but fails on the right; Annotator 3 is mostly unreliable on the left and reliable on the right; Annotator 4 alternates between high- and low-reliability regions; and Annotator 5 is the least reliable overall, except near the far right end of the domain. This construction provides a controlled setting for evaluating learning methods under input-dependent and correlated annotation noise.

On top of this synthetic multi-annotator setting, we consider different annotation sparsity levels $S\in \{0,0.2,0.4,0.6,0.8\}$. For each target sparsity level S, we generate annotator-specific sparsity rates ${\left\{s_w\right\}}_{w=1}^W$ by sampling from a Gaussian distribution with mean S and standard deviation $0.1$, and then clipping the sampled values to the interval $[0,1]$. Next, for each annotator w, a proportion $s_w$ of the samples is selected uniformly at random, and the corresponding labels are removed. This procedure yields incomplete annotator-response matrices with heterogeneous patterns of missingness. As sparsity increases, the amount of available supervision decreases and its distribution becomes more irregular; as a result, some classes may be represented by fewer observed annotations than others, thereby increasing the imbalance of the supervision signal. To ensure that every instance receives at least one annotation, the last worker provides additional labels when necessary. Consequently, in practice, annotator W may exhibit a realized sparsity level lower than its target value $s_W$.

This setting is especially relevant for the variational objective in Equation (14), which includes a focal-loss term controlled by the hyperparameter $\gamma$. In standard supervised learning, $\gamma$ is usually tuned by optimizing a validation metric such as accuracy or AUC. However, such a strategy is not directly applicable in real learning-from-crowds problems, because validation metrics require access to true labels, which are unavailable in practice. Here, the synthetic nature of the dataset enables analysis of $\gamma$ behavior in a controlled environment. Accordingly, for each sparsity level, we evaluate $\gamma \in \{0.1,0.2,0.4,0.6,0.8,1,2\}$ and compare the resulting test performance. This experiment is designed to characterize the sensitivity of the proposed focal-weighted objective to $\gamma$ and to assess its robustness under progressively sparser annotation regimes that yield weaker, potentially more imbalanced supervision.

4.2. Testing CCGPFL Under Different Conditions of Class Imbalance and Sparsity

Now, we test our CCGPFL using semi-synthetic and fully real datasets with varying levels of imbalance and sparsity. We consider semi-synthetic datasets to be those whose features come from real applications and whose labels are simulated using the approach proposed in [18]. Specifically, we use six datasets from the University of California Irvine machine learning (UCI) repository (Iris plant–(Iris), Occupancy Detection Data Set–(Occupancy), Image Segmentation–(Segmentation), Skin Segmentation Data Set–(Skin), Tic-Tac-Toe Endgame database–(tic-tac-toe), Wine Quality–(Wine)) and the bearing data collected by Case Western Reserve University, where we employed the parametrization process described in [50].

On the other hand, regarding the fully real datasets (where both the input features and the multi-label annotations come from real applications), we use two datasets. The first is a bio-signal database, where the ground truth is available and comprises the estimation of the presence/absence of voice pathologies. Each voice signal is parametrized to obtain an input space with $P=13$. The labels are obtained from a panel of up to four physicians, who measure voice quality using five perceptual features: Grade of dysphonia (G), Roughness (R), Breathiness (B), Asthenia (A), and Strain (S). For each perceptual scale, the specialist is required to provide a tag ranging from 0 (healthy) to 3 (severe); however, we discard scales (A) and (S) due to a low quality of annotators [51]. Thus, we identify three multiclass classification problems (one for each perceptual scale); however, since we only have the ground truth for the binary setting (pathological or normal voice), we redefine the multi-labeler annotations as follows: labels assigned to class 0 are considered as normal voice; otherwise, they are categorized as pathological voices. Moreover, the second dataset consists of a collection of song records categorized into 10 music genres. A total of 700 samples were published on the Amazon Mechanical Turk platform to obtain labels from 44 workers. The feature extraction is done by following the procedure in [15], from which we obtain an input space with $P=124$. Table 1 summarizes the tested datasets for the classification case.

Table 1. Tested databases under different conditions of class imbalance. Note that, for the voice dataset, we report the three values of R due to the considered perceptual scales (G, R, and B).

	Name	P	N	K	$\mathit{R}/\overline{\mathit{R}}$
fully synthetic	synthetic	1	100	3	$1/1.35$
semi-synthetic	Iris	4	150	3	1/1.28
	Occupancy	7	20,560	2	3.32/1.10
	Segmentation	18	2310	7	1/1.37
	Skin	4	245,057	2	3.81/1.26
	Tic-tac-toe	9	958	2	1.88/1.16
	Western	7	3413	4	2.42/1.40
	Wine	13	178	3	1.48/1.25
fully real	Voice	13	218	2	3.27/[1.58–2.20–5.94]
	Music	124	1000	10	1.19/1.75

4.3. Method Comparison and CCGPFL Training

The classification performance is evaluated using the overall accuracy (OA) and the area under the ROC curve (AUC). We use a hold-out validation scheme with 10 repetitions, where $70\%$ of the samples are used for training and the remaining $30\%$ for testing. This procedure is applied to all experiments, except for the fully synthetic and music datasets, where the training and testing sets are predefined.

In addition, we compare the performance of our CCGPFL with several state-of-the-art algorithms (see Table 2). We use the following abbreviations: Gaussian Processes—(GP), Ground Truth—(GT), Majority Voting—(MV), correlated chained Gaussian processes for multiple annotators—(CCGPMA), Deep Learning from Crowds—(DLFC), Chained Deep Learning—(CDL), Categorical Cross-Entropy—(CCE), and Generalized Cross-Entropy—(GCE).

Table 2. Short overview of the state-of-the-art methods tested.

Approach	Description
GPGT	A GP using the ground truth (upper bound). It uses a CCE-based loss function.
GPMV	A GP using the MV of the crowdsourced labels (lower bound). It employs a CCE-based loss function.
DLFC [7]	A Crowd Layer for deep learning; it assumes independent and homogeneous workers.
	We consider three variations: MW, VW, VW + B.
CDL-CCE [52]	A regularized chained deep neural network for multiple annotators, based on CCE.
	Assumes correlated workers with heterogeneous performance.
CDL-GCE [17]	An enhancement of CDL-CCE that includes a robust loss function based on the GCE.
CCGPMA [18]	A correlated chained Gaussian Processes for multiple annotators. It uses a CCE loss function
	to model the labelers’ performance as a function of the input features.
CCGP-GCE [28]	A correlated chained GP using an ELBO based on a GCE.

Conversely, to train our CCGPFL and all the GPs-based methods, we employ the well-known Squared Exponential Covariance Function [53], governed by the parameter $\vartheta \in {\mathbb{R}}^{+}$ that defines the characteristic length-scale, which is estimated by optimizing the corresponding ELBO. Further, according to the results in [18], we define the $D=W+K$ GPs for the SLFM. Now, regarding the inducing variables, we fix $M=10$ for the fully-synthetic experiments and $M=80$ for the remaining experiments. Gradient descent is used with the ADAM optimizer. All the GP-based models are trained for $It=100$ iterations. We utilize a mini-batch size of $m_b=100$ samples. However, for smaller datasets (with fewer than 500 samples), we adjust the mini-batch size to the number of training samples, $m_b=N$; furthermore, we employ $T=800$ samples for Monte Carlo approximations of the variational expectation. Finally, all experiments were conducted in Python 3.8, with the Pytorch 1.14 API, on a Google Colaboratory environment https://github.com/juliangilg/CCGPFL (accessed on 13 June 2026).

5. Results and Discussion

We first consider a controlled synthetic multi-annotator setting to examine how annotation sparsity affects the observed label distribution before analyzing the sensitivity of the proposed CCGPFL framework to the focal-loss parameter $\gamma$. The original synthetic dataset is fully balanced at the latent-class level. Therefore, any deviation of the annotation imbalance ratio $\overline{R}$ from one must arise from the annotation process rather than from the ground-truth class distribution.

Figure 4 reports the annotation imbalance ratio $\overline{R}$ for different sparsity levels S. Even at $S=0$, the observed imbalance ratio is $\overline{R}=1.38$, indicating that the empirical distribution of the observed labels is already mildly distorted in the absence of missing annotations. Since the latent class distribution is balanced, this initial deviation is explained by the labeling errors introduced by the noisy annotators. As the sparsity level increases, $\overline{R}$ also increases, showing that sparse annotation patterns further amplify this distortion. These results provide empirical evidence for one of the central claims of this work: annotation sparsity not only reduces the amount of available supervision but also aggravates the imbalance of the observed label distribution. This behavior, in turn, motivates the use of reweighting mechanisms such as focal loss in end-to-end learning from crowds.

Figure 4. Observed annotation imbalance ratio $\overline{R}$ for different annotation sparsity levels S in the synthetic multi-annotator setting. Higher sparsity levels tend to produce larger values of $\overline{R}$, indicating a stronger distortion of the empirical distribution of observed annotations.

Having established that increasing sparsity amplifies the distortion of the observed label distribution, we next analyze the sensitivity of the proposed objective to the focal-loss parameter $\gamma$. Figure 5 summarizes the AUC obtained for different combinations of sparsity levels S and $\gamma \in \{0,0.1,0.2,0.4,0.6,0.8,1,2\}$ over five independent hold-out folds, where $\gamma =0$ converges to the standard CE-based ELBO.

Figure 5. Sensitivity analysis of the focal-loss parameter $\gamma$ across different annotation sparsity levels S. Each panel reports the mean and standard deviation of the AUC over five independent hold-out folds for a fixed sparsity level and multiple values of $\gamma$, including $\gamma =0$, which corresponds to the standard categorical cross-entropy objective.

First, as shown in the results from Figure 5, under low sparsity levels ($S=0$ and $S=0.2$), there are no substantial differences between the standard CCE objective ($\gamma =0$) and the focal-weighted variants for the tested values of $\gamma$. In fact, the results suggest a slight performance degradation as $\gamma$ increases. Conversely, as sparsity increases, the relative behavior of $\gamma =0$ with respect to $\gamma >0$ becomes more informative. In particular, for intermediate sparsity regimes such as $S=0.4$ and $S=0.6$, the results suggest that values of $\gamma >0$ can yield more favorable AUC values than the CCE baseline, which is consistent with the role of focal reweighting under increasingly distorted annotation distributions.

At the same time, the figure also shows that the amount of available supervision bounds the benefits of focal loss. Under the most extreme sparse scenario, $S=0.8$, the AUC remains low for all tested values of $\gamma$, including both $\gamma =0$ and $\gamma >0$. To explain this behavior, recall that sparse annotations are generated by sampling an annotator-specific sparsity rate $s_w$ and then removing the corresponding proportion of labels at random. In addition, to ensure that every sample is labeled at least once, the last annotator (who performs worst in Figure 3) is used to provide additional labels when necessary. Therefore, under $S=0.8$, the problem is not only the severe reduction in available supervision but also the increased influence of a low-performing annotator, which further degrades the quality of the observed labels and limits the predictive performance of the model, regardless of the value of $\gamma$.

In summary, this experiment provides controlled empirical support for the main hypothesis underlying the proposed approach. As annotation sparsity increases, the observed supervision becomes not only weaker but also more distorted in its class distribution. Under low-to-intermediate sparsity regimes, focal reweighting appears more favorable than the standard CCE-based objective. In contrast, under extreme sparsity, its benefits are constrained by the degradation of quality labels.

The second set of experiments was conducted on both semi-synthetic and fully real datasets to evaluate the performance of our approach under various conditions of class imbalance and sparsity. Table 3 displays the classification results from these experiments, focusing on Overall Accuracy (OA) and Area Under the Curve (AUC). We used a nonparametric test, the Friedman test, to determine whether differences in classifier performance are statistically significant. This test ranks the algorithms for each dataset separately, with the best-performing algorithm receiving a rank of 1 and the second-best receiving a rank of 2 [54]. The null hypothesis of the Friedman test posits that all performances are equal. Additionally, we set a significance threshold of $\delta <0.01$; thus, a p-value below this threshold indicates rejection of the null hypothesis. Because GPGT is the theoretical upper bound, it is excluded from such statistical analysis. From the results, we obtain Chi-squared values of $34.18$ and $38.21$ with p-values of $3.7\times {10}^{-5}$ and $6.8\times {10}^{-6}$, respectively, for the AUC and OA. Therefore, we have sufficient evidence to conclude that significant differences exist between the multi-labeler classifiers.

Table 3. Classification results, in terms of the OA and AUC, for Semi-synthetic and fully-real datasets. In bold, the highest performance excluding the theoretical upper bound GPGT. The Friedman test returns a Chi-square value of $34.18$ and $38.21$ with p-value of $3.7\times {10}^{-5}$ and $6.8\times {10}^{-6}$, respectively, for the AUC and OA.

Method		Semisynthetic Datasets							Fully Real Datasets				Average
		TicTacToe	Skin	Occupancy	Iris	Wine	Segmentation	Western	Voice (G)	Voice (R)	Voice (B)	Music
GPGT	AUC [%]	87.90 ± 3.52	99.89 ± 0.02	99.43 ± 0.08	98.46 ± 1.71	99.89 ± 0.12	98.52 ± 0.11	93.03 ± 0.51	94.84 ± 2.74	94.84 ± 2.74	94.84 ± 2.74	94.51	96.01
	OA [%]	82.67 ± 2.58	98.90 ± 0.06	98.79 ± 0.11	88.22 ± 5.36	97.04 ± 1.89	89.08 ± 0.69	73.89 ± 1.49	86.97 ± 5.73	86.97 ± 5.73	86.97 ± 5.73	67.67	87.01
GPMV	AUC [%]	66.98 ± 3.96	88.86 ± 0.20	91.24 ± 0.38	97.76 ± 2.12	99.66 ± 0.37	98.40 ± 0.11	90.97 ± 0.40	93.01 ± 3.27	93.77 ± 2.41	79.62 ± 5.58	93.15	90.31
	OA [%]	62.50 ± 2.40	83.49 ± 0.18	81.57 ± 0.28	89.11 ± 5.21	92.04 ± 4.83	87.52 ± 0.80	65.89 ± 1.62	85.85 ± 4.10	81.95 ± 6.09	32.82 ± 7.82	58.00	74.61
DLFC-MW	AUC [%]	72.26 ± 4.07	86.36 ± 0.55	68.57 ± 4.21	98.85 ± 1.27	99.36 ± 0.49	99.56 ± 0.13	81.14 ± 2.10	89.57 ± 5.83	89.66 ± 5.83	81.23 ± 8.94	93.68	87.29
	OA [%]	65.03 ± 4.42	46.03 ± 0.25	50.60 ± 8.16	90.22 ± 4.79	93.52 ± 3.34	94.86 ± 1.09	61.87 ± 1.33	82.77 ± 4.81	81.23 ± 6.68	56.21 ± 6.35	65.33	71.60
DLFC-VW	AUC [%]	67.71 ± 2.58	86.56 ± 0.66	67.97 ± 3.87	99.03 ± 1.24	99.18 ± 0.67	99.58 ± 0.14	83.70 ± 2.36	89.42 ± 6.00	89.29 ± 5.87	80.92 ± 9.72	93.84	87.01
	OA [%]	64.42 ± 2.39	46.19 ± 0.28	65.19 ± 2.11	90.67 ± 4.85	91.67 ± 4.16	95.14 ± 1.12	66.64 ± 2.86	81.74 ± 5.35	80.41 ± 7.33	57.03 ± 6.89	67.33	73.31
DLFC-VW + B	AUC [%]	67.80 ± 3.30	86.68 ± 0.65	66.77 ± 2.58	98.65 ± 1.46	99.22 ± 0.67	99.52 ± 0.12	82.01 ± 1.85	90.30 ± 5.21	89.37 ± 6.32	82.18 ± 8.03	93.87	86.94
	OA [%]	62.52 ± 3.17	46.04 ± 0.27	52.74 ± 9.40	89.78 ± 4.79	92.59 ± 3.70	94.76 ± 1.08	63.5 ± 1.68	82.36 ± 5.99	80.62 ± 6.12	56.62 ± 6.03	63.67	71.38
CDL-CCE	AUC [%]	89.23 ± 4.88	98.71 ± 1.90	85.01 ± 3.39	97.03 ± 1.72	99.42 ± 0.40	99.79 ± 0.09	76.22 ± 12.62	92.64 ± 3.41	93.93 ± 2.93	90.75 ± 4.73	94.15	92.44
	OA [%]	82.53 ± 4.56	96.91 ± 2.99	83.46 ± 3.80	87.11 ± 4.31	94.63 ± 2.41	96.33 ± 1.07	61.66 ± 6.45	86.46 ± 4.55	86.62 ± 4.72	74.00 ± 9.77	65.00	83.15
CDL-GCE	AUC [%]	88.96 ± 3.51	97.65 ± 2.44	82.41 ± 0.65	96.82 ± 2.04	97.41 ± 1.96	99.77 ± 0.11	90.46 ± 2.76	92.40 ± 3.84	93.74 ± 3.13	90.39 ± 4.63	94.73	93.15
	OA [%]	82.19 ± 3.26	96.17 ± 3.07	83.79 ± 0.39	87.56 ± 3.61	91.30 ± 4.61	95.84 ± 1.23	68.78 ± 2.56	86.77 ± 4.97	86.15 ± 5.59	73.38 ± 8.84	63.00	83.17
CCGPMA	AUC [%]	73.12 ± 3.12	89.42 ± 2.12	97.75 ± 1.93	97.73 ± 0.00	99.43 ± 0.32	97.58 ± 0.42	93.15 ± 0.48	92.43 ± 3.29	93.83 ± 2.91	89.07 ± 5.01	94.56	92.55
	OA [%]	65.30 ± 3.01	86.84 ± 1.92	91.56 ± 5.45	91.11 ± 0.00	92.96 ± 3.19	83.56 ± 4.03	73.98 ± 1.50	86.97 ± 3.68	86.15 ± 4.70	54.36 ± 11.79	62.33	79.55
CCGP-GCE	AUC [%]	98.11 ± 1.34	98.44 ± 1.87	99.39 ± 0.08	99.08 ± 0.65	99.56 ± 0.27	99.13 ± 0.19	94.25 ± 0.37	90.52 ± 5.87	94.04 ± 2.90	90.14 ± 3.74	95.11	96.16
	OA [%]	92.78 ± 2.43	96.92 ± 2.77	88.98 ± 0.19	94.67 ± 2.27	95.19 ± 1.70	91.98 ± 0.97	76.66 ± 1.09	84.00 ± 6.85	88.15 ± 5.15	61.08 ± 10.68	70.33	85.52
CCGPFL $\gamma =0.1$	AUC [%]	99.01 ± 0.93	99.03 ± 1.39	97.5 ± 3.23	99.49 ± 0.54	99.10 ± 1.07	99.57 ± 0.05	94.11 ± 1.79	94.22 ± 2.28	93.95 ± 2.6	92.06 ± 3.12	95.37	96.67
	OA [%]	95.8 ± 2.76	96.28 ± 3.4	87.71 ± 1.49	94.67 ± 2.04	95.37 ± 3.73	94.34 ± 0.65	75.92 ± 4.69	88.31 ± 4.52	89.38 ± 3.26	88.31 ± 2.5	71.00	88.82

From Table 3, we see that the methods GPMV and DLFC (across all their variants) perform the worst in most cases. Notably, DLFC often falls below the simple GPMV baseline. At first glance, this is unexpected since majority voting (as in GPMV) is the most naive approach that assumes all annotators have similar expertise. However, two main factors help explain these results. First, DLFC—whether using MW, VW, or VW + B—offers a simple way to model how multiple workers label data. This approach is less flexible in representing labelers with heterogeneous reliabilities. Second, while GPMV is based on a Gaussian process framework, DLFC relies on deep learning networks, which can be at a disadvantage for the predominantly tabular datasets in this study [55]. Deep learning typically excels when there is a large volume of data or when it can learn complex feature representations, such as those found in images or text. In contrast, tabular datasets generally favor kernel-based approaches like Gaussian processes, which do not require large datasets to achieve good generalization.

We focus our analysis on multi-labeler models that consider correlated annotators, whose performance varies across input features. The models under consideration include CDL-CCE, CDL-GCE, CCGPMA, CCGP-CCE, and our CCGPFL. According to the results presented in Table 3, these methods outperform all other competing approaches. This is because these models provide a more realistic estimate of annotator behavior. Notably, our CCGPFL achieves the highest averages in both OA and AUC. Specifically, our method achieves the best OA on 7 out of 11 datasets and the best AUC on 6 out of 11 datasets. We attribute this success to the focal-loss-based variational objective presented in Equation (14), which enhances the model’s robustness against noisy, sparse, and imbalanced labels from multiple annotators. In comparison, CCGP-GCE shows the second-best average performance, thanks to its GCE-based loss function, which makes it robust to noisy labels. However, when we analyze the Voice (R), Voice (B), and Music datasets—identified as the most challenging due to their high class imbalance ratios (2.20, 5.94, and 1.75, respectively)—we observe that our CCGPFL achieves the best performance in both OA and AUC. This confirms that CCGPFL is less susceptible to noisy and imbalanced scenarios.

6. Conclusions

This work introduced CCGPFL, a probabilistic learning-from-crowds framework that combines Correlated Chained Gaussian Processes (CCGP) [26] with a focal-weighted variational objective. The proposed formulation models annotator behavior through input-dependent latent functions with explicit inter-annotator correlations, thereby capturing both heterogeneous reliabilities and dependencies among workers. Building on the ideas of [18,28], CCGPFL extends prior CCGP-based multi-annotator models by replacing standard CCE- or GCE-based variational formulations with an objective specifically tailored to sparse and imbalanced annotation regimes.

The experimental results provide support for the central hypothesis underlying the proposed approach. In the controlled synthetic setting, we showed that increasing annotation sparsity distorts the observed label distribution, even when the latent class distribution is perfectly balanced. This result is particularly relevant because it shows that sparsity affects not only the amount of available supervision, but also its empirical class distribution, thereby making the learning problem increasingly difficult for standard CE-based objectives. In this context, focal reweighting becomes more useful in low-to-intermediate sparse regimes, while its benefits are naturally constrained when the supervision signal becomes severely degraded.

The results on semi-synthetic and fully real datasets further show that CCGPFL achieves strong predictive performance in terms of both OA and AUC, while remaining competitive with or superior to existing multi-annotator baselines. In particular, the proposed framework attains the best average performance among the compared methods and shows clear advantages on the most challenging datasets, where label noise, sparsity, and class imbalance are more pronounced. Taken together, these findings highlight the importance of jointly modeling (i) dependencies among annotators, (ii) input-dependent annotator reliability, and (iii) imbalance-aware training objectives in realistic sparse multi-annotator settings.

A relevant limitation of the current formulation is that each annotator’s behavior is modeled through a binary latent variable ${\mathit{z}}^{\left(w\right)}$, representing reliability. Although this assumption is useful for capturing whether an annotator is reliable or not on a given instance, it does not explicitly represent structured biases in the labeling process, such as systematic confusion between specific classes. A natural direction for future work is therefore to extend the proposed framework with annotator-specific confusion matrices, following ideas explored in [10,11,16]. Another promising direction is to adapt the proposed loss formulation to late-fusion settings in multimodal learning, where multiple information sources are integrated at the decision level rather than through multiple annotators. Such extensions could be particularly relevant in clinical applications involving heterogeneous data sources, including the diagnosis and treatment of behavioral disorders such as attention-deficit/hyperactivity disorder [56].