A Survey and Taxonomy of Loss Functions in Machine Learning

Abstract

Most state-of-the-art machine learning techniques revolve around the optimization of loss functions, making the choice of an objective critical to model performance and reliability. Although recent reviews discuss loss functions in specific domains or in deep learning settings, there is still no single reference that presents widely used losses across major task families within a unified formal setting and with consistent optimization-relevant property annotations. In this survey, we compile and systematize the most widely adopted loss functions for regression, classification, generative modeling, ranking, energy-based modeling, and relational learning. Our selection procedure combines seeding from foundational textbooks and prior surveys with cross-checking of highly cited literature and common implementations in mainstream machine learning frameworks. We introduce 52 loss functions and organize them into an intuitive taxonomy, summarizing their theoretical motivation, key mathematical properties, and typical application contexts, with compact appendix tables for quick lookup. This survey is intended as a resource for undergraduate, graduate, and Ph.D. students, as well as researchers seeking a structured reference for selecting and comparing loss functions.

1. Introduction

In the last few decades, there has been an explosion in interest in machine learning [1,2]. This field focuses on the definition and application of algorithms that can be trained on data to model underlying patterns [3,4,5,6]. Machine learning approaches can be applied to many different research fields, including biomedical science [7,8,9,10], natural language understanding [11,12,13], anomaly detection [14], image classification [15], database knowledge discovery [16], robot learning [17], online advertising [18], time series forecasting [19], brain–computer interfacing [20], and many more [21]. To train these algorithms, it is necessary to define an objective function, which gives a scalar measure of the algorithm’s performance [3,22]. They can then be trained by optimizing the value of the objective function.

Within the machine learning literature, such objective functions are usually defined in the form of loss functions, which are optimal when they are minimized. The exact form of the loss function depends on the nature of the problem to be solved, the data available, and the type of machine learning algorithm being optimized. Finding appropriate loss functions is, therefore, one of the most important research endeavors in machine learning.

As machine learning has progressed, many loss functions have been introduced to address diverse tasks and application settings. Several surveys now provide systematic coverage of loss functions, particularly in deep learning contexts. For example, Terven et al. [23] survey loss functions together with evaluation metrics across modern deep learning pipelines, while Li et al. [24] provide a deep-learning-focused review with an explicit screening procedure and a dedicated category for metric losses. Earlier broad surveys (e.g., Wang et al. [25]) and domain-focused reviews (e.g., in face recognition or semantic segmentation [26,27]) also offer valuable summaries, but they differ in scope by either combining losses with evaluation metrics, focusing on narrower application domains, or predating several now-common objectives in modern generative and representation learning settings.

In this context, there remains a need for a single reference that focuses only on loss functions, presents them within a unified formal setting with consistent notation, and connects them through an interpretable taxonomy spanning both classical and modern machine learning objectives. The increasing fragmentation and specialization of the field means that new losses are often developed within specific communities, for example, generative modeling, ranking and retrieval, energy-based modeling, or graph representation learning, and are not always presented within a shared framework.

To improve transparency, we followed a structured selection procedure. We first seeded candidate losses from foundational textbooks and prior surveys, then expanded the list by inspecting highly cited papers associated with each task family and by checking which objectives are implemented and commonly used in major machine learning frameworks (e.g., scikit-learn [v 1.7], PyTorch [v 2.7], and TensorFlow [v 2.16]). We focused primarily on widely adopted objectives in the modern literature, while retaining seminal earlier works when they define canonical losses. This process resulted in 52 loss functions included in the main text and summarized in Appendix A (

Table A1. Summary of loss functions, classification, and mathematical properties.

Loss Function	Task	Taxonomy	Conv.	Diff.	Key Properties/Notes
Mean Bias Error (MBE)	Regression	Error-based	Yes	Yes	Errors may cancel out; used for evaluation.
Mean Absolute Error (MAE)	Regression	Error-based	Yes	No (at 0)	Robust to outliers; median-oriented estimate.
Mean Squared Error (MSE)	Regression	Error-based	Yes	Yes	Sensitive to outliers; MLE for Gaussian noise.
Root Mean Sq. Error (RMSE)	Regression	Error-based	Yes	Yes	Same units as target; widely used.
Huber Loss	Regression	Error-based	Yes	Yes	Robust hybrid of MAE and MSE; requires threshold $\delta$.
Log-cosh Loss	Regression	Error-based	Yes	Yes	Smooth approximation of MAE; no threshold hyperparameter.
RMSLE	Regression	Error-based	No	Yes	Works on log-transformed targets; emphasizes relative errors.
Zero-One Loss	Classification	Margin-based	No	No	Direct classification error; intractable.
Hinge Loss	Classification	Margin-based	Yes	No	Basis for SVMs; max-margin principle.
Perceptron Loss	Classification	Margin-based	Yes	No	No margin enforcement; strictly error-driven.
Smoothed Hinge	Classification	Margin-based	Yes	Yes	Differentiable variant of Hinge.
Quad. Smoothed Hinge	Classification	Margin-based	Yes	Yes	Piece-wise quadratic smoothing.
Modified Huber	Classification	Margin-based	Yes	Yes	Smooth; used for robust classification.
Ramp Loss	Classification	Margin-based	No	No	Capped Hinge; robust to outliers.
Cosine Similarity	Classification	Margin-based	No	Yes	Measures orientation; ignores magnitude.
Cross-Entropy (NLL)	Classification	Probabilistic	Yes	Yes	Standard for classification; MLE-based.
KL Divergence	Classification	Probabilistic	Yes	Yes	Measures information loss; asymmetric.
Focal Loss	Classification	Probabilistic	No	Yes	Focuses on hard examples; handles imbalance.
Dice Loss	Classification	Probabilistic	No	Yes	Overlap metric; for imbalance/segmentation.
Tversky Loss	Classification	Probabilistic	No	Yes	Generalizes Dice; tunable FP/FN balance.
VAE Loss (ELBO)	Generative	Probabilistic	No	Yes	Reconstruction + KL regularization.
Beta-VAE	Generative	Probabilistic	No	Yes	Trade-off for disentangled representations.
VQ-VAE	Generative	Probabilistic	No	No	Discrete latent codes; non-diff quantization.
Conditional VAE	Generative	Probabilistic	No	Yes	Conditioned generation (e.g., on labels).
Minimax Loss	Generative	Probabilistic	No	Yes	Original GAN loss; saddle point problem.
Wasserstein Loss	Generative	Probabilistic	No	Yes	Earth-Mover dist.; stable GAN training.
Diffusion (Simple)	Generative	Probabilistic	No	Yes	Noise prediction MSE; stable training.
Score-based Loss	Generative	Probabilistic	No	Yes	Fits score function (gradient of log-density).
CLIP Guidance	Generative	Margin-based	No	Yes	Aligns text/image embeddings (cosine sim).
Autoregressive LM	Generative	Probabilistic	Yes	Yes	Standard causal masking (GPT style).
Masked LM (MLM)	Generative	Probabilistic	Yes	Yes	Bidirectional context (BERT style).
Label Smoothing	Generative	Probabilistic	Yes	Yes	Prevents overconfidence; regularization.
Knwl. Distillation (KL)	Generative	Probabilistic	Yes	Yes	Compress teacher model info to student.
DPO Loss	NLP (LLM)	Probabilistic	No	Yes	Reparameterizes reward via LLM policy; requires frozen reference model.
SimPO Loss	NLP (LLM)	Probabilistic	No	Yes	Reference-free; uses length-normalized log-prob as reward with margin.
ORPO Loss	NLP (LLM)	Probabilistic	No	Yes	Reference-free alignment; optimizes odds ratio to penalize rejected outputs.
Pairwise Ranking	Ranking	Margin-based	Yes	No	Contrastive; minimizes pos/neg distance.
Triplet Loss	Ranking	Margin-based	Yes	No	Anchor-Pos-Neg structure; relative distance.
Listwise (Softmax)	Ranking	Probabilistic	No	Yes	Optimizes entire list order (top-1 prob).
Contrastive (NT-Xent)	Ranking	Margin-based	No	Yes	Self-supervised; uses multiple negatives.
LambdaLoss	Ranking	Margin-based	Yes	Yes	Directly optimizes IR metrics (NDCG).
Energy Loss	EBM	Margin-based	No	No	Direct mapping; prone to collapse.
Generalized Perceptron	EBM	Margin-based	Yes	No	Pushes down correct energy; no margin.
Energy NLL	EBM	Margin-based	Yes	Yes	Log-partition approx; probabilistic link.
Energy Hinge	EBM	Margin-based	Yes	No	Enforces margin between correct/incorrect.
Energy Log	EBM	Margin-based	Yes	Yes	Soft margin; smooth differentiable hinge.
MCE Loss	EBM	Margin-based	No	Yes	Approx. error count using sigmoid.
Square-square	EBM	Margin-based	Yes	Yes	Quadratically penalizes energy margins.
Square-exponential	EBM	Margin-based	Yes	Yes	Exponential penalty on incorrect energies.
Graph Reconstruction	Relational	Probabilistic	No	Yes	Link prediction; rebuilds adjacency.
Random Walk Loss	Relational	Probabilistic	No	Yes	Skip-gram for graphs (DeepWalk/Node2Vec).
Motif-based Loss	Relational	Error-based	No	Yes	Preserves higher-order substructures.
Graph Contrastive	Relational	Probabilistic	No	Yes	Invariance to graph augmentations.
Graph Laplacian	Relational	Error-based	Yes	Yes	Enforces smoothness among neighbors.
Mutual Info (DGI)	Relational	Probabilistic	No	Yes	Maximize local-global info agreement.
Distance Preservation	Relational	Error-based	No	Yes	Preserves structural/geodesic distances.

).

We included a loss function if it (i) has a clear mathematical definition, (ii) is widely used across multiple works or appears as a standard objective in common machine learning frameworks, and (iii) plays a central role in at least one major task family in our taxonomy (regression, classification, generative modeling, ranking, energy-based modeling, and relational learning). Highly specialized or very recent proposals with limited adoption were excluded unless they instantiate a general pattern that transfers across tasks.

For this reason, we have worked to build a taxonomy of loss functions that highlights the advantages and disadvantages of each technique. We hope this will be useful for new users who want to familiarize themselves with common loss functions and identify those most suitable for a problem they are trying to solve. We also hope this summary will serve as a comprehensive reference for advanced users, allowing them to quickly compare alternatives without having to search broadly across the literature. More generally, the survey is intended to help researchers contextualize newly proposed objectives by showing whether they fit within an existing category or instead represent a genuinely new direction.

Overall, we include 52 widely used loss functions. In each section, we organize the losses according to the broad class of tasks they address, define each loss mathematically, and discuss its most common applications, advantages, and drawbacks. Rather than isolating common machine learning challenges into a separate section, we discuss them directly within the presentation of each loss function whenever they motivate its design or use. In particular, issues such as class imbalance, sensitivity to outliers, and robustness to noisy data are addressed contextually alongside the mathematical definition and practical interpretation of the corresponding losses.

The main contribution of this work is the proposed taxonomy depicted in Figure 1. Each loss function is first divided according to the task on which it is exploited: regression, classification, ranking, generative modeling, energy-based modeling, and relational learning. Finally, we classify each loss function by its underlying strategy, such as error minimization, probabilistic formalization, or margin maximization.

This work is organized as follows: In Section 2, we provide a formal definition of a loss function and introduce our taxonomy. In Section 3, we describe the most common regularization methods used to reduce model complexity. In Section 4, we describe the regression task and the key loss functions used to train regression models. In Section 5, we introduce the classification problem and the associated loss functions. In Section 6, we present generative models and their losses. Ranking problems and their loss functions are introduced in Section 7, and energy-based models and their losses are described in Section 8. (While energy-based models (EBMs) are often grouped with generative models, we treat them separately because they learn unnormalized energy functions (rather than normalized likelihoods), use distinct losses (often contrastive or margin-based), and are applied in both generative and discriminative settings.) Relational learning losses are presented in Section 9. Finally, we draw conclusions in Section 10.

2. Definition of the Loss Function Taxonomy

In a general machine learning problem, the aim is to learn a function f that transforms an input, defined by the input space $\mathsf{\Phi }$, into a desirable output, defined by the output space $\mathcal{Y}$:

$$f:\mathsf{\Phi }\to \mathcal{Y}$$

(1)

where f is a function that can be approximated by a model, $f_{\mathbf{\Theta }}$, parameterized by the parameters $\mathbf{\Theta }$.

Given a set of inputs $\{{\mathbf{x}}_0,\dots ,{\mathbf{x}}_N\}\in \mathsf{\Phi }$, they are used to train the model with reference to target variables in the output space, $\{{\mathbf{y}}_0,\dots ,{\mathbf{y}}_N\}\in \mathcal{Y}$. Notice that, in some cases (such as autoencoders), $\mathcal{Y}=\mathsf{\Phi }$.

A loss function, L, is defined as a mapping of $f\left({\mathbf{x}}_i\right)$ with its corresponding ${\mathbf{y}}_i$ to a real number $l\in \mathbb{R}$, which captures the similarity between $f\left({\mathbf{x}}_i\right)$ and ${\mathbf{y}}_i$. Aggregating over all the points of the dataset, we find the overall loss, $\mathcal{L}$:

$$\begin{array}{c}\hfill \mathcal{L}\left(f\right|\{{\mathbf{x}}_0,\dots ,{\mathbf{x}}_N\},\{{\mathbf{y}}_0,\dots ,{\mathbf{y}}_N\})={\displaystyle \frac{1}{N}}\sum _{i=1}^{N}L(f\left({\mathbf{x}}_i\right),{\mathbf{y}}_i)\end{array}$$

(2)

The optimization function to be solved is defined as:

$$\begin{array}{c}\hfill \underset{f}{min}\mathcal{L}\left(f\right|\{{\mathbf{x}}_0,\dots ,{\mathbf{x}}_N\},\{{\mathbf{y}}_0,\dots ,{\mathbf{y}}_N\})\end{array}$$

(3)

Notice that it is often convenient to explicitly introduce a regularization term (R) that maps f to a real number $r\in \mathbb{R}$. This term is usually used for penalizng the complexity of the model in the optimization [3]:

$$\begin{array}{c}\hfill \underset{f}{min}{\displaystyle \frac{1}{N}}\sum _{i=1}^{N}L(f\left({\mathbf{x}}_i\right),{\mathbf{y}}_i)+R\left(f\right)\end{array}$$

(4)

In practice, the family of functions chosen for the optimization can be parameterized by a parameter vector $\mathbf{\Theta }$, which allows the minimization to be defined as an exploration in the parameter space:

$$\begin{array}{c}\hfill \underset{\mathbf{\Theta }}{min}{\displaystyle \frac{1}{N}}\sum _{i=1}^{N}L(f_{\mathbf{\Theta }}\left({\mathbf{x}}_i\right),{\mathbf{y}}_i)+R\left(\mathbf{\Theta }\right)\end{array}$$

(5)

2.1. Optimization Techniques for Loss Functions

2.1.1. Loss Functions and Optimization Methods

In this section, we list the most common mathematical properties that a loss may or may not satisfy, and then we briefly discuss the main optimization methods employed to minimize them. For the sake of simplicity, visualization, and understanding, we define such properties in a two-dimensional space, but they can be easily generalized to a d-dimensional one.

• Continuity (CONT): A real-valued function, that is a function from a subset of the real numbers to the real numbers, can be represented by a graph in the Cartesian plane; such a function is continuous if the graph is a single unbroken curve belonging to the real domain. A more mathematically rigorous definition can be given in terms of limits: a function f with variable x is continuous at the point c if

  $$\underset{x\to c}{lim}f\left(x\right)=f\left(c\right).$$

  (6)
• Differentiability (DIFF): A differentiable function f on a real variable is a function derivable at each point of its domain. A differentiable function is smooth, in the sense that it is locally well approximated by a linear function, and does not contain any break, angle, or cusp. A continuous function is not necessarily differentiable, but a differentiable function is necessarily continuous. More formally, in one dimension, f is differentiable at c if the following limit exists:

  $$f^{\prime }\left(c\right)=\underset{h\to 0}{lim}{\displaystyle \frac{f(c+h)-f\left(c\right)}{h}}.$$

  (7)
• Lipschitz Continuity (L-CONT): A Lipschitz continuous function is limited in how fast it can change. More formally, there exists a real constant $L\ge 0$ such that, for every pair of points in the domain,

  $$\left|f\right(x)-f(y\left)\right|\le L\parallel x-y\parallel ,\forall x,y\in \mathrm{dom}\left(f\right),$$

  (8)

  where L is called the Lipschitz constant of the function.
  To understand the robustness of a model, such as a neural network, some research papers [28,29] have tried to train the underlying model by defining an input–output map with a small Lipschitz constant. The intuition is that if a model is robust, it should not be too affected by perturbations in the input, $f(x+\delta x)\approx f\left(x\right)$, and this would be ensured by having f be ℓ-Lipschitz where ℓ is small [30].
• Convexity (CONVEX): A real-valued function f is convex if each segment between any two points on the graph of the function lies above the graph between the two points. More formally, f is convex if for all $x,y$ in its domain and all $\lambda \in [0,1]$,

  $$f(\lambda x+(1-\lambda \left)y\right)\le \lambda f\left(x\right)+(1-\lambda )f\left(y\right).$$

  (9)
  Convexity is a key feature since any local minimum of a convex function is also a global minimum. Whenever the second derivative exists, convexity is easy to check, since the Hessian of the function must be positive semi-definite.
• Strict Convexity (S-CONV): A real-valued function is strictly convex if the segment between any two points on the graph of the function lies strictly above the graph between the two points, except at the intersection points between the straight line and the curve. More formally, for all distinct $x,y$ in the domain and all $\lambda \in (0,1)$,

  $$f(\lambda x+(1-\lambda \left)y\right)<\lambda f\left(x\right)+(1-\lambda )f\left(y\right).$$

  (10)
  Strict convexity implies that, if a minimizer exists, it is unique. If the Hessian exists and is positive definite, this is a sufficient condition for strict convexity.

2.1.2. Relevant Optimization Methods

An optimization method is a technique that, given a formalized optimization problem with an objective function, returns the solution to obtain the optimal value of that optimization problem. Most of the optimization methods presented in this work rely on algorithms that may not guarantee the optimality of the solution, but imply a degree of approximation.

For each optimization method, we specify the mathematical properties that the loss function must satisfy, such as continuity, differentiability, or convexity. These requirements are listed in the headings of each method, using blue to indicate the necessary properties for the method’s usability and red for properties that provide optimality guarantees.

• Closed-Form Solutions (DIFF, S-CONV): These are systems of equations solvable analytically, where values of $\mathbf{\Theta }$ make the derivative of the loss function equal to zero. To guarantee a unique closed-form solution, the loss function must be differentiable (DIFF) and strictly convex (S-CONV), ensuring a single global minimum. Closed-form solutions are highly efficient and desirable where feasible; however, they are often impractical for complex models or high-dimensional parameter spaces. Therefore, closed-form solutions are primarily used in simpler, linear models or settings where the loss is quadratic or log-likelihood based, as in linear regression or Gaussian MLE problems.
• Gradient Descent (DIFF, CONVEX): Gradient descent is a first-order iterative optimization algorithm used to find a local minimum of a differentiable function. The loss function must be at least differentiable (DIFF) to compute gradients, and if the loss is convex (CONVEX), the local minimum is also the global minimum. Lipschitz continuity (L-CONT) can improve convergence guarantees, as it limits how quickly the function can change, but it is not strictly necessary for gradient descent. For non-differentiable losses, techniques like subgradients or gradient approximations can be employed [31,32]. The algorithm for gradient descent is formalized in Algorithm 1. In each iteration, it calculates the gradient of the loss function with respect to the current parameters, $\nabla \mathcal{L}\left({\mathbf{\Theta }}^{\left(t\right)}\right)$, and updates those parameters by taking a step of size $\alpha$ (the learning rate) in the opposite direction of the gradient to descend toward the minimum.

Algorithm 1 Gradient Descent

Input: initial parameters ${\mathbf{\Theta }}^{\left(0\right)}$, number of iterations T, learning rate $\alpha$

Output: final learning ${\mathbf{\Theta }}^{\left(T\right)}$

1: $t=0$ 2: while $t<T$ do 3:       estimate $\nabla \mathcal{L}\left({\mathbf{\Theta }}^{\left(t\right)}\right)$ 4:       compute $\Delta {\mathbf{\Theta }}^{\left(t\right)}=-\nabla \mathcal{L}\left({\mathbf{\Theta }}^{\left(t\right)}\right)$ 5:       ${\mathbf{\Theta }}^{(t+1)}:={\mathbf{\Theta }}^{\left(t\right)}+\alpha \Delta {\mathbf{\Theta }}^{\left(t\right)}$ 6: end while 7: return  ${\mathbf{\Theta }}^{\left(T\right)}$

• Stochastic Gradient Descent (SGD) (DIFF, CONVEX): SGD [3] is a stochastic approximation of gradient descent that computes the gradient from a randomly selected subset of the data instead of the entire dataset. This reduces the computational cost in high-dimensional problems, such as neural networks, and helps avoid local minima due to the stochastic nature of the updates. Like gradient descent, SGD requires the loss function to be differentiable (DIFF). Convexity (CONVEX) ensures that the global minimum is reachable, but even for non-convex functions, SGD can often find useful minima in practice. Lipschitz continuity (L-CONT) can improve the convergence rate, but is not required.
• Derivative-Free Optimization: In some cases, the derivative of the objective function may not exist or be difficult to compute. Derivative-free optimization methods, such as simulated annealing, genetic algorithms, and particle swarm optimization, can be employed [33,34]. While these methods do not strictly require continuity (CONT), having a continuous function typically improves the stability of the optimization process. Derivative-free methods can handle non-differentiable and non-convex losses, but they may struggle to scale to high-dimensional problems and can be computationally expensive.
• Zeroth-Order Optimization (ZOO): ZOO optimization is a subset of derivative-free optimization that approximates gradients using function evaluations rather than direct computation of derivatives [35]. These methods are useful in black-box scenarios where the gradient is not accessible but can be estimated through perturbations. While continuity (CONT) is not required, it improves the accuracy of gradient approximations and helps achieve better convergence rates. ZOO methods are effective for non-differentiable and non-convex losses, and they have been applied in adversarial attack generation, model-agnostic explanations, and other black-box scenarios [36,37].

2.2. The Proposed Taxonomy

Our taxonomy is summarized in Figure 1. To define it, we started by categorizing the losses depending on which machine learning problem they are best suited to solve. We have identified the following categories:

• Regression (Section 4);
• Classification (Section 5);
• Generative modelling (Section 6);
• Ranking (Section 7);
• Energy-based modelling (Section 8);
• Relational learning (Section 9).

We also made a distinction based on the mathematical concepts used to define the loss, obtaining the following sub-categories:

• Error-based;
• Probabilistic;
• Margin-based.

Using this approach, we developed a compact and intuitive taxonomy. We employed well-established terminology to ensure that users can intuitively navigate and understand the taxonomy.

3. Regularization Methods

Regularization methods can be applied to almost all loss functions. They are employed to reduce model complexity, simplifying the trained model and reducing its propensity to overfit the training data [38,39,40]. Model complexity is usually measured by the number of parameters and their magnitude [3,40,41]. Many techniques fall under the umbrella of regularization method and a significant number of them are based on the augmentation of the loss function [3,38]. An intuitive justification for regularization is that it imposes Occam’s razor on the complexity of the final model. More theoretically, many loss-based regularization techniques are equivalent to imposing certain prior distributions on the model parameters.

3.1. Regularization by Loss Augmentation

One can design the loss function to penalize the magnitude of model parameters, thus learning the best trade-off between bias and variance of the model and reducing the generalization error without affecting the training error too much. This prevents overfitting, while avoiding underfitting, and can be done by augmenting the loss function with a term that explicitly controls the magnitude of the parameters, or implicitly reduces the number of them. The general way of augmenting a loss function to regularize the result is formalized in the following equation:

$$\widehat{L}(f\left({\mathbf{x}}_i\right),{\mathbf{y}}_i)=L(f\left({\mathbf{x}}_i\right),{\mathbf{y}}_i)+\lambda \rho \left(\mathbf{\Theta }\right)$$

(11)

where $\rho \left(\mathbf{\Theta }\right)$ is called the regularization function and $\lambda$ defines the amount of regularization (the trade-off between fit and generalization).

This general definition makes it clear that we can employ regularization on any of the losses proposed in this paper.

We are now going to describe the most common regularization methods based on loss augmentation.

3.1.1. L2-Norm Regularization

In $L_2$ regularization, the loss is augmented to include the weighted $L_2$ norm of the weights [3,42], so the regularization function is $\rho \left(\mathbf{\Theta }\right)={∥\mathbf{\Theta }∥}_2^2$:

$$\widehat{L}(f\left({\mathbf{x}}_i\right),{\mathbf{y}}_i)=L(f\left({\mathbf{x}}_i\right),{\mathbf{y}}_i)+\lambda {∥\mathbf{\Theta }∥}_2^2$$

(12)

when this is employed in regression problems, it is also known as Ridge regression [3,43].

3.1.2. $L_1$-Norm Regularization

In $L_1$ regularization the loss is augmented to include the weighted $L_1$ norm of the weights [3,42], so the regularization function is $\rho \left(\mathbf{\Theta }\right)=∥\mathbf{\Theta }∥$

$$\widehat{L}(f\left({\mathbf{x}}_i\right),{\mathbf{y}}_i)=L(f\left({\mathbf{x}}_i\right),{\mathbf{y}}_i)+\lambda {∥\mathbf{\Theta }∥}_1$$

(13)

when this is employed in regression problems it is also known as Lasso regression [3,44].

3.2. Comparison Between $L_2$ and $L_1$ Norm Regularizations

$L_1$ and $L_2$ regularizations are both based on the same concept of penalizing the magnitude of the weights composing the models. Despite this, the two methods have important differences in their employability and their effects on the result.

One of the most crucial differences is that $L_1$, when optimized, can shrink weights to 0, while $L_2$ results in non-zero (smoothed) values [3,4,42,45,46]. This allows $L_1$ to reduce the dimensions of a model’s parameter space and perform an implicit feature selection. Indeed, it has been shown by [45] that by employing $L_1$ regularization on logistic regression, the sample complexity (i.e., the number of training examples required to learn “well”) grows logarithmically in the number of irrelevant features. On the contrary, the authors show that any rotationally invariant algorithm (including logistic regression) with $L_2$ regularization has a worst-case sample complexity that grows at least linearly in the number of irrelevant features. Moreover, $L_2$ is more sensitive to the outliers than $L_1$-norm since it squares the error.

$L_2$ is continuous, while $L_1$ is a piece-wise function. The main advantage of $L_2$ is that it is differentiable, while $L_1$ is non-differentiable at 0, which has some strong implications. Precisely, the $L_2$ norm can be easily trained with gradient descent, while $L_1$ sometimes cannot be efficiently applied. The first problem is the inefficiency of applying the $L_1$ penalty to the weights of all the features, especially when the dimensions of the feature space tend to be very large [47], producing a significant slowdown of the weight-updating process. Finally, the naive application of $L_1$ penalty in SGD does not always lead to compact models, because the approximate gradient used at each update could be very noisy, so the weights of the features can be easily moved away from zero by those fluctuations, and $L_1$ loses its main advantages for $L_2$ [47].

4. Regression Losses

4.1. Problem Formulation and Notation

A regression model aims to predict the value of a continuous target variable y (the dependent variable) from one or more predictor variables $\mathbf{x}$ (the independent variables). More precisely, let $f_{\mathbf{\Theta }}$ be a generic model parameterized by $\mathbf{\Theta }$, which maps an input ${\mathbf{x}}_i\in {\mathbb{R}}^D$ to a real-valued output,

$$f_{\mathbf{\Theta }}:{\mathbb{R}}^D\to \mathbb{R}.$$

Given a dataset of input–output pairs ${\left\{({\mathbf{x}}_i,y_i)\right\}}_{i=1}^N$, the goal is to estimate the parameters $\mathbf{\Theta }$ that best fit the data by minimizing a loss function $\mathcal{L}$.

All the losses considered for regression are based on functions of the residuals, i.e., the difference between the observed value $y_i$ and the predicted value $f_{\mathbf{\Theta }}\left({\mathbf{x}}_i\right)$. In the following, we denote by $f_{\mathbf{\Theta }}\left({\mathbf{x}}_i\right)$ the prediction corresponding to the input ${\mathbf{x}}_i$, and by $y_i$ the associated ground-truth target.

As highlighted by Figure 2, the mean bias error ($MBE$) can be regarded as a basic reference point for regression losses, from which several important variants can be derived. Among the most widely used are the mean absolute error ($MAE$), mean squared error ($MSE$), their regularized counterparts in linear modeling (lasso and ridge), and the root mean squared error ($RMSE$). In this section, we also introduce the Huber loss and the Smooth $L_1$ loss, which combine properties of both $MAE$ and $MSE$. Finally, we present the log-cosh loss and the root mean squared logarithmic error loss.

4.2. Error-Based Losses for Regression

Since regression models aim to minimize the discrepancy between actual and predicted values, the associated loss functions are typically classified as error-based. These losses directly measure the magnitude of the residuals in order to improve predictive accuracy.

4.2.1. Mean Bias Error Loss (CONT, DIFF, CONVEX)

The most straightforward regression loss is the mean bias error (MBE), defined in Equation (14). It measures the average signed residual and therefore captures whether a model tends, on average, to overestimate or underestimate the target variable. For this reason, MBE is mainly used as an evaluation measure rather than as a training objective. Indeed, positive and negative residuals may cancel each other out, leading to a deceptively small loss even when individual prediction errors are large [48,49,50].

The MBE loss is defined as

$${\mathcal{L}}_{MBE}={\displaystyle \frac{1}{N}}\sum _{i=1}^N\left(y_i-f\left({\mathbf{x}}_i\right)\right).$$

(14)

Several standard regression losses can be viewed as modifications of MBE that avoid error cancellation by transforming the residuals before aggregation. In particular, MAE, MSE, and log-cosh differ from MBE in how they penalize the signed prediction error.

4.2.2. Mean Absolute Error Loss (L-CONT, CONVEX)

The mean absolute error loss or L1 loss is one of the most basic loss functions for regression. It measures the average of the absolute bias in the prediction. The absolute value overcomes the problem of the $MBE$, ensuring that positive errors do not cancel the negative ones. Therefore, each error contributes to $MAE$ in proportion to the absolute value of the error. Notice that the contribution of the errors follows a linear behavior, meaning that many small errors are as important as a big one. This implies that the gradient magnitude is not dependent on the error size, and thus may lead to convergence problems when the error is small. A model trained to minimize the MAE is more effective when the target data conditioned on the input is symmetric. It is important to highlight that the derivative of the absolute value at zero is not defined.

As for MBE, MAE is also used to evaluate the performances of the models [51,52].

$${\mathcal{L}}_{MAE}={\displaystyle \frac{1}{N}}\sum _{i=1}^N\left|y_i-f\left({\mathbf{x}}_i\right)\right|$$

(15)

4.2.3. Mean Squared Error Loss (CONT, DIFF, CONVEX)

The mean squared error (MSE) loss, or L2 loss, is the average of the squared differences between observed values y and predicted values $\widehat{y}$. It is widely used in regression tasks. The squared term ensures that all errors are positive and amplifies the impact of outliers, making it suitable for problems where the noise in observations follows a normal distribution.

The MSE loss is defined as:

$${\mathcal{L}}_{MSE}={\displaystyle \frac{1}{N}}\sum _{i=1}^N{\left(y_i-f\left({\mathbf{x}}_i\right)\right)}^2$$

(16)

One key drawback of MSE is its sensitivity to outliers, as large errors have a disproportionately high influence due to the squaring of residuals.

Interpretation as Maximum Likelihood Estimation (MLE)

From a probabilistic viewpoint, MSE can be derived as a form of maximum likelihood estimation (MLE) under the assumption that the errors between the predicted and observed values follow a Gaussian (normal) distribution with constant variance [42,53]. Minimizing MSE is equivalent to maximizing the likelihood of the observed data under this Gaussian noise assumption. This probabilistic interpretation explains why MSE is commonly used when the residuals are expected to follow a Gaussian distribution and highlights its role as a standard regression loss.

4.2.4. Lasso Regression ($L_1$ Regularization)

Lasso regression is derived from augmenting the MSE loss with an $L_1$ regularization term, as detailed in Section 3.1.2. The regularized loss function penalizes the absolute magnitude of the model parameters, leading to sparsity in the learned model by driving some of the weights to zero. This makes Lasso particularly useful for feature selection in high-dimensional datasets [3,44].

The loss function for Lasso regression is:

$${\mathcal{L}}_{\mathrm{Lasso}}(f\left({\mathbf{x}}_i\right),{\mathbf{y}}_i)={\displaystyle \frac{1}{N}}\sum _{i=1}^N{\left(y_i-f\left({\mathbf{x}}_i\right)\right)}^2+\lambda {∥\mathbf{\Theta }∥}_1$$

(17)

The $L_1$ term encourages sparsity, shrinking irrelevant model weights to zero, which implicitly performs feature selection. However, as noted in Section 3.1.2, lasso can struggle with correlated features, often selecting only one from a group of correlated variables. Additionally, lasso is non-differentiable at zero, which may pose challenges in optimization, particularly when using gradient-based methods.

4.2.5. Ridge Regression ($L_2$ Regularization)

Ridge regression is an extension of MSE with an $L_2$ regularization term, as described in Section 3.1.1. The $L_2$ term penalizes the square of the model parameters, discouraging large coefficients and helping to mitigate overfitting in regression tasks, especially when there is multicollinearity (high correlation between features) [43].

The loss function for ridge regression is:

$${\mathcal{L}}_{\mathrm{Ridge}}(f\left({\mathbf{x}}_i\right),{\mathbf{y}}_i)={\displaystyle \frac{1}{N}}\sum _{i=1}^N{\left(y_i-f\left({\mathbf{x}}_i\right)\right)}^2+\lambda {∥\mathbf{\Theta }∥}_2^2$$

(18)

Interpretation as Maximum A Posteriori Estimation (MAP)

From a Bayesian perspective, ridge regression can be interpreted as maximum a posteriori estimation (MAP) [53]. In this context, the $L_2$ regularization term corresponds to a Gaussian prior on the model parameters, with the objective being to maximize the posterior distribution of the parameters given the data. This probabilistic interpretation shows that ridge regression shrinks the model coefficients towards zero without eliminating them entirely, as occurs in lasso.

Ridge regression is particularly effective when dealing with correlated features, as it distributes the coefficient weights more smoothly across them. However, unlike lasso, ridge does not perform feature selection, retaining all input features with non-zero weights.

4.2.6. Root Mean Squared Error Loss (CONT, DIFF, CONVEX)

The root mean squared error (RMSE) loss is directly related to the MSE and differs from it only by the presence of the square root term. Its main advantage is that it preserves the same units and scale as the target variable, which often makes its value easier to interpret in practical applications. Since the square root is a monotonic transformation, minimizing RMSE yields the same minimizer as minimizing MSE. However, the two objectives may induce different optimization dynamics because their gradients are scaled differently.

As with the previously presented losses, RMSE is also widely used as an evaluation metric for regression models [50,52], and it inherits the main limitation of MSE, namely its sensitivity to large residuals and outliers.

$${\mathcal{L}}_{RMSE}=\sqrt{{\displaystyle \frac{1}{N}}\sum _{i=1}^N{\left(y_i-f\left({\mathbf{x}}_i\right)\right)}^2}$$

(19)

Strictly speaking, RMSE is not differentiable when all residuals are exactly zero, although this is usually a negligible corner case in practice.

4.2.7. Huber Loss and Smooth $L_1$ Loss (L-CONT, DIFF, CONVEX)

The Huber loss [54] is a regression loss that combines the main advantages of the MAE and the MSE. It behaves quadratically for small residuals and linearly for large residuals, making it less sensitive to outliers than the MSE while remaining differentiable at zero, unlike the MAE. It is parameterized by $\delta$, which determines the transition point between the quadratic and linear regimes.

More precisely, when $\left|y_i-f\left({\mathbf{x}}_i\right)\right|\le \delta$, the Huber loss behaves like the MSE, whereas for larger residuals it behaves like the MAE. This makes it particularly useful in settings where robustness to outliers is needed without completely discarding the smooth optimization properties of squared-error losses.

$$L_{\mathrm{Huber}}=\left\{\begin{array}{cc}{\displaystyle \frac{1}{2}}{\left(y_i-f\left({\mathbf{x}}_i\right)\right)}^2\hfill & \mathrm{if}\left|y_i-f\left({\mathbf{x}}_i\right)\right|\le \delta ,\hfill \\ \delta \left(\left|y_i-f\left({\mathbf{x}}_i\right)\right|-{\displaystyle \frac{1}{2}}\delta \right)\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(20)

The choice of $\delta$ is crucial, since it controls the balance between sensitivity to small residuals and robustness to large ones. The main limitation of the Huber loss is therefore the presence of this additional hyperparameter, which must be chosen according to the scale of the noise or the notion of outlier relevant to the application.

A specific case of the Huber loss is the smooth $L_1$ loss, obtained when $\delta =1$. This loss has been shown to be particularly useful in tasks where a balance is needed between sensitivity to small residuals and robustness to outliers, such as object detection [55] and bounding-box regression in frameworks such as faster R-CNN [56].

4.2.8. Log-Cosh Loss (L-CONT, DIFF, S-CONV)

The log-cosh loss is defined as the logarithm of the hyperbolic cosine of the residual between the observed value $y_i$ and the predicted value $f\left({\mathbf{x}}_i\right)$. It behaves similarly to the MSE for small residuals and to the MAE for large ones. Indeed, for small values of r,

$$log(cosh\left(r\right))\approx {\displaystyle \frac{r^2}{2}},$$

whereas for large values of r,

$$log(cosh(r\left)\right)\approx \left|r\right|-log\left(2\right).$$

This makes the log-cosh loss a smooth alternative that combines the local quadratic behavior of the MSE with the increased robustness of the MAE for large residuals.

$${\mathcal{L}}_{\mathrm{logcosh}}={\displaystyle \frac{1}{N}}\sum _{i=1}^{N}log\left(cosh\left(y_i-f\left({\mathbf{x}}_i\right)\right)\right)$$

(21)

The log-cosh loss shares many of the advantages of the Huber loss, but without requiring the manual selection of a threshold hyperparameter. On the other hand, it is computationally more expensive because of the hyperbolic cosine and logarithmic terms, and it is less flexible than the Huber loss since it does not allow explicit control of the transition between the quadratic and linear regimes.

4.2.9. Root Mean Squared Logarithmic Error Loss (CONT, DIFF)

The root mean squared logarithmic error (RMSLE) loss, formalized in Equation (22), is the root mean squared error computed on the log-transformed observed values and predictions. By applying the logarithm to both $y_i$ and $f\left({\mathbf{x}}_i\right)$, this loss emphasizes relative rather than absolute discrepancies, making it particularly useful when the target variable spans several orders of magnitude. The addition of 1 inside the logarithm ensures that zero-valued targets and predictions can be handled.

The RMSLE loss is defined as

$${\mathcal{L}}_{RMSLE}=\sqrt{{\displaystyle \frac{1}{N}}\sum _{i=1}^N{\left(log(y_i+1)-log(f\left({\mathbf{x}}_i\right)+1)\right)}^2}$$

(22)

Compared with RMSE, RMSLE dampens the effect of large residuals when both the predicted and observed values are large, due to the compressive effect of the logarithm. For this reason, it is often preferred when relative errors are more important than absolute ones, or when the target values exhibit a large dynamic range. However, RMSLE is not suitable when target values may be negative, since the logarithm is undefined in that case.

This loss is particularly useful in regression problems where proportional differences matter more than exact magnitude matching. On the other hand, it is less appropriate when accurate prediction of large absolute values is critical, since the logarithmic transformation reduces the influence of large deviations [57,58].

5. Classification Losses

5.1. Problem Formulation and Notation

Classification is a class of supervised learning problems in which the goal is to assign an input $\mathbf{x}$ to one or more discrete classes. This can be achieved by training a model $f_{\mathbf{\Theta }}$ with parameters $\mathbf{\Theta }$ through the minimization of a suitable loss function L.

A first, general way to represent classification is to consider models that return a discrete label vector. In this case, the output space can be written as

$$\begin{array}{cc}\hfill f:& \mathsf{\Phi }\to {\mathsf{\Lambda }}^K\hfill \\ \hfill & \mathsf{\Lambda }=\{0,1\}.\hfill \end{array}$$

This formulation also covers multi-label classification, since more than one label may be associated with the same sample, e.g., $f\left(\mathbf{x}\right)=[0,1,0,1,0,0]$. To restrict the setting to single-label classification, it is sufficient to impose that exactly one label is active, i.e.,

$$\sum _{k=1}^K{\lambda }_k=1.$$

We can also consider models with continuous outputs, where for each class $k\in \{1,\dots ,K\}$, the model returns a confidence score or probability $p_k\left(\mathbf{x}\right)\in [0,1]$:

$$\begin{array}{cc}\hfill f:& \mathsf{\Phi }\to P^K\hfill \\ \hfill & P=[0,1].\hfill \end{array}$$

As before, to move from the multi-label to the single-label setting, the outputs must satisfy

$$\sum _{k=1}^K{p}_k\left(\mathbf{x}\right)=1,$$

so that the prediction can be interpreted as a probability distribution over the classes.

A more specific notation can be introduced for binary classification problems. This is useful here because many classical margin-based losses are originally defined in the binary setting. In this case, the label space is

$$\mathcal{Y}=\{-1,1\},$$

while the predictor is typically a real-valued scoring function

$$f:\mathsf{\Phi }\to \mathbb{R},$$

with prediction given by $\widehat{y}=\mathrm{sign}\left(f\left(\mathbf{x}\right)\right)$. This formulation is natural because margin-based losses depend on the quantity $yf\left(\mathbf{x}\right)$.

Although these losses are most naturally introduced in the binary setting, they can be generalized to multi-class classification either by decomposing the problem into multiple binary tasks (e.g., one-vs.-rest or one-vs.-one) or by adopting direct multiclass formulations [59,60,61]. Multi-label classification is instead usually handled by learning one binary predictor per label or by using probabilistic objectives independently across labels.

In this survey, we divide classification losses into two main macro-categories according to the underlying optimization strategy, namely margin-based losses and probabilistic losses, as illustrated in Figure 3. We first introduce the margin-based family, beginning with the most basic and intuitive formulation, the zero-one loss. We then discuss the hinge loss and its variants, including the smoothed hinge and quadratically smoothed hinge losses, followed by the modified Huber loss, the ramp loss, and the cosine similarity loss.

We then present the probabilistic losses, starting with cross-entropy and negative log-likelihood, which coincide from a mathematical point of view. Next, we introduce the Kullback–Leibler (KL) divergence, followed by additional losses commonly used in classification and segmentation. These include focal loss, which addresses class imbalance by focusing on hard examples; Dice loss, which measures overlap between predicted and target segmentations; and Tversky loss, which extends Dice loss by explicitly controlling the trade-off between false positives and false negatives in highly imbalanced settings.

5.2. Margin-Based Loss Functions

In this section, we introduce the most known margin-based loss functions.

5.2.1. Zero-One Loss

The most basic and intuitive margin-based classification loss is the zero-one loss, which assigns a value of 1 to a misclassified observation and 0 to a correctly classified one:

$$L_{\mathrm{ZeroOne}}(f\left(\mathbf{x}\right),y)=\left\{\begin{array}{cc}1\hfill & \mathrm{if}f\left(\mathbf{x}\right)·y<0\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(23)

Zero-one loss is not directly usable since it lacks convexity and differentiability. However, it is possible to derive employable surrogate losses that are classification calibrated, meaning they are a relaxation of $L_{\mathrm{ZeroOne}}$, an upper bound, or an approximation of this loss. A significant achievement of the recent literature on binary classification has been the identification of necessary and sufficient conditions under which such relaxations yield Fisher consistency (Fisher consistency, in this context, means that minimizing the surrogate loss also minimizes the expected zero-one loss under the true data distribution, ensuring reliable classification outcomes) [62,63,64,65,66,67]. All the following losses in this section satisfy such conditions.

5.2.2. Hinge Loss and Perceptron Loss (L-CONT, CONVEX)

The most famous surrogate loss is the hinge loss [68], which linearly penalizes every prediction where the resulting agreement is $\le 1$.

$$L_{\mathrm{Hinge}}(f\left(\mathbf{x}\right),y)=max(0,1-(f\left(\mathbf{x}\right)·y))$$

(24)

The hinge loss is not strictly convex, but it is Lipschitz continuous and convex, so many of the usual convex optimizers used in machine learning can work with it. The hinge loss is commonly employed to optimize the support vector machine (SVM [69,70]).

To train the perceptron [71], a variation of this loss, the perceptron loss, is employed. This loss slightly differs from the hinge loss, because it does not penalize samples inside the margin, surrounding the separating hyperplane, but just the ones that are mislabeled by this hyperplane with the same linear penalization.

$$L_{\mathrm{Perceptron}}(f\left(\mathbf{x}\right),y)=max(0,-(f\left(\mathbf{x}\right)·y))$$

(25)

There are two main drawbacks to using the hinge loss. Firstly, its adoption is used to make the model sensible to outliers in the training data. Secondly, due to the discontinuity of the derivative at $\left(f\right(\mathbf{x})·y)=1$, i.e., the fact that is not continuously differentiable, hinge loss results tend to be difficult to optimize.

5.2.3. Smoothed Hinge Loss (L-CONT, CONVEX, DIFF)

A smoothed version of the hinge loss was defined in [72] with the goal of obtaining a function that is easier to optimize, as shown by the following equation:

$$L_{\mathrm{SmoothedHinge}}(f\left(\mathbf{x}\right),y)=\left\{\begin{array}{cc}{\displaystyle \frac{1}{2}}-(f\left(\mathbf{x}\right)·y)\hfill & \left(f\right(\mathbf{x})·y)\le 0\hfill \\ {\displaystyle \frac{1}{2}}{(1-(f\left(\mathbf{x}\right)·y))}^2\hfill & 0<\left(f\right(\mathbf{x})·y)<1\hfill \\ 0\hfill & \left(f\right(\mathbf{x})·y)\ge 1\hfill \end{array}\right.$$

(26)

This smoothed version of the hinge loss is differentiable. Clearly, this is not the only possible smooth version of the hinge loss. However, it is a canonical one that has the important property of being zero for z ≥ 1, and it has a constant (negative) slope for z ≤ 0. Moreover, for $0<z<1$, the loss smoothly transitions from a zero slope to a constant negative one. This loss inherits sensitivity to outliers from the original hinge loss.

5.2.4. Quadratically Smoothed Hinge Loss (L-CONT, CONVEX, DIFF)

With the same goal as the smoothed hinge loss, a quadratically smoothed version was introduced in [73] to make optimization easier while preserving the main properties of the original hinge loss. Let $z=f\left(\mathbf{x}\right)·y$. A common formulation is:

$$L_{\mathrm{QSmoothedHinge}}(f\left(\mathbf{x}\right),y)=\left\{\begin{array}{cc}0\hfill & z\ge 1\hfill \\ {\displaystyle \frac{{(1-z)}^2}{2\gamma }}\hfill & 1-\gamma \le z<1\hfill \\ 1-z-{\displaystyle \frac{\gamma }{2}}\hfill & z<1-\gamma \hfill \end{array}\right.$$

(27)

where the hyperparameter $\gamma >0$ determines the degree of smoothing. As $\gamma \to 0$, the loss approaches the original hinge loss. In contrast to the hinge loss, this formulation is differentiable over the whole domain, while remaining convex and Lipschitz continuous. The quadratic region around the margin makes the optimization smoother, although the loss still inherits some sensitivity to outliers from the original hinge loss.

5.2.5. Modified Huber Loss (L-CONT, DIFF, CONVEX)

The modified Huber loss is a slight variation of the Huber loss for regression and a special case of the quadratic smoothed hinge loss with $\gamma =2$ (for more details refer to Section 4.2.7):

$$L_{\mathrm{ModHuber}}(f\left(\mathbf{x}\right),y)=\left\{\begin{array}{cc}{\displaystyle \frac{1}{4}}max{(0,-(f\left(\mathbf{x}\right)·y))}^2\hfill & \left(f\right(\mathbf{x})·y)\ge -1\hfill \\ -\left(f\right(\mathbf{x})·y)\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(28)

5.2.6. Ramp Loss (CONT)

The ramp loss, also known as the truncated hinge loss, is a piecewise linear and continuous loss obtained by truncating the hinge loss [74]. Unlike the hinge loss, the ramp loss is bounded, which makes it more robust to outliers and mislabeled examples. This robustness comes at the price of non-convexity, which makes the optimization problem more challenging.

A common formulation is:

$$L_{\mathrm{Ramp}}(f\left(\mathbf{x}\right),y)=\left\{\begin{array}{cc}1\hfill & \mathrm{if}\left(f\right(\mathbf{x})·y)\le 0\hfill \\ 1-\left(f\right(\mathbf{x})·y)\hfill & \mathrm{if}0<\left(f\right(\mathbf{x})·y)<1\hfill \\ 0\hfill & \mathrm{if}\left(f\right(\mathbf{x})·y)\ge 1\hfill \end{array}\right.$$

(29)

Equivalently, the ramp loss can be written as

$$L_{\mathrm{Ramp}}(f\left(\mathbf{x}\right),y)=min\left(1,L_{\mathrm{Hinge}}(f\left(\mathbf{x}\right),y)\right).$$

When employed in SVM-like classifiers, truncated-hinge objectives have been shown to yield more robust classifiers and, in some cases, smaller and more stable support-vector sets than standard hinge loss formulations [74]. Extensions to multicategory settings have also been studied in the literature [61].

5.2.7. Cosine Similarity Loss (CONT, DIFF)

Cosine similarity is commonly used to compare vectors when their orientation is more important than their magnitude [3,42]. A typical example arises in text representation, where samples are encoded through word counts or embeddings and the angular agreement between vectors is more informative than their norm. When both the target and the model output can be interpreted as vectors, cosine similarity can be adapted into a loss function as follows:

$$L_{\text{cos-sim}}(f\left(\mathbf{x}\right),\mathbf{y})=1-{\displaystyle \frac{\mathbf{y}·f\left(\mathbf{x}\right)}{∥\mathbf{y}∥∥f\left(\mathbf{x}\right)∥}}.$$

(30)

This loss is differentiable whenever the norms are non-zero, and it is useful when the objective is to align directions rather than magnitudes. It is important to note that cosine similarity itself lies in the interval $[-1,1]$, whereas the corresponding loss defined above lies in the interval $[0,2]$. This normalization makes the loss insensitive to scale, which can be advantageous in some applications but may be undesirable when vector magnitude also carries meaningful information.

Although cosine similarity loss is not a classical margin-based surrogate in the SVM sense, it can be interpreted as a geometric objective that promotes angular separation and alignment in representation space, and is therefore included here as a related non-probabilistic classification loss.

5.3. Probabilistic Loss Functions

Let q be the probability distribution underlying the dataset and $f_{\mathbf{\Theta }}$ the function generating the output. Probabilistic loss functions provide some distance function between q and $f_{\mathbf{\Theta }}$. By minimizing that distance, the model output distribution converges to the ground-truth one. Usually, models trained with probabilistic loss functions can provide a measure of how likely a sample is labeled with one class instead of another [3,42,75], providing richer information compared to margin-based options.

5.3.1. Cross-Entropy Loss and Negative Log-Likelihood Loss (CONT, DIFF, CONVEX)

Maximum likelihood estimation (MLE) is a method to estimate the parameters of a probability distribution by maximizing the likelihood [3,42,76]. From the point of view of Bayesian inference, MLE can be considered a special case of maximum a posteriori estimation (MAP) that assumes a uniform prior distribution over the parameters.

Formally, in binary classification, assuming $y_i\in \{0,1\}$ and that $f_{\mathbf{\Theta }}\left({\mathbf{x}}_i\right)\in [0,1]$ is the predicted probability of class 1, given a dataset of samples $\mathcal{D}$, we maximize the following quantity:

$$P\left(\mathcal{D}\right|\mathbf{\Theta })=\prod _{i=1}^N{f}_{\mathbf{\Theta }}{\left({\mathbf{x}}_i\right)}^{y_i}·{(1-f_{\mathbf{\Theta }}\left({\mathbf{x}}_i\right))}^{1-y_i}.$$

(31)

The aim is to find the maximum likelihood estimate by minimizing a loss function. To maximize Equation (31), we can turn it into a minimization problem by employing the negative log-likelihood. To achieve this goal, we define:

$$log\left(P\left(\mathcal{D}\right|\mathbf{\Theta })\right)=\sum _{i=1}^N\left(y_{i}log\left(f_{\mathbf{\Theta }}\left({\mathbf{x}}_i\right)\right)+(1-y_i)log(1-f_{\mathbf{\Theta }}\left({\mathbf{x}}_i\right))\right)$$

(32)

Then, we obtain the loss function by taking the negative of the log:

$${\mathcal{L}}_{NLL}=-\sum _{i=1}^N\left(y_{i}log\left(f_{\mathbf{\Theta }}\left({\mathbf{x}}_i\right)\right)+(1-y_i)log(1-f_{\mathbf{\Theta }}\left({\mathbf{x}}_i\right))\right).$$

(33)

Often, the above loss is also called the cross-entropy loss, because it can be derived by minimizing the cross-entropy between the target distribution q and the model distribution $f_{\mathbf{\Theta }}$. In the continuous case, cross-entropy is defined as

$$H(q,f_{\mathbf{\Theta }})=-\int q\left(\mathbf{x}\right)log\left(f_{\mathbf{\Theta }}\left(\mathbf{x}\right)\right)d\mathbf{x},$$

(34)

while in the discrete case (which is the one of interest here), it becomes

$$H(q,f_{\mathbf{\Theta }})=-\sum _{i=1}^{N}q\left({\mathbf{x}}_i\right)log\left(f_{\mathbf{\Theta }}\left({\mathbf{x}}_i\right)\right).$$

(35)

Moreover, cross-entropy and Kullback–Leibler divergence are related through

$$\begin{array}{cc}\hfill H(q,f_{\mathbf{\Theta }})& =H\left(q\right)+{\mathcal{L}}_{\mathrm{KL}}\left(q\right||f_{\mathbf{\Theta }}),\hfill \end{array}$$

(36)

where $H\left(q\right)$ is the entropy of the target distribution and does not depend on the model parameters. Therefore, minimizing the cross-entropy with respect to $\mathbf{\Theta }$ is equivalent to minimizing the negative log-likelihood in Equation (33) and, more generally, to minimizing the Kullback–Leibler divergence.

The classical approach to extend this loss to the multi-class scenario is to add as a final activation of the model a softmax function, defined according to the number of (K) classes considered. Given a score $f_k\left(\mathbf{x}\right)$ for each class, its output can be squashed to sum up to 1 by means of a softmax function $f_S$, obtaining

$${\widehat{f}}_k\left({\mathbf{x}}_i\right)=f_S\left(f_k\left({\mathbf{x}}_i\right)\right),$$

(37)

where the softmax is defined as follows:

$$f_S\left(s_i\right)={\displaystyle \frac{e^{s_i}}{{\sum }_{j=1}^K{e}^{s_j}}}.$$

(38)

The final loss, usually called categorical cross-entropy, is

$$L_{CCE}=-\sum _{i=1}^N\sum _{k=1}^K{y}_{ik}log\left({\widehat{f}}_k\left({\mathbf{x}}_i\right)\right),$$

(39)

where $y_{ik}$ is the one-hot encoded target for sample i and class k.

5.3.2. Kullback–Leibler Divergence (CONT, CONVEX, DIFF)

The Kullback–Leibler (${\mathcal{L}}_{\mathrm{KL}}$) divergence is an information-based measure of disparity among probability distributions. Precisely, it is a non-symmetrical measurement of how one probability distribution differs from another one [3,42,77]. Technically speaking, ${\mathcal{L}}_{\mathrm{KL}}$ divergence is not a distance metric because it does not obey symmetry nor the triangle inequality, i.e., ${\mathcal{L}}_{\mathrm{KL}}\left(q\right||f_{\mathbf{\Theta }})\ne {\mathcal{L}}_{\mathrm{KL}}(f_{\mathbf{\Theta }}\left|\right|q)$. It is important to notice that, in the classification use case, minimizing the ${\mathcal{L}}_{\mathrm{KL}}$ divergence is equivalent to minimizing the cross-entropy.

Precisely, the ${\mathcal{L}}_{\mathrm{KL}}$ divergence between two continuous distributions is defined as:

$${\mathcal{L}}_{\mathrm{KL}}\left(q\right||f_{\mathbf{\Theta }})=\int q\left(\mathbf{x}\right)log\left({\displaystyle \frac{q\left(\mathbf{x}\right)}{f_{\mathbf{\Theta }}\left(\mathbf{x}\right)}}\right)d\mathbf{x}=-\int q\left(\mathbf{x}\right)log\left(f_{\mathbf{\Theta }}\left(\mathbf{x}\right)\right)d\mathbf{x}+\int q\left(\mathbf{x}\right)log\left(q\left(\mathbf{x}\right)\right)d\mathbf{x}.$$

(40)

If we want to minimize ${\mathcal{L}}_{\mathrm{KL}}$ with respect to the parameter $\mathbf{\Theta }$, since the second integral is independent of $\mathbf{\Theta }$, we obtain:

$$\underset{\mathbf{\Theta }}{min}{\mathcal{L}}_{\mathrm{KL}}\left(q\right||f_{\mathbf{\Theta }})=\underset{\mathbf{\Theta }}{min}-\int q\left(\mathbf{x}\right)log\left(f_{\mathbf{\Theta }}\left(\mathbf{x}\right)\right)d\mathbf{x}=\underset{\mathbf{\Theta }}{min}H(q,f_{\mathbf{\Theta }}).$$

(41)

For this reason, in classification problems, cross-entropy is often used in practice instead of explicitly writing the Kullback–Leibler divergence: the two objectives differ only by an additive term independent of the model parameters. However, ${\mathcal{L}}_{\mathrm{KL}}$ divergence remains particularly useful when the target is itself a soft distribution rather than a one-hot label, as in knowledge distillation, variational inference, and probabilistic regularization.

5.3.3. Focal Loss (L-CONT)

Focal loss [78] is a modification of the cross-entropy loss designed to address class imbalance by down-weighting the contribution of easy examples and focusing more on hard, misclassified examples. This is particularly effective in tasks such as object detection, where imbalanced datasets are common.

The focal loss is defined as:

$${\mathcal{L}}_{\mathrm{Focal}}=-{(1-f_{\mathbf{\Theta }}\left({\mathbf{x}}_i\right))}^{\gamma }log\left(f_{\mathbf{\Theta }}\left({\mathbf{x}}_i\right)\right),$$

(42)

where $f_{\mathbf{\Theta }}\left({\mathbf{x}}_i\right)$ is the predicted probability for the true class, and $\gamma \ge 0$ is the focusing parameter that controls the strength of the modulation. When $\gamma =0$, focal loss reduces to the standard cross-entropy loss in the binary scenario.

Focal loss is Lipschitz continuous (L-CONT) for the predicted probabilities $f_{\mathbf{\Theta }}\left({\mathbf{x}}_i\right)$.

5.3.4. Dice Loss (CONT, DIFF)

Dice loss [79] is widely used in image segmentation tasks to handle class imbalance. It is based on the Dice coefficient, a measure of overlap between the predicted and true segmentation. Dice loss is particularly effective when there is a significant imbalance between foreground and background classes.

The Dice loss is defined as:

$${\mathcal{L}}_{\mathrm{Dice}}=1-{\displaystyle \frac{2{\sum }_i{f}_{\mathbf{\Theta }}\left({\mathbf{x}}_i\right)y_i}{{\sum }_i{f}_{\mathbf{\Theta }}\left({\mathbf{x}}_i\right)+{\sum }_i{y}_i}},$$

(43)

where $f_{\mathbf{\Theta }}\left({\mathbf{x}}_i\right)$ is the predicted probability for pixel i, and $y_i$ is the corresponding ground truth.

Dice loss is continuous (CONT) and differentiable (DIFF), but not convex. It is commonly employed in medical image segmentation tasks, where accurately measuring the overlap between the predicted and true segmentation is crucial.

5.3.5. Tversky Loss (CONT, DIFF)

Tversky loss [80] extends the Dice loss by introducing parameters that control the trade-off between false positives and false negatives. This makes Tversky loss especially useful in segmentation tasks with high class imbalance, where one class dominates the other.

The Tversky loss is defined as:

$${\mathcal{L}}_{\mathrm{Tversky}}=1-{\displaystyle \frac{{\sum }_i{f}_{\mathbf{\Theta }}\left({\mathbf{x}}_i\right)y_i}{{\sum }_i{f}_{\mathbf{\Theta }}\left({\mathbf{x}}_i\right)y_i+\alpha {\sum }_i{f}_{\mathbf{\Theta }}\left({\mathbf{x}}_i\right)(1-y_i)+\beta {\sum }_i(1-f_{\mathbf{\Theta }}\left({\mathbf{x}}_i\right))y_i}},$$

(44)

where $f_{\mathbf{\Theta }}\left({\mathbf{x}}_i\right)$ is the predicted probability for pixel i, $y_i$ is the corresponding ground truth, and $\alpha$ and $\beta$ are parameters that balance false positives and false negatives.

Tversky loss is continuous (CONT) and differentiable (DIFF), and by adjusting $\alpha$ and $\beta$, it can emphasize reducing false positives or false negatives, depending on the task’s requirements. This flexibility makes it a powerful tool in imbalanced segmentation problems.

6. Generative Losses

In recent years, generative models have become particularly valuable for modeling complex data distributions and regenerating realistic samples from them [81,82]. In this section, as illustrated in Figure 4, we describe the primary losses associated with variational autoencoders (VAEs) (Section 6.1), generative adversarial networks (GANs) (Section 6.2), diffusion models (Section 6.3), and Transformers with a focus on LLMs (Section 6.4).

While this survey focuses on the core generative models, other architectures like pixel RNNs [83], realNVP [84], flow-based models [85], and WaveNet [86] are also impactful but fall beyond the scope of this work.

6.1. Variational Autoencoders (VAEs)

Variational autoencoders (VAEs) [87,88] are generative models that learn a latent representation of data through probabilistic encoding and decoding. By modeling the underlying structure of data, VAEs aim to produce a latent space that follows a known prior distribution, typically Gaussian.

A VAE consists of two main components: an encoder and a decoder. The encoder maps the input data $\mathbf{x}$ to a latent variable $\mathbf{z}$, capturing key data characteristics in a compact form through the approximate posterior distribution $q\left(\mathbf{z}\right|\mathbf{x})$. The decoder then reconstructs the original data $\mathbf{x}$ from $\mathbf{z}$, modeling the conditional distribution $p\left(\mathbf{x}\right|\mathbf{z})$. This probabilistic framework allows VAEs to generate new samples by sampling from the learned latent space.

VAEs find applications in image generation, semi-supervised learning, and anomaly detection. In image generation, they facilitate smooth latent representations, useful for denoising and sample synthesis [89,90]. In semi-supervised learning, VAEs leverage both labeled and unlabeled data to improve classification [91]. For anomaly detection, VAEs model normal data distributions, identifying anomalies via high reconstruction errors for out-of-distribution samples [92]. Despite successes, VAEs face challenges such as image blurriness, often addressed by discrete latent spaces like those in VQ-VAE [93].

6.1.1. VAE Loss (ELBO) (CONT, DIFF, L-CONT)

The VAE loss function is derived from the evidence lower bound (ELBO), which provides a lower bound on the data log-likelihood. It comprises two main components: the reconstruction loss ${\mathcal{L}}_{\mathrm{recon}}$ and the Kullback–Leibler (KL) divergence ${\mathcal{L}}_{\mathrm{KL}}$ (see Section 5.3.2).

The reconstruction loss encourages the decoder to accurately reproduce the input data $\mathbf{x}$ from the latent variable $\mathbf{z}$ by minimizing the difference between the original data and its reconstruction. Depending on the data type, ${\mathcal{L}}_{\mathrm{recon}}$ can be the mean squared error (MSE, see Section 4.2.3) for real-valued data or the binary cross-entropy (negative log-likelihood, see Section 5.3.1) for binary data. In a unified expression, we can represent ${\mathcal{L}}_{\mathrm{recon}}$ as the expected log-likelihood with respect to the approximate posterior distribution of the latent variable:

$${\mathcal{L}}_{\mathrm{recon}}=-{\mathbb{E}}_{q\left(\mathbf{z}\right|\mathbf{x})}[log{f}_{\mathbf{\Theta }}\left(\mathbf{x}\right|\mathbf{z})].$$

The full VAE loss, often referred to as the negative ELBO, is defined as:

$${\mathcal{L}}_{VAE}=-{\mathbb{E}}_{q\left(\mathbf{z}\right|\mathbf{x})}[log{f}_{\mathbf{\Theta }}\left(\mathbf{x}\right|\mathbf{z})]+{\mathcal{L}}_{\mathrm{KL}}(q\left(\mathbf{z}\right|\mathbf{x})\parallel f_{\mathbf{\Theta }}\left(\mathbf{z}\right)),$$

(45)

where the first term is the reconstruction loss and the second term, the KL divergence, regularizes the latent space to follow the prior distribution.

While ${\mathcal{L}}_{\mathrm{recon}}$ and ${\mathcal{L}}_{\mathrm{KL}}$ may individually exhibit convexity under certain conditions, their combined interaction, particularly with neural network parameterization, results in an overall non-convex loss function [42,87,88].

The reconstruction loss alone focuses on achieving high-fidelity reconstructions of the input data. However, without the KL divergence term, the model does not impose any structure on the latent space, leading to unstructured representations that generalize poorly to generating new samples [87,88].

In contrast, the ELBO, which combines the reconstruction loss with the KL divergence term, introduces a trade-off between accurate reconstruction and regularization of the latent space. The KL divergence term encourages the latent space to follow a smooth prior distribution, facilitating sample generation by ensuring structured, regularized representations. However, if the KL divergence term dominates, it can lead to poorer reconstructions as the model prioritizes matching the prior over precise data recovery [94]. Excessive weight on the KL term can also cause posterior collapse, where the model ignores the latent variables entirely, resulting in suboptimal generative performance [95,96].

In summary, while ${\mathcal{L}}_{\mathrm{recon}}$ alone enhances data fidelity, combining it with ${\mathcal{L}}_{\mathrm{KL}}$ (in the ELBO) enables proper generative modeling. Achieving realistic data generation requires balancing accurate reconstruction with a regularized latent space [97].

6.1.2. Extensions of VAE Losses

Several extensions of the VAE loss have been proposed to improve the model’s performance or flexibility. The most notable variants include:

• Beta-VAE (CONT, DIFF, L-CONT): The beta-VAE [94] introduces a hyperparameter $\beta$ to weight the KL divergence term. The loss function becomes:

  $${\mathcal{L}}_{\beta \text{-}\mathrm{VAE}}={\mathcal{L}}_{\mathrm{recon}}+\beta {\mathcal{L}}_{\mathrm{KL}}.$$

  (46)
  By adjusting $\beta$, the balance between reconstruction accuracy and latent space regularization can be controlled [97]. This variant retains the continuity, differentiability, and Lipschitz continuity but it is non-convex due to the interaction between terms.
  Beta-VAE improves interpretability by encouraging the model to learn more disentangled representations. This is particularly useful in applications where distinct, independent latent factors are beneficial, as in unsupervised learning tasks or when a well-structured latent space is desired [94]. The introduction of $\beta$ can lead to a trade-off where increasing regularization may harm reconstruction quality. Excessively high values of $\beta$ can also cause posterior collapse, where the model ignores the latent variables, resulting in reduced generative performance [95,96,97].
• VQ-VAE (CONT): In vector quantized VAEs [93], the latent space is discrete, rather than continuous, and a codebook is used to quantize the latent variables. The key difference in VQ-VAE compared to traditional VAEs is the use of a discrete latent representation, which introduces the following steps:
  Given an input $\mathbf{x}$, the encoder produces a continuous latent variable ${\mathbf{z}}_e\left(\mathbf{x}\right)\in {\mathbb{R}}^D$. However, instead of passing ${\mathbf{z}}_e\left(\mathbf{x}\right)$ directly to the decoder, VQ-VAE performs a vector quantization by mapping ${\mathbf{z}}_e\left(\mathbf{x}\right)$ to the nearest vector in a learned codebook $\mathcal{E}={\left\{{\mathbf{e}}_k\right\}}_{k=1}^K$, where each ${\mathbf{e}}_k\in {\mathbb{R}}^D$ is a learned embedding vector. This process can be written as:

  $${\mathbf{z}}_q\left(\mathbf{x}\right)={\mathbf{e}}_k,\mathrm{where}k=arg\underset{j}{min}{\parallel {\mathbf{z}}_e\left(\mathbf{x}\right)-{\mathbf{e}}_j\parallel }_2.$$

  (47)
  The quantized latent variable ${\mathbf{z}}_q\left(\mathbf{x}\right)$ is then passed to the decoder, which reconstructs the input as $\widehat{\mathbf{x}}=p\left(\mathbf{x}\right|{\mathbf{z}}_q)$.
  The VQ-VAE loss function consists of three terms:

  $${\mathcal{L}}_{VQ-VAE}={\mathcal{L}}_{\mathrm{recon}}+\parallel \mathrm{sg}\left[{\mathbf{z}}_e\left(\mathbf{x}\right)\right]-{\mathbf{e}}_k{\parallel }_2^2+\beta {\parallel {\mathbf{z}}_e\left(\mathbf{x}\right)-\mathrm{sg}\left[{\mathbf{e}}_k\right]\parallel }_2^2,$$

  (48)

  where:

  −

  ${\mathcal{L}}_{\mathrm{recon}}$ is the reconstruction loss (typically mean squared error or binary cross-entropy).

  −

  The second term $\parallel \mathrm{sg}\left[{\mathbf{z}}_e\left(\mathbf{x}\right)\right]-{\mathbf{e}}_k{\parallel }_2^2$ (the codebook loss) ensures that the codebook vector ${\mathbf{e}}_k$ is close to the encoder output ${\mathbf{z}}_e\left(\mathbf{x}\right)$.

  −

  The third term $\parallel {\mathbf{z}}_e\left(\mathbf{x}\right)-\mathrm{sg}\left[{\mathbf{e}}_k\right]{\parallel }_2^2$ (the commitment loss) encourages the encoder to commit to a particular codebook vector, where $\mathrm{sg}[·]$ denotes the stop gradient operator, ensuring gradients only flow through the appropriate part of the network.

  −

  $\beta$ is a hyperparameter controlling the weight of the commitment loss.

  While VQ-VAE remains continuous (CONT) overall, the discrete quantization step introduces non-differentiability in the loss function, as the mapping from encoder outputs to codebook vectors is non-differentiable.
  The key advantage of VQ-VAE is that the use of a discrete latent space can produce sharper and higher-quality generated samples, addressing some of the issues with blurry outputs often observed in continuous VAEs [93]. VQ-VAE is particularly beneficial in tasks where the data have inherently discrete characteristics, such as in audio and image generation [98,99].
• Conditional VAE (CVAE) (CONT, DIFF, L-CONT): Conditional VAEs [100] condition both the encoder and decoder on auxiliary information (e.g., class labels), modifying the ELBO to learn the conditional distribution $p\left(\mathbf{x}\right|\mathbf{y})$. The CVAE loss is:

  $${\mathcal{L}}_{CVAE}=-{\mathbb{E}}_{q\left(\mathbf{z}\right|\mathbf{x},\mathbf{y})}[log{f}_{\mathbf{\Theta }}\left(\mathbf{x}\right|\mathbf{z},\mathbf{y})]+{\mathcal{L}}_{\mathrm{KL}}(q\left(\mathbf{z}\right|\mathbf{x},\mathbf{y})\parallel f_{\mathbf{\Theta }}\left(\mathbf{z}\right|\mathbf{y}))$$

  (49)
  The CVAE loss is continuous, differentiable, and Lipschitz continuous, and the KL divergence remains convex, but the overall loss is still non-convex due to the interaction between terms.
  The advantage of using CVAE is its ability to generate conditional outputs based on specific attributes or labels. This makes CVAE particularly useful in scenarios where controlled generation is required, such as in image generation conditioned on class labels or text generation based on input attributes [91,101]. By incorporating auxiliary information, CVAEs allow for more structured and interpretable latent spaces, improving the ability to generate targeted samples in a variety of applications [91].

6.2. Generative Adversarial Networks

Generative adversarial networks (GANs) are used to create new data instances that are sampled from the training data. GANs have two main components:

• The generator, referred to as $G\left(\{{\mathbf{z}}_0,\cdots ,{\mathbf{z}}_N\}\right)$, which generates data starting from random noise and tries to replicate real data distributions.
• The discriminator, referred to as $D\left(\{{\mathbf{x}}_0,\cdots ,{\mathbf{x}}_N\}\right)$, learns to distinguish the generator’s fake data from the real one. It applies penalties in the generator loss for producing distinguishable fake data compared with real data.

The GAN architecture is relatively straightforward, although one aspect remains challenging: GAN loss functions. Precisely, the discriminator is trained to provide the loss function for the generator. If generator training goes well, the discriminator gets worse at telling the difference between real and fake samples. It starts to classify fake data as real, and its accuracy decreases.

Both the generator and the discriminator components are typically neural networks, where the generator output is connected directly to the discriminator input. The discriminator’s classification provides a signal that the generator uses to update its weights through back-propagation.

As GANs try to replicate a probability distribution, they should use loss functions that reflect the distance between the distribution of the data generated by the GAN and the distribution of the real data.

Two common GAN loss functions are typically used: minimax loss [82] and Wasserstein loss [102]. The generator and discriminator losses derive from a single distance measure between the two aforementioned probability distributions. The generator can only affect one term in the distance measure: the term that reflects the distribution of the fake data. During generator training, we drop the other term, which reflects the real data distribution. The generator and discriminator losses look different, even though they derive from a single formula.

In the following, both minimax and Wasserstein losses are written in a general form. The properties of the loss function (CONT, DIFF, etc.) are identified based on the function chosen for the generator or discriminator.

6.2.1. Minimax Loss

The generative model G learns the data distributions and is trained simultaneously with the discriminative model D. The latter estimates the probability that a given sample is identical to the training data rather than G. G is trained to maximize the likelihood of tricking D [82]. In other words, the generator tries to minimize the following function while the discriminator tries to maximize it:

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\mathrm{minimax}}(D,G)& =E_{\{{\mathbf{x}}_0,\cdots ,{\mathbf{x}}_N\}}[log\left(D\left(\{{\mathbf{x}}_0,\cdots ,{\mathbf{x}}_N\}\right)\right)+\hfill \\ \hfill & E_{\{{\mathbf{z}}_0,\cdots ,{\mathbf{z}}_N\}}[log(1-D\left(G\left(\{{\mathbf{z}}_0,\cdots ,{\mathbf{z}}_N\}\right)\right))],\hfill \end{array}$$

(50)

where:

• $D\left(\{{\mathbf{x}}_0,\cdots ,{\mathbf{x}}_N\}\right)$ is the discriminator’s estimate of the probability that real data instance {${\mathbf{x}}_0,\cdots ,{\mathbf{x}}_N$} is real;
• $E_{\{{\mathbf{x}}_0,\cdots ,{\mathbf{x}}_N\}}$ is the expected value over all real data instances;
• $G\left(\{{\mathbf{z}}_0,\cdots ,{\mathbf{z}}_N\}\right)$ is the generator’s output when given noise $\{{\mathbf{z}}_0,\cdots ,{\mathbf{z}}_N\}$;
• $D\left(G\right(z\left)\right)$ is the discriminator’s estimate of the probability that a fake instance is real;
• $E_{\{{\mathbf{z}}_0,\cdots ,{\mathbf{z}}_N\}}$ is the expected value over all random inputs to the generator (in effect, the expected value over all generated fake instances $G\left(\{{\mathbf{z}}_0,\cdots ,{\mathbf{z}}_N\}\right))$.

The loss function above directly represents the cross-entropy between real and generated data distributions. The generator cannot directly affect the $log\left(D\left(\{{\mathbf{x}}_0,\cdots ,{\mathbf{x}}_N\}\right)\right)$ term in the function, and it only minimizes the term $log(1-D\left(G\left(\{{\mathbf{z}}_0,\cdots ,{\mathbf{z}}_N\}\right)\right))$. A disadvantage of this formulation of the loss function is that the above minimax loss function can cause the GAN to get stuck in the early stages of the training when the discriminator received trivial tasks. Therefore, a suggested modification to the loss [82] is to allow the generator to maximize $log\left(D\left(G\left(\{{\mathbf{z}}_0,\cdots ,{\mathbf{z}}_N\}\right)\right)\right)$.

6.2.2. Wasserstein Loss

The Wasserstein distance gives an alternative method of training the generator to better approximate the distribution of the training dataset. In this setup, the training of the generator itself is responsible for minimizing the distance between the distribution of the training and generated datasets. The possible solutions are to use distribution distance measures, like Kullback–Leibler (KL) divergence, Jensen–Shannon (JS) divergence, and the earth-mover (EM) distance (also called Wasserstein distance). The main advantage of using Wasserstein distance is due to its differentiability and having a continuous linear gradient [102].

A GAN that uses a Wasserstein loss, known as a WGAN, does not discriminate between real and generated distributions in the same way as other GANs. Instead, the WGAN discriminator is called a “critic,” and it scores each instance with a real-valued score rather than predicting the probability that it is fake. This score is calculated so that the distance between scores for real and fake data is maximized.

The advantage of the WGAN is that the training procedure is more stable and less sensitive to model architecture and selection of hyperparameters.

The two loss functions can be written as:

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\mathrm{critic}}& =D\left(\{{\mathbf{x}}_0,\cdots ,{\mathbf{x}}_N\}\right)-D\left(G\left(\{{\mathbf{z}}_0,\cdots ,{\mathbf{z}}_N\}\right)\right)\hfill \\ \hfill {\mathcal{L}}_{\mathrm{generator}}& =D\left(G\left(\{{\mathbf{z}}_0,\dots ,{\mathbf{z}}_N\}\right)\right)\hfill \end{array}$$

(51)

The discriminator tries to maximize ${\mathcal{L}}_{critic}$. In other words, it tries to maximize the difference between its output on real instances and its output on fake instances. The generator tries to maximize ${\mathcal{L}}_{Generator}$. In other words, It tries to maximize the discriminator’s output for its fake instances.

The benefit of Wasserstein loss is that it provides a useful gradient almost everywhere, allowing for the continued training of the models. It also means that a lower Wasserstein loss correlates with better generator image quality, meaning that it explicitly seeks a minimization of generator loss. Finally, it is less vulnerable to getting stuck in a local minimum than minimax-based GANs [102]. However, accurately estimating the Wasserstein distance using batches requires unaffordable batch size, which significantly increases the amount of data needed [103].

6.3. Diffusion Models

Diffusion models are generative models that rely on probabilistic likelihood estimation to generate data samples. Originally inspired by the physical phenomenon of diffusion, these models systematically add noise to the data and then learn to reverse this process. The forward process corrupts the data by progressively adding noise, and the reverse process reconstructs the data by removing the noise, a procedure learned through a neural network that models conditional probability densities [104,105,106].

The forward diffusion process adds Gaussian noise to the data over T time steps, transforming it into pure noise. The reverse diffusion process then uses a neural network to learn the conditional probabilities of each time step, gradually denoising the data to reconstruct the original input.

• Forward Diffusion Process

Given a data point ${\mathbf{x}}_0$ sampled from a real data distribution $q\left(\mathbf{x}\right)$, the forward process adds small amounts of Gaussian noise iteratively T times, resulting in a sequence of noisy samples ${\mathbf{x}}_1,\cdots ,{\mathbf{x}}_T$. The noise distribution is Gaussian, and because the forecasted probability density at time t depends only on the previous state $t-1$, the conditional probability density is:

$$q\left({\mathbf{x}}_t\right|{\mathbf{x}}_{t-1})=\mathcal{N}({\mathbf{x}}_t;\sqrt{1-{\beta }_t}{\mathbf{x}}_{t-1},{\beta }_t\mathbf{I}),$$

(52)

with ${\beta }_t\in (0,1)$, a hyperparameter that can be constant or variable, and $t\in [1,T]$. As $T\to \infty$, ${\mathbf{x}}_T$ converges to a pure Gaussian distribution. The forward process does not require neural network training but involves iteratively applying Gaussian noise to the data.

• Reverse Diffusion Process

In the reverse process, given the noisy state ${\mathbf{x}}_T$, the goal is to estimate the probability density of the original data by gradually removing the noise. The neural network, parameterized by $\mathbf{\Theta }$, is trained to approximate $f_{\mathbf{\Theta }}\left({\mathbf{x}}_{t-1}\right|{\mathbf{x}}_t)$, enabling the reconstruction of ${\mathbf{x}}_0$ from ${\mathbf{x}}_T$. The reverse diffusion process is modeled as:

$$f_{\mathbf{\Theta }}\left({\mathbf{x}}_{t-1}\right|{\mathbf{x}}_t)=\mathcal{N}({\mathbf{x}}_{t-1};{\mathit{\mu }}_{\mathbf{\Theta }}({\mathbf{x}}_t,t),{\mathbf{\Sigma }}_{\mathbf{\Theta }}({\mathbf{x}}_t,t)),$$

(53)

where ${\mu }_{\mathbf{\Theta }}({\mathbf{x}}_t,t)$ is a learned mean function, and ${\mathbf{\Sigma }}_{\mathbf{\Theta }}({\mathbf{x}}_t,t)$ is a learned variance term. Sampling from this distribution allows the model to reconstruct the data by successively denoising the noisy sample from time step T down to $t=0$ [106].

Recent work has demonstrated that diffusion models can outperform other generative models, such as GANs, in certain tasks [107,108], though they tend to be computationally more expensive due to the large number of forward and reverse steps involved.

6.3.1. Diffusion Model Loss Function (CONT, DIFF)

The objective of diffusion models is to minimize the variational lower bound (VLB) on the data likelihood, which is formulated as:

$${\mathcal{L}}_{\mathrm{diffusion}}={\mathbb{E}}_t\left[logp\left({\mathbf{x}}_T\right)-\sum _{t\ge 1}log{\displaystyle \frac{f_{\mathbf{\Theta }}\left({\mathbf{x}}_{t-1}\right|{\mathbf{x}}_t)}{q\left({\mathbf{x}}_t\right|{\mathbf{x}}_{t-1})}}\right].$$

(54)

A simplified version of this loss is often used to train denoising diffusion probabilistic models (DDPMs) [106]:

$${\mathcal{L}}_{\mathrm{diffusion}}^{\mathrm{simple}}={\mathbb{E}}_{t,{\mathbf{x}}_0,{\mathit{ϵ}}_t}\left[\parallel {\mathit{ϵ}}_t-{\mathit{ϵ}}_{\mathbf{\Theta }}(\sqrt{{\overline{\alpha }}_t}{\mathbf{x}}_0+\sqrt{1-{\overline{\alpha }}_t}{\mathit{ϵ}}_t,t){\parallel }^2\right],$$

(55)

where the model learns to predict the noise ${\mathit{ϵ}}_{\mathbf{\Theta }}$ added at each step. This simplified loss connects closely to score matching, as the model learns to predict the noise, which is essential for reconstructing the data.

The simplified loss function (also called noise prediction loss) in DDPMs is computationally efficient and relatively straightforward to implement. It uses a mean squared error (MSE) objective, which is cheap to compute. Each training iteration requires the model to predict the added noise and minimize the difference between the actual and predicted noise. This formulation is easy to optimize with standard gradient-based methods, leading to faster convergence and more stable training compared to more complex alternatives. As a result, DDPM’s simplified loss provides a good trade-off between computational efficiency and sample quality, making it suitable for a wide range of generative tasks [106].

6.3.2. Other MSE-Based Losses in Diffusion Models (CONT, DIFF)

While the noise prediction loss is central to diffusion models, other MSE-based losses are also employed to optimize different aspects of the generation process. These losses extend the basic MSE formulation to handle higher-level tasks and representations:

Perceptual Loss

In tasks such as super-resolution and image synthesis, where pixel-level differences may fail to capture perceptual quality, perceptual loss [109] is applied. This loss leverages high-level feature representations extracted from a pre-trained convolutional neural network (e.g., VGG). The MSE objective is then applied to these feature maps, rather than pixel values, ensuring that the generated images align more closely with human perception.

Latent Space Regularization

In latent diffusion models [109], MSE is applied to regularize the latent space during the diffusion process. This regularization ensures that the latent representations of the input data remain smooth and structured, improving the stability and consistency of the generation process. Latent space regularization helps to prevent issues such as mode collapse and ensures higher-quality outputs.

6.3.3. Score-Based Generative Model Loss (CONT, DIFF)

In score-based generative models (SBGMs) [110], the diffusion process is generalized to a continuous-time framework using stochastic differential equations (SDEs). The model learns the score function, i.e., the gradient of the log probability density at different noise levels.

The loss function for learning the score function is derived from denoising score matching:

$${\mathcal{L}}_{\mathrm{score}}={\mathbb{E}}_{{\mathbf{x}}_0,\sigma }\left[\lambda \left(\sigma \right)\parallel {\nabla }_{\mathbf{x}}log{f}_{\mathbf{\Theta }}\left({\mathbf{x}}_t\right|\sigma )+{\displaystyle \frac{{\mathbf{x}}_t-{\mathbf{x}}_0}{{\sigma }^2}}{\parallel }^2\right],$$

(56)

where ${\mathbf{x}}_t$ is the noisy sample, $log{f}_{\mathbf{\Theta }}\left({\mathbf{x}}_t\right|\sigma )$ is the predicted score function, and $\lambda \left(\sigma \right)$ is a weighting function for different noise levels.

In contrast with the simplified loss function in DDPMs, the score-based loss used in score-based generative models (SBGMs) is computationally more intensive. The model must learn the gradient of the log probability (the score function), which involves higher-order derivatives and sampling from various noise scales. Each iteration requires calculating and backpropagating through the score function, significantly increasing the computational cost. While this approach can provide richer generative performance and flexibility, particularly in continuous-time frameworks, the added complexity often results in slower training and higher resource requirements [110].

6.3.4. Cosine Similarity in Multimodal Context (CONT, DIFF)

Cosine similarity plays a crucial role in aligning representations across different modalities, particularly in models like CLIP (contrastive language-image pretraining [111]), which embeds both text and images in a shared feature space. In multimodal tasks, such as text-to-image generation, this approach ensures that representations from different domains are semantically consistent.

In text-to-image diffusion models, a common use of cosine similarity is the CLIP guidance loss [109], where the goal is to align the generated image with a given text prompt. The loss leverages a pre-trained CLIP model that computes embeddings for both the text and the generated image. By maximizing the cosine similarity between the text and image embeddings, the model enhances the semantic coherence of the generated images with respect to the input text.

The CLIP guidance loss is defined as:

$${\mathcal{L}}_{\mathrm{CLIP}}=1-cos({\varphi }_{\mathrm{text}}\left(\mathbf{y}\right),{\varphi }_{\mathrm{image}}\left(\widehat{\mathbf{x}}\right)),$$

(57)

where ${\varphi }_{\mathrm{text}}\left(\mathbf{y}\right)$ is the CLIP embedding of the input text $\mathbf{y}$, and ${\varphi }_{\mathrm{image}}\left(\widehat{\mathbf{x}}\right)$ is the CLIP embedding of the generated image $\widehat{\mathbf{x}}$.

This loss is continuous (CONT) and differentiable (DIFF) with respect to the generated image, as both the CLIP encoders and cosine similarity function are composed of standard neural network operations that are differentiable. Typically, the pre-trained CLIP model is frozen during the training of diffusion models, allowing gradients to flow back through the generated image while keeping the CLIP parameters unchanged.

6.4. Transformers and LLM Loss Functions

Transformer has revolutionized machine learning, particularly in natural language processing (NLP), by enabling the parallel processing of sequential data using self-attention mechanisms [112]. This architecture has been the foundation for many powerful models, including large language models (LLMs) such as GPT [113] and BERT [114], which excel at generative tasks like text generation, translation, and summarization.

The Transformer model is built around an encoder–decoder structure, with the encoder processing the input data and the decoder generating the output sequence. An attention mechanism is a technique used in machine learning and artificial intelligence to improve model performance by focusing on relevant information. In Transformers, the attention mechanism allows the model to selectively assign varying importance to different elements in the sequence, enabling it to capture long-range dependencies and context effectively. Within each encoder and decoder layer, attention mechanisms assign scores to tokens based on relevance, ensuring that the model attends to the most pertinent information for each output. After attention, feedforward layers further refine these representations, capturing complex patterns and allowing for high parallelization, which enables Transformers to efficiently handle long sequences and excel at tasks involving sequential data.

Transformer-based models, particularly LLMs, are trained using a variety of loss functions depending on the task. This section explores the primary loss functions used in Transformer and LLM training.

6.4.1. Probabilistic Losses in LLMs

Probabilistic losses are central to training LLMs, as they help models learn from probability distributions over sequences, tokens, and classes. These losses are primarily extensions or modifications of cross-entropy and KL divergence, as introduced in Section 5.3.1 and Section 5.3.2. Even though the original losses are generally convex, the overall optimization landscape becomes non-convex due to the complex architecture of Transformers [115,116]. Below, we outline key probabilistic losses used in training popular LLM architectures.

Autoregressive Language Modeling Loss (CONT, DIFF, CONVEX)

In autoregressive language models, such as GPT [113], the model is trained to predict the next token in a sequence based on the preceding tokens. This approach is crucial for generative tasks like text generation, where the model generates text one token at a time.

The loss function used in autoregressive models is typically the cross-entropy loss, which measures the negative log-likelihood of the correct token given the model’s predictions. For a sequence of tokens $({\mathbf{x}}_1,{\mathbf{x}}_2,\cdots ,{\mathbf{x}}_T)$, the loss is defined as:

$${\mathcal{L}}_{\mathrm{AR}}=-\sum _{t=1}^{T}log{p}_{\theta }\left({\mathbf{x}}_t\right|{\mathbf{x}}_{<t}),$$

(58)

where $p_{\theta }\left({\mathbf{x}}_t\right|{\mathbf{x}}_{<t})$ is the predicted probability of token ${\mathbf{x}}_t$ conditioned on previous tokens ${\mathbf{x}}_{<t}$.

Autoregressive cross-entropy loss is continuous (CONT), differentiable (DIFF), and convex (CONVEX) when considered in isolation. In LLMs, it effectively aligns model outputs with target sequences but, due to architectural complexity, contributes to an overall non-convex optimization landscape.

Masked Language Modeling (MLM) Loss (CONT, DIFF, CONVEX)

Masked language modeling, used in models like BERT [114], trains the model to predict randomly masked tokens within a sequence. This allows the model to leverage bidirectional context, as it relies on both left and right tokens for prediction.

The loss function for masked language modeling is also cross-entropy. For an input sequence $\mathbf{x}=({\mathbf{x}}_1,{\mathbf{x}}_2,\cdots ,{\mathbf{x}}_T)$ with masked tokens ${\mathbf{x}}_m$, the MLM loss is:

$${\mathcal{L}}_{\mathrm{MLM}}=-\sum _{m\in \mathrm{Masked}}log{p}_{\theta }\left({\mathbf{x}}_m\right|{\mathbf{x}}_{\setminus m}),$$

(59)

where $p_{\theta }\left({\mathbf{x}}_m\right|{\mathbf{x}}_{\setminus m})$ denotes the predicted probability of a masked token given the unmasked context.

The cross-entropy loss applied here is continuous (CONT), differentiable (DIFF), and convex (CONVEX), which suits MLM tasks well for aligning token predictions with target distributions.

Label Smoothing Loss (CONT, DIFF, CONVEX)

Label smoothing [112] is a regularization technique that distributes a small probability mass over incorrect classes, thus preventing the model from becoming overconfident. It is particularly helpful in LLMs for generative tasks like machine translation and summarization.

The adjusted target labels $y_i$ in label smoothing are given by:

$$y_i=(1-ϵ)·y_i+{\displaystyle \frac{ϵ}{K}},$$

(60)

where $ϵ$ is the smoothing parameter and K is the number of classes. The loss can then be expressed as:

$${\mathcal{L}}_{\text{label\_smoothing}}=-\sum _{i=1}^K{y}_{i}log{f}_{\mathbf{\Theta }}\left({\mathbf{x}}_i\right).$$

(61)

The label-smoothing cross-entropy remains continuous (CONT), differentiable (DIFF), and convex (CONVEX) in itself. By moderating confidence, it helps improve model generalization in tasks prone to overfitting.

KL Divergence Loss for Knowledge Distillation (CONT, DIFF)

Knowledge distillation, commonly applied in Transformer-based models like DistilBERT [117], transfers knowledge from a larger teacher model to a smaller student model. The loss function combines standard cross-entropy with a KL divergence term that minimizes the difference between the output distributions of the teacher and student models:

$${\mathcal{L}}_{\mathrm{KD}}=(1-\alpha ){\mathcal{L}}_{\mathrm{CE}}(p_{\mathrm{student}},\mathbf{y})+\alpha {\mathcal{L}}_{\mathrm{KL}}(p_{\mathrm{student}},p_{\mathrm{teacher}}),$$

(62)

where ${\mathcal{L}}_{\mathrm{CE}}$ is the cross-entropy loss, ${\mathcal{L}}_{\mathrm{KL}}$ is the KL divergence between the student and teacher outputs, and $\alpha$ controls the balance between the two terms.

The KL divergence loss is continuous (CONT) and differentiable (DIFF) in itself. In knowledge distillation, it allows smaller models to approximate the behavior of larger models, improving efficiency while maintaining performance.

6.4.2. Ranking Losses in LLM

In large language models (LLMs), ranking losses such as triplet loss and contrastive loss are widely used to structure the embedding space for tasks like semantic search, question answering, and sentence clustering. These losses aim to pull semantically similar sentences closer together while pushing dissimilar sentences farther apart, thus learning meaningful sentence embeddings. They will be discussed in Section 7 (in particular, in Section 7.2 and Section 7.4).

6.4.3. Alignment and Preference Optimization Losses

As LLMs have grown in capabilities and popularity among everyday users, aligning their outputs with human preferences has become a critical step, often replacing or supplementing complex reinforcement learning from human feedback (RLHF) pipelines. Recent advancements have translated this alignment directly as a loss optimization problem over preference data, where the dataset consists of a prompt $\mathbf{x}$, a chosen response ${\mathbf{y}}_w$, and a rejected response ${\mathbf{y}}_l$.

Direct Preference Optimization (DPO) Loss (CONT, DIFF)

DPO [118] bypasses the need for a separate reward model by reparameterizing the reward implicitly through the language model policy. The DPO loss optimizes the policy network $f_{\mathbf{\Theta }}$ relative to a frozen reference model $f_{\mathrm{ref}}$:

$${\mathcal{L}}_{\mathrm{DPO}}=-{\mathbb{E}}_{(\mathbf{x},{\mathbf{y}}_w,{\mathbf{y}}_l)}\left[log\sigma \left(\beta log{\displaystyle \frac{f_{\mathbf{\Theta }}\left({\mathbf{y}}_w\right|\mathbf{x})}{f_{\mathrm{ref}}\left({\mathbf{y}}_w\right|\mathbf{x})}}-\beta log{\displaystyle \frac{f_{\mathbf{\Theta }}\left({\mathbf{y}}_l\right|\mathbf{x})}{f_{\mathrm{ref}}\left({\mathbf{y}}_l\right|\mathbf{x})}}\right)\right]$$

(63)

where $\sigma$ is the logistic function and $\beta$ controls the deviation from the reference model. DPO operates as a continuous (CONT) and differentiable (DIFF) contrastive ranking loss for generative text.

Simple Preference Optimization (SimPO) Loss (CONT, DIFF)

While DPO is highly effective, it requires maintaining a frozen reference model in memory. Recent advancements like SimPO [119] and ORPO [120] eliminate this requirement. SimPO, for example, uses the length-normalized log probability as an implicit reward and introduces a target reward margin $\gamma >0$. The SimPO loss is defined as:

$${\mathcal{L}}_{\mathrm{SimPO}}=-{\mathbb{E}}_{(\mathbf{x},{\mathbf{y}}_w,{\mathbf{y}}_l)}\left[log\sigma \left(\beta \left({\displaystyle \frac{1}{|{\mathbf{y}}_w|}}log{f}_{\mathbf{\Theta }}\left({\mathbf{y}}_w\right|\mathbf{x})-{\displaystyle \frac{1}{|{\mathbf{y}}_l|}}log{f}_{\mathbf{\Theta }}\left({\mathbf{y}}_l\right|\mathbf{x})\right)-\gamma \right)\right]$$

(64)

SimPO is continuous (CONT) and differentiable (DIFF). By incorporating a margin-based approach directly into the log-likelihoods without a reference model, it significantly reduces memory overhead and has been shown to outperform DPO in generating high-quality, aligned text [119].

7. Ranking Losses

Machine learning can be employed to solve ranking problems, which have important industrial applications, especially in information retrieval systems. These problems can be typically solved by employing supervised, semi-supervised, or reinforcement learning [121,122].

The goal of ranking losses, in contrast to other loss functions like the cross-entropy loss or MSE loss, is to anticipate the relative distances between inputs rather than learning to predict a label, a value, or a set of values given an input. This is also sometimes called metric learning. Nevertheless, the cross-entropy loss can be used in the top-one probability ranking. In this scenario, given the scores of all the objects, the top-one probability of an object in this model indicates the likelihood that it will be ranked first [123].

Ranking loss functions for training data can be highly customizable because they require only a method to measure the similarity between two data points, i.e., similarity score. For example, consider a face verification dataset: pairs of photographs that belong to the same person will have a high similarity score, whereas those that do not will have a low score [124]. Different tasks, applications, and neural network configurations use ranking losses (like Siamese nets or triplet nets). Because of this, there are various losses can be used, including contrastive loss, margin loss, hinge loss, and triplet loss.

In general, ranking loss functions require a feature extraction for two (or three) data instances, which returns an embedded representation for each of them. A metric function can then be defined to measure the similarity between those representations, such as the Euclidean distance. Finally, the feature extractors are trained to produce similar representations for both inputs in case the inputs are similar or distant representations in case of dissimilarity.

Similar to Section 6, both pairwise and triplet ranking losses are presented in a general form, as shown in Figure 5. The properties of the loss function (CONT, DIFF, etc.) are identified based on the metric function chosen.

7.1. Pairwise Ranking Loss

In the context of pairwise ranking loss, positive and negative pairs of training data points are used [121,125,126,127]. Positive pairs are composed of an anchor sample ${\mathbf{x}}_a$ and a positive sample ${\mathbf{x}}_p$, which is similar to ${\mathbf{x}}_a$ in the metric. Negative pairs are composed of an anchor sample ${\mathbf{x}}_a$ and a negative sample ${\mathbf{x}}_n$, which is dissimilar to ${\mathbf{x}}_a$ in that metric. The objective is to learn representations with a small distance d between them for positive pairs and a greater distance than some margin value m for negative pairs. Pairwise ranking loss forces representations to have a 0 distance for positive pairs and a distance greater than a margin for negative pairs.

Given ${\mathbf{r}}_a$, ${\mathbf{r}}_p$, and ${\mathbf{r}}_n$; the embedded representations (the output of a feature extractor) of the input samples ${\mathbf{x}}_a$, ${\mathbf{x}}_p$, and ${\mathbf{x}}_n$, respectively; and d as a distance function, the loss function can be written as:

$$L_{pairwise}=\left\{\begin{array}{cc}d({\mathbf{r}}_a,{\mathbf{r}}_p)\hfill & \mathrm{if}\mathrm{positive}\mathrm{pair}\hfill \\ max(0,m-d({\mathbf{r}}_a,{\mathbf{r}}_n))\hfill & \mathrm{if}\mathrm{negative}\mathrm{pair}\hfill \end{array}\right.$$

(65)

For positive pairs, the loss will vanish if the distance between the embedding representations of the two elements in the pair is 0; instead, the loss will increase as the distance between the two representations increases. For negative pairs, the loss will vanish if the distance between the embedding representations of the two elements is greater than the margin m. However, if the distance is less than m, the loss will be positive, and the model parameters will be updated to provide representations for the two items that are farther apart. When the distance between ${\mathbf{r}}_a$ and ${\mathbf{r}}_n$ is 0, the loss value will be at most m. The purpose of the margin is to create representations for negative pairs that are far enough, thus implicitly stopping the training on these pairs and allowing the model to focus on more challenging ones. If ${\mathbf{r}}_0$ and ${\mathbf{r}}_1$ are the pair elements representations, y is a binary flag equal to 0 for a negative pair and to 1 for a positive pair, and the distance d is the Euclidean distance:

$$L_{pairwise}({\mathbf{r}}_0,{\mathbf{r}}_1,y)=y\left|\right|{\mathbf{r}}_0-{\mathbf{r}}_1\left|\right|+(1-y)max(0,m-||{\mathbf{r}}_0-{\mathbf{r}}_1\left|\right|)$$

(66)

Unlike typical classification learning, this loss requires more training data and time because it requires access to all the data of all potential pairs during training. Additionally, because training involves pairwise learning, it will output the binary distance from each class, which is more computationally expensive if there is incorrect classification [128].

7.2. Triplet Loss

Employing triplets of training data instances instead of pairs can produce better performance [122,124,129]. This approach is called triplet ranking loss. A triplet consists of an anchor sample ${\mathbf{x}}_a$, a positive sample ${\mathbf{x}}_p$, and a negative sample ${\mathbf{x}}_n$. The objective is for the distance between the anchor sample and the negative sample representations, $d({\mathbf{r}}_a,{\mathbf{r}}_n)$, to be greater (by a margin m) than the distance between the anchor and positive representations $d({\mathbf{r}}_a,{\mathbf{r}}_p)$. Here, d typically represents the Euclidean distance, but other distance metrics can be applied.

The triplet ranking loss is defined as:

$${\mathcal{L}}_{\mathrm{triplet}}({\mathbf{r}}_a,{\mathbf{r}}_p,{\mathbf{r}}_n)=max(0,m+d({\mathbf{r}}_a,{\mathbf{r}}_p)-d({\mathbf{r}}_a,{\mathbf{r}}_n))$$

(67)

This loss function considers three types of triplets based on the values of ${\mathbf{r}}_a$, ${\mathbf{r}}_p$, ${\mathbf{r}}_n$, and m:

• Easy Triplets: $d({\mathbf{r}}_a,{\mathbf{r}}_n)>d({\mathbf{r}}_a,{\mathbf{r}}_p)+m$. Here, the distance between the negative sample and the anchor sample is already large enough. The model parameters are not updated, and the loss is 0.
• Hard Triplets: $d({\mathbf{r}}_a,{\mathbf{r}}_n)<d({\mathbf{r}}_a,{\mathbf{r}}_p)$. In this case, the negative sample is closer to the anchor than the positive sample. The loss is positive (and $>m$), leading to updates in the model’s parameters.
• Semi-Hard Triplets: $d({\mathbf{r}}_a,{\mathbf{r}}_p)<d({\mathbf{r}}_a,{\mathbf{r}}_n)<d({\mathbf{r}}_a,{\mathbf{r}}_p)+m$. Here, the negative sample is farther away from the anchor than the positive sample, but the margin constraint is not yet satisfied. The loss remains positive (and $<m$), prompting parameter updates.

Triplet ranking loss is sensitive to small changes in the input samples, making it less generalizable across datasets [128]. This sensitivity arises because the model learns relative distances based on the specific sample distribution in the training data, meaning that transferring these learned distances to new data may not always yield the same effectiveness.

Triplet loss is also used in models like Sentence-BERT (SBERT) [130] to learn sentence embeddings that preserve semantic similarity. A triplet consists of an anchor sentence, a positive sentence (semantically similar to the anchor), and a negative sentence (semantically dissimilar). The objective is for the anchor’s embedding to be closer to the positive sentence than to the negative sentence by a margin m. This loss improves the alignment of sentence embeddings with semantic meaning, making it particularly useful for sentence similarity tasks.

7.3. Listwise Ranking Loss (CONT, DIFF)

Unlike pairwise and triplet ranking losses, which operate on pairs or triplets of data points, listwise ranking losses focus on ranking the entire list of items. One common listwise ranking loss is based on the softmax cross-entropy loss, which is used to predict the top-one ranking probability [123].

Given a list of items with scores, the softmax cross-entropy loss aims to maximize the probability of ranking the correct item at the top:

$$L_{listwise}=-\sum _{i=1}^N{y}_{i}log{\displaystyle \frac{e^{s_i}}{{\sum }_{j=1}^N{e}^{s_j}}},$$

(68)

where $y_i$ is the ground-truth relevance for item i, and $s_i$ is the predicted score for item i in a list of N items.

This loss is continuous (CONT) and differentiable (DIFF), and it is particularly useful in information retrieval tasks, where the quality of the entire ranking list matters.

7.4. Contrastive Ranking Loss: NT-Xent (CONT, DIFF, L-CONT)

The normalized temperature-scaled cross-entropy loss (NT-Xent) is a generalization of pairwise ranking losses, typically used in self-supervised learning tasks [131]. This loss enables efficient learning by comparing the similarity between an anchor and multiple negative samples in a batch.

Given an anchor sample ${\mathbf{x}}_a$ and a positive sample ${\mathbf{x}}_p$, along with N negative samples ${\mathbf{x}}_n^i$, the NT-Xent loss is computed as:

$$L_{\text{NT-Xent}}=-log{\displaystyle \frac{e^{\mathrm{sim}({\mathbf{x}}_a,{\mathbf{x}}_p)/\tau }}{{\sum }_{i=1}^N{e}^{\mathrm{sim}({\mathbf{x}}_a,{\mathbf{x}}_n^i)/\tau }}},$$

(69)

where $\mathrm{sim}(·,·)$ is a similarity measure (e.g., cosine similarity), and $\tau$ is a temperature scaling parameter.

This loss is continuous (CONT), differentiable (DIFF), and Lipschitz continuous (L-CONT), and it is often used in contrastive learning tasks where learning representations from large-scale data with many negatives is necessary.

This loss has also been used in language models like SimCSE [132] to maximize the similarity between an anchor and a positive sample while minimizing the similarity with negative samples. One of the primary advantages of contrastive losses in the context of language modelling is their ability to improve the generalization of sentence embeddings by providing a strong supervision signal, particularly in tasks requiring semantic similarity [132]. Additionally, using multiple negatives in the mini-batch, as with NT-Xent, improves training efficiency by leveraging more negative examples [130], leading to faster convergence and more stable training compared to methods with fewer negatives.

LambdaLoss (CONT, DIFF)

LambdaLoss [133,134] is a specialized loss function designed for learning-to-rank tasks, where the objective is to optimize ranking metrics such as normalized discounted cumulative gain (NDCG) or mean reciprocal rank (MRR). It is particularly effective in applications like search engines and recommendation systems, where errors in top-ranked items are more costly.

LambdaLoss differs from traditional pairwise losses by directly optimizing ranking metrics like NDCG, and weighting errors by their impact on the final ranking. The loss function considers the relative change in the ranking metric when swapping the predicted rankings of two items.

Given two documents i and j, with ground truth labels $y_i$ and $y_j$, and predicted scores $f\left({\mathbf{x}}_i\right)$ and $f\left({\mathbf{x}}_j\right)$, LambdaLoss is defined as:

$$L_{\mathrm{Lambda}}=|\Delta {\mathrm{NDCG}}_{ij}|·\sigma (f\left({\mathbf{x}}_i\right)-f\left({\mathbf{x}}_j\right)),$$

(70)

where:

• $\Delta {\mathrm{NDCG}}_{ij}$ represents the change in NDCG if the documents i and j are swapped in the ranking.
• $\sigma (f\left({\mathbf{x}}_i\right)-f\left({\mathbf{x}}_j\right))$ is the pairwise ranking loss, typically modeled as a logistic function $\sigma \left(z\right)={\displaystyle \frac{1}{1+e^{-z}}}$ or a hinge loss.

The total LambdaLoss ${\mathcal{L}}_{\mathrm{Lambda}}$ for a dataset is computed as the sum of all pairwise losses across all document pairs:

$${\mathcal{L}}_{\mathrm{Lambda}}=\sum _{i<j}{L}_{\mathrm{Lambda}},$$

(71)

where i and j index the document pairs in the dataset.

$L_{\mathrm{Lambda}}$ is continuous (CONT) and differentiable (DIFF), making it suitable for gradient-based optimization. The loss function directly optimizes ranking metrics such as NDCG, ensuring that the model’s training objective aligns with the final evaluation metric. LambdaLoss emphasizes higher-ranked items, as errors in top positions are penalized more heavily due to their greater impact on the overall ranking.

LambdaLoss is commonly used in systems that require high-quality rankings, such as search engines, ad placement, and recommendation systems.

8. Energy-Based Losses

An energy-based model (EBM) is a probabilistic model that uses a scalar energy function to describe the dependencies of the model variables [135,136,137,138,139,140,141]. An EBM can be formalized as $F:\mathcal{X}\times \mathcal{Y}\to \mathbb{R}$, where $F(\mathbf{x},\mathbf{y})$ stands for the relationship between the $(\mathbf{x},\mathbf{y})$ pairings.

Given an energy function and the input $\mathbf{x}$, the best fitting value of $\mathbf{y}$ is computed with the following inference procedure:

$$\tilde{\mathbf{y}}=argmi{n}_{\mathbf{y}}\left\{F(\mathbf{x},\{{\mathbf{y}}_0,\cdots ,{\mathbf{y}}_N\})\right\}$$

(72)

Energy-based models provide fully generative models that can be used as an alternative to probabilistic estimation for prediction, classification, or decision-making tasks [135,139,140,142,143].

The energy function $E(\mathbf{x},\mathbf{y})\equiv F(\mathbf{x},\mathbf{y})$ can be explicitly defined for all the values of $\mathbf{y}\in \mathcal{Y}=\{{\mathbf{y}}_0,\cdots ,{\mathbf{y}}_N\}$ if and only if the size of the set $\mathcal{Y}$ is small enough. In contrast, when the space of $\mathcal{Y}$ is sufficiently large, a specific strategy, known as the inference procedure, must be employed to find the $\mathbf{y}$ that minimizes $E(\mathbf{x},\{{\mathbf{y}}_0,\cdots ,{\mathbf{y}}_N\})$.

In many real situations, the inference procedure can produce an approximate result, which may or may not be the global minimum of $E(\mathbf{x},\{{\mathbf{y}}_0,\cdots ,{\mathbf{y}}_N\})$ for a given $\mathbf{x}$. Moreover, it is possible that $E(\mathbf{x},\{{\mathbf{y}}_0,\cdots ,{\mathbf{y}}_N\})$ has several equivalent minima. The best inference procedure to use often depends on the internal structure of the model. For example, if $\mathcal{Y}$ is continuous and $E(\mathbf{x},\{{\mathbf{y}}_0,\cdots ,{\mathbf{y}}_N\})$ is smooth and differentiable everywhere concerning $\mathbf{y}$, a gradient-based optimization algorithm can be employed [144].

In general, any probability density function $p\left(\mathbf{x}\right)$ for $\mathbf{x}\in {\mathbb{R}}^D$ can be rewritten as an EBM:

$$p_{\mathit{\theta }}\left(\mathbf{x}\right)={\displaystyle \frac{exp(-E_{\mathit{\theta }}\left(\mathbf{x}\right))}{{\int }_{{\mathbf{x}}^{\prime }}exp(-E_{\mathit{\theta }}\left({\mathbf{x}}^{\prime }\right))d{\mathbf{x}}^{\prime }}},$$

(73)

where the energy function ($E_{\mathit{\theta }}$) can be any function parameterized by $\mathit{\theta }\in \mathbf{\Theta }$ (such as a neural network). In these models, a prediction (e.g., finding $p\left({\mathbf{x}}_{\mathbf{0}}\right|{\mathbf{x}}_{\mathbf{1}},{\mathbf{x}}_{\mathbf{2}},\dots )$) is made by fixing the values of the conditional variables and estimating the remaining variables, (e.g., ${\mathbf{x}}_{\mathbf{0}}$), by minimizing the energy function [145,146,147].

An EBM is trained by finding an energy function that associates low energies to values of $\mathbf{x}$ drawn from the underlying data distribution, $p_{\theta }\left(\mathbf{x}\right)\sim p_D\left(\mathbf{x}\right)$, and high energies for values of $\mathbf{x}$ not close to the underlying distribution.

8.1. Training EBM

Given the aforementioned conceptual framework, the training can be thought of as finding the model parameters that define the good match between the output ($\mathbf{y}\in \mathcal{Y}$) and the input ($\mathbf{x}\in \mathcal{X}$) for every step. This is achieved by estimating the best energy function from the set of energy functions ($\mathcal{E}$) by scanning all the model parameters $\mathbf{\Theta }$ [148,149], where $\mathcal{E}=\left\{F\right(\mathbf{\Theta },\mathbf{X},\mathbf{Y}):\mathbf{\Theta }\in \mathcal{W}=\mathbf{\Theta }\}$. A loss function should behave similarly to the energy function described in Equation (72) (i.e., lower energy for a correct answer must be modeled by low losses, instead; higher energy, to all incorrect answers, by a higher loss).

Considering a set of training samples $\mathcal{S}=\{({\mathbf{x}}_i,{\mathbf{y}}_i):i=1,\cdots ,N\}$, during the training procedure, the loss function should have the effect of pushing down $E(\mathbf{\Theta },{\mathbf{y}}_i,{\mathbf{x}}_i)$ and pulling up $E(\mathbf{\Theta },{\tilde{\mathbf{y}}}_i,{\mathbf{x}}_i)$, i.e., finding the parameters $\mathbf{\Theta }$ that minimize the loss:

$${\mathbf{\Theta }}^{*}=mi{n}_{\mathbf{\Theta }\in \mathcal{W}}{\mathcal{L}}_{ebm}(\mathbf{\Theta },\mathcal{S})$$

(74)

The general form of the loss function ${\mathcal{L}}_{ebm}$ is defined as:

$${\mathcal{L}}_{ebm}(\mathbf{\Theta },\mathcal{S})={\displaystyle \frac{1}{N}}\sum _{i=1}^N{L}_{ebm}({\mathbf{y}}_i,E(\mathbf{\Theta },\mathcal{Y},{\mathbf{x}}_i))+\mathrm{regularizer}\mathrm{term}$$

(75)

where:

• ${\mathcal{L}}_{ebm}({\mathbf{y}}_i,E(\mathbf{\Theta },\mathcal{Y},{\mathbf{x}}_i))$ is the per-sample loss;
• ${\mathbf{y}}_i$ is the desired output;
• $E(\mathbf{\Theta },\mathcal{Y},{\mathbf{x}}_i))$ is energy surface for a given ${\mathbf{x}}_i$; as $\mathbf{y}\in \mathcal{Y}$ varies.

This is an average of a per-sample loss function, denoted as $L_{ebm}({\mathbf{y}}_i,E(\mathbf{\Theta },\mathcal{Y},{\mathbf{x}}_i))$, over the training set. This function depends on the desired output ${\mathbf{y}}_i$ and on the energies derived by holding the input sample ${\mathbf{x}}_i$ constant and changing the output scanning over the sample $\mathcal{Y}$. With this definition, the loss remains unchanged when the training samples are mixed up and when the training set is repeated numerous times [135]. As the size of the training set grows, the model is less likely to overfit [150].

8.2. Loss Functions for EBMs

Following Figure 6, in this section, we introduce the energy loss first since it is the most straightforward. Then, we present common losses in machine learning that can be adapted to the energy-based models, such as the negative log-likelihood loss, the hinge loss, and the log loss. Subsequently, we introduce more sophisticated losses like the generalized perceptron loss and the generalized margin loss. Finally, the minimum classification error loss, the square-square loss, and its variation square-exponential loss are presented.

8.2.1. Energy Loss

The so-called energy loss is the most straightforward loss due to its simplicity. It can be simply defined by using the energy function as the per-sample loss:

$$L_{energy}({\mathbf{y}}_i,E(\mathbf{\Theta },\mathcal{Y},{\mathbf{x}}_i))=E(\mathbf{\Theta },{\mathbf{x}}_i,{\mathbf{y}}_i)$$

(76)

This loss is often used in regression tasks. According to its definition, it pulls the energy function down for values that are close to the correct data distribution. However, the energy function is not pulled up for incorrect values. The assumption is that, by lowering the energy function in the correct location, the energy for incorrect values is left higher as a result. Due to this assumption, the training is sensitive to the model design and may result in energy collapse, leading to a largely flat energy function.

8.2.2. Generalized Perceptron Loss (L-CONT, CONVEX)

The generalized perceptron loss is defined as:

$$L_{perceptron}({\mathbf{y}}_i,E(\mathbf{\Theta },\mathcal{Y},{\mathbf{x}}_i))=E(\mathbf{\Theta },{\mathbf{y}}_i,{\mathbf{x}}_i)-[\underset{\mathbf{y}\in \mathcal{Y}}{min}]\left\{E(\mathbf{\Theta },\{{\mathbf{y}}_0,\cdots ,{\mathbf{y}}_N\},{\mathbf{x}}_i)\right\}$$

(77)

This loss is positive definite as the second term is the lower bound of the first one, i.e., $E(\mathbf{\Theta },{\mathbf{y}}_i,{\mathbf{x}}_i)-\left[mi{n}_{\mathbf{y}\in \mathcal{Y}}\right]\left\{E(\mathbf{\Theta },\{{\mathbf{y}}_0,\cdots ,{\mathbf{y}}_N\},{\mathbf{x}}_i)\right\}\ge 0$. By minimizing this loss, the first term is pushed down and the energy of the model prediction is raised. Although it is widely used [151,152], this loss is suboptimal as it does not detect the gap between the correct output and the incorrect ones, it does not restrict the function from assigning the same value to each wrong output ${\mathbf{y}}_i$, and it may produce flat energy distributions [135].

8.2.3. Negative Log-Likelihood Loss (CONT, DIFF, CONVEX)

In analogy with the description in Section 5.3.1, the negative log-likelihood loss (NLL) in the energy-based context is defined as:

$${\mathcal{L}}_{NLL}(\mathbf{\Theta },\mathcal{S})={\displaystyle \frac{1}{N}}\sum _{i=1}^N\left(E(\mathbf{\Theta },{\mathbf{y}}_i,{\mathbf{x}}_i)+{\displaystyle \frac{1}{\beta }}log{\int }_{\mathbf{y}\in \mathcal{Y}}{e}^{\beta E(\mathbf{\Theta },\mathbf{y},{\mathbf{x}}_i)}\right)$$

(78)

where $\mathcal{S}$ is the training set.

This loss reduces to the perceptron loss when $\beta \to \infty$ and to the log loss in case $\mathcal{Y}$ has only two labels (i.e., binary classification). Since the integral above is intractable, considerable efforts have been devoted to finding approximation methods, including Monte Carlo sampling methods [153] and variational methods [154]. While these methods have been devised as approximate ways of minimizing the NLL loss, they can be viewed in the energy-based framework as different strategies for choosing each $\mathbf{y}$ whose energies will be pulled up [135,155]. The NLL is also known as the cross-entropy [156] loss and is widely used in many applications, including energy-based models [157,158]. This loss function formulation is subject to the same limitations listed in Section 5.3.1.

8.2.4. Generalized Margin Loss

The generalized margin loss is a more reliable version of the generalized perceptron loss. The general form of the generalized margin loss in the context of energy-based training is defined as:

$$L_{margin}(\mathbf{\Theta },{\mathbf{x}}_i,{\mathbf{y}}_i)=Q_m(E(\mathbf{\Theta },{\mathbf{x}}_i,{\mathbf{y}}_i),E(\mathbf{\Theta },{\mathbf{x}}_i,{\overline{\mathbf{y}}}_i))$$

(79)

where ${\overline{\mathbf{y}}}_i$ is the so-called “most offending incorrect output”, which is the output that has the lowest energy among all possible outputs that are incorrect [135]; m is a positive margin parameter; and $Q_m$ is a convex function that ensures that the loss receives low values for $E(\mathbf{\Theta },{\mathbf{x}}_i,{\mathbf{y}}_i)$ and high values for $E(\mathbf{\Theta },{\mathbf{x}}_i,{\overline{\mathbf{y}}}_i)$. In other words, the loss function can ensure that the energy of the most offending incorrect output is greater by some arbitrary margin than the energy of the correct output.

This loss function is written in the general form, and a wide variety of losses that use specific margin function $Q_m$ to produce a gap between the correct output and the wrong output are formalized in the following part of the section.

Hinge Loss (L-CONT, CONVEX)

Already explained in Section 5.2.2, the hinge loss can be rewritten as:

$$L_{hinge}(\mathbf{\Theta },{\mathbf{x}}_i,{\mathbf{y}}_i)=max(0,m+E(\mathbf{\Theta },{\mathbf{x}}_i,{\mathbf{y}}_i)-E(\mathbf{\Theta },{\mathbf{x}}_i,{\overline{\mathbf{y}}}_i))$$

(80)

This loss enforces that the difference between the correct answer and the most offending incorrect answer be at least m [159,160]. Individual energies are not required to take a specific value because the hinge loss depends on energy differences. This loss function shares limitations with the original hinge loss defined in Equation (24).

Log Loss (DIFF, CONT, CONVEX)

This loss is similar to the hinge loss, but it sets a softer margin between the correct output and the most offending outputs. The log loss is defined as:

$$L_{log}(\mathbf{\Theta },{\mathbf{x}}_i,{\mathbf{y}}_i)=log(1+e^{E(\mathbf{\Theta },{\mathbf{x}}_i,{\mathbf{y}}_i)-E(\mathbf{\Theta },{\mathbf{x}}_i,{\overline{\mathbf{y}}}_i)})$$

(81)

This loss is also called soft hinge and it may produce overfitting on high-dimensional datasets [161].

Minimum Classification Error Loss (CONT, DIFF, CONVEX)

A straightforward function that roughly counts the total number of classification errors while being smooth and differentiable is known as the minimum classification error (MCE) loss [162]. The MCE is written as a sigmoid function:

$$L_{mce}(\mathbf{\Theta },{\mathbf{x}}_i,{\mathbf{y}}_i)=\sigma (E(\mathbf{\Theta },{\mathbf{x}}_i,{\mathbf{y}}_i)-E(\mathbf{\Theta },{\mathbf{x}}_i,{\overline{\mathbf{y}}}_i))$$

(82)

where $\sigma$ is defined as $\sigma \left(x\right)={(1+e^{-x})}^{-1}$. While this function lacks an explicit margin, it nevertheless produces an energy difference between the most offending incorrect output and the correct output.

Square-Square Loss (CONT, CONVEX)

Square-square loss deals differently with the energy of the correct output $E(\mathbf{\Theta },{\mathbf{x}}_i,{\mathbf{y}}_i)$ and the energy of the most offensive output $E(\mathbf{\Theta },{\mathbf{x}}_i,{\overline{\mathbf{y}}}_i)$ as:

$$L_{sq-sq}(\mathbf{\Theta },{\mathbf{x}}_i,{\mathbf{y}}_i)=E{(\mathbf{\Theta },{\mathbf{x}}_i,{\mathbf{y}}_i)}^2+{(max(0,m-E(\mathbf{\Theta },{\mathbf{x}}_i,{\overline{\mathbf{y}}}_i)))}^2$$

(83)

The combination aims to minimize the energy of the correct output while enforcing a margin of at least m on the most offending incorrect outputs. This loss is a modified version of the margin loss. This loss can only be used when there is a lower bound on the energy function [126,155].

Square-Exponential Loss (CONT, DIFF, CONVEX)

This loss is similar to the square-square loss function, and it only differs in the second term:

$$L_{sq-exp}(\mathbf{\Theta },{\mathbf{x}}_i,{\mathbf{y}}_i)=E{(\mathbf{\Theta },{\mathbf{x}}_i,{\mathbf{y}}_i)}^2+\gamma e^{-E(\mathbf{\Theta },{\mathbf{x}}_i,{\overline{\mathbf{y}}}_i)}$$

(84)

While $\gamma$ is a positive constant, the combination aims to minimize the energy of the correct output while pushing the energy of the most offending incorrect output to an infinite margin [125,142,155]. This loss is considered a regularized version of the aforementioned square-square loss. This loss, as for the square-square loss, can only be used when there is a lower bound on the energy function [126,155].

9. Relational Learning

While regression and classification losses focus on point-wise predictions, and ranking losses focus mainly on pairwise relationships, relational learning losses aim to capture the topological dependencies and global properties of graph-structured data. These losses are predominant in graph neural networks (GNNs) and manifold learning methods, where the goal is to embed nodes such that the geometric and relational structure of the original graph is preserved in the latent space. A graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$ consists of a set of nodes $\mathcal{V}$ and edges $\mathcal{E}$ representing connections (e.g., in social networks, molecular graphs, knowledge graphs, or recommendation systems). Unlike traditional neural networks that assume Euclidean input structure (grids or sequences), GNNs directly incorporate graph topology and node features to model complex relational dependencies [163].

Common loss functions in graph learning include probabilistic losses like cross-entropy (see Section 5.3.1), which is well suited for supervised node and graph classification as well as link prediction tasks [164,165], and hinge-based, used in ranking scenarios, such as knowledge graph embedding, where relative distances between correct and incorrect edges are optimized [166,167]. In generative settings (see Section 6), reconstruction losses are employed by graph autoencoders (GAEs) and variational GAEs to learn node embeddings that can accurately reconstruct the graph’s adjacency matrix, effectively modeling the underlying graph distribution [165,168]. Relational learning losses can thus be categorized by their underlying strategy: probabilistic losses and error-based losses.

Many graph learning objectives, especially in self-supervised settings, explicitly leverage the structure and topology of the graph beyond what standard pointwise supervision provides. In this section, we describe several relational learning losses that encode distinct inductive biases. Some enforce smoothness over local neighborhoods, others preserve higher-order structural relationships or role-based similarities, and yet others use contrastive objectives to learn embeddings invariant to graph perturbations. These losses can be used standalone or as regularizers alongside supervised objectives, guiding GNNs to capture both local and global patterns and yielding embeddings that are structurally meaningful and generalize well across downstream tasks.

9.1. Graph Reconstruction Loss (CONT, DIFF, L-CONT)

Graph reconstruction losses aim to rebuild the graph from learned embeddings, typically treating the presence or absence of edges as a prediction target. A common choice is a binary cross-entropy loss over the adjacency matrix. For an unweighted graph with adjacency matrix A, the model (e.g., a GAE) produces an estimated probability ${\widehat{A}}_{ij}$ for each potential edge $(i,j)$. The loss sums over all node pairs:

$${\mathcal{L}}_{\mathrm{recon}}=-\sum _{i,j}\left[A_{ij}log{\widehat{A}}_{ij}+(1-A_{ij})log\left(1-{\widehat{A}}_{ij}\right)\right],$$

(85)

often with negative sampling or weighting to handle class imbalance (since sparse graphs have far more absent edges than present ones). By directly optimizing the likelihood of observed links, this loss encourages embeddings to encode information sufficient to predict graph connectivity. Graph reconstruction loss is widely used in unsupervised network embedding and link prediction scenarios, providing a simple yet effective way to train on graph structure when explicit labels are unavailable. It underpins approaches like VGAE [165] and ARGA (adversarially regularized GAE) [168], which have demonstrated strong performance on link prediction benchmarks.

One advantage of the reconstruction objective is its directness and interpretability: low reconstruction error implies that important connections in the original graph are preserved in the embedding space. However, reconstructing every edge can lead to overfitting to noisy or uninformative links and over-emphasizing high-degree nodes; regularization techniques (e.g., adversarial training [168]) and negative sampling are often necessary to mitigate these issues. Moreover, this loss focuses on immediate local structure (adjacent relationships), so it may be complemented by additional terms that capture broader graph properties for a more holistic representation.

9.2. Random Walk-Based Loss (CONT, DIFF, L-CONT)

Random walk-based methods, such as DeepWalk [169] and node2vec [170], learn node embeddings by leveraging the co-occurrence statistics of nodes in short random walks over the graph. The key idea is analogous to word embeddings in NLP: nodes that frequently appear together within a random walk window should have similar vector representations. The loss is typically formulated as a skip-gram objective with negative sampling. For a node v, let $N_w\left(v\right)$ denote the multiset of nodes encountered within a window of length w in random walks starting from v. Let ${\mathbf{h}}_u$ denote the embedding of node u. A common objective is:

$${\mathcal{L}}_{\mathrm{RW}}=-\sum _{v\in V}\sum _{u\in N_w\left(v\right)}\left[log\sigma \left({\mathbf{h}}_v^{\top }{\mathbf{h}}_u\right)+\sum _{u^{\prime }\sim P_n}log\left(1-\sigma \left({\mathbf{h}}_v^{\top }{\mathbf{h}}_{u^{\prime }}\right)\right)\right],$$

(86)

where $\sigma$ is the sigmoid function, $P_n$ is a noise distribution over nodes for drawing negative samples $u^{\prime }$, and ${\mathbf{h}}_v^{\top }{\mathbf{h}}_u$ represents the similarity (inner product) between node embeddings. This loss pushes co-occurring nodes to have high similarity, while pulling randomly sampled node pairs apart, effectively capturing both local and some higher-order proximity patterns in the graph. Random-walk-based losses are unsupervised and have been widely used for node representation learning, link prediction, and even as a pretraining mechanism for GNNs. They can incorporate vertex proximity information without requiring labels.

A notable advantage of this approach is its scalability and simplicity, as it reduces graph mining to repeated local context generation and word2vec-style optimization. However, these methods rely purely on graph structure (ignoring node attributes) and are typically transductive; embeddings for new nodes cannot be obtained without retraining. Subsequent inductive methods like GraphSAGE [171] address this limitation by learning a mapping function that can generate embeddings for unseen nodes. Hyperparameters such as walk length, context window size, and the number of negative samples must be carefully tuned, as they significantly influence the quality of the learned embeddings.

9.3. Motif-Based Loss (CONT, DIFF, L-CONT)

Motif-based losses aim to capture higher-order graph structures by encouraging node embeddings to preserve specific recurrent subgraph patterns (motifs) such as triangles, cliques, or cycles. Motifs often carry meaningful functional information—for instance, triangles in social networks may indicate strong friend groups, while specific graphlets in protein interaction networks signify biological complexes. A standard formulation, introduced by Benson et al. [172], involves constructing a motif-adjacency matrix $A^M$, where $A_{ij}^M$ denotes the count of motifs containing both nodes i and j. The objective is to learn embeddings that can reconstruct this higher-order proximity matrix, effectively treating motif co-occurrence as a stronger form of connectivity. Formally, one can minimize the reconstruction error between the motif adjacency and the embedding similarity:

$${\mathcal{L}}_{\mathrm{motif}}=\sum _{i,j}\parallel A_{ij}^M-{\mathbf{h}}_i^{\top }{\mathbf{h}}_j{\parallel }^2,$$

(87)

where ${\mathbf{h}}_i,{\mathbf{h}}_j$ are the node embeddings. By optimizing this loss, the model forces nodes that frequently participate in the same motifs to have highly similar representations. This approach provides a powerful inductive bias, as demonstrated in works on higher-order spectral clustering [172] and motif-based graph convolution [173], which show that incorporating motif structures significantly improves performance on tasks sensitive to local topology. It has been applied in social networks (to enhance community detection or model friend groups) [174,175], in biological networks (to capture functional substructures like regulatory loops or protein complexes) [176,177], and in knowledge graphs (to incorporate common relational subgraph patterns) [178,179]. A practical consideration is the computational cost of motif enumeration, which can be high for large graphs; thus, one must carefully select motifs relevant to the specific downstream task. Nonetheless, when appropriate motifs are identified, this loss provides a powerful inductive bias, as demonstrated in motif-based graph convolutions [173], significantly improving performance on tasks sensitive to local topology.

9.4. Graph Contrastive Loss (CONT, DIFF, L-CONT)

Graph contrastive loss is a self-supervised objective designed to learn informative node or graph embeddings by comparing different augmented views of a graph. The central idea is that the representation of an entity (a node, subgraph, or entire graph) should be similar under different data augmentations, while remaining distinct from representations of other entities. In practice, one constructs two views of the graph (for instance, by randomly dropping or perturbing edges, features, or nodes) and obtains embeddings for the same node in both views. A contrastive loss then maximizes agreement between embeddings of corresponding entities (positives) and minimizes agreement with embeddings of different entities (negatives). A typical formulation of the contrastive loss for node embeddings uses an InfoNCE-based objective [131,180]:

$${\mathcal{L}}_{\mathrm{CL}}=-\sum _{i\in V}log{\displaystyle \frac{e^{\mathrm{sim}({\mathbf{h}}_i,{\mathbf{h}}_i^{+})/\tau }}{e^{\mathrm{sim}({\mathbf{h}}_i,{\mathbf{h}}_i^{+})/\tau }+{\sum }_{j\ne i}{e}^{\mathrm{sim}({\mathbf{h}}_i,{\mathbf{h}}_j^{-})/\tau }}},$$

(88)

where ${\mathbf{h}}_i$ is the embedding of node i in the original graph, ${\mathbf{h}}_i^{+}$ is the embedding of the same node in an augmented version of the graph (a positive pair), ${\mathbf{h}}_j^{-}$ are embeddings of negative samples (other nodes or nodes from different graphs), $\mathrm{sim}(·,·)$ is a similarity function (e.g., cosine similarity), e is the Euler number, and $\tau$ is a temperature hyperparameter. By maximizing this InfoNCE objective, the model learns embeddings that are invariant to the chosen graph perturbations yet discriminative among different nodes. Graph contrastive learning has proven effective for self-supervised representation learning on graphs, yielding embeddings that perform well on downstream tasks such as node classification, graph classification, link prediction, and transfer learning without requiring any labels. Notable examples include methods like GraphCL [181] and GRACE [182], which differ in the types of augmentations used (edge removals, attribute masking, node dropout, subgraph sampling, etc.), and deep graph infomax (DGI) [183], which employs a global–local contrastive scheme.

A major advantage of contrastive losses is their ability to extract rich features by maximizing mutual information between different views of the data [183]. However, their performance can be sensitive to the choice of graph augmentations and negative sampling strategy: inappropriate or overly strong augmentations might destroy important structural information, whereas trivial augmentations can lead to degenerate solutions (e.g., collapsed embeddings that lack useful information). Careful design of augmentations and tuning of the temperature and negative sample pool are required to ensure that the contrastive task provides a meaningful learning signal. Despite these challenges, contrastive objectives offer a flexible framework to leverage abundant unlabeled graph data for representation learning, and they have quickly become a cornerstone of state-of-the-art GNN pretraining techniques.

9.5. Graph Laplacian (Smoothness) Loss (CONT, DIFF, L-CONT)

The graph Laplacian loss (also known as a smoothness or homophily regularization) is a fundamental graph-specific loss that encourages connected nodes to have similar embeddings. The intuition is that in many real-world graphs, adjacent nodes are likely to share properties (the homophily assumption). Given the adjacency matrix A of the graph, the Laplacian smoothness loss is typically defined as:

$${\mathcal{L}}_{\mathrm{lap}}={\displaystyle \frac{1}{2}}\sum _{i,j}{A}_{ij}{\parallel {\mathbf{h}}_i-{\mathbf{h}}_j\parallel }^2,$$

(89)

where ${\mathbf{h}}_i$ and ${\mathbf{h}}_j$ are the embeddings of nodes i and j respectively. This objective penalizes large differences between the representations of any two connected nodes, encouraging the learned embedding function to vary smoothly across edges. Graph Laplacian-based losses are widely used in semi-supervised learning on graphs (where only a small subset of nodes have labels) as a regularizer to complement a supervised loss, as well as in purely unsupervised manifold learning and spectral embedding methods [184]. By enforcing local smoothness, this loss helps models generalize from limited labeled data and preserves local structural patterns in the learned embeddings. For example, graph convolutional networks (GCNs) implicitly incorporate this principle in their message-passing layers and often benefit from an explicit Laplacian regularization term during training to further align neighboring node representations [165].

A benefit of ${\mathcal{L}}_{\mathrm{lap}}$ is that it is convex and easy to compute. However, if over-emphasized, a strong smoothness constraint can lead to the well-known over-smoothing effect, wherein node embeddings become nearly indistinguishable across large regions of the graph after repeated averaging [185]. This is especially problematic in graphs with heterophily, where connected nodes have disparate labels or attributes, and forcing them to be similar can harm performance. In practice, the Laplacian loss term is often given a relatively small weight or applied only to certain layers, to avoid washing out important differences while still promoting consistency among neighbors. Despite this caveat, Laplacian smoothing is a key component in many graph learning approaches, as it exploits network connectivity to impose a useful inductive bias of local consistency.

9.6. Mutual Information Maximization Loss (CONT, DIFF)

Another paradigm in self-supervised graph learning is to maximize the mutual information (MI) between local and global representations of a graph. The principle is that a good node embedding should capture not only its local neighborhood features but also be informative about the broader subgraph or graph it resides in. Deep graph infomax (DGI) [183] is a representative approach that introduces a global summary vector for the graph and trains the model to distinguish between embeddings from the true graph and those from a corrupted graph. Concretely, one computes a readout or summary representation $\mathbf{s}$ of the entire graph (e.g., by average or pooling over all node embeddings), and uses a discriminator to classify node embeddings as either coming from the original graph (positive) or from a “negative” graph where the input features or structure have been shuffled. The loss can be written as:

$${\mathcal{L}}_{\mathrm{MI}}=-\sum _{i\in V}log\sigma \left({\mathbf{h}}_i^{\top }\mathbf{s}\right)-\sum _{j\in V}log\left(1-\sigma \left({\tilde{\mathbf{h}}}_j^{\top }\mathbf{s}\right)\right),$$

(90)

where ${\mathbf{h}}_i$ is the embedding of node i in the real graph, ${\tilde{\mathbf{h}}}_j$ is the embedding of node j in a corrupted version of the graph (the negative sample), $\mathbf{s}$ is the summary embedding of the real graph, and $\sigma$ is the sigmoid function. By maximizing the mutual information between $\left\{{\mathbf{h}}_i\right\}$ and the global descriptor $\mathbf{s}$, the model is encouraged to produce node representations that are globally coherent and sensitive to the graph’s overall structure. This enables effective self-supervised training of GNNs, allowing them to generate high-quality embeddings without any labels and to achieve strong performance on tasks like node classification, clustering, and transfer learning [183].

The advantage of MI-based losses like DGI is that they force the encoder to capture not just immediate neighbor information but also higher-level structural patterns that correlate with global graph properties. A potential drawback is the need to design a good graph corruption strategy (e.g., feature shuffling or graph sampling) to generate challenging negatives. If the negative examples are too trivial (e.g., completely random graphs), the discriminator’s task becomes too easy and the learned embeddings may not be very informative; if negatives are too similar to the real graph, the task may be overly difficult. Despite these considerations, mutual information maximization loss has proven to be a powerful driver for unsupervised graph representation learning and is complementary to purely local objectives: it encourages the preservation of global graph information that might otherwise be overlooked.

9.7. Distance/Structural Preservation Loss (CONT, DIFF, L-CONT)

Distance-preserving losses are designed to maintain certain graph-theoretic distances or structural similarities between nodes in the learned embedding space. Unlike local adjacency-based losses (e.g., reconstruction or Laplacian smoothing), these losses aim to preserve more global and mesoscopic structure—such as community structure, centrality, or role equivalence—by embedding nodes such that graph-derived distances are reflected as geometric distances. The general idea is to ensure that if two nodes are structurally similar in the original graph (not necessarily directly connected, but e.g., having similar connectivity patterns or occupying analogous positions), their embeddings should be close in vector space. A typical formulation uses a penalty on differences between graph distances and Euclidean distances of embeddings. For instance:

$${\mathcal{L}}_{\mathrm{dist}}=\sum _{i,j}{\left(d_{\mathrm{G}}(i,j)-\parallel {\mathbf{h}}_i-{\mathbf{h}}_j\parallel \right)}^2,$$

(91)

where $d_{\mathrm{G}}(i,j)$ is a chosen graph-based distance or similarity measure between nodes i and j (e.g., shortest-path distance, random walk distance, or similarity based on graph kernels), and $\parallel {\mathbf{h}}_i-{\mathbf{h}}_j\parallel$ is the distance between the corresponding node embeddings. This objective penalizes discrepancies between distances in the graph and distances in the embedding space, ensuring that nodes with similar structural roles or positions remain close in latent space. Distance-preservation losses have been applied to tasks such as role discovery and structural node embedding (capturing nodes with similar graph roles even if they are far apart) [186,187], anomaly detection in networks (where structurally unusual nodes should stand out in the embedding space) [188], and improving link prediction by accounting for long-range dependencies beyond immediate neighbors [170,186].

A benefit of these losses is that they incorporate global information that purely local losses might miss; for example, they can group together nodes that are analogously positioned in different parts of the graph (like hubs in different communities). On the downside, computing and enforcing all-pairs distance preservation can be computationally expensive for large graphs (scaling roughly with ${\left|\mathcal{V}\right|}^2$ pairs). In practice, methods often rely on sampling strategies or truncated distance measures to remain tractable. Additionally, if the chosen distance metric $d_{\mathrm{G}}$ does not align with the target task (for instance, preserving exact shortest-path distances might be less useful for a classification task that only depends on local structure), this loss can introduce noise. Therefore, distance-preserving objectives are most beneficial when the structural patterns themselves are of interest (as in understanding network roles or anomalies), or as complementary losses to ensure that an embedding captures multi-scale structure alongside other task-specific objectives.

10. Conclusions

The choice of an appropriate loss function is fundamental to the success of machine learning applications. In this survey, we have comprehensively reviewed and detailed 52 of the most commonly used loss functions (summarized in Appendix A), covering a diverse range of machine learning tasks, including classification, regression, generative modeling, ranking, energy-based modeling, and relational learning. Each loss function has been contextualized within its domain of application, highlighting both its advantages and limitations to guide informed selection based on specific problem requirements. This survey is intended as a resource for machine learning practitioners at all levels, including undergraduate, graduate, and Ph.D. students, as well as researchers working to deepen their understanding of loss functions or to develop new ones. By structuring each loss function within a novel taxonomy, presented in Section 2, we have created an intuitive framework that categorizes these functions based on their underlying principles, problem domains, and methodological approaches. This taxonomy provides readers with a clear view of the relationships among different techniques, enabling them to quickly identify the most suitable loss function for their task while also facilitating comparison across methods. We hope that this structured view of loss functions will not only streamline the selection process for practitioners but also support the conceptual development of this field.

Looking ahead, current research increasingly treats the loss function not as a fixed component, but as a design variable that can be adapted to the task, the data, and the training objective. Important trends include automated objective design through meta-learning and search-based approaches [189,190,191], adaptive and composite objectives for multi-task and self-supervised learning [192,193], and robustness-oriented objectives that explicitly address noisy labels, class imbalance, calibration, adversarial robustness, and distribution shift [194,195,196,197,198,199,200]. In addition, large language models are increasingly trained with preference-based and alignment-oriented objectives that extend beyond next-token prediction, including RLHF-style pipelines and direct preference optimization methods [118,201,202]. These developments suggest that the study of loss functions is progressively shifting from cataloguing standard objectives toward the broader problem of objective design.

These trends also indicate several natural directions for extending the present survey. A first improvement would be to include a more systematic analysis of the computational cost of different loss functions, including time and memory requirements, optimization overhead, and training stability. A second extension would be to enrich the appendix and section summaries with more detailed comparative tables, including typical frameworks and deployment considerations. Future versions of this survey could also expand the coverage of domain-specific dense-prediction losses, multi-loss weighting strategies, loss design under concept drift and non-stationary data, and automatically discovered or data-adaptive objectives in modern machine learning pipelines [203,204]. In this sense, the present work can also serve as a basis for future iterations tracking how loss function design continues to evolve across machine learning.