Cross Entropy in Deep Learning of Classifiers Is Unnecessary—ISBE Error Is All You Need

Abstract

In deep learning of classifiers, the cost function usually takes the form of a combination of SoftMax and CrossEntropy functions. The SoftMax unit transforms the scores predicted by the model network into assessments of the degree (probabilities) of an object’s membership to a given class. On the other hand, CrossEntropy measures the divergence of this prediction from the distribution of target scores. This work introduces the ISBE functionality, justifying the thesis about the redundancy of cross-entropy computation in deep learning of classifiers. Not only can we omit the calculation of entropy, but also, during back-propagation, there is no need to direct the error to the normalization unit for its backward transformation. Instead, the error is sent directly to the model’s network. Using examples of perceptron and convolutional networks as classifiers of images from the MNIST collection, it is observed for ISBE that results are not degraded with SoftMax only but also with other activation functions such as Sigmoid, Tanh, or their hard variants HardSigmoid and HardTanh. Moreover, savings in the total number of operations were observed within the forward and backward stages. The article is addressed to all deep learning enthusiasts but primarily to programmers and students interested in the design of deep models. For example, it illustrates in code snippets possible ways to implement ISBE functionality but also formally proves that the SoftMax trick only applies to the class of dilated SoftMax functions with relocations.

1. Introduction

A deep model is a kind of mental shortcut [1], broadly understood as a model created in deep learning of a certain artificial neural network $\mathcal{N}$, designed for a given application. What, then, is an artificial neural network [2], its deep learning [3,4], and what applications [5] are we interested in?

From a programmer’s perspective, an artificial neural network is a type of data processing algorithm [6], in which subsequent steps are carried out by configurable computational units, and the order of processing steps is determined by (dynamically created) computing graph. The computing graph is always directed and acyclic (DAG). Interestingly, even recurrent neural networks, such as the LSTM (Long Short-Term Memory) nets, which are trained using gradient methods, have DAG-type computational graphs defined.

At the training stage, each group of input data X, i.e., each group of training examples, technically each batch of training examples, first undergoes the inference (forward) phase on the current model, i.e., processing through the network $\mathcal{N}$ at its current parameters W. As a result, network outputs $Y\leftarrow {\mathcal{F}}_{\mathcal{N}}(X;W)$ are produced [7]. The result of the entire network $\mathcal{N}$ with the functionality ${\mathcal{F}}_{\mathcal{N}}$ and joined set of parameters W is the result of combining the results of the activities for individual units $\mathcal{U}$ with individual functionalities ${\mathcal{F}}_{\mathcal{U}}$ and with possible individual parameters $W_{\mathcal{U}}$, as well.

$$\begin{array}{c}{\displaystyle \stackrel{\mathrm{Inference}}{\overbrace{\stackrel{X;W}{\to }\begin{array}{|c|}\hline {\mathcal{F}}_{\mathcal{N}}\\ \hline\end{array}\stackrel{Y\leftarrow {\mathcal{F}}_{\mathcal{N}}(X;W)}{\to }}}\equiv \stackrel{\mathrm{Inference}}{\overbrace{\begin{array}{|c|}\hline \cdots \stackrel{X_u;W_u}{\to }\begin{array}{|c|}\hline {\mathcal{F}}_{\mathcal{U}}\\ \hline\end{array}\stackrel{Y_u\leftarrow {\mathcal{F}}_{\mathcal{U}}(X_u;W_u)}{\to }\cdots \\ \hline\end{array}}}}\hfill \end{array}$$

After the inference phase comes the model update phase, where the current model is modified (improved) according to the selected optimization procedure [8]. The model update phase begins with calculating the loss (cost) value $Z\leftarrow \mathcal{L}(Y,Y^{\circ })$ defined by the chosen loss function $\mathcal{L}$ as well as the inference outcome Y and the target result $Y^{\circ }$.

$$\begin{array}{c}\stackrel{\mathrm{Inference}}{\overbrace{{\displaystyle \stackrel{X;W}{\to }\begin{array}{|c|}\hline {\mathcal{F}}_{\mathcal{N}}\\ \hline\end{array}\stackrel{Y\leftarrow {\mathcal{F}}_{\mathcal{N}}(X;W)}{\to }}}}\cdots \stackrel{\mathrm{Model}\mathrm{update}\text{-}\mathrm{start}}{\overbrace{\stackrel{Y,Y^{\circ }}{\to }\begin{array}{|c|}\hline \mathcal{L}\\ \hline\end{array}\stackrel{Z\leftarrow \mathcal{L}(Y,Y^{\circ })}{\to }}}\hfill \end{array}$$

The loss Z depends (indirectly through Y) on all parameters W and what conditions the next step of the update phase is the determination of sensitivity $\overline{W}$ of the loss function $\mathcal{L}$ to their changes. The mathematical model of sensitivity is the gradient $\overline{W}\doteq \frac{\partial \mathcal{L}}{\partial W}$. Knowing this gradient, the optimizer will make the actual modification of W in a direction that also takes into account the values of gradients obtained for previous training batches.

Calculating the gradient with respect to parameters actually assigned to different computational units required the development of an efficient algorithm for its propagation in the opposite direction to inference [9,10].

Just as in the inference phase, each unit $\mathcal{U}$ has its formula $Y_u\leftarrow {\mathcal{F}}_{\mathcal{U}}(X_u,W_u)$ for processing data from input $X_u$ to output $Y_u$ with parameters $W_u$, so in the backward gradient propagation phase, it must have a formula ${\overline{X}}_u,{\overline{W}}_u\leftarrow {\overline{\mathcal{F}}}_{\mathcal{U}}\left({\overline{Y}}_u\right)$ for transforming the gradients assigned to its outputs ${\overline{Y}}_u$ into gradients assigned to its inputs ${\overline{X}}_u$ and its parameters ${\overline{W}}_u$.

$$\begin{array}{c}\stackrel{\mathrm{BackPropagation}}{\overbrace{\stackrel{\overline{X};\overline{W}\leftarrow {\overline{\mathcal{F}}}_{\mathcal{N}}(\overline{Y};X,Y,W)}{\leftarrow }\begin{array}{|c|}\hline {\overline{\mathcal{F}}}_{\mathcal{N}}\\ \hline\end{array}\stackrel{\overline{Y}}{\leftarrow }}}\cdots \stackrel{\mathrm{Loss}\mathrm{function}\mathrm{gradient}\frac{\partial L}{\partial Y}}{\overbrace{\stackrel{\overline{Y}\leftarrow \overline{\mathcal{L}}(\overline{Z};Y,Z)}{\leftarrow }\begin{array}{|c|}\hline \overline{\mathcal{L}}\\ \hline\end{array}\stackrel{\mathbf{1}=\frac{\partial Z}{\partial Z}}{\leftarrow }}}\\ \stackrel{\mathrm{Gradient}\mathrm{BackPropagation}}{\overbrace{\begin{array}{|c|}\hline \cdots \stackrel{{\overline{X}}_u;{\overline{W}}_u\leftarrow {\overline{\mathcal{F}}}_{\mathcal{U}}({\overline{Y}}_u;X_u,Y_u,W_u)}{\leftarrow }\begin{array}{|c|}\hline {\overline{\mathcal{F}}}_{\mathcal{U}}\\ \hline\end{array}\stackrel{{\overline{Y}}_u}{\leftarrow }\cdots \\ \hline\end{array}}}\end{array}$$

Based on such local rules of gradient backpropagation and the created computation graph, the backpropagation algorithm can determine the gradients of the cost function with respect to each parameter in the network. The computation graph is created during the inference phase and is essentially a stack of links between the arguments and results of calculations performed in successive units [10,11].

Deep learning is precisely a concert of these inference and update phases in the form of gradient propagation, calculated for randomly created groups of training examples. These phases, intertwined, operate on multidimensional, deep tensors (arrays) of data, processed with respect to network inputs, and on deep tensors of gradient data, processed with respect to losses, determined for the output data of the trained network.

Here, by a deep tensor, we mean a multidimensional data array that has many feature maps, i.e., its size along the feature axis is relatively large, e.g., 500, which means 500 scalar feature maps. We then say that at this point in the network, our data has a deep representation in a 500-dimensional space.

As for the applications we are interested in this work, the answer is those that have at least one requirement for classification [12]. An example could be crop detection from satellite images [13], building segmentation in aerial photos [14], but also text translation [15]. Classification is also related to voice command recognition [16], speaker recognition [17], segmentation of the audio track according to speakers [18], recognition of speaker emotions with visual support [19], but also classification of objects of interest along with their localization in the image [20].

It may be risky to say that after 2015, in all the aforementioned deep learning classifiers, the cost function takes the form of a composition of the $SoftMax$ function [21] and the $CrossEntropy$ function, i.e., cross-entropy [22]. The SoftMax unit normalizes the scores predicted by the classifier model for the input object into SoftMax scores that sum up to one, which can be treated as an estimation of the conditional probability distribution of classes. Meanwhile, cross-entropy measures the divergence of this estimation from the target probability distribution (class scores). In practice, the target score may be taken from a training set prepared manually by a so-called teacher [23] or may be calculated automatically by another model component, e.g., in the knowledge distillation technique [24].

For K classes and $n_b$ training examples, the $SoftMax$ function is defined for the raw score matrix $X\in {\mathbb{R}}^{n_b\times K}$ as:

$$\left[Y\leftarrow SoftMax\left(X\right)\right]⟶\left[Y_{bi}\leftarrow \frac{e^{X_{bi}}}{{\displaystyle \sum _{j\in \left[K\right]}{e}^{X_{bj}}}},b\in \left[n_b\right],i\in \left[K\right]\right],$$

where the notation $\left[K\right]$ denotes any K-element set of indices—in this case, they are class labels.

The $CrossEntropy$ function on the matrix $Y,Y^{\circ }\in {\mathbb{R}}^{n_b\times K}$ is defined by the formula:

$$\left[Z\leftarrow CrossEntropy(Y,Y^{\circ })\right]⟶\left[{\displaystyle Z_b\leftarrow -\sum _{j\in \left[K\right]}{Y}_{bj}^{\circ }{log}_e{Y}_{bj},b\in \left[n_b\right],z\in {\mathbb{R}}^{n_b}}\right]$$

$$\stackrel{\mathrm{Classifier}\mathrm{loss}\text{function}:\mathrm{Separated}\mathrm{Implementation}}{\overbrace{\begin{array}{c}\stackrel{\mathrm{Scores}\mathrm{Inference}}{\overbrace{\stackrel{\mathrm{classified}\mathrm{object}}{\to }\begin{array}{|c|}\hline {\mathcal{F}}_{\mathcal{N}}\\ \hline\end{array}\stackrel{\mathrm{raw}\mathrm{scores}X}{\to }}}\hfill \\ \stackrel{\mathrm{Loss}\mathrm{Estimation}\mathcal{L}}{\overbrace{\stackrel{\mathrm{raw}\mathrm{scores}X}{\to }\begin{array}{|c|}\hline \mathtt{SoftMax}\\ \hline\end{array}\stackrel{\mathrm{soft}\mathrm{scores}Y,Y^{\circ }}{\to }\begin{array}{|c|}\hline \mathtt{CrossEntropy}\\ \hline\end{array}\stackrel{\mathrm{losses}Z}{\to }}}\hfill \end{array}}}$$

(1)

When classifiers began using a separated implementation of the combination of the SoftMax normalization and the CrossEntropy loss, it quickly became evident in practice that its implementation had problems with scores close to zero, both in the inference phase and in the backward propagation of its gradient. In formulas of properties 1–3 in Theorem 1 of Section 2, we see from where the problem comes. Only the integration of CrossEntropy with normalization SoftMax eliminated these inconveniences. The integrated approach has the following form:

$$\stackrel{\mathrm{Classifier}\mathrm{loss}\mathrm{function}\text{-}\mathrm{Integrated}\mathrm{Implementation}}{\overbrace{\begin{array}{c}\stackrel{\mathrm{Inference}}{\overbrace{\stackrel{\mathrm{classified}\mathrm{object}}{\to }\begin{array}{|c|}\hline {\mathcal{F}}_{\mathcal{N}}\\ \hline\end{array}\stackrel{\mathrm{raw}\mathrm{scores}X}{\to }}}\hfill \\ \stackrel{\mathrm{Loss}\mathrm{Estimation}\mathcal{L}}{\overbrace{\stackrel{\mathrm{raw}\mathrm{scores}X,\mathrm{soft}\mathrm{scores},Y^{\circ }}{\to }\begin{array}{|c|}\hline \mathtt{CrossEntropy}\circ \mathtt{SoftMax}\\ \hline\end{array}\stackrel{\mathrm{losses}Z}{\to }}}\hfill \end{array}}}$$

The integrated functionality of these two features has the following redundant mathematical notation:

$$\begin{array}{c}Z\leftarrow \left[CrossEntropy\circ SoftMax\right](X,Y^{\circ })⟶\hfill \\ {\displaystyle Z_b\leftarrow -\sum _{j\in \left[K\right]}{Y}_{bj}^{\circ }{log}_e\frac{e^{X_{bj}}}{{\displaystyle \sum _{i\in \left[K\right]}{e}^{X_{bi}}}},b\in \left[n_b\right]}\hfill \end{array}$$

This redundancy in notation was helpful in deriving the equation for the gradient backpropagation for the integrated loss function $CrossEntropy\circ SoftMax$. Later, we will also use for such composition the name $SoftCE$.

The structure of this paper is as follows:

• The second section is devoted to mathematics of the SoftMax trick. Its validity is proved in two ways:

  (a)

  Using gradient formulas for the composition of differentiable functions;

  (b)

  Using the concept of the Jacobian matrix and linear algebra calculus.

• In the third section titled ISBE Functionality, the conditions that a normalization unit must meet for its combination with a cross-entropy unit to have a gradient at the input equal to the difference in soft scores: $\overline{X}=Y-Y^{\circ }$ are analyzed. Then the definition of ISBE functionality is introduced, which in the inference phase (I) normalizes the raw score to a soft score (S), and in the backward propagation phase (B) returns an error (E), equal to the difference in soft scores. It is also justified why, in the case of the $SoftMax$ normalization function, the ISBE functionality has, from the perspective of the learning process, the functionality of the integrated element CrossEntropy ∘ SoftMax.
• In the first subsection of the fourth section, using the example of the problem of recognizing handwritten digits and the standard MNIST(60K) image collection [25], numerous experiments show that in addition to the obvious savings in computational resources, in the case of five activations serving as normalization functions, the classifier’s effectiveness is not lower than that of the combination of the normalization SoftMax and Cross Entropy. This ISBE property was verified for the activation units SoftMax, Sigmoid, Hardsigmoid, and Tanh and Hardtanh. The second subsection of the fourth section reports on how ISBE behaves for a more demanding dataset CIFAR-10 and a more complex architecture VGG-16.
• The final fifth section contains conclusions.
• In Appendix A, the class of functions leading to the dilated SoftMax trick is fully characterized using concepts of dilation and relocation of function domain.
• In Appendix B the ISBE functionality is integrated with PyTorch class torch.autograd. Function.

The main contributions of this research are:

• Introducing ISBE functionality as simplification and, at the same time, extension of the functional combination of SoftMax with CrossEntropy.
• Verification of ISBE feasibility and efficiency on two datasets and three CNN architectures.
• Enhancement of theoretical background for the concept of SoftMax trick via its generalization and full characterization of normalization functions which exhibit this property.

Concluding this introduction, I would like to emphasize that this work is not intended to depreciate the concept of entropy in the context of machine learning. It has played and continues to play a key role as a loss function. Its form appears naturally in many data modeling tasks. For example, in the case of multi-class logistic regression, when computing optimal weights, maximizing the negative logarithm of the likelihood function directly leads to the cross-entropy function. In the context of modeling, cross-entropy will remain an important research tool. The context of the ISBE functionality concerns only the specific method of calculating the gradient of network parameters for the needs of SGD (Stochastic Gradient Descent) optimizers. Only this, nothing more.

2. Discrete Cross-Entropy and the SoftMax Trick Property

This section is devoted to the mathematics of the SoftMax trick. Its validity will be proved in two ways:

• Using gradient formulas for the composition of differentiable functions,
• Using the concept of Jacobian matrix and linear algebra calculus.

The discrete cross-entropy function $CE$ of a target discrete probability distribution $y^{\circ }\in {[0,1]}^K,{\sum }_i{y}_i^{\circ }=1$, relative to the calculated by the classifier the probability distribution $y\in {(0,1)}^K,$${\sum }_i{y}_i=1$, is defined by the formula:

$$CE(y^{\circ },y)\doteq -\sum _{i\in \left[K\right]}{y}_i^{\circ }{log}_e{y}_i$$

(2)

The notation $\left[K\right]$ refers to a sequence of K indexes, such as $(1,\dots ,K)$ or $(0,1,\dots ,K-1)$.

It appears that the gradient of the cross entropy $CE(y^{\circ },y)$ with respect to y is not defined at the zero components:

$${\nabla }_{y}CE(y^{\circ },y)=-y^{\circ }÷y,$$

(3)

where the operation ÷ for the expression $w=u÷v$, denotes the division of vector u components by the components of vector v, i.e., $w_i\doteq u_i/v_i$, $i\in \left[K\right]$.

In the special case when the vector $y\in {(0,1)}^K$ is calculated based on the vector $x\in {\mathbb{R}}^K$ according to the formula $y_i=SoftMax{\left(x\right)}_i=e^{x_i}/{\sum }_k{e}^{x_k}$, the gradient of $CE$ with respect to x has a particularly simple resultant formula. Its simplicity was the reason for the term SoftMax trick:

$$\begin{array}{c}y\left(x\right)\doteq y\doteq SoftMax\left(x\right),SoftCE(y^{\circ },x)\doteq CE(y^{\circ },SoftMax\left(x\right))\hfill \\ ⟶{\nabla }_{x}SoftCE(y^{\circ },x)\stackrel{SoftMaxtrick}{=}y-y^{\circ }\hfill \end{array}$$

(4)

Some authors [26] also use the term SoftMax trick for that part of the proof showing that the derivative of the natural logarithm of the sum of functions $e^{x_i}$ equals to the $SoftMax$ function.

The SoftMax trick can be described as a theorem and proved in two ways: the elementary one and via the matrix calculus. The following theorem includes elementary properties of the cross-entropy function, optionally preceded by the $SoftMax$ normalization.

Theorem 1.

Let $y^{\circ }\in {[0,1]}^K,{\sum }_{k\in \left[K\right]}{y}_k^{\circ }=1,$ be the target probability distribution and $y\in (0,1),$${\sum }_{k\in \left[K\right]}{y}_k=1,$ be the predicted probability distribution. Then

 1.

For any $i\in \left[K\right],$$\frac{\partial CE(y^{\circ },y)}{\partial y_i}=\frac{y_i^{\circ }}{y_i}$.

 2.

${\nabla }_{y}CE(y^{\circ },y)=y^{\circ }÷y\doteq {\left[\frac{y_1^{\circ }}{y_1},\dots ,\frac{y_K^{\circ }}{y_K}\right]}^{⊺}$

 3.

The range of $CE$ covers the positive part of the real axis:

$$\left\{CE(y^{\circ },y):y^{\circ }\in {[0,1]}^K,y\in {(0,1)}^K\right\}=(0,\infty )$$

 4.

Let $x\in {\mathbb{R}}^K$ be the vector of raw scores, and $y^{\circ }$ be the target probability distribution. Then $SoftMax$ normalization followed by $CE$, is defined as follows

$${\displaystyle SoftCE(y^{\circ },x)\doteq -\sum _{j\in \left[K\right]}{y}_j^{\circ }{log}_e\frac{e^{x_j}}{{\sum }_{k\in \left[K\right]}{e}^{x_k}}.}$$

Contrary to $CE$ only, $SoftCE$ exhibits the bounded gradient:

 (a)

The Jacobian of $SoftMax$ equals to:

$$Jacobian\left(SoftMax\right)\left(x\right)=\frac{\partial SoftMax\left(x\right)}{\partial x}=\mathtt{diag}\left[y\right]-y{y}^{⊺}wherey=SoftMax\left(x\right).$$

 (b)

For any $i\in \left[K\right],$ $\frac{\partial \left[SoftCE(y^{\circ },x)\right]}{\partial x_i}=y_i-y_i^{\circ }$, where $y_i={\left(SoftMax\left(x\right)\right)}_i$ .

 (c)

${\nabla }_{x}SoftCE(y^{\circ },x)=y-y^{\circ }$, where $y=SoftMax\left(x\right)$ .

 (d)

The range of $SoftCE$ covers the interval $(-1,1):$

$$\left\{SoftCE(y^{\circ },x):y^{\circ }\in {[0,1]}^K,x\in {\mathbb{R}}^K\right\}=(-1,+1)$$

Elementary proof of SoftCE properties.

$$\begin{array}{c}{\displaystyle y_i\doteq \frac{e^{x_i}}{{\sum }_k{e}^{x_k}}⟶\frac{\partial \left[{log}_e\left({\sum }_k{e}^{x_k}\right)\right]}{\partial x_i}=y_i⟶}\hfill \\ \begin{array}{ccc}\hfill {\displaystyle \frac{\partial SoftCE(y^{\circ },x)}{\partial x_i}}& =& {\displaystyle \frac{\partial \left[{\sum }_j{y}_j^{\circ }{log}_e\left({\sum }_k{e}^{x_k}\right)-{\sum }_j{y}_j^{\circ }\stackrel{x_j}{\overbrace{{log}_e{e}^{x_j}}}\right]}{\partial x_i}}\hfill \\ & =& y_i\stackrel{=1}{\overbrace{{\sum }_j{y}_j^{\circ }}}-y_i^{\circ }=y_i-y_i^{\circ }\hfill \end{array}\hfill \end{array}$$

   □

Proof of SoftCE property using matrix calculus. 

 

In matrix notation [27], the property of SoftMax trick has a longer proof, as we first need to calculate the Jacobian of the $SoftMax$ function [28].

If ${\displaystyle y_j\doteq \frac{e^{x_j}}{{\sum }_{k\in \left[K\right]}{e}^{x_k}}}$, then ${\displaystyle \frac{\partial y_j}{\partial x_i}=\left\{\begin{array}{c}{\displaystyle \frac{-e^{x_j}·e^{x_i}}{{\left({\sum }_{k\in \left[K\right]}{e}^{x_k}\right)}^2}=-y_i·y_j,\mathrm{when}i\ne j}\hfill \\ {\displaystyle \frac{e^{x_i}·\left({\sum }_{k\in \left[K\right]}{e}^{x_k}\right)-e^{x_i}·e^{x_i}}{{\left({\sum }_{k\in \left[K\right]}{e}^{x_k}\right)}^2}=y_i-y_i^2=(1-y_i)·y_i,}\hfill \\ \mathrm{when}i=j\hfill \end{array}\right.}$

The general formula is ${\displaystyle \frac{\partial y_j}{\partial x_i}=({\delta }_{ij}-y_i)y_j.}$ Therefore:

$${\left(\frac{\partial y}{\partial x}\right)}_{ij}\doteq \frac{\partial y_j}{\partial x_i}={\delta }_{ij}{y}_j-y_i{y}_j={\left(\mathtt{diag}\left[y\right]\right)}_{ij}-{\left(y{y}^{⊺}\right)}_{ij}.$$

Hence, $\frac{\partial y}{\partial x}=\mathtt{diag}\left[y\right]-y{y}^{⊺}$. From the chain rule

$$\frac{\partial \left[SoftCE(y^{\circ },x)\right]}{\partial x}\doteq \frac{\partial CE(y^{\circ },y\left(x\right))}{\partial x}={\left(\frac{\partial y}{\partial x}\right)}^{⊺}·\frac{\partial \left[CE(y^{\circ },y)\right]}{\partial y},\mathrm{where}y\left(x\right)\doteq SoftMax\left(x\right)$$

and the symmetry of $SoftMax$ Jacobian matrix $\frac{\partial y}{\partial x}$, we obtain:

$$\begin{array}{ccc}\hfill \frac{\partial \left[SoftCE(y^{\circ },x)\right]}{\partial x}& =& \left(\mathtt{diag}\left[y\right]-y{y}^{⊺}\right)\left(-y^{\circ }÷y\right)\hfill \\ & =& y\underset{{\mathbf{1}}_K^{⊺}{y}^{\circ }=1}{\underbrace{{(y÷y)}^{⊺}{y}^{\circ }}}-\underset{I_K}{\underbrace{\mathtt{diag}\left[y÷y\right]}}{y}^{\circ }=y-y^{\circ }\hfill \end{array}$$

(5)

   □

While looking at the above two proofs for the Theorem 1, a question can be raised: Is it only the $SoftMax$ function that has SoftMax trick property? The answer to this problem can be found in Appendix A. You can understand why the second proof using Jacobian matrix and matrix calculus has been presented here.

3. ISBE Functionality

The ISBE functionality is a proposed simplification of the cost function, combining the SoftMax normalization function with the cross-entropy function, hereafter abbreviated as CE${}_{all}$. Its role is to punish those calculated probability distributions that significantly differ from the distributions of scores proposed by the teacher.

To understand this idea, let us extend the inference diagram for CE${}_{all}$ with the backward propagation part for the gradient. We consider this diagram in its separate version, omitting earlier descriptions for the diagram (1):

$$\begin{array}{c}\stackrel{\mathrm{Loss}\mathrm{Inference}\mathcal{L}}{\overbrace{\stackrel{X}{\to }\begin{array}{|c|}\hline \mathtt{SoftMax}\\ \hline\end{array}\stackrel{Y,Y^{\circ }}{\to }\begin{array}{|c|}\hline \mathtt{CrossEntropy}\\ \hline\end{array}\stackrel{Z}{\to }}}\hfill \\ \stackrel{\mathrm{BackPropagation}}{\overbrace{\stackrel{\overline{X}\stackrel{Theorem1}{\leftarrow }Y-Y^{\circ }}{\leftarrow }\begin{array}{|c|}\hline \overline{\mathtt{SoftMax}}\\ \hline\end{array}\stackrel{\overline{Y};Y,Y^{\circ }}{\leftarrow }\begin{array}{|c|}\hline \overline{\mathtt{CrossEntropy}}\\ \hline\end{array}\stackrel{\overline{Z}=\mathbf{1}}{\leftarrow }}}\hfill \end{array}$$

(6)

The meaning of variables $X,Y,Y^{\circ },Z$ and $\overline{Z},\overline{Y},\overline{X}$ appearing in the above diagram (6):

$\begin{array}{cc}X\hfill & \mathrm{raw}\mathrm{score}\mathrm{at}\mathrm{the}\mathrm{input}\mathrm{of}\mathrm{the}\mathrm{normalization}\mathrm{function}\mathrm{preceding}\text{cross-entropy}\mathrm{CE},X\in {\mathbb{R}}^K,\hfill \\ Y\hfill & \mathrm{normalization}\mathrm{result},\text{so-called}\mathrm{soft}\mathrm{score},Y\in {(0,1)}^K,\hfill \\ Y^{\circ }\hfill & \mathrm{target}\mathrm{soft}\mathrm{score},\mathrm{assigned}\mathrm{to}\mathrm{the}\mathrm{classified}\mathrm{example},\hfill \\ Z\hfill & \mathrm{output}\mathrm{of}\text{cross-entropy}\mathrm{CE},Z\in \mathbb{R},\hfill \\ \overline{Z}\hfill & \mathrm{formal}\mathrm{gradient}\mathrm{at}\mathrm{the}\mathrm{input}\mathrm{of}\mathrm{the}\mathrm{backward}\mathrm{propagation}\mathrm{algorithm},\overline{Z}=1,\hfill \\ \overline{Y}\hfill & \mathrm{gradient}\mathrm{of}\text{cross-entropy}\mathrm{CE}\mathrm{with}\mathrm{respect}\mathrm{to}\mathrm{Y}:\overline{Y}=\frac{\partial Z}{\partial Y}=-\frac{Y^{\circ }}{Y},\hfill \\ \overline{X}\hfill & \mathrm{gradient}\mathrm{of}\text{cross-entropy}\mathrm{CE}\mathrm{with}\mathrm{respect}\mathrm{to}\mathrm{X}:\overline{X}\stackrel{Theorem1}{\leftarrow }(Y-Y^{\circ }).\hfill \end{array}$ 

The key formula here is $\overline{X}\leftarrow (Y-Y^{\circ })$. Its validity comes from the mentioned Theorem 1 which includes the proof for the Formula (4) associated with the SoftMax trick property.

The generalized form of this property is given in the Appendix A within the Theorem A1 which includes interesting observations on necessary and sufficient conditions for the SoftMax trick.

For instance, the Equation (A2) on the form of the Jacobian of the normalization unit is both a sufficient and necessary condition for its combination with the cross-entropy unit to ensure the equality (A3). Moreover, this condition implies that an activation function with a Jacobian of the SoftMax type is a SoftMax function with optional relocation.

Theorem A1 leads us to a seemingly pessimistic conclusion: it is not possible to seek further improvements by changing the activation and at the same time expect the SoftMax trick property to hold. Thus, the question arises: what will happen if, along with changing the activation unit, we change the cross-entropy unit to another or even omit it entirely?

In the ISBE approach, the aforementioned simplification of the CE${}_{all}$ cost function involves precisely omitting the cross-entropy operation in the inference stage and practically omitting all backward operations for this cost function. So what remains? The answer is also an opportunity to decode the acronym ISBE again:

• In the inference phase (I), we normalize the raw score X to $Y=SoftMax\left(X\right)$, characterized as a soft score (S).
• In the backward propagation phase (B), we return an error (E) equal to the difference between the calculated soft score and the target score, i.e., $\overline{X}\doteq Y-Y^{\circ }$.

Why can we do this and still consider that in the case of the SoftMax activation function, the value of the gradient transmitted to the network is identical: ${\overline{X}}_{C{E}_{all}}={\overline{X}}_{ISBE}\doteq Y-Y^{\circ }$?

The answer comes directly from the property ${\overline{X}}_{C{E}_{all}}=Y-Y^{\circ }$, formulated in Equation (4), which as it was already mentioned, was proved in the Theorem 1 as the SoftMax trick property.

We thus have on the left the following diagram of data and gradient backpropagation through such a unit. On the right, we have its generalization to a ScoreNormalization unit instead of SoftMax unit.

$$\left.\begin{array}{c}\stackrel{\mathrm{ISBE}\mathrm{Inference}}{\overbrace{\stackrel{X}{\to }\begin{array}{|c|}\hline \mathtt{SoftMax}\\ \hline\end{array}\stackrel{Y,Y^{\circ }}{\to }}}\hfill \\ \stackrel{\overline{ISBE}\mathrm{BackPropagation}}{\overbrace{\stackrel{\overline{X}\leftarrow Y-Y^{\circ }}{\leftarrow }\begin{array}{|c|}\hline \mathtt{Subtract}\\ \hline\end{array}\stackrel{Y,Y^{\circ }}{\leftarrow }}}\hfill \end{array}\right\}\stackrel{\mathrm{generalize}}{\to }\left\{\begin{array}{c}\stackrel{\mathrm{ISBE}\mathrm{Inference}}{\overbrace{\stackrel{X}{\to }\begin{array}{|c|}\hline \mathtt{Score}\mathtt{Normalization}\\ \hline\end{array}\stackrel{Y,Y^{\circ }}{\to }}}\hfill \\ \stackrel{\overline{ISBE}\mathrm{BackPropagation}}{\overbrace{\stackrel{\overline{X}\leftarrow Y-Y^{\circ }}{\leftarrow }\begin{array}{|c|}\hline \mathtt{Subtract}\\ \hline\end{array}\stackrel{Y,Y^{\circ }}{\leftarrow }}}\hfill \end{array}\right.$$

Which activation functions should we reach for in order to test them in the ISBE technique?

• The SoftMax activation function should be the first candidate for comparison, as it theoretically guarantees behavior comparable to the system containing cross-entropy.
• Activations should be monotonic so that the largest value of the raw score remains the largest score in the soft score sequence.
• Soft scores should be within a limited range, e.g., $[0,1]$ as in the case of SoftMax and Sigmoid, or $[-1,+1]$ as for Tanh.
• The activation function should not map two close scores to distant scores. For example, normalizing a vector of scores by projecting onto a unit sphere in the p-th Minkowski norm meets all the above conditions. However, it is not stable around zero. Normalization $\frac{x}{{\parallel x\parallel }_p}$ maps, for example, two points $ϵ,-ϵ$ distant by $2·{\parallel ϵ\parallel }_p$ to points distant exactly by 2, thus changing their distance $\frac{1}{{\parallel ϵ\parallel }_p}$ times, e.g., a million times, when ${\parallel ϵ\parallel }_p={10}^{-6}$. This operation is known in Pytorch library as normalize function.

The experiments conducted confirm the validity of the above recommendations. The Pytorch library functions SoftMax, sigmoid, tanh, hardsigmoid, hardtanh meet the above three conditions and provide effective classification at a level of effectiveness higher than $99.5\%$, comparable to CrossEntropy ∘ SoftMax. In contrast, with the function normalize, the optimizer failed to converge on the same MNIST(60K) collection and with the same architectures.

What connects these good normalization functions $F:{\mathbb{R}}^K\to {\mathbb{R}}^K$, of which two are not even fully differentiable? Certainly, it is the Lipschitz condition occurring in a certain neighborhood of zero [29]:

$$x\in {\mathbb{R}}^K{,\parallel x\parallel }_p\le ϵ⟶{\parallel F\left(x\right)\parallel }_p\le c{\parallel x\parallel }_p,\mathrm{where}c\mathrm{is}\mathrm{a}\mathrm{certain}\mathrm{constant}.$$

Note that the Lipschitz condition meets the expectations of the fourth requirement on the above list of recommendations for ISBE. Moreover, we do not expect here that the constant c be less than one, i.e., that the function F has a narrowing character.

We also need a recommendation for teachers preparing class labels, which we represent as vectors blurred around the base vectors of axes $I_K=[e_1,\dots ,e_K],e_i\left[j\right]\doteq {\delta }_{ij}$:

• example blurring value $\mu$, e.g., $\mu ={10}^{-6}$:

  $${\tilde{e}}_i\left[j\right]\leftarrow (1-\mu ){\delta }_{ij}+\frac{\mu }{K-1}(1-{\delta }_{ij})$$
• when the range of activation values is other than the interval $[0,1]$, we adjust the vector ${\tilde{e}}_i$ to the new range, e.g., for tanh the range is the interval $(-1,+1)$ and then the adjustment has the form:

  $${\tilde{e}}_i\leftarrow 2·{\tilde{e}}_i-1,i=1,\dots ,K$$

Finally, let us take a look at the code for the main loop of the program implemented on the Pytorch platform.

• This is what the code looks like when loss_function is chosen as nn.CrossEntropyLoss:
  for (labels,images) in tgen:
       outputs = net(images)
       loss = loss_function(outputs, labels)
       optimizer.zero_grad()
       loss.backward()
       optimizer.step()
• Now we introduce the ISBE option for SoftMax activation and replace the call for loss function by soft error calculation:
  for (labels,images) in tgen:
       outputs = net(images)
       soft_error = SoftMax(outputs) - labels
       optimizer.zero_grad()
       outputs.backward(soft_error)
       optimizer.step()

More options, including the definition of ISBE functionality, can be found in Appendix B. Of course, the above code snippets are only intended to illustrate how easy it is to add the functionality of ISBE to an existing application.

4. Experiments

What do we want to learn from the planned experiments? We already know from theory that in the case of the SoftMax activation, we cannot worsen the parameters of the classifier using cross-entropy, both in terms of success rate and learning time.

Therefore, we first want to verify whether theory aligns with practice, but also to check for which normalization functions the ISBE functionality does not degrade the model’s effectiveness compared to CE${}_{all}$.

The learning time $t_{ISBE}$ should be shorter than $t_{CE}$. Still, to be independent of the specific implementation, we will compare the percentage of the backpropagation time in the total time of inference and backpropagation:

$$\tau \doteq \frac{\mathrm{backpropagation}\mathrm{time}}{\mathrm{inference}\mathrm{time}+\mathrm{backpropagation}\mathrm{time}}\times 100\%$$

(7)

From many quality metrics, for simplicity, we choose the success rate (also called accuracy), defined as the percentage of correctly classified elements from the test collection MNIST(10K)

$$\alpha =\frac{\mathrm{number}\mathrm{of}\mathrm{correct}\mathrm{classifications}}{\mathrm{size}\mathrm{of}\mathrm{the}\mathrm{test}\mathrm{collection}}\times 100\%$$

(8)

We want to know how this value changes when we choose different architectures and different activations in the ISBE technique, as well as different options for aggregating cross-entropy over the elements of the training batch.

4.1. Experiments with MNIST Dataset

Firstly, we evaluate the efficiency of the ISBE idea on the standard MNIST(60K) image collection and the problem of their classification.

We have the following degrees of freedom in our experiments:

• Two architecture options
  • Architecture ${\mathcal{N}}_0$ consists of two convolutions   $\mathbb{C}$ and two linear units   $\mathbb{F}$, of which the last one is a projection from the space of deep feature vectors of dimension 512 to the space of raw scores for each of the $K=10$ classes:

    $$\stackrel{\mathrm{image}}{\to }{\mathcal{I}}_{{28}_{yx}}^1{\mathbb{C}}_{3^k{2}^s}^{32}{\mathbb{C}}_{3^k{2}^s}^{64}{\mathbb{D}}_{20}{\mathbb{F}}^{512}{\mathbb{F}}^{10}\stackrel{\mathrm{class}\mathrm{scores}}{\to }$$

    as by STNN notation [30], for instance
    $\left\{\begin{array}{cc}{\mathbb{C}}_{3^k{2}^s}^{64}& \mathrm{means}32\mathrm{convolutions}\mathrm{with}3\times 3\mathrm{masks},\mathrm{sampled}\mathrm{with}\mathrm{a}\mathrm{stride}\mathrm{of}2,\hfill \\ {\mathbb{D}}_{20}& \mathrm{DropOut}—\mathrm{a}\mathrm{unit}\mathrm{zeroing}\mathrm{20\%}\mathrm{of}\mathrm{tensor}\mathrm{elements,},\hfill \\ {\mathbb{F}}^{512}& \mathrm{a}\mathrm{linear}\mathrm{unit}\mathrm{with}\mathrm{a}\mathrm{matrix}A\in {\mathbb{R}}^{?\times 512},\hfill \\ & \mathrm{here}?=64—\mathrm{it}\mathrm{is}\mathrm{derived}\mathrm{from}\mathrm{the}\mathrm{shape}\mathrm{of}\hfill \\ & \mathrm{the}\mathrm{tensor}\mathrm{produced}\mathrm{by}\mathrm{the}\mathrm{previous}\mathrm{unit}.\hfill \end{array}\right.$
  • Architecture ${\mathcal{N}}_1$ consists of two blocks, each with three convolutions—it is a purely convolutional network, except for the final projection:

    $$\stackrel{\mathrm{image}}{\to }{\mathcal{I}}_{{28}_{yx}}^1{\mathbb{C}}_{3^k}^{32}{\mathbb{C}}_{3^k}^{32}{}_p\mathbb{C}_{5^k{2}^s}^{32}{\mathbb{D}}_{40}{\mathbb{C}}_{3^k}^{64}{\mathbb{C}}_{3^k}^{64}{}_p\mathbb{C}_{5^k{2}^s}^{64}{\mathbb{D}}_{40}{\mathbb{C}}_{4^k}^{128}{\mathbb{F}}^{10}\stackrel{\mathrm{class}\mathrm{scores}}{\to }$$
    Note that the last convolution in each block has a p requirement for padding, i.e., filling the domain of the image with additional lines and rows so that the image resolution does not change.
• Three options for reducing the vector of losses in the CrossEntropyLoss element: none, mean, sum.
• Five options for activation functions used in the ISBE technique:
  • SoftMax: ${\displaystyle y_i\leftarrow \frac{e^{x_i}}{{\displaystyle \sum _{j\in \left[K\right]}{e}^{x_j}}},i\in \left[K\right],}$
  • Tanh: ${\displaystyle y_i\leftarrow \frac{e^{x_i}-e^{-x_i}}{e^{x_i}+e^{-x_i}},i\in \left[K\right],}$
  • HardTanh: ${\displaystyle y_i\leftarrow \left\{\begin{array}{cc}-1& \mathrm{if}{x}_i\le -1\hfill \\ x_i& \mathrm{if}-1<x_i<+1\hfill \\ +1& \mathrm{if}+1\le x_i\hfill \end{array}\right\},i\in \left[K\right],}$
  • Sigmoid: ${\displaystyle y_i\leftarrow \frac{1}{1+e^{-x_i}},i\in \left[K\right],}$
  • HardSigmoid: ${\displaystyle y_i\leftarrow \left\{\begin{array}{cc}0& \mathrm{gdy}{x}_i\le -2\hfill \\ \frac{x_i+2}{4}& \mathrm{gdy}-2<x_i<+2\hfill \\ +1& \mathrm{gdy}+2\le x_i\hfill \end{array}\right\}=\frac{HardTanh(x_i/2)+1}{2},i\in \left[K\right].}$

The results of the experiments, on the one hand, confirm our assumption that the conceptual Occam’s razor, i.e., the omission of the cross-entropy unit, results in time savings $\tau$, and on the other hand, the results are surprisingly positive with an improvement in the metric of success rate $\alpha$ in the case of hard activation functions $HardTanh$ and $HardSigmoid$. It was observed that only the option of reduction by none behaves exactly according to theory, i.e., the success rate is identical to the model using $SoftMax$ normalization. Options mean and sum for the model with entropy are slightly better than the model with SoftMax.

The consistency of models in this case means that the number of images incorrectly classified out of 10 thousand is the same. The experiments did not check whether it concerns the same images. A slight improvement, in this case, meant that there were less than a few or a dozen errors, and the efficiency of the model above $99.6\%$ meant at most 40 errors per 10 thousand of test images.

4.1.1. Comparison of Time Complexity

We compare time complexity according to the metric given by the Formula (7).

In the context of time,

Table 1. Comparison of the metric $\tau$, i.e., the percentage share of backpropagation time in the total time with inference. The share ${\tau }_{CE}$ of cross-entropy with three types of reduction is compared with five functions of soft normalization. The analysis was performed for architectures ${\mathcal{N}}_0$ and ${\mathcal{N}}_1$.

Net	Mean	None	Sum	Hsigmoid	Htanh	Sigmoid	SoftMax	Tanh
${\mathcal{N}}_0$	60.61%	59.56%	59.98%	58.31%	58.21%	57.38%	57.45%	59.07%
${\mathcal{N}}_1$	54.89%	53.92%	53.98%	52.68%	52.33%	51.75%	51.95%	52.11%
${\mathcal{N}}_1^r$	54.45%	53.92%	54.00%	52.78%	52.30%	51.67%	51.73%	52.11%

clearly shows that the total timeshare of backpropagation, obviously depending on the complexity of the architecture, affects the time savings of the ISBE technique compared to CrossEntropyLoss—

Table 2. Metric $\Delta \tau \doteq {\tau }_{ISBE}-{\tau }_{CE}$, i.e., the decrease in the percentage share of backpropagation time in the total time with inference. The analysis was performed for architectures ${\mathcal{N}}_0$ and ${\mathcal{N}}_1$.

Net	CE Loss	Hsigmoid	Htanh	Sigmoid	SoftMax	Tanh
${\mathcal{N}}_0$	mean	−2.30%	−2.40%	−3.23%	−3.16%	−1.54%
${\mathcal{N}}_0$	none	−1.25%	−1.35%	−2.18%	−2.11%	−0.50%
${\mathcal{N}}_0$	sum	−1.67%	−1.77%	−2.60%	−2.53%	−0.92%
${\mathcal{N}}_1$	mean	−2.21%	−2.56%	−3.14%	−2.94%	−2.79%
${\mathcal{N}}_1$	none	−1.24%	−1.59%	−2.17%	−1.97%	−1.82%
${\mathcal{N}}_1$	sum	−1.30%	−1.65%	−2.23%	−2.03%	−1.87%

. The absence of pluses in this table, i.e., the fact that all solutions based on ISBE are relatively faster in the learning phase, is an undeniable fact.

The greatest decrease in the share of backpropagation, over $3\%$, occurs for the $Sigmoid$ and $SoftMax$ activations. The smallest decrease in architecture ${\mathcal{N}}_0$ is noted for the soft (soft) normalization function $Tanh$ and its hard version $HardTanh$. This decrease refers to cross-entropy without reduction, which is an aggregation of losses calculated for all training examples in a given group into one numerical value.

Inspired by the Theorem A1, which states that the relocation of the $SoftMax$ function preserves the SoftMax trick property, we also add data to the Table 1 for the network ${\mathcal{N}}_1^r$. This network differs from the ${\mathcal{N}}_1$ network only because the normalization unit has a trained relocation parameter. In practice, we accomplish training with relocation for normalization by training with the relocation of the linear unit immediately preceding it. This is done by setting its parameter: bias = True.

As we can see, the general conclusion about the advantage of the ISBE technique in terms of time reduction for the model with the relocation of the normalization function is the same.

4.1.2. Comparison of Classifier Accuracy

Comparison of classifier accuracy and differences in this metric are contained in

Table 3. In the table, the success rate of three classifiers based on cross-entropy with different aggregation options is compared with the success rate determined for five options of soft score normalization functions. The analysis was performed for architectures ${\mathcal{N}}_0$ and ${\mathcal{N}}_1$.

Net	Mean	None	Sum	Hsigmoid	Htanh	Sigmoid	SoftMax	Tanh
${\mathcal{N}}_0$	99.45%	99.41%	99.47%	99.50%	99.50%	99.56%	99.41%	99.45%
${\mathcal{N}}_1$	99.61%	99.58%	99.59%	99.64%	99.66%	99.64%	99.62%	99.63%
${\mathcal{N}}_1^r$	99.55%	99.64%	99.64%	99.61%	99.66%	99.69%	99.63%	99.57%

and

Table 4. Change in success rate between models with cross-entropy and models with soft score normalization function. The analysis was performed for architectures ${\mathcal{N}}_0$ and ${\mathcal{N}}_1$.

Net	CE Loss	Hsigmoid	Htanh	Sigmoid	SoftMax	Tanh
${\mathcal{N}}_0$	mean	0.05%	0.05%	0.11%	−0.04%	0.00%
${\mathcal{N}}_0$	none	0.09%	0.09%	0.15%	0.00%	0.00%
${\mathcal{N}}_0$	sum	0.13%	0.03%	0.09%	−0.06%	−0.02%
${\mathcal{N}}_1$	mean	0.03%	0.05%	0.03%	0.01%	0.02%
${\mathcal{N}}_1$	none	0.06%	0.08%	0.06%	0.04%	0.05%
${\mathcal{N}}_1$	sum	0.05%	0.07%	0.05%	0.03%	0.04%

. The accuracy is computed according to the Formula (8).

The number of pluses on the side of ISBE clearly exceeds the number of minuses. The justification for this phenomenon requires separate research. Some light will be shed on this aspect by the analysis of learning curves—the variance in the final phase of learning is clearly lower. The learning process is more stable.

In Table 4, we observe that, with the exception of the function $SoftMax$, which on several images of digits performed worse than the model with cross-entropy, the soft activations have an efficiency slightly or significantly better. However, we are talking about levels of tenths or hundredths of a percent here. The largest difference noted for the option SoftMax was 15-hundredths of a percent, meaning 15 more images correctly classified. Such differences are within the margin of statistical error.

The use of relocation for the normalization function does not provide a clear conclusion—for some models, there is a slight improvement; for others, there is a slight deterioration. It is true that the ISBE functionality with sigmoid activation achieved the best efficiency of $99.69\%$, but this is only a matter of a few images.

Within the limits of statistical error, we can say that the ISBE functionality gives the same results in recognizing MNIST classes. Its advantages are:

• of decrease time in the total time,
• simplification of architecture, and therefore playing the philosophical role of Occam’s razor.

4.1.3. Visual Analysis

Further analysis of the results will be based on the visual comparison of learning curves.

First, let us see on three models cross-entropy-mean, SoftMax, sigmoid their loss and efficiency curves obtained on training data MNIST(54K) and on data intended solely for model validation MNIST(6K). These two loss curves are calculated after each epoch. We supplement them with a loss curve calculated progressively after each batch of training data (see Figure 1).

Let us note the correct course of the train loss curve with respect to the progressive loss curve—both curves are close. The correct course is also for the validation loss curve—the validation curve from about epoch 30 is below the training curve, maintaining a significant distance. This effect was achieved only after applying a moderate input image augmentation procedure via random affine transformations in the pixel domain.

Correct behavior of learning curves was recorded both for the models with entropy and for models with the ISBE functionality. This also applies to classifier performance curves.

• Curves of loss functions can appear together as long as the type of function is identical, which entails a similar range of variability for loss function values. One might wonder what measure of loss to adopt in the case of ISBE since this technique, in fact, does not calculate loss values. We opt for a natural choice of mean square error for errors in soft scores:

  $${\mathcal{L}}_{ISBE}=MSE(Y,Y^{\circ })\doteq \frac{1}{n_b}·{\parallel Y-Y^{\circ }\parallel }_2^2$$

  where $n_b$ is the batch size.
  For such defined measures, it turns out that only the option of reduction by summing has a different range of variability, and therefore it is not on the Figure 2.
• In the case of classifier accuracy, a common percentage scale does not exclude placing all eight curves for each considered architecture. However, due to the low transparency of such a figure, it is also worth juxtaposing different groups of curves of the dependency $\alpha \left(e\right)$. The accuracy $\alpha$ of the classifier MNIST(60K) is calculated on the test set MNIST(10K).

Sets of curves, which we visualize separately for architectures ${\mathcal{N}}_0$, ${\mathcal{N}}_1$ are:

• All options for loss functions (3) and soft score functions (5),
• CE none, CE mean, CE sum versus SoftMax,
• CE none, CE mean, CE sum versus tanh, hardtanh,
• SoftMax versus sigmoid, hardsigmoid,
• SoftMax versus tanh, hardtanh,
• SoftMax versus sigmoid, tanh.

Due to space constraints, we show learning curves and classifier effectiveness graphs only for architecture ${\mathcal{N}}_1$ in Figure 2 and Figure 3.

In Figure 2 we can clearly observe four clusters of models:

• CrossEntropyLoss based with reduction option sum (as out of common range it was not shown),
• CrossEntropyLoss based with reduction options none, and mean,
• ISBE based with normalizations to range $[0,1]$ including functions
  $SoftMax,$ $Sigmoid,$ and $HardSigmoid$,
• ISBE based with normalizations to range $[-1,1]$ including functions $Tanh,$ and $HardTanh$.

Within a cluster, the loss curves behave very similarly. Interestingly, the loss curves in ISBE-based clusters tend to the same value greater than zero. In contrast, cross-entropy-based curves also tend to the same limit. However it is clearly greater than ISBE one.

Now, we will pay more attention to test learning curves. We generate test learning curves on the full set of test data MNIST(10K). After each epoch, one point is scored towards the test learning curve. We will show these curves in several comparative contexts.

Accuracy charts of learning (see Figure 3) were obtained to compare cross entropy (CE) performances versus ISBE performance. We have:

• Comparison of CE versus soft options:
  • all options for loss functions and soft score functions
  • CE none, CE mean versus SoftMax,
  • CE none, CE mean versus tanh, hardtanh.
• Comparison of SoftMax versus other soft options:
  • SoftMax versus sigmoid, hardsigmoid,
  • SoftMax versus tanh, hardtanh,
  • SoftMax versus sigmoid, tanh.

In the case of classifier accuracy curves, the variances in the clusters described above are smaller than in the union of clusters. Close to the final epochs, all curves tend to be chaotic within the range of $(99.4,99.7)$.

Visualizing the effectiveness of classifiers for different architectures of different complexities, although more obvious, also has research value (see Figure 4):

• CE none, CE mean, CE sum from ${\mathcal{N}}_0$ versus CE none, CE mean, CE sum from ${\mathcal{N}}_1$,
• CE none, SoftMax from ${\mathcal{N}}_0$ versus CE none, SoftMax from ${\mathcal{N}}_1$,
• SoftMax, sigmoid from ${\mathcal{N}}_0$ versus SoftMax, sigmoid from ${\mathcal{N}}_1$,
• sigmoid, tanh from ${\mathcal{N}}_0$ versus sigmoid, tanh from ${\mathcal{N}}_1$,
• sigmoid, hardsigmoid from ${\mathcal{N}}_0$ versus sigmoid, hardsigmoid from ${\mathcal{N}}_1$,
• tanh, hardtanh from ${\mathcal{N}}_0$ versus tanh, hardtanh from ${\mathcal{N}}_1$.

Figure 4 shows the better performance of ${\mathcal{N}}_1$ compared to ${\mathcal{N}}_0$. Moreover, we can observe slightly more stable behavior for ISBN-based curves than for cross-entropy-based.

4.2. Experiments with CIFAR-10 Dataset

In this subsection, the CIFAR-10—the more demanding than MNIST dataset is considered in the context of ISBE functionality. Moreover, the VGG feature extractor with more than 14 M parameters, i.e., more than 10 times larger model than ${\mathcal{N}}_1$, is joined to make further tests. In Figure 5, we can compare sample images from MNIST dataset and CIFAR-10 dataset. What is immediately observed is the background of objects classified—the uniform black for MNIST and the natural scene in case of CIFAR-10. It is the main reason that despite the almost perfect fit achieved by VGG-16 on the training set CIFAR-10 of 50 thousand images,the best results on the independent testing dataset of 10 thousand images are near $93\%$. The best results known w CIFAR-10 for all architectures attempted so far are near $95\%$—about one percent more than the record achieved by human beings.

The architecture VGG-16 was presented by Simonyan and Zisserman in their seminal paper [31], Very Deep Convolutional Networks for Large-Scale Image Recognition. VGG-16 model now serves the community as the universal image feature extractor. Its structure has the following sequential form:

Like for the two architectures ${\mathcal{N}}_0,{\mathcal{N}}_1$ tested for MNIST, the optimizer used for model updates is still AdaM with exponential decay of learning rate with respect to epochs. However, now the initial learning rate is $0.1$, not $0.01$.

In Figure 6, we can observe better convergence for all ISBE options than for the cross-entropy. Moreover, during testing, the loss value for CE is slowly increasing, starting at about epoch 30, while for all ISBE options, it is stabilizing on the fixed level.

From the results presented in Figure 7, it is visible that in the training and testing stages, there are different clusterings for ISBE options:

• In training, there are three groups of ISBE options: {hardtanh}, {tanh, hardsigmoid}, {sigmoid, SoftMax}.
• In testing there are two groups: {tanh, hardtanh} and {SoftMax, sigmoid, hardsigmoid}.

The significant gap between tanh, hardtanh and other ISBE options can be explained by different ranges for the first group and for the second one, i.e., $(-1,+1)$ versus $(0,1)$. It is not fully clear why in the training stage hardtanh is separate to tanh.

In Figure 8, the accuracy for cross-entropy and all ISBE options can be compared. It is observed that hard versions are inferior to others. However, while testing, a slight advantage is achieved by the hyperbolic tangent tanh.

Ultimately, we have bad news on time savings when using autograd interface. Contrary to MNIST experiments where ISBE functionality was implemented by the direct replacement of CE loss in the main learning loop, the CIFAR-10 experiments were using the definition of ISBE_func class being the extension to torch.autograd.Function class. It seems that the general mechanism of interfacing to C++ used by PyTorch in this case, is less efficient than for cross_entropy function. This is perhaps the reason that functionality with fewer operations takes slightly more time while the same functionality without explicit use of autograd mechanism gives always time savings up to $3\%$.

5. Conclusions

Cross-entropy CE as a loss function owes much to normalization performed by the SoftMax activation function. In the backward gradient backpropagation phase, only this activation, through perfect linearization, can prevent the explosion or suppression of the gradient originating from CE. What we call the SoftMax trick, as a mathematical phenomenon, is explained by the theory presented in the second section and its extension in Appendix A. There is proof that such linearization can only be realized by a function $F:{\mathbb{R}}^K\to {\mathbb{R}}^K$ with a Jacobian identical to that of the SoftMax function. In turn, such a Jacobian can only be derived for the dilated and relocated versions of the SoftMax function.

For further research, there remain practical aspects of a more general Theorem A1 implying that dilated and relocated versions of SoftMax are the only ones having the property of dilated SoftMax trick. However, it is quite intuitive that the dilation vector could be used to deal with class unbalanced datasets.

Should we, therefore, celebrate this unique relationship between activation and cost function? In this work, we have shown that it is rather beneficial to use the final effect of the action of this pair, namely the linear value equal to $Y-Y^{\circ }$, which can be calculated without their participation. This is exactly what the ISBE functionality does—it calculates the soft score vector in the forward step to return in the backward step its error from the target score.

To determine the normalized score, the ISBE functionality can use not only the SoftMax function, as it is not necessary to meet the unity condition, i.e., to ensure a probability distribution as scores of the trained classifier. At least four other activation functions sigmoid, tanh and their hard versions HardSigmoid and HardTanh perform no worse. The choice of these final activations was rather a matter of chance, so researchers face further questions. How do we normalize raw scores and appropriately represent (encode) class labels in relation to this normalization to not degrade the classifier’s results? What properties should such normalization functions have? Experiments suggest that meeting the Lipschitz condition in the vicinity of zero may be one of these properties.

The theoretical considerations presented prove that the ISBE functionality in the process of deep model learning correctly simulates the behavior of the CrossEntropy unit preceded by the SoftMax normalization.

The experiments showed that the ISBE functionality saves the time of forward and backward stages up to $3\%$, and the effectiveness of the classifier model remains unchanged within the margin of statistical error. Obviously, those gains are strongly dependent on datasets and network architectures.

In turn, a more complex case of integrating ISBE functionality with AD tools (AutoGrad) of a given platform can be solved for PyTorch by copying the proven code from Appendix B. However, as we described in the section on experiments with CIFAR-10, the time savings were consumed by this kind of interfacing to autograd system.