An n-Sigmoid Activation Function to Improve the Squeeze-and-Excitation for 2D and 3D Deep Networks

Abstract

The Squeeze-and-Excitation (SE) structure has been designed to enhance the neural network performance by allowing it to execute positive channel-wise feature recalibration and suppress less useful features. SE structures are generally adopted in a plethora of tasks directly in existing models and have shown actual performance enhancements. However, the various sigmoid functions used in artificial neural networks are intrinsically restricted by vanishing gradients. The purpose of this paper is to further improve the network by introducing a new SE block with a custom activation function resulting from the integration of a piecewise shifted sigmoid function. The proposed activation function aims to improve the learning and generalization capacity of 2D and 3D neural networks for classification and segmentation, by reducing the vanishing gradient problem. Comparisons were made between the networks with the original design, the addition of the SE block, and the proposed n-sigmoid SE block. To evaluate the performance of this new method, commonly used datasets, CIFAR-10 and Carvana for 2D data and Sandstone Dataset for 3D data, were considered. Experiments conducted using SE showed that the new n-sigmoid function results in performance improvements in the training accuracy score for UNet (up 0.25% to 99.67%), ResNet (up 0.9% to 95.1%), and DenseNet (up 1.1% to 98.87%) for the 2D cases, and the 3D UNet (up 0.2% to 99.67%) for the 3D cases. The n-sigmoid SE block not only reduces the vanishing gradient problem but also develops valuable features by combining channel-wise and spatial information.

1. Introduction

Activation functions (AFs) refer to the idea that neurons in the brain form input–output dynamic systems [1]. The choice of such computational transfer function mappings is important for artificial neural networks (ANN). Sigmoid functions are commonly used as activation functions, due to their derivatives being adaptable to gradient descent algorithms [2]. Additionally, they possess intrinsic output boundedness and are continuously differentiable C∞(R), monotonic with nonlinearity properties [3,4], which fulfills important Lyapunov stability conditions [5].

Remarkably, for the best function approximation and faster convergence, sigmoid functions ought to be symmetric about the origin of the function’s input–output map [6]. In the literature, the function approximation power of the sigmoid has also been explored exhaustively. For instance, Ohn I. and Kim Y. [7] investigated the smooth function approximation capacity of deep neural networks and grouped the activation functions into two groups: piecewise linear [8] (rectified linear unit (ReLU) variants) and locally quadratic (sigmoid variants).

Sigmoid functions remain the favorite choice in architectures such as recurrent neural networks (RNNs) [9,10], mostly because parallel computing techniques (such as the GPU power) have made these computations faster. Moreover, the use of the computing techniques of function approximation has drastically reduced the computational complexity to adequate levels of accuracy [11].

Nevertheless, two important problems affect the sigmoid functions in general. First, representations of sigmoid functions hold a fundamental learning or optimization downside whereby their derivatives exponentially move towards zero, particularly as the networks grows deeper. This conundrum is termed the vanishing (exponentially decreasing) gradient problem.

The second problem is the numerical indeterminacy of the log-sum-exp [10]. In naïve computation, one encounters underflows or even overflows depending on the scale of the input. The proposed n-sigmoid in this paper aims to resolve both issues.

To decrease both the computational and model complexity, several neural network architectures [12,13] centered on the network channels in many recent works have demonstrated that the efficacy of very deep networks is very low [14,15]. In contrast, the Squeeze-and-Excitation Network (SENet) [16], winner of the ILSVRC 2017 classification challenge, centers on channel-wise feature recalibration to refine the network’s accuracy.

The SE block has two major steps, squeeze followed by excitation. The squeeze part relies on global average pooling and brings about the channel descriptor. As for the excitation segment, it has a straightforward gating structure using a sigmoid activating function. A scalar magnified by the feature map in a channel-wise manner forms the output.

In summary, the key contributions of this proposed method are as follows:

(1)

This paper proposes a novel SE block with the n-sigmoid activating function. The proposed n-sigmoid function satisfies the requirements of an activating function, such as boundedness, smoothness, and sensitivity to input values.

(2)

The proposed SE block boosts the system representation capacity by clearly modeling the interconnections between the channels.

(3)

The proposed n-sigmoid activating function is in the range of [0, 1], which allows it to preserve meaningful features and subdue less beneficial features.

(4)

The proposed SE structure can be simply merged with existing models and boost their performance.

This paper is arranged as follows. Section 1 is the introduction. In Section 2, the related works are discussed to first introduce the squeeze-and-excitation networks as seen in the vision fraternity, secondly to explore the various activation functions and their enhancements, and finally to briefly investigate the channel and spatial correlations in the convolutional neural network (CNN). In Section 3, the proposed methodology is discussed, where the traditional sigmoid activation function and the vanishing gradient problem are explained before the new proposed n-sigmoid is introduced together with a solution to the numerical stability of the log-sum-exp and to the vanishing gradient. In Section 4, the experiments with their results are explained in detail, and, finally, in Section 5, we discuss the conclusions and future works.

2. Related Works

2.1. Squeeze-and-Excitation (SE)

The Squeeze-and-Excitation Network has been accepted in the vision fraternity. The SE structure recalibrates channel-wise features and reduces the less valuable ones. At each layer of the network, a set of filters displays neighborhood spatial connectivity designed through input channels—combining channel-wise and spatial information around neighboring receptive fields.

The convolution part (the main building block of convolutional neural networks) allows networks to build valuable features by combining channel-wise and spatial information in the internal receptive fields. A large category of previous research has explored the spatial relationship by trying to reinforce the representational strength of a convolutional neural network (CNN) by improving the nature of spatial encodings throughout its feature ladder. Conversely, the “Squeeze-and-Excitation” (SE) block (Figure 1) is based on the channel relationship and introduces a new building unit to recalibrate adaptively channel-wise features. SE blocks significantly improve current state-of-the-art convolutional neural networks’ performance, adding a small additional computational cost.

The Squeeze-and-Excitation formula is as follows:

$$\mathrm{S}\mathrm{E}=\sigma \left(W_2\gamma \right(W_{1}z+b_1)+b_2)$$

(1)

where $\sigma$ is the sigmoid, $\gamma$ is the ReLU function, and {$W_1$ $\in {\mathfrak{R}}^{\frac{C}{1}xC}$, $b_1$} and {$W_2$ $\in {\mathfrak{R}}^{\frac{C}{1}xC}$, $b_2$} are the weights connecting two fully connected layers.

Inside the SE, the computational cost and capacity are controlled by a reduction ratio r. The input X and the scalar SE are multiplied channel-by-channel to obtain the final output:

$$X_{\mathrm{S}\mathrm{E}}=X\times \mathrm{S}\mathrm{E}$$

(2)

where $“\times ”$ symbolizes channel-wise multiplication. The SE structure will retain useful features and reduce unnecessary features to boost the network’s performance.

Figure 1 shows that the sigmoid function plays an important role inside the squeeze segment of an SE structure, where it acts as a gating mechanism with a sigmoid activation function to increase the representational ability of the network’s dynamic channel-wise feature recalibration.

The different uses of the sigmoid activation function in artificial neural networks are intrinsically limited by gradients that are vanishing. The word “vanishing” means that, in a feedforward network (FFN), the backpropagated error signal naturally declines (or rises) exponentially in relation to the distance from the last layer [17]. The vanishing gradient arises when the artificial neural network is trained with backpropagation and gradient-based learning methods. In the worst case, this would stop the neural network from training more and better.

2.2. Activation Functions

Activation functions have been interminably the cause of nonlinearity in neural networks [18]. Conventional activation functions (such as tanh and sigmoid) are differentiable and continuous, but the sigmoid has only a positive value, while the tanh has a negative one. Most enhancements of the sigmoid function normally focus on changing the slope of the sigmoid or shifting the original sigmoid, as opposed to the new proposed sigmoid that is a piecewise log-shifted function in a finite input–output space.

The sigmoid activation function is often used in feed-forward neural networks (FFNN) [19] to introduce nonlinearity. To accelerate network convergence, CNNs mostly use the hyperbolic tangent as an activation function [20]. One of the latest advances in activation functions is the non-negative rectified linear unit (ReLU) [21], where the identity map in the positive portion solves the gradient vanishing challenge.

To conquer the challenge of the inability of ReLU to multiply negative values because of the constant zero in its negative parts, the parametric rectified linear unit (PReLU) [22] includes a learnable variable in the negative portion. New exponential linear unit (ELU) enhancements [23] similarly insert additional learnable variables to expand the significance of the ELU. These variables/parameters are either channel-wise or channel-shared. Lately, the addition of the learnable parameters has been the new direction in activation function research. The p-sigmoid [24] showcases parameters related exclusively to function boundaries (which allows positive, zero, or negative contributions from each hidden unit), horizontal displacement, and the steepness of the curve.

2.3. Channel or Spatial Correlations

With the revival of the convolutional neural network (CNN), the vision fraternity prioritized two hyperparameters, which are the depth [8,12] of the network and the filter sizes [25]. The depth has continued to increase, to the extent that it is now possible to train very deep networks.

Aside from the depth of the network, Wide ResNet [26] has shown that the channel width is also a very important consideration when it comes to performance. Old convolutions have integrated channel and spatial correlations, while more recent work has attempted to dissociate them to improve the efficiency. The Inception module [27] first discovered cross-channel interaction and later led to a set of spatial correlations.

ResNext [28] investigated the trade-offs between width (the number of channels in a layer), depth, and group. Xception [29] introduced the depth-wise separable convolution, where the number of group convolutions equals the number of channels. ShuffleNet [30] introduced a channel shuffle task to improve the depth-wise separable convolution. Shi et al. [31] introduced a spatial and channel attention unit. The input features were improved using spatial attention and channel attention separately, where the features were acquired with hybrid attention.

2.4. Adaptation and Improvements of the SE Block

Since its inception, due to its large success, various attempts have been made to improve the SE block. Several avenues have been exploited, such as the reduction of its architecture [32], optimization of the reduction ratio [33], alteration of the global pooling operation [34], combination of the squeeze and excitation stages [35], an improved gating mechanism by introducing a more powerful activation function [24], etc., as can be seen in

Table 1. Summary of various SE methods.

SE Methods	Dataset Used	Result %	Limitations
PSigmoid [24]	CIFAR10	93.03	Over-reliance on the learned attention scores, computational complexity, and overfitting if not properly generalized
SE SqueezeNext [32]	CIFAR10	92.6	Tradeoff between model’s size, accuracy, and speed, and no change in the accuracy even when the model depth is reduced.
LRSE-Net [33]	Deep Stenosis Detection	96.05	Need for further improvements, such as evaluating for an object-based recognition system.
CaSE EfficientNetBO [34]	VTAB + MD	98.02	Requires iterative hardware-dependent gradient updates and falls short on structured datasets.
FuSENet [35]	IP, UP, and SA HSI	98.64%	Needs addition to a residual learning network to improve learning efficiency.

. The proposed n-sigmoid aims to challenge the latter by presenting a very competitive activation function.

p-sigmoid [24] (PSE) aims to enhance the SE and introduces the notion of combined channel-wise and channel-shared parameters. In [32], the SE SqueezeNext presents a more sophisticated block with a condensed architecture for higher accuracy, but it is found that there is a trade-off between the model’s accuracy, size, and speed for various resolution and width multipliers, even though there is no improvement in the model accuracy when reducing the depth of the model. Ovalle-Magallanes et al. [33] presented a Lightweight Residual Squeeze-and-Excitation Network (LRSE-Net) for stenosis classification in X-ray coronary angiography images, where the reduction ratios of individual Squeeze-and-Excitation modules are optimized to enhance the feature recalibration. In [34], the authors present a new adaptive block called Contextual Squeeze-and-Excitation (CaSE) that regulates a pretrained neural network to drastically enhance the performance with a user data single forward pass as context. The execution of SENet relies on the global pooling squeeze operation, which could lead to reduced performance. In [35], the authors propose a bilinear fusion mechanism over various types of squeeze operation (max pooling and global pooling). The excitation operation is executed with the fused output of the squeeze operation.

3. Proposed Method

3.1. Squeeze-and-Excitation (SE) Block with n-Sigmoid

The proposed methodology seeks to improve the SE block by using the n-sigmoid (explained in Section 3.2) as a gating mechanism instead of the classic sigmoid. Figure 2 shows the schematic of the proposed solution, which will allow the SE block to keep meaningful features and suppress less valuable ones, as well as increase its capacity to learn and generalize 2D and 3D networks for overall enhanced performance.

The idea of replacing the sigmoid activation function with the n-sigmoid presents several advantages:

(1)

The network performance is improved. The training and validation loss and accuracy are slightly upgraded.

(2)

The famous gradient vanishing/exploding challenge is resolved.

(3)

The spatial and channel relationships in the network are enhanced since the n-sigmoid allows the SE block to further model channel interdependencies.

3.2. The Proposed n-Sigmoid Activation Function

The proposed generalized form of the n-sigmoid is

$$\begin{array}{cc}\hfill f\left(\mathrm{x}\right)& ={\int }_0^{\infty }{\sum }_{k=1}^n\frac{\alpha }{(e^{\left(\alpha x\pm \beta \right)}+1)}\hfill \\ & =\pm \alpha \ast {\int }_0^{\infty }{\sum }_{k=1}^n\frac{1}{\left(e^{\left(\alpha x\pm \beta \right)}+1\right)}dx\hfill \\ & =\pm a\ast \frac{{\sum }_1^n{\log }_n(e^{\left(\alpha \alpha x\pm \beta \right)}+1)}{b}\hfill \end{array}$$

(3)

where $a \mathrm{i}\mathrm{s} \mathrm{a} \mathrm{c}\mathrm{o}\mathrm{n}\mathrm{s}\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{t}$ that depends on $\alpha$ and results from the computation of the integration of the derivative, and b is a constant depending on $\beta$ and resulting from the computation of the integration of the derivative.

$\beta$ is the offset parameter that centers the input domain x, while $\alpha (\in \mathfrak{R})$ is the logistic growth rate parameter. The finite input–output space that is contained within $x_{\mathrm{m}\mathrm{i}\mathrm{n}}-x_{\mathrm{m}\mathrm{a}\mathrm{x}}$, and $y_{\mathrm{m}\mathrm{i}\mathrm{n}}-y_{\mathrm{m}\mathrm{a}\mathrm{x}}$ defines the boundaries of the function.

An n-sigmoid (Figure 3) is presented in this finite input–output surface, which yields the resulting integration of an n-equal sigmoid (Equation (3)). However, the naïve computation of the logarithm sum may lead to numerical challenges that will need a solution, as explained in Section 3.2.1.

The novel n-sigmoid function has a derivative a series of n shifted sigmoids in a defined and finite input–output space. This results in restricting the vanishing gradient problem, as explained in Section 3.2.2.

3.2.1. Numerical Stability of the Log-Sum-Exp

A naïve implementation of the log-sum-exp can result in numerical indeterminacy even for moderate values. In deep learning and statistical modeling, it is possible to work in a logarithmic scale. There are many possible situations for this numerical explosion. For instance, if x and y are small numbers, their multiplication may result in underflow.

Moreover, if x and y are small numbers, their multiplication may result in identity

$$\log \sum _{n=1}^N\mathrm{e}\mathrm{x}\mathrm{p}\left\{x_n\right\}=a+\log \sum _{n=1}^N\mathrm{e}\mathrm{x}\mathrm{p}\{x_n-a\}$$

(4)

for any $a$. It is now possible to shift the center of the exponentiated variables up and down as needed. A typical action to take is to perform

$$a=\underset{n}{\max }{x}_n$$

(5)

which ensures that the largest value that we exponentiate will be zero. This means that the result will not overflow, and even if the rest underflow, one will still obtain a reasonable value.

3.2.2. Solution to the Vanishing/Exploding Gradient

Presenting the novel activation function in a finite input–output could reduce drastically the vanishing gradient limitation and enhance the accuracy of a fully convolutional neural network [36].

As an activation function, the maximum gradient is set to or falls within a value that ensures that the gradient neither explodes nor vanishes. This is achieved since, in this case, the maximum value of the derivative is 0.5. Hence, the maximum gradient is solely dependent on the input/output limits of the derivative.

Therefore, with the n-sigmoid activation function definition, gradients do not vanish, except at the input value x, which charts the maximum or minimum boundary of the output f(x) space, as proven in Figure 4. Consequently, due to the way in which the gradient is restricted and presented piecewise over the input–output map, the common occurrence of vanishing gradients will be far diminished compared to the traditional sigmoid use case.

Figure 4 shows the derivative of the proposed n-sigmoid, which is a piecewise shifted sigmoid, and its integration results in the new proposed activation function (Figure 3).

The derivative g of the n-sigmoid is a piecewise n shifted sigmoid function defined as

$$g=\pm \sum _{k=1}^n\frac{\alpha }{(e^{(\pm \alpha x\pm \beta )})}$$

(6)

where $n$ is the number of the subdivisions of the sigmoid function.

3.3. The Advantages of Using the n-Sigmoid over Other Logistic Activation Functions

The n-sigmoid present many advantages compared to existing sigmoid activation functions such as the n-logistic sigmoid [36]:

• The peak of the derivative is present at the most sensitive point (middle at zero).
• This novel activation function presents a lower risk of gradient vanishing problems since the derivative glides only in the defined space, which results from the definition of the derivative piecewise sigmoid function.
• It is also opposite to the n-logistics sigmoid [36], which has multiple peaks that cannot be physically motivated or explained, since the value of the activation function is not constantly at the mid-length of the nth sigmoid, as can be seen in the graph presented in the paper. The question is what will cause the presence of so many peaks to be relevant, especially because there are no peaks at zero. There is a lack of physical motivation.

4. Experiments and Results

In this section, the datasets used are described and the n-sigmoid is compared to other traditional activation functions, and finally a performance comparison of the n-sigmoid is established when used in 2D and 3D CNN models and when adding the Squeeze-and-Excitation Network.

The vanishing gradient problem is resolved by the restriction of the input–output bounded n-sigmoid function in a finite map.

4.1. Datasets and Evaluation Metrics

The datasets used for the 2D implementation are the MNIST [37], CIFAR10 [38], and Carvana [39] datasets. As for the 3D implementation, the 3D Sandstone dataset have been used.

The MNIST dataset (Figure 5) is a vast handwritten digit dataset that is frequently utilized to train different image processing structures. The database is extensively utilized for training and testing in several machine learning tasks.

The CIFAR10 dataset is composed of 60,000 (32 × 32) color images divided into 10 classes. It has 50,000 training images and 10,000 test images. The dataset has one test batch and five training batches, each with 10,000 images in random order.

The Carvana dataset has been designed for a successful online used car start-up to develop long-term trust with clients and streamline the online buying operation. The dataset consists of a custom revolving car photo studio of 16 standard pictures of each vehicle in their inventory.

Sandstone 2015 is a 3D model series having both 3D and 2D geoscientific information that supplements the GSWA Record, while the lung CT scan dataset comprises 20 CT scans of COVID-19 patients gathered from Radiopaedia and coronavirus cases.

To evaluate the proposed approach, the metrics considered were the accuracy, loss, and dice similarity coefficient (DSC) or dice score, also known as the F1 score. In the case of image segmentation, DSC is one of the most used metrics and is proposed to evaluate the models. In general, the DSC is meant to indicate the results’ similarity regarding the segmentation area and the predicted result and the ground truth. In addition to using DSC, this study also used a classification metric, accuracy, which measures the ratio of correctly identified predicted pixels to all predicted pixels.

4.2. n-Sigmoid Compared to Traditional Activation Functions

A common convolutional model (details of the model) was implemented and used with the CIFAR dataset to compare the activation functions and show that the n-sigmoid activation function performs better.

A simple convolutional neural network (CNN) consists of a pile of Conv2D and MaxPooling2D layers. As input, a CNN has tensors of shape image_height, image_width, and color_channels.

To analyze and access the various training results, different activation functions, such as the sigmoid function, the hyperbolic tangent function, the ReLU function, the leaky-ReLU function, and the n-sigmoid, were selected. In this case, the n-sigmoid function was configured as a hidden layer activated with the values of n = 4. The main metric used to compare their performance on the convolutional neural network was the loss (validation and training) for the different activation functions (Figure 6 and Figure 7) using the same CIFAR dataset on a simple deep learning framework. It can be observed from Figure 6 that the n-sigmoid (purple line) ensured a low validation loss throughout the experiment and only ReLU (green line) performed better. The same can be observed in Figure 7, where the training loss is lower than for the other activation functions for the simple convolutional network throughout the training.

Another evaluation aimed to compare the effects of changing the sigmoid activation function in the SE block with other activation functions such as the rectified linear unit (ReLU), exponential linear unit (ELU), sigmoid, and p-sigmoid (

Table 2. Effects of other activation functions on the SE structure using CIFAR10.

Model	Activation Function	CIFAR10 Results (%)
Resnet	Original	92.41
	ReLU	*
	ELU	85.46
	Sigmoid	92.97
	p-sigmoid	93.03
	n-sigmoid	95.1

* indicates failure to converge.

) when using the Resnet network on the CIFAR10 dataset.

It is observable that the n-sigmoid greatly improves the performance of the SE structure, because it enhances the network attributes by specifically modeling the interdependencies between channels.

4.3. Implementation Details

All training and validations were performed using either the TensorFlow backend with the Keras library or the Pytorch library. The idea was to experiment and assess the performance of the existing leading architectures in both 2D and 3D when using the Squeeze-and-Excitation networks with the n-sigmoid as the activation function.

In the 2D area, the architectures selected were ResNet (classification), DenseNet (classification and segmentation), and UNet (segmentation). The purpose was to compare the accuracy training results to confirm that the Squeeze-and-Excitation network with the new proposed sigmoid activation function (n-sigmoid SE) would outclass the high-performing Squeeze-and-Excitation network (SE).

The training replicated the parameters of the existing leading networks but added a few necessary changes related to the specifications and hardware used. The comparison process consisted of first training the model as originally designed, and then later with the addition of an SE block or layer, and, finally, the n-sigmoid_SE block was added (Table 2), and all results were compared mostly in terms of the training accuracy score.

For the training process, the 2D UNet [40] employed the Adam optimization method with an initial learning rate of 1e-4, while ResNet [12] and DenseNet [41] used Stochastic Gradient Descent (SGD) with a momentum of 0.9 and weight_decay of 5e-4. The 3D UNet was trained using Keras, where the kernel_initializer was set to ‘he_uniform’ and the model was compiled with the Adam optimizer with a learning rate of 1e-3. In these experiments, all models used simple data augmentation and conventional parameter settings.

As for the loss function, the 2D UNet was trained with binary cross-entropy (BCE) with logits loss, while ResNet and DenseNet used the cross-entropy loss function. Binary cross-entropy was used as a loss function for the 3D UNet.

4.4. Squeeze-and-Excitation Block Using n-Sigmoid Analysis

The Squeeze-and-Excitation block using n-sigmoid is the essential improvement in our proposed methodology. To demonstrate its effect on different networks in 2D and 3D, several ablation experiments were carried out.

4.4.1. Results Analysis of the 2D Squeeze-and-Excitation Block with n-Sigmoid

Selected Models’ Definition

UNet is a convolutional neural network developed initially for biomedical image segmentation but has gained a great deal of popularity. The network architecture was improved and expanded to work with even small training images and to give more accurate segmentations.

ResNet (Residual Network) [12] is a neural network that allows the training of very deep neural networks with over 150 layers with success. Before ResNet (Figure 8), it was very difficult to train very deep neural networks because of vanishing gradients. ResNet is the first to introduce the idea of a skip connection, where the first input is added to the output of the convolution block.

A DenseNet [41] is a convolutional neural network with dense connections between layers (dense blocks), where all layers (with matching feature-map sizes) are connected and used for segmentation (Figure 9).

Comparative Analysis

Various parameters (learning rate, epochs, the number of repetitions in the input–output space of the sigmoid, etc.) were used, but the comparison is confined within the network, keeping the design characteristics unchanged.

This comparison work was performed on an NVIDIA Tesla K80 GPU.

Table 3. Performance comparison of the state-of-the-art models for training accuracy on classification (a) and segmentation (b).

Model	Category	Accuracy Results (%)
Resnet20 (Cifar10)	Original	92.7
	SE	94.2
	Parametric SE	93.03
	n-sigmoid SE (n = 4)	95.5
ResNeXt (Cifar10)	Original	96.4
	SE	96.47
	Parametric SE	96.54
	n-sigmoid SE (n = 4)	97.41
(a)
Model	Category	Accuracy Results (%)
Unet (Carvana)	Original	99.3
	SE	99.48
	Parametric SE	--
	n-sigmoid SE (n = 4)	99.67
DenseNet (Cifar10)	Original	99.69
	SE	99.77
	Parametric SE	--
	n-sigmoid SE (n = 4)	98.87
(b)

shows that there was a performance improvement when the models added a Squeeze-and-Excitation (SE) block using the n-sigmoid as the activation function.

The 2D UNet training accuracy improved by 0.25% from 99.67%, while ResNet20 with the n-sigmoid SE increased by 0.9% from 94.2% to 95.1%, and DenseNet with the n-sigmoid SE was enhanced by 1.1% from 96.7% to 98.87%. The training and validation curves of DenseNet and ResNet for both the SE models and the n-sigmoid SE models are shown in Figure 10 and Figure 11, and the proposed model obtained the highest accuracy curves and the lowest loss.

Although the difference in the results was not of great importance, the experiments showed that a variation in the value of $\alpha$ slightly affected the results obtained. For instance, at $\alpha$ equal to 0.06, the training accuracy result was 95.360% for ResNet with n-sigmoid, which is a decrease from 95.500% when $\alpha$ = 0.03. This shows that the lower the value chosen for $\alpha$, the lower the accuracy becomes.

The compared testing results showcase that the n-sigmoid SE improves the testing accuracy results when compared to the parametric SE from 93.03% to 95.1%.

n = number of repetitions (sigmoid) in the input–output map of the n-sigmoid activation function.

4.4.2. Results Analysis of the 3D Squeeze-and-Excitation Block with n-Sigmoid

Selected Model Definition

Firstly, 3D-UNet [42] has a contractive and expanding path to build a bottleneck in its pivot by using a combination of convolution and pooling operations. After this bottleneck, the image is reconstructed through an amalgamation of convolutions and upsampling. Skip connections are brought in to help the backward flow of gradients and improve the training (Figure 12).

Comparative Analysis

This work also was performed on a Tesla K80 GPU. As discussed in the previous sub-section, even in 3D data, the 3D n-sigmoid affects positively the performance of the neural network model. Here, 3D state-of-the-art deep learning models were tested, and the results are shown in Table 4. The results shown in Table 4 illustrate that the proposed n-sigmoid SE UNet achieved accuracy of 99.67% and DSC of 88.91. Essentially, when using the n-sigmoid SE, the DenseNet achieved 99.71% (from 99.12% on original 3D DenseUNet) accuracy and 95.26 DSC (from 92.54 on original 3D DenseUNet), while ResUNet achieved 99.01% (from 97.93% on original 3D ResUNet) accuracy and 89.52 DSC (from 89.09 on original 3D ResUNet).

Table 4 also shows the Dice coefficients of all models used. It is worth noting that there are limited improvements in the DSC results over the 3D UNet baseline using Carvana dataset [43], which improves only by 0.8 from 88.08 to 88.93. The same is true for the 3D ResUNet DSC, which only improves by 0.43 from the 3D ResUNet baseline. It is only for the 3D Dense UNet that the improvement is sizeable, possibly due to its structure of dense connections between layers.

Table 4. Segmentation accuracy and Dice similarity coefficient (DSC) results comparison.

Model	Dataset	Dice Similarity Coefficient	Accuracy (%)
3D UNet	Sandstone	88.08 [44]	99.43 [45]
3D SE UNet		88.62	99.48
3D n-sigmoid SE UNet		88.91	99.67
3D DenseUNet	LiTS	92.54 [44]	99.12 [46]
3D SE DenseUNet		94.56 [47]	99.35
3D n-sigmoid SE DenseUNet		95.26	99.71
3D ResUNet	Lung CT scan dataset	89.09 [44]	97.93 [46]
3D SE ResUNet		89.31 [44]	98.29
3D n-sigmoid SE ResUNet		89.52	99.01

Table 4. Segmentation accuracy and Dice similarity coefficient (DSC) results comparison.

Model	Dataset	Dice Similarity Coefficient	Accuracy (%)
3D UNet	Sandstone	88.08 [44]	99.43 [45]
3D SE UNet		88.62	99.48
3D n-sigmoid SE UNet		88.91	99.67
3D DenseUNet	LiTS	92.54 [44]	99.12 [46]
3D SE DenseUNet		94.56 [47]	99.35
3D n-sigmoid SE DenseUNet		95.26	99.71
3D ResUNet	Lung CT scan dataset	89.09 [44]	97.93 [46]
3D SE ResUNet		89.31 [44]	98.29
3D n-sigmoid SE ResUNet		89.52	99.01

One can see that the n-sigmoid SE model performs well in all models studied, with an acceptable margin for both accuracy and DSC. Note that all the results shown here are from three different datasets, namely the Sandstone, LiTS, and lung CT scan datasets. Fairness will dictate that all the models be trained with the same dataset, but the large difference in performance shows that n-sigmoid SE will still perform better, even if using similar datasets.

The training and validation curves of the 3D UNet together with the 3D n-sigmoid SE UNet’s accuracy and loss are shown in Figure 13. In essence, our proposed n-sigmoid SE yields the best performance throughout the training, while it presents the lowest curve for the loss too.

4.5. Discussion

The performance results validate the capability of the proposed method in the different datasets used, such as the CIFAR10, Carvana, LiTS, and Lung CT scan datasets. Several performance metrics were used to evaluate the performance, such as the accuracy and loss, as well as the Dice similarity coefficient (DSC), where the n-sigmoid has consistently shown better results compared to the other models throughout the experiments. By visualizing the training and validation curves of the 2D and 3D experiments (Figure 10, Figure 11, and Figure 13), it is worth noting that the network performance is directly enhanced when using the n-sigmoid SE. The data augmentation approach and hyperparameter settings were set as described in Section 4.3.

4.5.1. Impact of Using Other Activation Functions

The authors have evaluated the effect of replacing the sigmoid activation function in the SE block with other activation functions. They considered four other activation functions—the original sigmoid, ReLU, ELU, and p-sigmoid—and compared them to the n-sigmoid. Replacing the sigmoid in the SE block with other well-known activation functions will greatly reduce the performance. This results from the triumph of the SE block, which has significantly improved the quality of network representation by clearly modeling the interdependencies between channels.

4.5.2. The Vanishing Gradient Solution

The most important challenge of the sigmoid activation function is the vanishing/exploding gradients. It essentially creates a situation wherein a deep neural network does not have the ability to propagate useful gradient information from the output end of the model back to the layers near the input end of the model. This paper has demonstrated that by constraining the n-sigmoid in a finite input–output space, the problem of vanishing gradients could be drastically limited. The proposed SE with n-sigmoid has drastically lowered the risk of gradient vanishing, as explained in Section 3.2.2.

4.5.3. Future Research Directions

In the future, although the n-sigmoid activation function has shown good results in the SE, there is still a need for further improvements, such as overcoming robustly numerical instability for extreme values of the input, evaluating the proposed activation function in a Transformer as the backbone for a self-attention mechanism, and detecting scenes or partial scenes in 3D indoor segmentation. Additionally, a future direction of this work regarding model reduction could be to investigate more methodologies, such as their decomposition, quantization, etc. Finally, another research direction could be to consider using n-sigmoid in more complex tasks such as 2D–3D object detection, feature extraction for video identification, etc.

5. Conclusions

In this paper, the authors introduced and defined an n-sigmoid Squeeze-and-Excitation block for use as an activation function. It introduces the log of shifted sigmoid functions to be n-times summed up over a finite input–output map. This function largely overcomes the famous vanishing gradient problem of sigmoid functions since it is constrained in a defined space.

The n-sigmoid used in the SE network has shown that it is possible to resolve the vanishing or exploding gradient dilemma to allow deep neural networks to train adequately. The vanishing gradient problem was essentially constrained with the use of the n-sigmoid function in a finite input–output bounded space.

Furthermore, the authors integrated the novel n-sigmoid into the SE structure to improve the SE’s performance in its feature recalibration task. Experiments conducted on four different leading networks using a set of benchmark datasets (CIFAR10, Carvana, 3D Sandstone) show that this new module is an advancement from the SE block as well as the parametric SE.