Scale-Aware Network with Scale Equivariance

Abstract

The convolutional neural network (CNN) has achieved good performance in object classification due to its inherent translation equivariance, but its scale equivariance is poor. A Scale-Aware Network (SA Net) with scale equivariance is proposed to estimate the scale during classification. The SA Net only learns samples of one scale in the training stage; in the testing stage, the unknown-scale testing samples are up-sampled and down-sampled, and a group of image copies with different scales are generated to form the image pyramid. The up-sampling adopts interpolation, and the down-sampling adopts interpolation combined with wavelet transform to avoid spectrum aliasing. The generated test samples with different scales are sent to the Siamese network with weight sharing for inferencing. According to the position of the maximum value of the classification-score matrix, the testing samples can be classified and the scale can be estimated simultaneously. The results on the MNIST and FMNIST datasets show that the SA Net has better performance than the existing methods. When the scale is larger than 4, the SA Net has higher classification accuracy than other methods. In the scale-estimation experiment, the SA Net can achieve low relative RMSE on any scale. The SA Net has potential for effective use in remote sensing, optical image recognition and medical diagnosis in cytohistology.

1. Introduction

The convolutional neural network (CNN) has achieved good performance in object detection and classification due to its inherent translation equivariance [1,2]. $\Phi :U\to V$ represents the mapping from the image field to the feature field. For an input image $x\in U$, the sliding window convolution and weight sharing in the CNN ensure that the feature can always be detected no matter where it is. While reducing the number of network parameters, the position of the maximum response on the feature map will also shift accordingly with the translation of the input feature, as shown in (1).

$$\Phi (Tx)=T^{\prime }\Phi (x)$$

(1)

where $T$ and $T^{\prime }$ represent the translation transformation of the image filed and feature field, respectively, and Equation (1) shows that the CNN has translation equivariance [3,4,5,6].

In addition, the conventional CNN is resilient to slight deformation (translation, rotation, and isotropic scaling) due to the pooling operation. As long as the translation of pixels does not exceed the pooling kernel size, the output of max-pooling remains unchanged. However, the pooling kernel size is limited, so the translation invariance is not evident in the shallow layer of the network [7,8]. With the deepening of network layers, the translation invariance of the deep CNN increases with the increase in the receptive field. No matter where the input feature is, the CNN can detect the feature, predict the same label, and the data distribution is approximately unchanged. At this time, the network has global translation invariance. When $T^{\prime }$ is an identity transformation, invariance can be regarded as a special case of equivariance [3]. At this time, Equation (1) can be rewritten as:

$$\Phi (Tx)=\Phi (x)$$

(2)

Translation equivariance and invariance are the inherent properties of the CNN and are dependent on its architecture. However, in many practical applications, the scale equivariance of CNN is more critical. For example, in cytohistology, the size of a tissue or cell reflects whether the cell is normal or cancerous [9], as shown in Figure 1. Different cell sizes must be distinguishable among the outputs, and the network that meets this requirement is the scale-equivariant CNN. In addition, in the real world, the change in the distance between the object and the observer will directly result in the scale variation of the object’s projection on the retina, thereby resulting in the characteristics of ‘near large and far small’ in human vision, which shows the depth and volume of the object in relation to the size of the object.

Scale variation is as common as translation, whereby the existing CNN adopts both a sliding-window convolution and pooling in order to adapt to the scale change, but its scale equivariance is poor [10]. For example, after the input image is scaled, the learned convolution kernel will mismatch the scaled images, as shown in Figure 2, resulting in weak equivariance for scaling.

In fact, for single-layer convolution pooling, since the convolution operation is carried out by moving the sliding window on the image line by line, the pattern in the image can be detected before and after the translation. However, the image scaling and the translation of the sliding-window convolution belong to different spatial transformations, which will hinder the equivariance. In addition, for the max-pooling layer, the pixels on the image before and after scaling are generally not in the same pooling window, so the response of the image before and after scaling is often different, which also hinders the invariance of the network. In single-layer convolution pooling, the scaling operation will hinder the equivariance and invariance. With the deepening of the network, the receptive field gradually increases. Unlike the gradual strengthening of translation invariance and equivariance, the degradations of the CNN scale invariance and equivariance will be more obvious [11]. Therefore, the CNN is quite vulnerable to scaling operations, and scaling disturbance may cause misclassification. Due to the lack of equivariance, the CNN cannot estimate scale.

To solve these problems, we proposed a Scale-Aware Network (SA Net) with scale equivariance in order to estimate scale during classification. The specific methods are as follows: only one scale pattern is learned in the training stage. In the testing stage, the testing sample with an unseen scale is zoomed in and out on a group of images with different scales that form an image pyramid. The image-zooming-in channels are up-sampled by bilinear interpolation. The image-zooming-out channels are down-sampled. Unlike using Gaussian derivatives to limit bandwidth [12], the combination of dyadic discrete-wavelet transform (DWT) and bilinear interpolation is used in order to avoid spectrum aliasing.

Dyadic zooming out uses the discrete-wavelet transform, and non-dyadic zooming out uses the result of the next-level discrete-wavelet transform that is zoomed in by bilinear interpolation. Then, the image pyramid with different scales is sent to the multi-channel Siamese CNN with weight sharing for inferencing. A two-dimensional classification-score matrix is obtained. The classification and scale estimation can be simultaneously carried out by the position of the maximum value of the classification-score matrix.

Our main contributions can be summarized as follows.

1. In the testing stage, we scale the input image to form an image pyramid, in which the down-sampling of the image adopts dyadic DWT to avoid spectrum aliasing.

2. The weights between channels are shared, and multi-channel inferencing is simultaneously carried out. Learning only one scale can obtain the generalization performance of multi-scale images in the multi-scale target recognition, thus reducing the model’s parameters.

3. Different from the existing methods, the SA Net is sensitive to different scales of the same class during the testing stage. The SA Net establishes a scale-related inferencing branch without an additional scale-learning branch in order to enable the network to estimate unknown scales. Therefore, we can estimate the scale of the image while predicting the class.

The remainder of this paper is organized as follows. In Section 2, we introduce the related work. In Section 3, we will present our proposed SA Net, including the details of the training and testing stages and how the image-feature pyramid is informed. In Section 4, we present the experimental results and implementation details in order to validate the effectiveness of our proposed method. Finally, in Section 5, we conclude our work.

2. Related Work

To improve the scale equivariance of the CNN, there are mainly two families: multi-scale training and single-scale training. The SA Net belongs to the second family.

2.1. Multi-Scale Training

Multi-scale training can be divided into transform feature representations [11,13,14,15,16] and transform convolution kernels [10,17,18,19].

For transforming feature representations, it is equivalent to increasing the number of training samples in a disguised form, and the most representative way is the conventional-data-augmentation (CDA) strategy, which can be executed online or offline. The purpose of online CDA and offline CDA is the same, i.e., to improve the model’s generalization performance by learning as many samples as possible. Given abundant random-scaling training samples, a CNN might score the same sample at different scales. Besides, Kanazawa et al. [13] proposed a Locally Scale-Invariant CNN (LSI-ConvNet), which scales the input image into multiple scales in a specified way, then convolutes the images of different scales with the same convolution kernel, and finally normalizes the feature image through undo-scaling, adopting max-pooling over scales at each spatial location to obtain the locally scale invariant representations. However, some useful information such as scale variations was discarded, which will reduce the model’s performance in some specific tasks. For example, in cytohistology, the sizes of cells reflect whether their function is normal, and the above methods are not scale sensitive and cannot be used in these fields.

For transforming convolution kernels, Xu et al. [20] proposed a representative Scale-Invariant ConvNet (SI-ConvNet) to transform convolution kernels, which can be regarded as a set of filters. Although different filters have different sizes, they share the same parameters. Each filter corresponds to a channel and simultaneously detects patterns at different scales, and the last layer outputs of different channels are stacked into a feature vector. Then, the feature vector is input into the softmax layer for classification. The results show that the SI-ConvNet is not sensitive to scale changes and susceptible to object-scale variations. Ghosh et al. [21] proposed the Scale-Steerable ConvNet (SS-ConvNet), in which log-radial harmonics are introduced as the steerable basis, and the filters of each convolution layer are a linear combination of basis filters. These methods provide a new paradigm for enhancing a model’s scale invariance, but they also focus on scale invariance. In addition, they are too dependent on the training process, which will inevitably lead to the increase in the complexity of the network structure.

2.2. Single-Scale Training

Single-scale training uses single-scale samples instead of multi-scale samples in the training stage and thus reduces the number of parameters and the computational cost. Lindeberg et al. proposed FovMax and FovAvg networks [22], which adopted weight sharing between different channels together with global max or average pooling over the outputs from all channels in order to achieve scale invariance for digit classification on MNIST Large-Scale dataset. In addition, Lindeberg presents a theoretical analysis of the invariance and covariance properties of his proposed networks and, for the first time, explores their ability to generalize to scales that are not present in the training set. His proposed network reduces the number of parameters and computational costs, but using a Gaussian derivative to avoid spectrum aliasing is contrary to the simplicity of the network structure.

Although Lindeberg’s method lacks the estimation function for unseen scales as the result of the global pooling, as mentioned above, it provides some inspiration for scale equivariance. Based on single-scale training, the SA Net uses wavelet transform and combines all channels’ outputs to achieve scale equivariance, thus estimating previously unseen scales. The results on the MNIST and FMNIST show that it has good classification and scale-estimation performance.

3. Methods

In order to highlight the differences between the training stage and the testing stage, the SA Net is divided into two stages: training and testing stages, as shown in Figure 3.

3.1. Training Stage

The SA Net only needs to learn samples with one scale in the training stage and save the model, which reduces the scale diversity of the training dataset. According to Figure 3, the classification-score vector is obtained after the testing sample passes through the CNN. For the MNIST dataset, the length of $f$ is 10, corresponding to 0–9, which is a total of 10 Arabic numbers. The predicted class:

$$\widehat{c}=\arg \max (f)$$

(3)

In the training stage, the cross-entropy loss is calculated according to the prediction class and ground-truth, and then the parameters are updated by back-propagation.

3.2. Testing Stage

3.2.1. Discrete-Wavelet Transform

To avoid spectrum aliasing caused by direct down-sampling, we use dyadic DWT, and for a two-dimensional image $f(x,y)$. with an input size of $P\times Q$, we define the two-dimensional (2D) DWT pair as:

$$W_{\phi }\left(a_0,p,q\right)=\frac{1}{\sqrt{PQ}}\sum _{x=0}^{P-1}\sum _{y=0}^{Q-1}f(x,y){\phi }_{a_0,p,q}(x,y)$$

(4)

$$W_{\psi }^b(a,p,q)=\frac{1}{\sqrt{PQ}}\sum _{x=0}^{P-1}\sum _{y=0}^{Q-1}f(x,y){\psi }^b{}_{a,p,q}(x,y)$$

(5)

where ${\phi }_{a_0,p,q}(x,y)$ is the scale function and ${\psi }_{a,p,q}^b(x,y)$ is the wavelet function, which is defined as follows:

$${\phi }_{a,p,q}(x,y)=2^{a/2}\phi \left(2^{a}x-p,2^{a}y-q\right)$$

(6)

$${\psi }_{a,p,q}^b(x,y)=2^{a/2}{\psi }^b\left(2^{a}x-p,2^{a}y-q\right)$$

(7)

where $a_0$ is the original scale of the image, $W_{\phi }\left(a_0,p,q\right)$ is the low-frequency coefficient, which represents the approximate value of the image at the scale $a_0$, and $W_{\psi }^b(a,p,q)$ is the high-frequency coefficient, including the horizontal, vertical, and diagonal details attached to the scale $a(a\ge a_0)$, $b=\left\{H,V,D\right\}$. Equation (4) shows that the approximate representation of the input image can be obtained after using DWT, which is the same as the traditional pooling method. Equation (5) still retains the high-frequency component and can restore the original image with the low-frequency component. The SA Net only keeps the low-frequency information after the DWT first-level decomposition, as shown in Equation (4), thereby effectively avoiding spectrum aliasing in down-sampling.

Fast-wavelet transform (FWT) [23] can be expressed as follows:

$$W_{\psi }(a,q)=\sum _p{h}_{\psi }(p-2l)W_{\phi }(a+1,p)$$

(8)

$$W_{\phi }(a,q)=\sum _p{h}_{\phi }(p-2l)W_{\phi }(a+1,p)$$

(9)

The Equations (8) and (9) show a DWT coefficient’s relationship between adjacent scales. It is observed that the scale approximation and detail coefficients of scale $a$ can be iteratively computed by convolving $W_{\phi }(a+1,p)$ with the temporally inverted scaling and wavelet vectors $h_{\psi }(p-2l)$ and $h_{\phi }(p-2l)$. The 2D FWT is similar to 1D FWT, except that 2D FWTs have three sets of detail coefficients occurring in the horizontal, vertical, and diagonal directions [24,25].

3.2.2. Image Pyramid

The testing sample $x$ with unseen scale is zoomed in and zoomed out on a group of images with different scales that form an image pyramid. The methods are as follows: scaling $x$ according to the geometric scale set $S=\{s_1,s_2,\dots ,s_L\}$ to generate an image pyramid $\{x_{s_1},x_{s_2},\dots ,x_{s_L}\}$ composed of multi-scale samples $L$ is the number of channels (i.e., the number of layers of the image pyramid). For $s_i$ > 1, that is, when the testing sample passes through the image-zooming-in channels, the testing sample is up-sampled by bilinear interpolation, which is the same as [22]. For $s_i$ < 1, that is, when the testing sample passes through the image-zooming-out channels, the testing sample is down-sampled, which is different from using a Gaussian derivative to limit the bandwidth [22]. Dyadic discrete wavelet transform (DWT) combined with interpolation is used to avoid spectrum aliasing caused by down-sampling. Only low-frequency components are retained for the results of dyadic DWT, as shown in Figure 4. Dyadic zooming-out uses DWT, and non-dyadic zooming-out uses the result of next-level DWT and zooms in it by bilinear interpolation.

After the testing sample passes through the image pyramid in the previous step, $L$ images of different scales $\{x_{s_1},x_{s_2},\dots ,x_{s_L}\}$ are generated. These images are sent to the Siamese CNNs with weight sharing for inferencing. The Siamese CNNs have corresponding $L$ channels, and the output of channelin $i$ is as follows:

$$f_i=f{(x_{s_i})}^T$$

(10)

The output of each channel $f_i$ is a flattened vector, and the output of channel $i$ for class $j$ is $f_{ij}$. The flattened vectors are stacked into a two-dimensional matrix:

$$F=\left[f_1,f_2,\dots ,f_i,\dots ,f_L\right]\in {ℝ}^{L\times C}$$

(11)

where $C$ is the number of classes of the corresponding dataset.

Softmax is widely used in the CNN for multi-classification tasks. It maps multiple inputs into the $(0,1)$. However, in the SA Net, the input is a two-dimensional array with various elements. When the input elements are too large or too small, they will exceed the range of the float, resulting in overflow and underflow. To ensure numerical stability and speed up the computation at the same time, we use the logsoftmax to obtain the classification-score matrix $F^{\prime }$, which is defined as follows:

$$\begin{array}{l}{F}^{\prime }=\mathrm{logsoftmax}(F)=\log \left(\frac{e^{f_{ij}}}{{\displaystyle \sum _i^L{\displaystyle \sum _j^C{e}^{f_{ij}}}}}\right)=\log \left(\frac{\frac{e^{f_{ij}}}{e^M}}{{\displaystyle \sum _i^L{\displaystyle \sum _j^C\frac{e^{f_{ij}}}{e^M}}}}\right)\\ =\log \left(\frac{e^{\left(f_{ij}-M\right)}}{{\displaystyle \sum _i^L{\displaystyle \sum _j^C{e}^{\left(f_{ij}-M\right)}}}}\right)==\left(f_{ij}-M\right)-\log \left({\displaystyle \sum _i^L{\displaystyle \sum _j^C{e}^{\left(f_{ij}-M\right)}}}\right)\end{array}$$

(12)

where $M=\underset{i,j}{\max }(F)$, that is, the maximum value of all elements in $F$.

3.2.3. Classification and Scale Estimation

As shown in Figure 3, according to the maximum of the two-dimensional classification-score matrix $F^{\prime }$, we can obtain the predicted class $\widehat{c}$ and the estimated scale $\widehat{s}$ of the testing sample.

The predicted class $\widehat{c}$ and scale estimation $\widehat{s}$ are related to coordinate $(i,j)$, corresponding to the maximum of $F^{\prime }$, and are represented as follows:

$$(i,j)=\arg \max (F^{\prime })$$

(13)

$$\widehat{c}=j, \widehat{s}=1/i$$

(14)

For the data set $D={\left\{x^{(n)},y^{(n)},s^{(n)}\right\}}_{n=1}^N$ containing $N$ testing samples, the GT class of $x^{(n)}$ is $y^{(n)}$ and the GT scale of $x^{(n)}$ is $s^{(n)}$. The estimated scale of the $x^{(n)}$ is ${\widehat{s}}^{(n)}$. The RMSE shown in Equation (15) is generally used to evaluate the estimation ability. However, to evaluate the performance of scale estimation under different scales, we use the relative value $\delta$ of RMSE shown in Equation (16):

$$\mathrm{RMSE}=\sqrt{\frac{1}{N}{\displaystyle \sum _n^N{\left(s^{(n)}-{\widehat{s}}^{(n)}\right)}^2}}$$

(15)

$$\delta =\frac{\mathrm{RMSE}}{\widehat{s}}\times 100\%$$

(16)

The detailed method is shown in the pseudo code of Algorithm 1.

Algorithm 1: Classification and scale estimation algorithm of SA Net

Input: Testing set $D={\left\{x^{(n)},y^{(n)},s^{(n)}\right\}}_{n=1}^N$

For $n=1toN$ do Select a sample $(x^{(n)},y^{(n)},s^{(n)})$ in order;  For $i=1toL$ do   $x_{s_i}^{(n)}\leftarrow T_i(x^{(n)})$;   $f_i\leftarrow f{(x_{s_i}^{(n)})}^T$;  end  // 2-D classification-score matrix  $F^{L\times C}\leftarrow \mathrm{logsoftmax}{\left[f_1,f_2,\cdot \cdot \cdot ,f_L\right]}^T$;  $\left(i,j\right)\leftarrow \arg \max F^{L\times C}$;  ${\widehat{c}}^{(n)}\leftarrow j$;…………//label prediction  ${\widehat{s}}^{(n)}\leftarrow 1/i$;…………//scale estimation  if ${\widehat{c}}^{(n)}=y^{(n)}$ then   $cls\leftarrow cls+1$;  end end $Ac{c}_{cls}\leftarrow \frac{cls}{N}\times 100\%$ $\mathrm{RMSE}\leftarrow \sqrt{\frac{1}{N}{\displaystyle \sum _n^N{\left(s^{(n)}-{\widehat{s}}^{(n)}\right)}^2}}$ $\delta \leftarrow \frac{\mathrm{RMSE}}{\widehat{s}}\times 100\%$

Output: $Ac{c}_{cls}$, $\delta$

4. Experiments and Discussion

4.1. Scale Datasets

The traditional MNIST dataset [2] is only composed of grayscale images of handwritten characters with a resolution of 28 × 28 at the same scale. It cannot evaluate the scale-estimation ability and classification ability of the SA Net for unknown-scale samples. We use the MNIST Large-Scale dataset [22] proposed by Jansson.

The MNIST Large-Scale dataset is a variant of the original MNIST dataset. The image of the original MNIST dataset is firstly rescaled to 112 × 112 using bicubic interpolation, and the rescaled image is then smoothed and threshold-softened to eliminate the image-edge aliasing. The processed image is then resized within a certain scale such that the scaling ratio of width and height is the same. Then, the resulting image is embed into a 112 × 112 resolution image using zero-padding or cropping on the borders. After the above processing, the MNIST Large-Scale dataset with several scale changes is obtained, as shown in Figure 5. The MNIST Large-Scale dataset is different from the MNIST dataset containing only one scale, and its scale scaling ratio is $s_D\in \left[1/2,8\right]$.

The MNIST Large-Scale dataset consists only of images of hand-written digits, and the existing performances on the MNIST Large-Scale dataset leave little room from improvement. To validate and evaluate our proposed method’s performance, we use the dataset of the variation of the Fashion MNIST–FMNIST Large-Scale dataset. The raw Fashion MNIST is a dataset that contains 70 K grayscale images (the pixel value is an integer between 0 and 255), associated with labels from 10 classes [26]. We processed the FMNIST Large-Scale dataset in the same way as the MNIST Large-Scale dataset, as shown in Figure 6.

4.2. Implementation Details and Network Parameters

The deep-learning framework was Pytorch 1.8.1 (CUDA 11.0), and the workstation was equipped with 32 GB memory, one Nvidia GeForce RTX 3090 graphics card(ASUS, Taiwan, China) and 24GB of video memory. The model was trained with 50,000 canonical images of the MNIST Large-Scale dataset for 20 epochs using the Adam [27] optimizer, and 10,000 images of different scales were used for testing. The training batch size was 128. The learning rate started at $3\times 10^{-3}$ and decayed with a factor 0.1 every 2 epochs until it reduced to $5\times 10^{-5}$. In this experiment, we used the standard CNN as the benchmark. The CNN’s feature extractor included four Conv-BN [28]-ReLU [29]-Max-pooling blocks. The number of features of each block was 16, 16, 32 and 32, respectively. After passing through the feature extractor, a group of feature maps was sent to the two-layer fully connected (FC) layers for classification, followed by logsoftmax. The number of FC layers was 100 and 10, respectively, and the first FC layer adopted Dropout [30] with a probability of 15%.

4.3. Classification Experiment

CNN, FovMax and FovAvg were trained on a single-scale-training dataset of scale 2 from the MNIST and FMNIST Large-Scale datasets with the same training settings and strategy. Figure 7 shows the loss curve during the training procedure on the MNIST Large-Scale dataset.

We use FovMax, FovAvg and the SA Net with nine channels for digit classification and the standard CNN as the benchmark to evaluate the classification performance for different testing scales on different datasets, as shown in

Table 1. Classification accuracy (%) comparison on different scales with different networks.

Datasets	Methods	Wavelet Basis	${\mathbf{2}}^{\mathbf{-}\mathbf{1}}$	${\mathbf{2}}^{\mathbf{-}\frac{\mathbf{1}}{\mathbf{2}}}$	$\mathbf{1}$	${\mathbf{2}}^{\frac{\mathbf{1}}{\mathbf{2}}}$	$\mathbf{2}$	${\mathbf{2}}^{\mathbf{3}\mathbf{/}\mathbf{2}}$	$\mathbf{4}$	${\mathbf{2}}^{\mathbf{5}\mathbf{/}\mathbf{2}}$	8
MNIST Large Scale	CNN	—	20.86	38.67	68.35	96.60	99.38	94.98	45.83	21.84	12.42
	FovMax	—	98.51	99.26	99.31	99.32	99.32	99.29	99.30	97.40	77.75
	FovAvg	—	98.75	98.81	98.36	98.85	99.33	98.84	99.31	98.73	88.44
	SA Net	bior	98.81	99.26	99.30	99.38	>99.41	99.37	99.31	99.12	90.15
		haar	98.68	99.17	99.25	99.34	99.39	99.35	99.28	98.96	89.74
		db	98.74	99.20	99.27	99.38	99.41	99.36	99.29	99.12	90.13
FMNIST Large Scale	CNN	—	15.47	17.12	37.64	58.56	90.64	61.30	36.33	14.78	9.62
	SA Net	bior	82.31	82.75	83.46	82.60	90.64	88.86	86.12	68.32	41.20
		haar	82.07	82.36	83.18	82.08	90.64	88.73	85.92	68.03	41.04
		db	82.24	82.79	83.39	82.23	90.64	88.81	86.03	68.16	41.13

. In this paper, bold data represent the best results of different methods under the same conditions.

We can see that the CNN has good testing-classification performance only when the test scale is consistent with the training scale, and the test classification accuracy decreases with the increase of the difference between the testing scale and the training scale both on MNIST and FMNIST Large-Scale datasets. The scale of the training-set samples is 2. When the testing scale is 2, the classification accuracy on the MNIST Large-Scale dataset is 99.38%. However, when the testing scale is 8, the classification accuracy reduces to 12.42%, which is slightly better than the random selection of the MNIST Large-Scale dataset. The standard CNN does not learn scale-related features in the training procedure, and it has poor adaptability to scale changes. FovAvg and FovMax greatly improve the network’s resilience to scale variation. When the training and testing scales are inconsistent, FovAvg and FovMax still have good classification performance, but when the testing scale is large, the test-classification accuracy degrades seriously. For example, the training scale is 2.0. On the test scale of $2^{-1}$, the classification accuracies of FovAvg and FovMax are 98.51% and 98.75%, respectively. On the test scale of 8.0, the classification accuracies of FovMax and FovAvg are 77.75% and 88.44%, respectively, and the classification performance declines. However, the classification accuracy of the SA Net is 90.15%, which is 12.5% and 7.71% higher than FovMax and FovAvg, respectively. The SA Net overcomes the existing problems of FovMax and FovAvg due to the down-sampling method of combination of dyadic DWT and interpolation.

Experiments on different datasets show that the SA Net has better robustness on the MNIST Large-Scale dataset, because handwritten digits are more regular than the images in the FMNIST Large-Scale dataset and are better for feature extraction.

Using different wavelet families, wavelet decomposition will yield different results. In order to select the optimal wavelet basis, we compare the classification experimental results of the SA Net using Haar (haar), Daubechies (db) and Biorthogonal (bior), and we can see that bior has the greatest benefit to the improvement of classification performance.

4.4. Scale-Estimation Experiment

4.4.1. Dataset

To explore the generalization ability of the SA Net for unknown scale images in a larger scale range, we generated digits with a scale range of $\left[1/4,4\right]$ and padded them with zeros to a 224 × 224 resolution, following the production mechanism of the MNIST Large-Scale dataset.

4.4.2. Receptive Field

The definition of the receptive field in the CNN is the area size mapped by the pixels on the feature map by each layer on the original image. The convolution layer and pooling layer will affect the size of the receptive field. The definition of the receptive field is as follows:

$$R{F}_{l+1}=R{F}_l+(k-1)\times S_l$$

(17)

$$S={\displaystyle \prod _{i=1}^l{s}_i}$$

(18)

where $R{F}_l$ and $R{F}_{l+1}$ represent the receptive fields of layer $l$ and layer $l+1$, respectively, $k$ represents the size of convolution kernel of layer $l+1$, $s_i$ represents stride of layer $i$ and $S$ represents the product of $s_i$ of the previous $l$ layer. The stride of the current layer does not affect the receptive field of the current layer. In order to ensure that all patterns in the image can be detected, the receptive field should match the size of the original image [31]. However, the actual receptive field is often smaller than the theoretical receptive field [32]. For an image of 224 × 224 resolution, the receptive field of the network described in Section 4.2 does not meet the requirements above, so we added two Conv-BN-ReLU-Max-pooling blocks in front of the FC layer of the SA Net to increase the network depth and to increase the receptive field. The parameters of the additional blocks are:

• the number of feature channels are 32 and 64, respectively.
• the size of the convolution kernels is three for two blocks.
• the other parameters of the blocks are suitable for the convolution layer.

The receptive field of the final layer of the network can be obtained through Equations (17) and (18), and result is $R{F}_{final}=226$, which meets the requirements of detecting all objects.

4.4.3. Experiment Results

According to Section 3.2.3, the SA Net uses the training set with a canonical scale; to estimate the scale of the testing sample with unknown scale, we use the dataset in Section 4.1.

When the number of channels $L$ is 9, the common ratio is $\sqrt{2}$, and we obtain the performance for scale estimation against different scales $\widehat{s}$ of 10,000 testing samples, as shown in

Table 2. Performance for scale estimation against different scales on different datasets.

	Datasets	Wavelet Basis	${\mathbf{2}}^{-\mathbf{2}}$	${\mathbf{2}}^{-\frac{\mathbf{3}}{\mathbf{2}}}$	${\mathbf{2}}^{\mathbf{-}\mathbf{1}}$	${\mathbf{2}}^{\mathbf{-}\frac{\mathbf{1}}{\mathbf{2}}}$	$\mathbf{1}$	${\mathbf{2}}^{\frac{\mathbf{1}}{\mathbf{2}}}$	$\mathbf{2}$	${\mathbf{2}}^{\frac{\mathbf{3}}{\mathbf{2}}}$	4
$\delta$/%	MNIST Large Scale	bior	7.1	23.2	22.7	28.3	26.5	21.6	15.7	17.2	13.9
		haar	9.3	25.9	27.1	30.2	28.3	24.6	17.4	19.5	14.2
		db	7.2	21.8	23.9	29.5	26.5	21.3	15.9	17.4	14.1
	FMNIST Large Scale	bior	13.65	24.51	36.60	42.13	51.19	39.83	58.90	59.91	53.68
		haar	14.10	23.16	36.81	42.31	51.32	40.32	60.80	62.30	55.73
		db	13.74	24.59	36.60	42.07	51.23	40.02	59.18	60.08	53.71

. It can be seen that the SA Net has good performance for scale estimation; for example, when the $\widehat{s}$ is 4, $\delta$ can reach 13.9% on the MNIST Large-Scale dataset. Besides, the SA Net also works on the FMNIST Large-Scale dataset, which are grayscale images and closer to real-world images compared to the MNIST [33].

Similar to the classification experiment, in order to select the optimal wavelet families, we comprehensively compare the scale estimation experimental results of the SA Net using Haar (haar), Daubechies (db) and Biorthogonal (bior). It can be seen that, like the conclusion in Section 4.3, bior is also the optimal wavelet family.

In order to verify the effectiveness of the SA Net for scale estimation, we visualize the multi-column’s output response and analyze the mis-estimated samples. For the testing sample with class ‘5′ and scale ‘$2^{-\frac{1}{2}}$’, the SA Net’s 3D response is show Figure 8.

In order to more intuitively evaluate the effect of scale estimation, we draw slices figures of a three-dimensional output response of the MNIST Large-Scale dataset according to the max response, as shown in Figure 9. In class dimension $\widehat{c}$, the curve is not smooth, resulting in several sidelobes. Sidelobes will interfere with the classification task. The coordinate corresponding to sidelobes is the wrong class, indicating the correlation between adjacent classes is weak in the classification task. However, the amplitude of sidelobes is always smaller than the main lobe. The coordinate corresponding to the maximum response in the class dimension always corresponds to the GT class, verifying that the SA Net can always achieve high classification performance in the classification task in Section 4.3. In the scale dimension $\widehat{s}$, the curve is relatively smooth. It has only one peak without sidelobe interference, which is conducive to scale estimation, so the testing sample’s true scale can always be obtained.

Compared with the single-channel network, the multi-channel network will certainly increase the inferencing time. However, the inferencing time of the SA Net for one frame of an image costs about 26 ms (using one GeForce RTX 3090 GPU). Therefore, the SA Net has the potential to achieve real-time scale estimation and classification.

4.4.4. Visualization of Scale Estimation

Rescaling the testing samples according to the estimated scales is equivalent to the inverse process of generating an image pyramid in Section 3.2.2, which can correct the testing sample to the canonical scale. Several restored examples in the MNIST Large-Scale dataset are chosen for visualization, as shown in Figure 10.

It can be seen that for testing images of any scale, the SA Net can correct the testing images to the normal scale with the scale-estimation value. In practical application, the images collected by the optical imaging sensor can be easily processed after the above correction process.

5. Conclusions

The existing methods can only be applied to the classification of scaled images and cannot estimate unknown-scale images. We propose a scale-aware network with scale equivariance, which can estimate the scale while classifying without adding additional scale-related learning tasks. According to experimental results on different datasets and wavelet families, we verify the practicability of our proposed method for scale estimation and find the optimal wavelet basis for our network. In addition, the classification experiments on the MNIST and FMNIST Large-Scale datasets show that the classification performance of the SA Net on unknown-scale samples is better than other existing methods, which is due to the down-sampling of the wavelet transform combined with interpolation to avoid spectrum aliasing.

The proposed method can be used in remote sensing, optical image recognition and medical diagnosis in cytohistology.