Scale and Background Aware Asymmetric Bilateral Network for Unconstrained Image Crowd Counting

Abstract

This paper attacks the two challenging problems of image-based crowd counting, that is, scale variation and complex background. To that end, we present a novel crowd counting method, called the Scale and Background aware Asymmetric Bilateral Network (SBAB-Net), which is able to handle scale variation and background noise in a unified framework. Specifically, the proposed SBAB-Net contains three main components, a pre-trained backbone convolutional neural network (CNN) as the feature extractor and two asymmetric branches to generate a density map. These two asymmetric branches have different structures and use features from different semantic layers. One branch is densely connected stacked dilated convolution (DCSDC) sub-network with different dilation rates, which relies on one deep feature layer and can handle scale variation. The other branch is parameter-free densely connected stacked pooling (DCSP) sub-network with various pooling kernels and strides, which relies on shallow feature and can fuse features with several receptive fields to reduce the impact of background noise. Two sub-networks are fused by the attention mechanism to generate the final density map. Extensive experimental results on three widely-used benchmark datasets have demonstrated the effectiveness and superiority of our proposed method: (1) We achieve competitive counting performance compared to state-of-the-art methods; (2) Compared with baseline, the MAE and MSE are decreased by at least $6.3\%$ and $11.3\%$, respectively.

1. Introduction

Crowd counting, a fundamental task in many public safety monitoring systems, aims to estimate the number of people in a still image. There are many studies have been devoted to solve this problem and they have made some progress [1,2]. Conventional methods solve the crowd counting problem by using hand-crafted features to detect each individual person, predict the count of people via regression or density estimation [3,4,5,6,7]. Due to insufficient semantic representation of hand-crafted features, these methods usually have low performance. Recently, benefiting from the powerful feature representation of CNN, CNN-based methods become dominant in crowd counting. In view of different types of network architectures, CNN-based crowd counting models can be divided into two categories: single column based methods [8,9] and multi-column based methods [10,11].

However, there are still several challenges preventing the computer vision community from designing a model capable of performing precise and robust crowd counting, such as occlusion, complex background, scale variation, non-uniform distribution, perspective distortion, rotation, illumination variation and weather changes. In this paper, we make an effort to address two of these challenges: scale variation and background noise. As illustrated in Figure 1, the scales of person vary as the distance from the camera, the background regions may have similar colors or textures with the people.

The scale variation is the primary problem that should be addressed in the crowd counting models. In the era of deep learning, researchers have made efforts to integrate semantic features of different scales to solve this problem. The main algorithm framework can be divided into three categories: (1) Multi-column. Some works [10,12,13] adopted “multi-column” architecture, where each branch has a different filter kernel size to handle a specific scale; (2) Nonstandard convolution. Some nonstandard convolution operations like dilated convolution [14,15,16] or deformable convolution [17,18], are exploited to model multi-scale information; (3) Feature pyramid networks (FPN). Several works [19,20,21] assumed features from different levels can capture different scale information and utilized FPN to fuse multi-level features. To eliminate noises caused by cluttered backgrounds, semantic segmentation or visual attention operations [22,23,24,25] are two common ways to suppress the response of the background regions. These methods guide the network focus on person instance via a mask image. Moreover, transformers are a new genre of deep neural networks that stack multiple self-attended layers. Unsurprisingly, we see an increasing effort on re-purposing such types of transformer models for crowd counting [26,27,28,29,30,31]. However, all methods mentioned above usually require millions of extra to-be-learned parameters (e.g., learnable multi-column, learnable segmentation/attention branch, multi-head transformer structure), which results in a higher computational burden.

Recently, there have been some works [32,33,34] that attempt to model scale and background problems in a unified framework. The structures of these models are basically the same: a backbone to extract the image feature, and two multi-scale fusion branches to predict a density map and a background mask, respectively. These two branches share the image feature in a dual/symmetric way, that is, they utilize one or multiple common layers to learn scale and background aware features respectively. However, the latest research [35] shows that, compared with deep features, shallow features have a better ability to distinguish background and people. In addition, ref. [36] demonstrates the effectiveness of asymmetric module.

In this paper, we propose a new method for unconstrained image crowd counting, termed Scale and Background aware Asymmetric Bilateral Network (SBAB-Net), which is able to handle scale variation and background noise in a unified framework. Specifically, the proposed SBAB-Net contains three main components—a pre-trained backbone CNN as the feature extractor and two asymmetric branches to generate a density map. These two asymmetric branches have different structures and use features from different semantic layers. One branch is a densely connected stacked dilated convolution (DCSDC) sub-network with different dilation rates, which relies on one deep feature layer and can handle scale variation. The other branch is a densely connected stacked pooling (DCSP) sub-network with various pooling kernels and strides, which relies on shallow features and can fuse features with several receptive fields to reduce the impact of background noise. Compared with the existing semantic segmentation or visual attention based background aware methods, the proposed DCSP is learnable, parameters-free. Two sub-networks are fused by an attention mechanism to generate the final density map. Comprehensive experimental results on three widely-used benchmark datasets demonstrate that the proposed method is effective at the crowd counting task.

We note that Huang et al. [37] also apply the stacked pooling structure for crowd counting. The main differences between our method and [37] are three-fold. First, feature fusion strategies are different. We present a densely connected manner to fuse features with several receptive fields while [37] utilizes channel-wise average operation. Second, inspired by [35], we utilize shallow features to suppress the background noise while [37] use deep features to model scale information. Third, we handle scale and background information simultaneously while [37] only considers the scale problem.

Moreover, it should be noted that although there are several existing methods (e.g., CSRNet [16], DSACA [14], ADNet [15], DADNet [38]) that also utilize a dilation convolution structure to model scale variation, the differences between these existing dilation approaches and the proposed approach are three fold—Firstly, the proposed densely connected stacked dilated convolution (DCSDC) sub-network is a new and effective module to handle scale variation. Compared with the most famous dilation solution, CSRNet [16], our DCSDC fuses features from three different convolution layers with different dilation rates (i.e., 1, 2, 3) while CSRNet only stacks six dilated convolution layers with the same dilation rate (i.e., 2). Although ADNet [15] presents an adaptive dilated network in which the dilation rate can be learned online, only a specific dilation rate is estimated for an input feature map. DSACA [14] uses different dilation rates to capture multi-scale information by fusing features from different layers of the backbone. On the contrary, the proposed DCSDC only utilizes the last feature layer of the backbone, and captures multi-scale information by a densely connected stacked dilated convolution sub-network with different dilation rates. The multi-dilation convolution branch in DADNet [38] is a parallel structure while in our DCSDC it is a serial structure. The disadvantage of the parallel structure is that it neglects the small scale visual context when modeling a large scale visual context. Extensive experiments on three widely-used benchmarks demonstrate that the proposed method is better than these existing dilation approaches. Secondly, the scale aware feature $f_s$ is refined by the background aware feature $f_b$ in our framework. Finally, the dual branches in our SBAB-Net are asymmetric while most existing frameworks are symmetric, for example, SFANet [32].

In summary, this paper makes the following contributions.

• We propose a novel asymmetric bilateral network to handle scale variation and background noise in a unified framework. The scale-aware feature is captured by a multi-convolutional branch based on one deep feature layer. The background-aware feature is captured by a multi-pooling branch based on one shallow feature layer. These two features are fused in an attention manner. On the contrary, most existing methods organize scale and background branches in a symmetric/dual structure, that is, share the image feature from one or multiple common layers.
• A new DCSP sub-network is proposed to capture background-aware feature, which can fuse features with several receptive fields by several pooling layers with different strides to reduce the impact of background noise, and without any extra learnable parameters.
• We propose a new DCSDC sub-network to extract multi-scale information based on one deep feature layer, which densely connected several stacked dilated convolution layers with different dilation rates to capture the scale-aware feature. Moreover, the scale-aware feature is refined by the background-aware feature to ensure that the final feature can handle scale and background information simultaneously.
• We conducted extensive experiments on three challenging crowd counting datasets, that is, ShanghaiTech [10], UCF-QNRF [39], NWPU-Crowd [40]. Both quantitative and qualitative results demonstrate the effectiveness of the proposed method.

The remainder of this paper is organized as follows. Section 2 reviews the related work of CNN-based crowd counting methods for addressing scale variation and background noise challenges. Section 3 presents the proposed method in detail. Section 4 conducts comparisons on benchmarks and provides the experimental analysis. Finally, we conclude this paper in Section 5.

2. Related Work

The research on crowd counting can be roughly divided into three different dimensions: detection-based, regression-based and density estimation approaches.

Conventional methods [6,41] count the number of people in an image by using the handcraft feature to detect head or body, which performs worse for high crowd images. Thus, the direct regression methods have been proposed, which aims to predict the number of people directly from the image feature. These regression methods can also be divided into traditional and deep learning algorithms according to the utilization of low-level [4,42] or deep [9,43,44] features. However, direct regression methods ignore the spatial context information, which leads to the results not being robust. Recently, deep learning based density map estimation methods become dominant in crowd counting area. Each value in the density map represents the count of the person which corresponding to the original image region. Apparently, the sum of the density map is the number of people appearing in the whole image.

We shortly review some representative density estimation works related to our framework in the following.

2.1. Handle Scale Variation

Early work [10,11,16] for scale-aware feature extraction is via the multi-column or multi-network structure; each column or sub-network handles specific scale information via different convolution types, which is more complicated for optimization and computation wasting. Thus, single-column architecture becomes more popular. There are three common strategies modeling multi-scale in a single-column CNN: (1) Fuse features extracted from different layers [45], in which they assume that a different layer could capture different object scales; (2) Integrate features via pyramid operation, which may be an image pyramid [46] or a feature pyramid [19,20]; and (3) Learn scale from the annotations’ distribution directly [47,48,49]. Different from the above methods, we propose a novel densely connected stacked dilated convolution sub-network with different dilation rates, which can handle multi-scale variation only based on one deep feature layer.

2.2. Handle Background Noise

Attention structure is widely-used to handle background noise. Liu et al. [22] present a detection module to guide the network counting of the crowd region. Gao et al. [50] introduce spatial-wise and channel-wise attention modules to constrain the network’s focus on the head region—a similar idea is reported in [51]. Sindagi et al. [52] propose an inverse attention mechanism which can infuse segmentation information into the counting network efficiently. Shi et al. [53] consider the pixels near to annotations as foreground and design a branch to predict these foreground regions to learn attention factors. Liu et al. [54] propose a recurrent attentive zooming network to detect ambiguous regions recurrently.

2.3. Handle Scale Variation and Background Noise Simultaneously

Recently, some works have attempted to handle scale variation and background noise simultaneously in a unified framework. Zhang et al. [55] propose a cross layer attentional neural field. Guo et al. [38] and Liu et al. [17] use both attention and deformable convolutional structures in a model. All methods with the attention structure mentioned above have a common shortcoming, which is that these attention modules need millions of parameters to be optimized, which makes the model become more complex.

In addition, there are some works that attempt to improve the performance of the counting model by using auxiliary tasks, for example, a classification task is used in [21,56,57], a segmentation task is utilized in [58,59,60], and a depth estimation task is adopted in [59,61]. These methods improve the performance significantly, which indicates that modeling related tasks simultaneously can improve the crowd counting accuracy. Inspired by this, in this paper, we deem the foreground mask prediction as the auxiliary task, and achieve it efficiently by feeding the shallow feature into the proposed novel parameter-free densely connected stacked pooling sub-network.

3. Methodology

In this section, we introduce the proposed SBAB-Net for crowd counting in detail. Our goal is to estimate the number of persons in a crowd image. To this end, following [62], we formulate the crowd counting task as a density map regression and distribution matching problem. In SBAB-Net, the input is the crowd image and the output is the corresponding density map.

3.1. Framework

As shown in Figure 2, our proposed SBAB-Net is composed of three modules, that is, backbone, densely Connected Stacked Dilated Convolution (DCSDC) and densely Connected Stacked Pooling (DCSP) modules. The backbone is a pre-trained CNN to extract both shallow and deep features of the input image. Let $I\in R^{H\times W\times 3}$ denote the input crowd image of width W and height H. The extracted shallow and deep features are represented as $S_I\in R^{\frac{H}{d_1}\times \frac{W}{d_1}\times c_1}$ and $D_I\in R^{\frac{H}{d_2}\times \frac{W}{d_2}\times c_2}$, respectively, where $d_1$ and $d_2$ are output strides, $c_1$ and $c_2$ are channels of the feature maps. $D_I$ is fed into the DCSDC module to obtain scale aware feature $f_s$, and $S_I$ is fed into the DCSP module to obtain background aware feature $f_b$. We ensure the dimensions of the two features $f_s$ and $f_b$ to be equal by assigning proper convolutional kernel numbers in the DCSDC module. Then we use a foreground mask regression head (a light weight CNN to predict the probability of pedestrians at each location) to distinguish background and person instance based on feature $f_b$. To handle scale and background information simultaneously, we present an element-wise multiple based attention mechanism which refines the feature $f_s$ based on the feature $f_b$, formulation is as,

$$f=f_s\otimes f_b.$$

(1)

Apparently, according to Equation (1), the features of background regions in $f_s$ will be suppressed. Finally, a density map regression head (a light weight CNN to predict the number of pedestrians at each location) is utilized to predict a person density map based on the refined feature f. Therefore, the sum of the density map is the estimated person numbers of the input image I. The structure of the proposed density map regression head and foreground mask regression head are illustrated in Figure 3; the learnable structures and input-output dimensions of these two sub-networks are the same, while the output semantics are different. In addition, both density map regression head and foreground mask regression head are mapping local features to semantics. Therefore, $f_s$ is sensitive to person scales and $f_b$ is sensitive to regions with pedestrians; the refined feature f is sensitive to scales and background regions simultaneously.

It should be noted that we should predict both foreground mask and density map in training, and only the density map needs to be estimated in the test.

Next, we will introduce the DCSDC and DCSP modules in detail.

3.2. Densely Connected Stacked Dilated Convolution (DCSDC)

The DCSDC implements a non-linear mapping function $H(·)$ which takes the deep feature $D_I$ as input, and produces a scale aware feature $f_s$, that is,

$$f_s=H\left(D_I\right).$$

(2)

As shown in Figure 2, DCSDC consists of three dilation convolution layers with different dilation rates, and these layers are connected in a densely connected way. It is able to handle scale variation based on one deep feature layer.

Specifically, all of the three convolution layers have $3\times 3$ kernels, the dilation rates are 1, 2, and 3 respectively. Meanwhile, since the feature dimensions of each convolution layer in the DCSDC shall be consistent, the corresponding padding sizes are also set to 1, 2, and 3 respectively. The non-linear functions of three dilation convolution layers are denoted as $h_1(·)$, $h_2(·)$, and $h_3(·)$ and the output of the first dilation convolution layer $H^1$ is formulated as,

$$H^1=[D_I,h_1\left(D_I\right)],$$

(3)

where $[·]$ represents the feature concatenation operation.

The output of the second dilation convolution layer $H^2$ is formulated as,

$$H^2=[D_I,h_2\left(H^1\right),H^1].$$

(4)

Consequently, the output of the third dilation convolution layer, that is, $f_s$, is formulated as,

$$f_s=[D_I,h_3\left(H^2\right),H^1,H^2].$$

(5)

3.3. Densely Connected Stacked Pooling (DCSP)

The DCSP implements a non-linear mapping function $G(·)$ which takes the deep feature $S_I$ as input, and produces a background aware feature $f_b$, that is,

$$f_b=G\left(S_I\right).$$

(6)

As shown in Figure 2, DCSP consists of three max pooling layers with different pooling kernels, and these layers are connected in a densely connected way. It is able to fuse features with several receptive fields to reduce the impact of background noise via a learnable parameter-free manner.

Specifically, the first two pooling kernels in DCSP are $2\times 2$ max pooling with stride 2, and the third pooling kernel is a $3\times 3$ max pooling with stride 1. The non-linear functions of three pooling layers are denoted as $g_1(·)$, $g_2(·)$, and $g_3(·)$, the output of the first pooling layer $G^1$ is formulated as,

$$G^1=g_1\left(S_I\right).$$

(7)

The output of the second pooling layer $G^2$ is formulated as,

$$G^2=[g_2\left(G^1\right),G^1].$$

(8)

Consequently, the output of the third pooling layer, that is, $f_b$, is formulated as:

$$f_b=[g_3\left(G^2\right),G^1,G^2].$$

(9)

3.4. Loss Function

In this section, we first introduce the ground-truth generation strategy, then detail the loss function.

To obtain the ground-truth density map $D_i^{GT}$ (here i represents the i-th image), we follow the method of generating density maps in [62], which does not need any Gaussian to preprocess ground truth annotations. The generation of density map $D_i^{GT}$ can be formulated as,

$$D_i^{GT}\left(x\right)=\delta \left(x\right),$$

(10)

where $\delta \left(x\right)$ means the annotated person numbers in the position x.

Based on ground-truth density map $D_i^{GT}$, we compute ground-truth foreground mask $F_i^{GT}$ as follows:

$$F_i^{GT}\left(x\right)=\left\{\begin{array}{cc}0\hfill & \delta \left(x\right)=0.\hfill \\ 1\hfill & \delta \left(x\right)>0.\hfill \end{array}\right.$$

(11)

Equation (11) ensures that position x with person instance will be deemed foreground.

To increase the generalization performance of the proposed model, inspired by [62], counting loss, optimal transport loss and total variation loss are utilized to measure the difference between ground-truth density map $D_i^{GT}$ and predict density map $D_i^P$.

Counting loss ($L_c$) is defined as follows:

$$L_c(D_i^P,D_i^{GT})=|\parallel D_i^P{\parallel }_1-|\parallel D_i^{GT}{\parallel }_1\left|\right|,$$

(12)

where ${\parallel \ast \parallel }_1$ denotes the $L_1$ norm of a vector.

Optimal transport loss ($L_{ot}$) is defined as,

$$L_{ot}(D_i^P,D_i^{GT})=\langle {\alpha }^{*},\frac{D_i^{GT}}{\parallel D_i^{GT}{\parallel }_1}\rangle +\langle {\beta }^{*},\frac{D_i^P}{\parallel D_i^P{\parallel }_1}\rangle ,$$

(13)

where ${\alpha }^{*}$ and ${\beta }^{*}$ are the solutions of Monge-Kantarovich Optimal Transport formulation [63].

Total variation loss ($L_{tv}$) is defined as,

$$L_{tv}(D_i^P,D_i^{GT})=\frac{1}{2}\parallel D_i^{GT}{\parallel }_1\parallel \frac{D_i^{GT}}{\parallel D_i^{GT}{\parallel }_1}-\frac{D_i^P}{\parallel D_i^P{\parallel }_1}{\parallel }_1.$$

(14)

To measure the difference between the ground-truth foreground mask $F_i^{GT}$ and predicted foreground mask $F_i^P$, BCE segmentation loss ($L_b$) is defined as,

$$L_b(F_i^P,F_i^{GT})=-F_i^{GT}log{F}_i^P+(F_i^{GT}-1)log(1-F_i^P).$$

(15)

The final loss is the weighted summation of the four loss terms mentioned above,

$$Loss=L_c+{\lambda }_1{L}_{ot}+{\lambda }_2{L}_{tv}+{\lambda }_3{L}_b,$$

(16)

where the hyper-parameters ${\lambda }_1$, ${\lambda }_2$ and ${\lambda }_3$ control the contributions of different components.

4. Experiments

In this section, we first introduce the experimental setting and implementation details, and then analyze in detail the effectiveness of each component in our framework through a set of ablation experiments on three public benchmarks. After that, we evaluate our method by comparing to the state-of-the-art methods.

4.1. Experimental Setting

4.1.1. Datasets

The experiments are conducted and evaluated on three common crowd counting benchmarks, including ShanghaiTech [10], UCF-QNRF [39] and NWPU-Crowd [40]; some samples are shown in Figure 4. Each person in a crowd image on these datasets is annotated with one point close to the center of the head.

The ShanghaiTech dataset consists of 1198 annotated crowd images and 330,165 annotated people. It is divided into two independent parts: Part-A and Part-B. Part-A contains 482 images which are collected from the Internet, and it is split into train and test subsets consisting of 300 and 182 images, respectively. Part-B contains 716 images which are collected from the busy streets, and it is split into training and test subsets consisting of 400 and 316 images, respectively.

UCF-QNRF is composed of 1535 images (1201 for training and 334 for testing) with more than 1.25 million label instances.

The NWPU-Crowd dataset is a large-scale congested dataset recently introduced in [40], which consists of 5109 images, with a total of 2,133,375 annotated heads with various illumination scenes and density ranges. The dataset is randomly split into training, validation and test subsets, which respectively contain 3109, 500 and 1500 images.

4.1.2. Evaluation Metrics

Follow the existing works [64,65,66,67], mean absolute error (MAE) and mean squared error (MSE) are used to evaluate the performance of different crowd counting methods.

The MAE is calculated by,

$$MAE=\frac{1}{N}\sum _{i=1}^N\parallel C_i-C_i^{GT}\parallel .$$

(17)

MSE is formulated as,

$$MSE=\sqrt{\frac{1}{N}\sum _{i=1}^N{\parallel C_i-C_i^{GT}\parallel }^2},$$

(18)

where N is the number of images in the test set, $C_i$ and $C_i^{GT}$ are the estimated and ground-truth individuals of i-th image, respectively.

4.2. Implementation Details

We set hyper-parameters following [62]. All empirical studies are conducted on an Nvidia GTX 3070 GPU in PyTorch. During training, the input size is $256\times 256$ on ShanghaiTech Part-A, $384\times 384$ on NWPU, $512\times 512$ on ShanghaiTech Part-B and UCF-QNRF; all of them are randomly cropped from the original image. During testing, the input image of the network maintains the original resolution. The AdamW [68] optimizer is used to train all models, and the learning rate and weight decay are set to 0.00001 and 0.0001 respectively. The output strides $d_1$ and $d_2$ are set to 4 and 8, respectively. The hyper-parameters ${\lambda }_1$, ${\lambda }_2$ and ${\lambda }_3$ in loss function are set to 0.1, 0.01 and 1, respectively. There is no augmentation during training and inference.

4.3. Ablation Studies

The purpose of this section is to demonstrate the effectiveness of the proposed solution rather than achieving a state-of-the-art performance in a certain dataset by all manner of means. Therefore, we utilize the light network MobileNetV2 [69] as our backbone.

4.3.1. Convergency

Figure 5 shows the value of the loss function of our method versus the different number of training epochs on the ShanghaiTech Part-A dataset. We can observe that the loss value decreases almost monotonously and converges smoothly during the entire training procedure, which indicates that the proposed framework can be efficiently trained by using the stochastic gradient descent optimization algorithm, AdamW. The model has reached convergence if the value of loss does not decrease in 20 consecutive epochs.

4.3.2. Impact of Different Components

Compared with the baseline [62], the proposed SBAB-Net has two novel terms, that is, DCSDC and DCSP. To investigate the impact of these terms on the performance of the proposed method, we developed and evaluated two variations of SBAB-Net: the one is SBAB-Net w/o DCSDC, that is, only backbone and DCSP are used to predict the density map directly. The other is SBAB-Net w/o DCSP, that is, it does not model the background region explicitly. The optimization procedure of these two cases is similar to the proposed SBAB-Net.

Table 1. Ablation studies. Performance comparison of the proposed SBAB-Net and its three variations in terms of MAE and MSE on benchmarks. The highest score is shown in boldface. (The backbone is MobileNetV2).

Components	ShanghaiTech Part-A		ShanghaiTech Part-B		UCF-QNRF		NWPU
	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE
Baseline	68.92	108.08	8.57	15.38	210.56	393.69	85.29	401.35
SBAB-Net w/o DCSDC	65.06	105.29	8.34	14.88	172.11	255.84	-	-
SBAB-Net w/o DCSP	62.50	100.31	8.17	13.72	-	-	82.87	389.29
SBAB-Net	60.57	95.89	8.03	12.24	108.58	185.38	69.91	237.14

shows the performance comparisons of SBAB-Net and its two variations on three benchmarks. From the results, we can see that the full SBAB-Net performs best on all datasets, which indicates that all of the two novel terms (DCSDC and DCSP) contribute to the final counting accuracy. We can also see that SBAB-Net w/o DCSP outperforms SBAB-Net w/o DCSDC by a large margin, which indicates that the importance of the DCSDC for the model to learn the scale aware representation.

We qualitatively evaluate the effectiveness of the proposed SBAB-Net by visualizing the density maps, as shown in Figure 6. We can see that there are obvious differences between density maps of the baseline and SBAB-Net on the following two aspects: (1) Robustness to noises, as shown in the second and fourth rows; (2) Robustness to scale variations, as shown in the first and third rows.

Both quantitative and qualitative experimental results (Table 1 and Figure 6) demonstrate that utilizing $f_b$ refines $f_s$ by element wise multiple operation can improve the counting performance (“SBAB-Net w/o DCSP” and “SBAB-Net” comparisons on Table 1) and suppress background regions (visualization results in Figure 6).

It should be noted that the major difference between our DCSP and the SP presented in [37] is that features from several different pooling kernels are fused by the densely connected method in our framework. Figure 7 shows the performance comparison of the proposed DCSP and SP in terms of MAE and MSE on the ShanghaiTech Part-A dataset. From this, we can see that DCSP performs better than SP. We think the reason is that channel-wise average operation in original SP loses many details.

4.4. Comparison with State-of-the-Art Methods

To verify the effectiveness of our SBAB-Net, we compare the proposed method with sixteen state-of-the-art methods in the experiments, namely, CSRNet [16], SFCN [70], CAN [71], Bayesian+ [65], S-DCNet [72], SANet + SPANet [73], SDANet [35], ADNet [15], ADSCNet [15], ASNet [74], AMRNet [75], AMSNet [76], DM-Count [62], P2PNet [64], DADNet [38] and SFANet [32]. As with the structure in DM-Count [62], the backbone of the proposed SBAB-Net is the pre-trained VGG-19 [77].

Table 2, Table 3 and Table 4 report the MAE and MSE of the proposed SBAB-Net and the compared methods (performance numbers directly cited from the original papers) of ShanghaiTech, UCF-QNRF, and NWPU datasets, from which we have the following observations:

• Although the structures of DCSDC and DCSP are simple, the proposed SBAB-Net is able to achieve a competitive counting performance compared to state-of-the-art methods on both small and large datasets. SBAB-Net, especially, significantly outperforms baseline (DM-Count [62]) while using the same backbone as the feature extractor.
• The proposed asymmetric bilateral network performs better than the commonly used dual structure (e.g., SFANet [32]). It is a further proof of the observation in [35], that is, the features of different layers in CNN have different abilities to model background noise.
• Although the same three dilation rates are utilized in the proposed framework and CSRNet [16], we achieve a much better performance, which indicates the advantage of our DCSDC structure.
• Our SBAB-Net is not the best approach. A better multi-task learning framework (e.g., P2PNet [64]), a better CNN structure (e.g., AMSNet [76]) or a better training strategy (e.g., ADSCNet [15]) can obtain a higher crowd counting performance.

Table 2. Performance comparison of the proposed in terms of MAE and MSE on ShanghaiTech Part-A dataset.

Methods	ShanghaiTech Part-A
	MAE	MSE
CSRNet [16]	68.2	115.0
SFCN [70]	64.8	107.5
CAN [71]	62.3	100.0
Bayesian+ [65]	62.8	101.8
S-DCNet [72]	58.3	95.0
SANet + SPANet [73]	59.4	92.5
SDANet [35]	63.6	101.8
ADSCNet [15]	55.4	97.7
ADNet [15]	61.3	103.9
ASNet [74]	57.78	90.13
AMRNet [75]	61.59	98.36
AMSNet [76]	56.7	93.4
DM-Count [62]	59.7	95.7
DADNet [38]	64.2	99.9
SFANet [32]	63.8	105.2
Ours	57.69	93.77

Table 2. Performance comparison of the proposed in terms of MAE and MSE on ShanghaiTech Part-A dataset.

Methods	ShanghaiTech Part-A
	MAE	MSE
CSRNet [16]	68.2	115.0
SFCN [70]	64.8	107.5
CAN [71]	62.3	100.0
Bayesian+ [65]	62.8	101.8
S-DCNet [72]	58.3	95.0
SANet + SPANet [73]	59.4	92.5
SDANet [35]	63.6	101.8
ADSCNet [15]	55.4	97.7
ADNet [15]	61.3	103.9
ASNet [74]	57.78	90.13
AMRNet [75]	61.59	98.36
AMSNet [76]	56.7	93.4
DM-Count [62]	59.7	95.7
DADNet [38]	64.2	99.9
SFANet [32]	63.8	105.2
Ours	57.69	93.77

Table 3. Performance comparison of the proposed in terms of MAE and MSE on the UCF-QNRF dataset.

Methods	UCF-QNRF
	MAE	MSE
CSRNet [16]	120.3	208.5
SFCN [70]	102	171
CAN [71]	107	183
Bayesian+ [65]	88.7	154.8
S-DCNet [72]	104.4	176.1
ASNet [74]	91.59	159.71
AMRNet [75]	86.6	152.2
AMSNet [76]	101.8	163.2
DM-Count [62]	85.6	148.3
SFANet [32]	100.8	174.5
P2PNet [64]	85.32	154.5
ADNet [15]	90.1	147.1
DADNet [38]	113.2	189.4
Ours	84.81	152.17

Table 3. Performance comparison of the proposed in terms of MAE and MSE on the UCF-QNRF dataset.

Methods	UCF-QNRF
	MAE	MSE
CSRNet [16]	120.3	208.5
SFCN [70]	102	171
CAN [71]	107	183
Bayesian+ [65]	88.7	154.8
S-DCNet [72]	104.4	176.1
ASNet [74]	91.59	159.71
AMRNet [75]	86.6	152.2
AMSNet [76]	101.8	163.2
DM-Count [62]	85.6	148.3
SFANet [32]	100.8	174.5
P2PNet [64]	85.32	154.5
ADNet [15]	90.1	147.1
DADNet [38]	113.2	189.4
Ours	84.81	152.17

Table 4. Performance comparison of the proposed in terms of MAE and MSE on the NWPU validation set.

Methods	NWPU Validation Set
	MAE	MSE
CSRNet [16]	104.8	433.4
SFCN [70]	95.4	608.3
CAN [71]	93.5	489.9
Bayesian+ [65]	93.6	470.3
DM-Count [62]	70.5	357.6
Ours	63.26	211.75

Table 4. Performance comparison of the proposed in terms of MAE and MSE on the NWPU validation set.

Methods	NWPU Validation Set
	MAE	MSE
CSRNet [16]	104.8	433.4
SFCN [70]	95.4	608.3
CAN [71]	93.5	489.9
Bayesian+ [65]	93.6	470.3
DM-Count [62]	70.5	357.6
Ours	63.26	211.75

5. Conclusions

In this paper, we presented a new framework (SBAB-Net) for unconstrained image crowd counting, which can handle scale variation and background noise in a unified framework. SBAB-Net achieved this goal with an asymmetric bilateral network, which consists of two novel branches. One branch is a densely connected stacked dilated convolution (DCSDC) sub-network with different dilation rates, which relies on one deep feature layer and can handle scale variation. The other branch is a parameter-free densely connected stacked pooling (DCSP) sub-network with various pooling kernels and strides, which relies on shallow features and can fuse features with several receptive fields to reduce the impact of background noise. Two sub-networks are fused by an attention mechanism to generate the final density map. Extensive experimental results on three widely-used benchmark datasets have demonstrated the effectiveness and superiority of our proposed method: (1) We achieve competitive counting performance compared to state-of-the-art methods; (2) Compared with baseline, the MAE and MSE are decreased by at least $6.3\%$ and $11.3\%$, respectively.