Joint Semantic Deep Learning Algorithm for Object Detection under Foggy Road Conditions

Abstract

Current mainstream deep learning methods for object detection are generally trained on high-quality datasets, which might have inferior performances under bad weather conditions. In the paper, a joint semantic deep learning algorithm is proposed to address object detection under foggy road conditions, which is constructed by embedding three attention modules and a 4-layer UNet multi-scale decoding module in the feature extraction module of the backbone network Faster RCNN. The algorithm differs from other object detection methods in that it is designed to solve low- and high-level joint tasks, including dehazing and object detection through end-to-end training. Furthermore, the location of the fog is learned by these attention modules to assist image recovery, the image quality is recovered by UNet decoding module for dehazing, and then the feature representations of the original image and the recovered image are fused and fed into the FPN (Feature Pyramid Network) module to achieve joint semantic learning. The joint semantic features are leveraged to push the subsequent network modules ability, and therefore make the proposed algorithm work better for the object detection task under foggy conditions in the real world. Moreover, this method and Faster RCNN have the same testing time due to the weight sharing in the feature extraction module. Extensive experiments confirm that the average accuracy of our algorithm outperforms the typical object detection algorithms and the state-of-the-art joint low- and high-level tasks algorithms for the object detection of seven kinds of objects on road traffics under normal weather or foggy conditions.

1. Introduction

Fog is a common weather occurrence and can severely damage the image quality captured by the outdoor equipment. There has been a large body of literature on object detection under inclement weather conditions, which include one-stage and combination approaches [1,2]. For example, one-stage approaches include the domain-based adaptive target detection algorithm under foggy conditions [3,4,5,6,7,8]; however, these methods also have limitations, for example, their performance is not guaranteed when the training and test data sets are vastly different, and these methods failed to take advantage of the image recovery potential information while combination ones include single image fog removal [9,10,11,12] and combined object detection [13,14,15,16] algorithms. For combination approaches specifically, their first step aims at fog removal for a single image [12], which is the algorithm of low-level vision tasks in order to improve the performance of subsequent high-level vision tasks [17]; their second step aims at object detection, which in turn is a metric for the performance of the low-level task. Recently, deep learning methods for single image fog removal have achieved superior image quality; however, due to the network design features of subsequent high-level tasks not being jointly taken into account, combination models are often cumbersome and computationally inefficient. To overcome these problems, in this paper, we propose the joint semantic deep learning algorithm for object detection under foggy road conditions, referred to as $JSFR$.

The deep neural network of JSFR is constructed by embedding attention mechanism modules (AM) and the UNet decoding modules into the same scale corresponding submodules of the feature extraction module of object detection algorithm $FasterRCNN$. The main idea of our proposed algorithm can be shown intuitionally in Figure 1. The novel algorithm improves the subsequent target detection performance through the joint representation of features before and after image recovery and is applied to detect seven types of objects on road traffics under foggy conditions.

The main contributions are as follows:

• The algorithm is proposed for the joint tasks of low-level dehazing and high-level target detection. On the one hand, the two networks are combined to reduce the parameter scale and to improve the test time. On the other hand, the joint feature representations can help improve the robustness of road target detection under foggy or fog-free conditions. In addition, the joint network architecture is beneficial for improving the performances of both low-level and high-level tasks;
• The embedded attention mechanism module in the backbone network is conducive to capturing the position information of the fog at any time. Since fog is different from other noises, fog has the characteristics of fast drift and rapid position change, so determining the position information of fog before dehazing is beneficial for restoring image quality;
• Comprehensive experiments confirm the effectiveness of the proposed algorithm. Regardless of normal or foggy weather, and irrespective of synthetic or real data, the test results show the superiority of our algorithm. Figure 2 shows an example of its detection effect in the real world.

The remaining sections are arranged as follows: Section 2 reviews the SOTA object detection methods and single image fog removal methods; Section 3 gives an overview and introduces the formulation and optimization of our method; Comprehensive experiments are given in Section 4; and Section 5 concludes.

2. Related Work

In this section, deep neural network frameworks for object detection and for single image fog removal are reviewed in detail.

2.1. Deep Neural Network Based Object Detection Methods

Object detection is a fundamental task in computer vision. Many researchers developed various deep neural networks to improve the performance of object detection. These object detection methods can in general fall into two categories [18]: regression-based [13,16,19,20,21] and region-based suggestion methods [14,22,23].

Regression-based methods include: Yolo family [16,19,20], SSD [21] and RetinaNet [13], or their variants. The key idea is to use CNN modules to extract the images feature maps and then predict bounding box coordinates and category probabilities. For example, Yolo is first proposed by Redmon et al. [19] in 2016, which is constructed by 24 convolutional layers and 2 fully connected layers to achieve real-time detecting through end-to-end training. In the Yolo family, Yolo V2 [20] and Yolo V3 [16] obtain competitive detection performance by modifying the backbone network and adding a multi-scale prediction network module. Ref. [21] introduces SSD, which adds an auxiliary module to extract multi-scale feature maps and achieve object detection of different scales. RetinaNet achieves higher detection accuracy by introducing a focal loss function to balance the small sample category and reduce the running time in all one-stage object detection methods. Although the detection speed of the one-stage target detection algorithms is much higher than that of the two-stage target detection algorithms, the detection accuracy is much more inferior than the latter since the positioning of the target is not very accurate.

Region-based suggestion methods are two-stage object detection methods, including the R-CNN family [14,22,23]: R-CNN [22], Fast RCNN [23] and Faster RCNN [14], and R-FNN [24] and their variants. The key component of these combination methods is that a region-based suggestion module is proposed. In other words, the corresponding regions are extracted from the feature maps and then input into the ROI pooling module to generate fixed-length feature vectors for the classification and further regression of bbox (Bounding Box). It is worth mentioning that Faster RCNN has many advantages: it achieves the highest precision in all popular object detection methods; it can solve multi-scale and small target problems; it is easy to migrate to other categories and problems. So, Faster RCNN was chosen as the backbone network in this paper.

2.2. Object Detection under Foggy Weather Conditions

To address object detection under foggy weather, many researchers have constructed various deep learning methods from domain-adaptive or image recovery viewpoints, respectively.

Domain adaptive-based object detection methods under foggy weather conditions include [3,4,5,6,8] etc. In particular, Ref. [25] proposes a multi-level domain adaptive Faster RCNN, which uses different domain classifiers to supervise multi-scale feature alignment and improve the recognition ability by increasing domain classifiers. Ref. [26] proposes a robust multi-scale adversarial learning method for cross-domain object detection, which reduces image-level domain differences by multiple expansion convolution kernels, and reduces the influence of negative migration by excluding images and instances with low transfer ability. Ref. [27] proposes a stacked complementary loss method to achieve domain adaptation, which learns comprehensive discerning representations by detaching the gradient into several auxiliary losses in different network stages. Domain adaptation is a special case of transfer learning (TL). The idea is to project the features of different domains into the same feature space so that the target domain training can be enhanced by using data from other domains. It is a pity that TL has difficulty working well when there is a huge difference between two domains.

Another line of research focuses on image recovery. These strategies consist of embedding an additional image recovery module into the object detection backbone network [7,28], or recovering images first and then detecting objects for recovered images [11,12,29,30,31]. Ref. [28] proposes a dual subnet network (DSNet) to jointly learn three tasks including visibility enhancement, object classification, and object localization through embedding an image recovery module into RetinaNet. In [7], Image-Adaptive YOLO (IA-YOLO) is proposed, which configures a small convolutional neural network and a differentiable image processing module before Yolo V3 to balance two tasks, i.e., image enhancement and object detection, under both foggy and low-light scenarios. Ref. [17] improves the detection performance by concatenating the dehazing module AOD-Net (All-in-One Dehazing Network) with the object detection network Faster RCNN by end-to-end training. Ref. [32] proposes a lightweight dehazing network PDR-Net.

The above-mentioned two groups of methods have greatly advanced the research of joint tasks including the low levels and high levels. However, the former often overlooks the image recovery potential information while the latter does not emphasize object detection performance and takes object detection as a task-driven evaluation index for image restoration.

Then a natural question arises about how to organically integrate the defogging and object detection modules to build an end-to-end joint tasks object detection algorithm under foggy weather. Can we design a network to complete the process of low-level image dehazing and high-level target recognition at the same system? In other words, can the network dealing with low-level vision tasks and the network dealing with high-level vision tasks be organically combined? Could it be possible not only to improve the generalization of subsequent high-level tasks, but also to save energy? To this end, the JSFR algorithm was proposed and applied to the detection of seven objects in road traffic scenarios.

3. Proposed Method

In this section, an overview of JSFR is first introduced in brief, and then the attention module and the parameters flow-chart of the noval fusion network are described in detail, and finally the formulation and the total loss function of JSFR are interpreted.

3.1. Framework Overview

For this paper, we designed JSFR to detect objects in road traffic under foggy weather conditions, as shown in Figure 3. The whole framework consists of four main modules: the image feature extraction module, the haze removal module, the multi-scale feature fusion module and the detection module. Faster RCNN embeds the four-level decoder of UNet [33] and the attention module [34] into the last, corresponding to the same scale sub-blocks of the feature extraction module. The haze removal module and the feature extraction module share weights to avoid additional computational burden. Finally, the four feature maps and the corresponding recovered feature maps are summed together and then input to the feature pyramid network (FPN ) [35] to learn the multi-scale joint semantic representation, and then the output results are input to the detection module separately to improve the target detection accuracy under hazy weather conditions. The attention module is shown in Figure 4 and the setup of the fusion sub-module is shown in Figure 5.

3.2. Attention Module (AM) and Parameters Flow-Chart

Considering that different channel features of the image degraded by the fog represent different concentrations and different locations of the fog, AM modules [34] are embedded between the feature extraction module and UNet decoder, which is used to learn the foggy distribution. AM consists of channel and spatial attention, see Figure 4. The channel attention pays more attention to the concentration information of the fog in the input image, while the spatial attention module mainly focuses on the location information of the fog. The fog information can be captured adequately by the combination of the two modules.

3.3. The Parameters Flow-Chart of the Novel Fusion Network

In our model, the UNet encoder part is not designed separately, but shares the feature extraction layers with Faster RCNN, viz. ResNet50. We divide the 16 residual blocks of ResNet50 into four residual block stages, denoted as C2, C3, C4 and C5, respectively. The output of each residual block is embedded into the attention module separately and passed to the UNet Decoder part, and then its output is cascaded with the results of the feature extraction layers to fuse the low-level detail features and high-level semantic features to enrich the single image information. The whole parameters flow-chart is shown in Figure 5, which is consistent with singl_u.py in our all codes.

3.4. Formalization and Loss Function

Let input image be $\mathcal{O}\in \mathcal{R}$${}^{w\times h\times n}$, which is a tensor, where w and h are the length and width of the respective image, and n is the number of the channel. $f_1\left(\mathcal{O}\right)$, $f_2\left(\mathcal{O}\right)$, $f_3\left(\mathcal{O}\right)$, $f_4\left(\mathcal{O}\right)$ are obtained from the last four feature extraction sub-blocks of Faster RCNN. Secondly, $f_1\left(\mathcal{O}\right)$, $f_2\left(\mathcal{O}\right)$ and $f_3\left(\mathcal{O}\right)$ are input into the output of the corresponding $AM$ module and we obtain $h\left(f_1\left(\mathcal{O}\right)\right)$, $h\left(f_2\left(\mathcal{O}\right)\right)$, and $h\left(f_3\left(\mathcal{O}\right)\right)$, respectively. Then, $f_4\left(\mathcal{O}\right)$ is fed into the first layer of the UNet decoder and outputs $u\left(f_4\left(\mathcal{O}\right)\right)$. Next, $u\left(f_4\left(\mathcal{O}\right)\right)$ and $h\left(f_3\left(\mathcal{O}\right)\right)$ are cascaded to pass the second layer of the UNet decoder, ${\mathcal{B}}_1=u(h\left(f_3\left(\mathcal{O}\right)\right);u\left(f_4\left(\mathcal{O}\right)\right))$ is obtained. Similarly, we obtained successively ${\mathcal{B}}_2=u(h\left(f_2\left(\mathcal{O}\right)\right);{\mathcal{B}}_1)$; $\widehat{\mathcal{B}}$ = $u(h\left(f_2\left(\mathcal{O}\right)\right);{\mathcal{B}}_2)$. So, the mean squared error ($MSE$ loss) is used as one fine-tuning objective function $L_{def}$ for recovering the quality of the foggy image, and $L_{def}$ is operated by the equation:

$$L_{def}=\frac{1}{N}\sum _{i=1}^N{∥\widehat{B}-B∥}^2,$$

(1)

where N is the number of foggy images.

Then, the element-wise sum of ${\mathcal{B}}_1$ and $f_3\left(\mathcal{O}\right)$, ${\mathcal{B}}_2$ and $f_2\left(\mathcal{O}\right)$, $\widehat{\mathcal{B}}$ and $f_1\left(\mathcal{O}\right)$ are sent to the corresponding scale sub-module of FPN, respectively. On the one hand, they are fused layer by layer from bottom to top in the FPN module and are outputted into the RPN network to generate regression detection boxes, and the detection boxes are classified into two categories (positive and negative). Therefore, the loss function of the RPN network is $L_{rpn}$:

$$\begin{array}{cc}\hfill L_{rpn}& =-\frac{1}{N_{cls}}{\sum }_i^{}{log}_{}\left[{p_i}^{*}{p}_i+\left(1-p_i^{*}\right)\left(1-p_i\right)\right]+\lambda \frac{1}{N_{reg}}{\sum }_i^{}{p}_i^{*}{\mathcal{L}}_{1smooth}\left(t_i-t_i^{*}\right)\hfill \\ \hfill \mathrm{where},\\ \hfill {\mathcal{L}}_{1smooth}& =\left\{\begin{array}{cc}0.5x^2\hfill & \mathrm{if}\left|x\right|<1\hfill \\ \left|x\right|-0.5\hfill & otherwise,\hfill \end{array}\right.\hfill \end{array}$$

(2)

where $t_i=\left\{t_x,t_y,t_w,t_h\right\}$ is a vector representing the four parameterized coordinates of the predicted bbox; $t_i^{*}$ is the predicted vector’ coordinate; $p_i$ is the probability of the predicted foreground and $p_i^{*}$ is the probability of the predicted background; $N_{cls}$ is the number of all objects.

On the other hand, the fused feature maps through $FPN$ and the proposal’s outputs through $RPN$ are first added element-wise, and then sent into the $ROI$ pooling layer. Next, these feature maps with $ROI$ are fed into the subsequent fully connected layers, whose outputs are employed to classify seven objects and to estimate these objects’ bboxes. Toward this purpose, we use the $L_{rcnn}$ as a loss function as follows:

$$L_{rcnn}=\frac{1}{N_{cls}}\sum _{j=1}^M{L}_{cls}(p_j,p_j^{*})+\lambda \frac{1}{N_{reg}}\sum _{j=1}^M{p}_j^{*}{L}_{reg}(t_j,t_j^{*})$$

(3)

$$L=\alpha L_{def}+L_{rpn}+L_{rcnn},$$

(4)

where $t_j$ is the four coordinates of the label bbox of each object; $t_j^{*}$ is the coordinate of its predicted bbox; $M$ is the number of all objects, and $M$ equals 7 in this paper; and $L_{reg}(t_j,t_j^{*})$ is the ${\mathcal{L}}_{1smooth}$ function. $p_j$ represents the probability of a label class, $p_j^{*}$ represents the predicted probability of each class, and $L_{cls}(p_j,p_j^{*})$ is the cross-entropy loss. The final network output is denoted as $\widehat{D}$. The whole network is trained by L shown in Equation (4). It is noted that $\alpha \in \left[0,1\right]$ is a hyperparameter to be fine-tuned to improve the performance of the object detection by recovering image quality. In ablation experiments, we set $\alpha$ equal to 1; in this way, the mAP of the proposed method achieves the highest value. See Section 4.4 for details.

4. Experimental Result

4.1. Experimental Setup

4.1.1. Implementation Details

To be fair, all algorithms were trained on the Foggy Cityscapes training subset again by end-to-end, and were tested on the synthetic and real images with fog or no-fog. Foggy Cityscapes was divided into a training subset, a validation subset and a testing subset at the ratio of 8:1:1. During training, we let the batchsize, epoch, confidence threshold be 2, 0.5, and 50, respectively. The initial learning rate was set to $5\times {10}^{-4}$ and descent mode to segmented constant decay. All experiments were run on Pytorch with an NVIDIA GeForce GTX3090.

4.1.2. Evaluation Metric

Precision [36], IoU [19], Recall [36], AP [37], and mAP [28] were used to evaluate the object detection performance for our experimental results, whose expressions are reviewed in brief as follows:

$$IoU=\frac{A\cap B}{A\cup B},$$

(5)

where A is the label of bbox and B is the predicted bbox. For each object, when the intersection over union ratio IoU is greater than 0.5, the detection is considered correct.

Additionally, precision, recall and AP are defined as:

$$Precision=\frac{TP}{TP+FP}$$

(6)

$$Recall=\frac{TP}{TP+FN}$$

(7)

$$AP={\int }_0^{1}P\left(R\right)dR,$$

(8)

where TP, FP, and FN are true positives, false positives, and false negatives, respectively, and $P/R\in [0,1]$ is Precision/Recall.

Finally, mAP is defined as

$$mAP=\frac{1}{N}\sum _{i=1}^{N}A{P}_i,$$

(9)

which is the average AP value over all object classes; N is the number of categories.

4.2. Datasets

Although there existed a few datasets for object detection under inclement weather, Foggy Cityscapes [38] was the synthetic dataset, while Foggy Driving Dataset [38], RTTS [39] and $FoggyZurichDataset$ [40] were the real datasets. In this paper, Foggy Cityscapes was used to train these competing methods; Foggy Cityscapes, Foggy Driving Dataset, and RTTS were used to test these methods.

Foggy Cityscapes Dataset.Foggy Cityscapes contains eight categories, namely person, rider, car, truck, bus, train, motorcycle, and bicycle. As we were studying bus road traffic situations, we re-constructed the Foggy Cityscapes with seven categories composed of person, rider, car, truck, bus, motorcycle, and bicycle. In addition, considering that Foggy Cityscapes is based on Cityscapes [41], we could use images from Cityscapes as the corresponding ground truth (or label) from Cityscapes.

Real Datasets. The Foggy Driving Dataset [38] consists of 101 color images depicting real-world foggy driving scenes, which are used to test the performance of object detection in foggy weather. It consists of 509 vehicle instances (cars, trucks, buses, trains, motorcycles, and bicycles) and 290 human instances (people and riders), which are used for testing generalization. RTTS is also a real-world unpaired foggy image, which is a subset of RESIDE [39]. The format of RTTS is the same as that of VOC2007, and there exist five classes (cars, bicycles, motorcycles, people and buses) of objects with labels for object detection.

4.3. Experiment and Analysis

In this section, we compare the proposed algorithm JSFR with the popular object detection algorithms, the combination approaches, and the domain adaptive approaches by testing on synthetic or real images with and without fog.

4.3.1. Synthetic Foggy Image

To discuss the object detection performance under foggy weather, a qualitative comparison between our proposed JSFR with popular object detection methods Faster RCNN [14], RetinaNet [14], Yolo V3 [16], Efficient-Det [15] is shown in Figure 6 and Figure 7, and quantitative evaluations can be seen in

Table 1. Comparison of AP and mAP between JSFR and object detection methods including Faster RCNN, RetinaNet, Yolo V3, and Efficient-Det.

Category	Faster RCNN	RetinaNet	Yolo V3	Efficient-Det	Ours
bicycle	0.36	0.28	0.36	0.23	0.49
bus	0.41	0.38	0.38	0.31	0.51
car	0.62	0.43	0.58	0.38	0.74
motorcycle	0.39	0.31	0.33	0.26	0.43
person	0.38	0.25	0.39	0.21	0.58
rider	0.44	0.32	0.43	0.26	0.56
truck	0.14	0.21	0.23	0.19	0.41
mAP	39.14%	31.14%	38.57%	37.86%	53.14%

. Figure 6 and Figure 7 present scenes with sparse and dense fog, respectively. Regarding Figure 6, our algorithm recognizes seven vehicles, one rider and one bicycle. However, all others did not recognize the two distant vehicles. In addition, the confidence of all recognized objects by JSFR is not less than that of the others. While the detected object is a small target and in a dense foggy scene, as in Figure 7, our algorithm and Faster RCNN can identify the vehicles in the distance, but the confidence in recognized objects by JSFR is 0.99, which is higher than that of Faster RCNN. The quantitative results show that JSFR outperforms by up to 23%, 31.14%, 23.6% and 36% higher than Faster RCNN, RetinaNet, Yolo V3, and Efficient-Det, as shown in in Table 1.

Figure 8 and Figure 9 and

Table 2. Comparison of AP and mAP between JSFR, combination methods including AODNet+Faster RCNN, FFANet + Faster RCNN, PSDNet + Faster RCNN, and domain adaptive methods including DA and ATF.

Category	AODNet	FFANet	PSDNet	DA	ATF	Ours
bicycle	0.27	0.24	0.32	0.45	0.39	0.49
bus	0.41	0.36	0.31	0.23	0.48	0.51
car	0.61	0.59	0.59	0.57	0.51	0.74
motorcycle	0.33	0.23	0.39	0.23	0.34	0.43
person	0.29	0.33	0.37	0.39	0.38	0.58
rider	0.38	0.37	0.45	0.46	0.48	0.56
truck	0.26	0.17	0.28	0.07	0.26	0.41
mAP	36.43%	32.71%	38.71%	32.85%	40.57%	53.14%

compare the test results between JSFR and combinations such as AODNet [17] + Faster RCNN, FFANet [42] + Faster RCNN, PSDNet [32] + Faster RCNN, and domain adaptive based methods such as DA [6] and ATF [43]. Under the scene with sparse fog (refer to Figure 8) JSFR and ATF identified more targets than others, but there is still one person and a car not detected by ATF in Figure 8g. In the scene with dense fog, only JSFR recognized the small objects such as vehicles, while the others did not work on them; the confidence scores of all objects recognized by our method are higher than the corresponding scores of other methods (refer to Figure 9). From Table 2, we can see that the mAP of JSFR is up to 53.14%, which is higher than that of the others.

From Figure 6, Figure 7, Figure 8 and Figure 9 and Table 1 and Table 2, we summarize that the object detection performances of our algorithm, and domain adaptive and combined algorithms are all higher than those of the popular object detection algorithms under foggy weather, and our proposed method outperforms the others. So, we conclude that it is very necessary for the subsequent high-level object detection task that a sub-block is embedded into the whole object detection network to recover the quality from images degraded by fog.

4.3.2. Real Foggy Image

Figure 10 and Figure 11 show the test results in real Foggy Driving Datasets. It can be concluded that the generalization of JSFR is better than that of others. For example, Figure 10b shows that our JSFR algorithm can correctly identify six cars, a rider, and a bicycle. However, Yolo V3 can recognize six cars in Figure 10d, in which a car is mis-predicted as a truck and a bicycle is not detected; Efficient-Det detected only three cars; only four cars were detected by RetinaNet. Referring to Figure 11, our algorithm detected a car at a distance in dense fog. The test results on RTTS are shown in Figure 12 and Figure 13. In Figure 12, JSFR, RetinaNet, and Yolo V3 detected the person with different confidence levels of 0.99, 0.97, and 0.52, respectively, so it is evident that our algorithm has the highest confidence. As seen in Figure 13, our algorithm had the ability to detect distant people which were not detected by other methods, and all confidence scores were also the highest.

4.3.3. Synthetic Image without Fog

In order to clarify that our method, JSFR, is also able to detect objects in normal weather, we performed a qualitative comparison and a quantitative evaluation by mAP between JSFR and Faster RCNN, RetinaNet, Yolo V3, Efficient-Det on Cityscapes [41].

As can be seen from Figure 14, JSFR can detect the person in the chair in normal weather, while the others cannot do it, and the confidence of all recognized targets is also the highest.

Table 3. Comparison of AP and mAP between JSFR and object detection methods including Faster RCNN, RetinaNet, Yolo V3, and Efficient-Det on Cityscapes.

Category	Faster RCNN	RetinaNet	Yolo V3	Efficient-Det	Ours
bicycle	0.31	0.24	0.33	0.24	0.42
bus	0.29	0.27	0.27	0.32	0.26
car	0.57	0.43	0.62	0.50	0.73
motorcycle	0.29	0.20	0.40	0.25	0.31
person	0.33	0.19	0.34	0.16	0.54
rider	0.35	0.17	0.34	0.12	0.53
truck	0.18	0.17	0.15	0.20	0.25
mAP	33.14%	28.86%	35.00%	25.57%	43.43%

shows that the mAP of JSFR is higher than that of the other models, being 10.29%, 19.57%, 8.43% and 17.86% higher than Faster RCNN, RetinaNet, Yolo V3 and Efficient-Det, respectively.

4.4. Ablation Study

To verify the fine-tuning effect of the restored image module, we conducted an ablation experiment for hyperparameter $\alpha$. Let $\alpha$ be 0, 1.0, and 0.5, respectively; we obtained four models such as Faster RCNN, JSFR1, JSFR2, and JSFR, which were tested on Foggy Cityscapes. From

Table 4. The ablation experiments for hyperparameter $\alpha$.

$\mathit{\alpha }$	Model	mAP
0	$Faster$$RCNN$	39.14%
0.1	$JSFR1$	50.39%
0.5	$JSFR2$	52.28%
1.0	$JSFR$	53.14%

it can be seen that the mAP values are the lowest if $\alpha =0$ and the highest if $\alpha =1$. So, we selected $\alpha$ equal to 1 in the final model, JSFR1.

4.5. Inference Time

We recorded the testing times of all models tested on Foggy Cityscapes, RTTS, Foggy Driving Dataset and Cityscapes in

Table 5. Testing speeds of JSFR, combination methods including AODNet+Faster RCNN, FFANet + Faster RCNN, and PSDNet + Faster RCNN, domain adaptive methods including DA and ATF, and Faster RCNN on the four datasets with GPU.

		Dataset	Foggy Cityscapes	RTTS	Foggy Driving Dataset	Cityscapes
	Speed(s)
Algorithm
AODNet + Faster RCNN			2.68	2.55	2.54	2.68
FFANet + Faster RCNN			6.22	6.31	6.36	6.22
PSDNet + Faster RCNN			3.45	3.22	3.27	3.45
DA			2.73	2.66	2.65	2.73
ATF			2.71	2.56	2.54	2.72
Faster RCNN			1.66	1.56	1.56	1.66
JSFR			1.69	1.54	1.55	1.69

. An image from the four datasets can be tested in about 1.6 s by JSFR on a single NVIDIA GeForce GTX 3090 GPU. Our testing time is shorter than that of the combination and domain adaptive methods, while it is comparable to the time of the Faster RCNN. In other words, JSFR still maintains the testing speed of Faster RCNN, even though the UNet decoder module is embedded into Faster RCNN.

5. Conclusions

In this paper, JSFR is proposed to address multiple object detection in bus traffic road scene under foggy weather, which is designed to be a deep neural network by embedding UNet decoder and attention mechanism sub-modules into Faster RCNN for joint low-level and high-level task. Though UNet decoder achieves fog removal and recove the degraded image quality, its running time is still saved because of sharing parameters of the feature extracted module. The joint semantic representation is learned from recovered image and the image with fog by FPN, thus the fused information is leveraged to improve the the target detection performance. Comprehensive experiments confirm that JSFR outperforms other methods for target detection on synthetic and real image in both foggy and normal conditions. In the future, we will further improve the accuracy of the algorithm and reduce network parameters to meet the needs of landing implementation [44]. Secondly, we will continue to deeply study the application scenarios such as video detection of bus traffic roads and vehicle or pedestrian tracking. Thirdly, we will consider fusing overlap functions and fuzzy (rough) sets (see [45,46,47,48,49]) to design fog-attention-module. To provide convenience for researchers, all codes will be available at http://github.com/mendy-2013.