All-Weather Pedestrian Detection Based on Double-Stream Multispectral Network

Abstract

Recently, advanced driver assistance systems (ADAS) have attracted wide attention in pedestrian detection for using the multi-spectrum generated by multi-sensors. However, it is quite challenging for image-based sensors to perform their tasks due to instabilities such as light changes, object shading, or weather conditions. Considering all the above, based on different spectral information of RGB and thermal images, this study proposed a deep learning (DL) framework to improve the problem of confusing light sources and extract highly differentiated multimodal features through multispectral fusion. Pedestrian detection methods, including a double-stream multispectral network (DSMN), were used to extract a multispectral fusion and double-stream detector with Yolo-based (MFDs-Yolo) information. Moreover, a self-adaptive multispectral weight adjustment method improved illumination–aware network (i-IAN) for later fusion strategy, making different modes complimentary. According to the experimental results, the good performance of this detection method was demonstrated in the public dataset KAIST and the multispectral pedestrian detection dataset FLIR, and it even performed better than the most advanced method in the miss rate (MR) (IoU@0.75) evaluation system.

1. Introduction

Advanced driver assistance systems (ADAS) are important technical systems to detect the external environment and make appropriate judgments using vehicle sensors. Nowadays, it has become a trend for ADAS to develop from assisted driving into automatic driving. Most current sensors in the automatic driving market are based on RGB images, radars [1], and lidars. Among all those sensors, the most advantageous feature of image-based sensors is their low costs, which can help easily control manufacturing costs compared with self-driving vendors. Additionally, the use of image sensors can not only enhance automatic driving but also reduce the rate of traffic accidents. Moreover, the life safety of pedestrians is most vital.

As a significant topic of ADAS, pedestrian detection has received much attention along with the development of the automatic driving industry. However, pedestrian detection still faces many challenges owing to light source intensity, object shading, or other changes and conditions. In regard to automatic driving vehicles (e.g., Tesla Model 3, Toyota Camry, Chevrolet Malibu, and Honda Accord), the American automobile association conducted a pedestrian knock-down test in daytime scenarios [2]. According to the results, when the test vehicle runs at 20 miles per hour, and an adult suddenly crosses the road, there was a 40% chance of the vehicle avoiding this pedestrian; when the test vehicle drives at 20 miles per hour and suddenly encounters a child showing up between two vehicles, there was an 89% chance of the vehicle hitting the child; finally, when the test vehicle runs at 30 miles per hour, there was a 100% chance of the vehicle hitting the pedestrian. The knock-down possibility in daytime scenarios is very high. However, when driving at night at low light, self-driving vehicles still face serious problems in detecting pedestrians.

In summary, the pedestrian detection task applied in ADAS requires robust methods unaffected by environmental factors such as light source intensity and real-time detection methods. However, the mono-spectral information provided by each type of image sensor has advantages and disadvantages [3]. For example, although the visible light image sensor provides high spatial resolution and high-definition texture details, it will be affected by low lighting or bad weather without providing useful information. Thermal image sensors can distinguish the target from the background based on radiation differences. Moreover, it can retain a good image-capturing ability even in bad scenarios such as low light intensity, heavy rain, or thick smoke, as shown in Figure 1. Although different sensors can obtain multispectral information from different images detected by varying wavelengths in the same scenario, the difficulty lies in effectively using the mono-spectral information from different sensors to fully use each type of sensor.

Previous studies have suggested that it is better to use deep learning (DL) to detect pedestrians than to use manual features (traditional methods). In previous years, most DL-based methods relied on mono-spectral images captured by single sensors. Moreover, the CMOS optical image capturing system is generally used in those detection methods, which may cause issues such as low light intensity, lack of color information, and low signal-to-noise ratio. There are many studies on the positioning and classification of an anchor. For the multiscale convolutional neural network (MS-CNN) detected by ADAS, Wei et al. [4] improved the scale changes of large targets, target shading, target classification, and target positioning. However, they have not proposed relevant resolutions to solve issues encountered by image sensors in unfavorable lighting conditions such as insufficient light, solar glare, and headlamp glare. Blin et al. [5] used polarimetric features to enhance the accuracy rate of object detection if the light was insufficient. However, their datasets ignored the strong reflection by light sources from heavy rain or snow. Based on the features of thermal-like images extracted from visible light images, Kruthiventi et al. [6] could detect barely visible pedestrians under challenging light intensity; however, the method still has its limitations for only depending on visible light images. Traditional mono-spectral methods cannot be applicable under different experimental environments, experimental equipment, and environmental light sources, and images can only be well detected under a specific environment or lighting conditions

In recent years, most studies have shown that the fusion of RGB and thermal images can improve pedestrian detection performance. Based on the histogram of oriented gradient (HOG) features of enhanced thermal images, Hwang et al. [7] put forward the multispectral aggregated channel features (ACF) and a multispectral pedestrian dataset KAIST for training and testing. Their results suggested that the detection performance of the multispectral method was evidently better than that of mono-spectral methods. After [7] establishing the first large-scale multispectral pedestrian baseline dataset, many studies further focused on the optimal multispectral fusion model besides using a neural convolution network to extract features. Wagner et al. [8] proposed a multispectral pedestrian detection method based on R-CNN to discuss the advantages and disadvantages of two multispectral fusion models. According to the results, later fusion is better than early fusion. Li et al. [9] put forward a unified network integrated framework that can significantly improve the performance, which integrates the outputs of different branches (including optimal segmentation and detection tasks). Chen et al. [10] used TV minimization based on structure transfer to combine infrared images and visible images, retain infrared intensity distribution and local appearance information. However, when the pedestrian targets are not prominent in infrared images, the detection performance of the proposed method will be affected. Considering the consistency between varied modes, Zhang et al. [11] proposed a module that can integrate and refine RGB and thermal features. Except for the refining of features, it is equally important to extract highly discriminative multispectral features. Lu et al. [12] revealed the importance of those features through the channel-level cross-modality interaction attention module.

Furthermore, Zhang et al. [13] employed the pixel-level attention module to focus on the localization guidance of inter-modality and intra-modality. However, although multispectral fusion can largely improve pedestrian detection performance, it is inevitable to detect pedestrians in low-resolution images. To solve this issue, Wolpert et al. [14] raised an anchor-free framework and employed a new data enhancement method. Additionally, Nataprawira et al. [15] merged multispectral image channels, and designed a compressed YoloV3 model, sacrificing minimal accuracy and greatly improving the processing time of the model. Even though many studies have discussed modality infusion, they have not fully investigated the complementarity of single modality, let alone the situation when the representative features of pedestrians cannot be extracted, owing to excessive irrelevant background information.

In addition to modality fusion, the complementarity of modality under different lighting conditions is also worthy of investigation. Therefore, studies [16,17] have proposed different lighting perception modules to self-adaptively re-weight different modalities. The reason is that the detection performance of RGB images is slightly better than that of thermal images in daytime or in sufficient lighting; however, at night or with insufficient lighting, due to the invisible light spectrum, the detection performance of thermal images is significantly better than that of RGB images. Moreover, Zhou et al. [18] discovered that the multispectral pedestrian detection model has not only illumination modality imbalance problems, but also feature modality imbalance. Accordingly, they proposed two solutions, namely a differential modality perception fusion module and a lighting perception module. However, only using daytime or night scenarios to train illumination networks cannot deliver good results in the shadow scenarios during the night or daytime. Most notably, there are still doubts with regard to temperature labels that measure temperature conditions which cannot even be obtained from existing multispectral baseline datasets, making the training ineffective. Thus, Zhuang et al. [19] further examined temperature conditions and employed an illumination lightweight temperature perception multispectral network (IT-MN), hoping to improve the performance of model detection through temperature conditions.

In summary, the study is organized as follows: Section 2 proposes the MFDs-Yolo and i-IAN modules to detect pedestrians; Section 3 focuses on experimental results and comparison, and the final section illustrates the conclusion.

2. Methodology

In this paper, we adopted different spectral information to achieve the pedestrian detection of multi-sensor applications in ADAS. Based on the reference [20,21], we used two independent Yolo-based detectors to extract and detect features of two different spectra to employ the RGB and thermal image information. Moreover, to allocate fusion weights of two different spectra and solve the light source fusion in actual scenarios, we proposed the aware illumination network to self-adaptively allocate fusion weights for different scenarios under varied light source intensity. Finally, the main contributions of this paper are as follows: (1) an innovative multispectral method MFDs-Yolo (a multispectral fusion and double-stream detector with Yolo-based methods) redesigned based on YoloV4, with the double-stream multispectral network (DSMN), which adopts a post-fusion strategy through RGB and thermal image information; and (2) a redesigned i-IAN module based on MB-Net with a framework of DSMN, which can self-adaptively allocate fusion weights of RGB-SubNet and thermal-SubNet for scenarios of varied light source intensity besides sufficient light in daytime or limited light at night.

2.1. Proposed Double-Stream Multispectral Network (DSMN)

The overall framework of the proposed method is shown in Figure 2, which is mainly based on the extension of the YoloV4 framework. Specifically, it consists of an MFDs-Yolo and an i-IAN module. MFDs-Yolo consists of two YoloV4 double-stream architectures, which generates individual detection results through the multispectral information provided by RGB images and thermal images. i-IAN uses RGB images to know the lighting situation of the environment and adaptively adjusts the weight parameters of RGB images and thermal images by using the illumination perception method. The image weights and the output of MFDs-Yolo are fused to produce the final detection results.

2.2. A Multispectral Fusion and Double-Stream Detector with Yolo-Based Detectors (MFDs-Yolo)

To establish a real-time pedestrian detection model to carry out tasks in scenarios, the choice of a neural network architecture should consider both the accuracy and calculation speed. Inspired by YoloV4, this work raised an improved network framework using multispectral fusion and double-stream technology to detect pedestrians in different scenarios.

The framework of DSMN is made up of two Yolo-based subnetworks, including RGB-SubNet and thermal-SubNet. Based on the confidence score of classification and the offset of a bounding box for objects, independent detection results can be generated. RGB and FIR images were especially taken as inputs to perform computer vision tasks. Finally, the detection results could be obtained by using calculated fusion weights to re-weight the detection results of the two subnetworks, as shown in Figure 3.

The structure of each Yolo-based subnetwork is similar to that of YoloV4 and can be divided into three parts: backbone, neck, and head. The backbone, acting as the preliminary feature extraction, uses CSPNet–CSPDarketnet53 to add CSP, based on each residual neural network (RNN) of Darketnet53. Therefore, before the input of each RNN, the feature map will be split into two parts by the first convolution, one of which will remain intact, and the other will go through a 1 × 1 convolution layer and pooling layer. Later, the two parts will join together to be input into the next RNN. In addition, Darketnet53 has 53 layers, each containing Conv2D, batch normalization, and Mish; using CSPNet can enhance CNN’s learning ability and remain accurate while keeping lightweight and reducing calculation costs. For the neck, it consists of spatial pyramid pooling (SPP) and path aggregation network (PAN). To enrich the input information to the head, features were infused. Among them, the core size of the maximum pooling layer in SPP k = {1 × 1, 5 × 5, 9 × 9, 13 × 13} and different feature maps of pooled cores were combined as the output. Compared with single pooling layers, the SPP can increase the receptive field of the backbone, and PAN can connect feature maps through more sampling to improve the overall calculation efficiency of the model. Regarding head, consisting of multiple convolution layers, it can be used for predicting the bounding box. It outputs three sizes of tensors for connecting and calculating the anchor to obtain the position of the prediction box.

Our concept of multispectral fusion originated from the comparison of the fusion framework carried out by halfway fusion [20] and IAF R-CNN [16], including input fusion, early fusion, halfway fusion, late fusion, and score fusion. Input fusion, as the most direct fusion method, only integrates multispectral images at the input layer; early fusion is performed after the first convolution layer and can continue to train the pre-trained backbone; halfway fusion begins the fusion after the fourth convolution layer and reduces the dimensions by NIN; later fusion usually uses the last full-connected layer of multi-spectrum to fuse; and score fusion can be classified into the cascade and non-cascade designs, the former of which submits the results to the subnetwork of other modalities and carries out fusion through equal weights, while the latter carries out the re-weighting based on prediction scores and bounding box regression output by two subnetworks. Finally, based on experimental results, we employed a low-error-rate-score fusion strategy to fuse two subnetworks.

2.3. An Improved Illumination Modality Module (i-IAN)

Insufficient and confusing lighting is inevitable for actual road scenarios. Although a multispectral fusion method combing RGB and thermal images was already used, those two kinds of spectral information cannot be fully applied. To allocate different weights to MFDs-Yolo, the lighting intensity should be estimated, and thus pedestrians in confusing lighting scenarios can be detected. This work estimated the lighting intensity based on RGB images, as thermal images cannot reveal luminosity or brightness information, while RGB images can deliver better results.

As shown in Figure 4, the overall framework of i-IAN includes input and feature extraction, illumination weight, and weight fusion. In the part of input and feature extraction, this work adjusted the tensor into (56,56,3) to improve the efficiency of training or testing; a series of convolution, fully connected, and maximum pooling layers followed. The maximum pooling layer consisted of two convolution blocks and a fully connected layer, each convolution block uses 3 × 3 convolution and connects with ReLU and a 2 × 2 maximum pooling layer. Passing through two flattened and dense layers ((256,) and (2,)), the fully connected layer adds a dropout between two dense layers to slow down the fitting. At last, the softmax loss was used to obtain the illumination value iv, while the weights $w_d$ and $w_n$ were obtained through Equation (1). $w_d$ and $w_n$ represent the illumination values of the day and night, and the network is optimized by minimizing the cross entropy loss function between illumination values and true labels.

$$w=\frac{iv}{1+\alpha exp(-\frac{iv-0.5}{\beta })}, iv\in [0, 1]$$

(1)

where α and β are both learnable parameters.

In regard to the illumination weight aspect, to better allocate weights, we distributed a high weight for RGB-SubNet, and low weight for thermal-SubNet, so the final detection results can benefit from two kinds of spectra. On the contrary, when the weight of the thermal-SubNet dominates under bad lighting conditions, detection effects can be improved as the information provided by RGB was the noise. Considering all the above, the initial weights of $w_d$ and $w_n$ can be re-calculated to obtain the weight of RGB $w^{rgb}$:

$$w^{rgb}=\left(\frac{w_d-w_n}{2}\right)\times \left({\alpha }_w\times \left|w\right|+{\gamma }_w\right)+\frac{1}{2}$$

(2)

where |w| ∈ [0, 1] denotes another independent prediction branch with one neuron as the output of fully connected layers at the end; and ${\alpha }_w$ and ${\gamma }_w$ denote the learnable parameters at initial values of 1 and 0, respectively. With 0.5 as the bias, the linear regression of |w| is used to re-parameterize $w_d$ and $w_n$. When the light source is sufficient, the weight of positive values will be enlarged and $w^{rgb}$ will be increased. Conversely, when the light source is insufficient, the weight of negative values will be reduced and $w^{rgb}$ will be decreased. Moreover, due to the complementarity of RGB and thermal images, the weight of the light source was defined as 1, and the weight of the thermal $w^{thermal}$ was obtained using Equation (3):

$$w^{thermal}=1-w^{rgb}$$

(3)

For weight fusion, the final prediction results were calculated using $w^{rgb}$ and $w^{thermal}$. Each Yolo-based subnetwork can produce two outputs, including K + 1 types of confidence scores s = (s₀, …, s_K) and the offset of bounding box regression t = (t₁, …, t_K) among K types of objects. Thus, according to $s^{rgb}$ and $t^{rgb}$ of RGB-SubNet and $s^{thermal}$ and $t^{thermal}$ of thermal-SubNet, we could obtain the final detection results using Equations (4) and (5):

$$s^{final}=w^{rgb}\times s^{rgb}+w^{thermal}\times s^{thermal}$$

(4)

$$t^{final}=w^{rgb}\times t^{rgb}+w^{thermal}\times t^{thermal}$$

(5)

3. Discussion

We adopted two multispectral pedestrian baseline datasets in the experiments to evaluate the effectiveness of the proposed methods.

3.1. Dataset and Evaluation Metrics

In the experiments of this study, we used two public multispectral pedestrian baseline datasets, including KAIST [7] and FLIR [22], to evaluate the effectiveness of the proposed methods. All of the images in Figure 5 are RGB or thermal images. However, the pedestrians or objects captured by different image sensors will have different features. First of all, KAIST contains 95,328 RGB–thermal image pairs (640 $\times$ 480), which include 103,128 bounding boxes and 1182 distinct pedestrians. Because of the defects in the original training data, this work adopted the improved labeling dataset put forward by Zhang et al. [23]. There were 8963 images in the training dataset, while samples were taken every 20 frames in the testing dataset. Finally, 2252 RGB–thermal image pairs were obtained for testing, including 1455 daytime and 797 night images. For the public dataset FLIR, there were 10,228 frames of RGB–thermal image pairs (640 $\times$ 512), 9214 of which were image pairs with labeling boxes. There were five categories in FLIR, namely people (28,151), bicycles (4457), cars (46,692), dogs (240), and others (2228). A total of 8862 frames in the dataset were used for training, while the remaining 1366 frames were used for testing. However, due to the drawbacks of the original dataset, the re-aligned dataset to put forward by Zhang et al. [11] was used for the experiment.

In this experiment, log coverage miss rate (MR) [7] and mean average precision (mAP) were employed as indicators to evaluate pedestrian detection. Each negative sample sampled by MR within the range of $[{10}^{-2},{10}^0]$ was calculated by averaging the MR. The lower the MR value, the better the evaluation, while mAP was the opposite.

3.2. Implement Details and Quantitative Study

In this work, the model is realized under the PyTorch framework, in which the MFDs-Yolo module is modified based on the PyTorch version of Yolo V4; and the i-IAN module is modified based on the TensorFlow version of MB-Net. For the resolution, this research adjusted the size of an image into 416 $\times$ 416 through the double-stream interpolation method to meet the required input dimension; during the training process, the Adam optimizer was used with an initial learning rate of 1 × 10⁻³. Then, cosine annealing changed the learning rate, and 200 epochs were trained. Additionally, this paper employed focal loss [24] to train the model effectively to help the model learn semantic information. Finally, without using the pre-training model, our model was trained from scratch, and all the experiments were performed on the single NVIDIA GeForce RTX 3090 Ti GPU and Intel i7-10400 CPU.

Table 1. The effectiveness comparison of the proposed method in KAIST dataset with other methods.

Methods	MR (IoU@0.5) (%)			MR (IoU@0.75) (%)			Platform	Speed (s)
	All	Day	Night	All	Day	Night
ACF [7]	47.32	42.57	56.17	88.79	87.70	91.22	MATLAB	7.73
Proposed YOLOv3 [10]	43.25	46.99	35.84	—	—	—	1080 Ti	—
Halfway fusion [20]	25.75	24.88	26.59	81.29	78.43	86.80	TITAN X	0.43
Fusion RPN+BF [25]	18.29	19.57	16.27	72.97	68.14	81.35	MATLAB	0.80
IAF R-CNN [16]	15.73	14.55	18.26	75.50	72.34	81.12	TITAN X	0.21
IATDNN + IASS [17]	14.95	14.67	15.72	76.69	76.46	77.05	TITAN X	0.25
RFA [26]	14.61	16.78	10.21	—	—	—	TITAN X	0.08
CIAN [12]	14.12	14.77	11.13	—	—	—	1080 Ti	0.08
MSDS-RCNN [9]	11.34	10.53	12.94	70.57	67.36	79.25	TITAN X	0.22
AR-CNN [23]	9.34	9.94	8.38	64.22	57.87	76.82	1080 Ti	0.12
MBNet [18]	8.13	8.28	7.86	60.12	54.90	68.34	1080 Ti	0.07
This work	14.33	13.34	22.36	50.76	47.85	53.76	3090 Ti	0.76

and

Table 2. The mAP comparison of the proposed method in KAIST dataset with other methods.

Methods	mAP (%)
PiCA-Net [27]	65.80
R3Net [27]	70.85
tY [28]	63.00
SSTN101 [29]	73.22
MuFEm + ScoFA [30]	78.03
This work	80.30

compare results obtained from the proposed and the most advanced methods. In Table 1, MR (IoU@0.5) and MR (IoU@0.75) were adopted in all scenarios, including daytime and night, to evaluate and compare their differences. Based on the results, the evaluation of MR (IoU@0.75) obtained the best results when part of all scenarios reached 50.73%, daytime scenarios achieved 47.85%, and night scenarios fulfilled 53.76%. IoU@0.75 is closer to ground-truth, that is, its predictions were more accurate, which is nearly 10 MR lower than the best performance of the MBNet. Table 2 used mAP to evaluate, and referring to the results, the proposed method of this study delivered the best performance with an mAP of 80.30%. Moreover, the details of the mAP of each category are illustrated in

Table 3. The mAP details of the proposed method in KAIST dataset.

Category	mAP (%)	AP (%)	Precision (%)	Recall (%)	F1 Score	MR (%)
Person	80.30	93.21	92.89	86.82	0.90	4.62
People		64.29	89.47	44.74	0.60	28.94
Cyclist		83.41	88.99	74.05	0.81	11.45

, including AP, Precision, Recall, F1-score, and MR. It can be found that the AP value of people is lower than that of the other two categories. The KAIST dataset marked pedestrians in overlapping, occlusion or crowds, etc., as “People”. According to the formulas of Precision and Recall:

$$Precision=\frac{TP}{TP+FP}$$

(6)

$$Recall=\frac{TP}{TP+FN}$$

(7)

It can be inferred that the Precision of people reaches 89.47% while Recall is only 44.74%, so FP < FN, i.e., the value of those that are not people, yet detected by the model as people, is low, and the value of those that are people but were detected by the model as not people is high, so that the pedestrians who were obscured can be clearly labeled as person, that is, they can be identified as individual pedestrians. As a result, people’s FN increased, and AP was underestimated compared to the other two categories.

Table 4. The AP and mAP comparison of FLIR dataset and other methods.

Methods	mAP (%)	AP (%)			MR (%)
		Person	Bicycle	Car	Person	Bicycle	Car
Baseline	53.97	54.69	39.66	67.57	—	—	—
MMTOD-CG [31]	61.40	63.31	50.26	70.63	—	—	—
MMTOD-UNIT [31]	61.54	64.47	49.43	70.72	—	—	—
TD(T, T) [32]	71.40	75.50	51.90	86.90	—	—	—
BU(AT, T) [32]	73.10	76.10	56.10	87.00	—	—	—
BU(LT, T) [32]	73.20	75.60	57.40	86.50	—	—	—
LGADet [33]	—	74.52	—	—	28.43	—	—
This work	70.66	70.13	65.01	77.03	21.53	22.43	17.00

compares the results obtained from the proposed method and other most advanced methods. Except for the results of mAP, it also revealed the results of AP (%) and MR (%) for three categories (person, bicycle, and car). Moreover, good results have been realized in recent studies. Bicycle performs better than person and car, which have relatively more data. This result is based on the feature of dataset re-alignment. Taking the thermal image as the benchmark, the RGB image is aligned toward the thermal image. If the RGB image is smaller than the original image size after alignment, the edge will be filled with 0, which will affect the determinant of i-IAN and mislead the weight distribution of multispectral double-stream network. In contrast, the images with the bicycle category do not need to be filled with 0 after alignment, i.e., there is no black frame on the edges of the images, so bicycle has the best performance among the three categories. It can be deduced that if RGB images are used as the benchmark and thermal images are aligned to RGB images, i-IAN can obtain the complete illumination information of the images and is expected to achieve better results.

Figure 6 illustrates the output results of the proposed method in KAIST and FLIR datasets. The upper and lower corners of each image show the detection results of RGB and thermal images, respectively.

4. Conclusions

Due to instabilities such as lighting changes and weather conditions, image sensors still face challenges in all-weather multispectral pedestrian detection. Considering all the above, this study proposed a new DL network framework to carry out multispectral pedestrian detection. We used multispectral fusion technology combined with an illumination perception method to adjust multispectral image information adaptively to compensate for the shortcomings of mono-spectral methods. Based on the Yolo structure, a double-stream network of MFDs-Yolo suitable for extracting multispectral information was redesigned, and a self-adaptive modality weight adjustment method i-IAN was used in the later fusion strategy of the framework. It can accurately identify road objects ahead in the harsh environment of low light at night, while retaining abundant color information during the day when light is sufficient. Further research plans include exploring the application of our DL network framework in real-time pedestrian detection systems for autonomous vehicles. To address the limitation of computing speed, cloud computing can play a crucial role. By offloading the computational burden to powerful cloud-based infrastructure, we can enhance the overall system efficiency and enable the efficient processing of large amounts of data and complex algorithms. The availability of future 5G technology and the emergence of the internet of vehicle (IoV) ecosystem will be instrumental in supporting the seamless integration of cloud computing into pedestrian detection systems. However, considering the complementarity of modalities, highly discriminative multispectral features, and confusing illumination, this work employed two public multispectral pedestrian datasets, including KAIST and FLIR, and achieved a good detection accuracy along with a low MR, the results of which can be applied in self-driving image systems. Compared with the state-of-the-art detection methods in this study, MR was reduced by 9.36% and 6.9%, respectively.