Semi-Supervised Object Detection with Multi-Scale Regularization and Bounding Box Re-Prediction

Abstract

Semi-supervised object detection has become a hot topic in recent years, but there are still some challenges regarding false detection, duplicate detection, and inaccurate localization. This paper presents a semi-supervised object detection method with multi-scale regularization and bounding box re-prediction. Specifically, to improve the generalization of the two-stage object detector and to make consistent predictions related to the image and its down-sampled counterpart, a novel multi-scale regularization loss is proposed for the region proposal network and the region-of-interest head. Then, in addition to using the classification probabilities of the pseudo-labels to exploit the unlabeled data, this paper proposes a novel bounding box re-prediction strategy to re-predict the bounding boxes of the pseudo-labels in the unlabeled images and select the pseudo-labels with reliable bounding boxes (location coordinates) to improve the model’s localization accuracy based on its unsupervised localization loss. Experiments on the public MS COCO and Pascal VOC show that our proposed method achieves a competitive detection performance compared to other state-of-the-art methods. Furthermore, our method offers a multi-scale regularization strategy and a reliably located pseudo-label screening strategy, both of which facilitate the development of semi-supervised object detection techniques and boost the object detection performance in autonomous driving, industrial inspection, and agriculture automation.

1. Introduction

Object detection is used to identify and locate objects of interest (such as humans, animals, vehicles, and other objects) in images or videos. In recent years, deep learning-based object detection methods have been rapidly developed. However, the main deep learning-based object detection methods [1,2,3,4,5,6,7] are supervised and heavily rely on large-scale manually annotated datasets, such as MS COCO. When the dataset samples are insufficient, the model generalization is compromised, resulting in lower detection accuracy. On the other hand, the manual annotation of a large-scale dataset is highly labor-intensive and expensive. Therefore, semi-supervised learning [8,9,10,11,12,13,14,15,16], which involves training with a small amount of labeled data and a large amount of unlabeled data, is being investigated by more and more researchers. By incorporating both labeled and unlabeled data, semi-supervised learning can exploit the rich information in the unlabeled data and generate a more robust and accurate model, reducing its dependence on a large amount of labeled data.

Semi-supervised object detection applies semi-supervised learning to object detection, exploiting the information from unlabeled data. The semi-supervised object detection methods mainly fall into two categories: the methods based on consistency regularization [17,18,19,20,21] and the methods based on pseudo-labels [22,23,24,25,26,27,28,29,30,31,32,33]. The former uses consistency regularization to constrain the model to obtain the same prediction results for a given image and its perturbed counterpart. The latter, as a mainstream version of the semi-supervised object detection method, first utilizes a teacher model to predict pseudo-labels of unlabeled data and then employs the labeled data and unlabeled data to train a student model. During training, the teacher model’s parameters are updated using the Exponential Moving Average [34] (EMA) based on the student model’s parameters.

In recent years, many researchers have explored semi-supervised object detection methods. Jeong et al. [17] proposed a consistency loss to produce the same predictions for an image and its horizontally flipped counterpart. Zhou et al. [18] replaced the sparse pseudo-boxes with dense prediction as a united and straightforward form of the pseudo-label. Guo et al. [19] proposed Scale-Equivalent Distillation (SED) to alleviate the noise problem that arises from the false negative samples and inaccurate bounding box regression. Li et al. [20] proposed Multi-view Scale-invariant Learning (MSL) with mechanisms of both label- and feature-level consistency to achieve feature consistency by aligning the shifted feature pyramids in two varied scaled images. Miyato et al. [21] proposed a new regularization method based on virtual adversarial loss, which is a new measure of the local smoothness of the conditional label distribution. Sohn et al. [22] presented a self-training and augmentation-driven consistency regularization framework, which trained a detector with a limited number of annotated samples and generated pseudo-labels from unlabeled samples. To further improve the quality of pseudo-labels, Zhou et al. [23] proposed the Instant-Teaching framework, which employed a co-rectify method to rectify erroneous predictions made by two structurally identical but independently trained models. Tang et al. [24] proposed the Humble Teacher framework that utilized the EMA method and soft-label mechanism for improving the accuracy of semi-supervised models. Li et al. [25] proposed a novel self-correcting pseudo-label module and pseudo-label-guided copy–paste technology to generate more reliable predictions and enhance instance representation learning within diverse complex scenes. Liu et al. [26] introduced the Unbiased Teacher framework to employ a real-time pseudo-label generation method, which addressed the class-imbalance issue in object detection using focal loss [27] and the EMA. Xu et al. [28] proposed the Soft Teacher framework, which introduced classification loss to address the imbalanced foreground–background samples and utilized a bounding box discrepancy filter to fully leverage the bounding box regression information from the unlabeled data. Feng et al. [29] proposed a semi-supervised object detection method based on position confidence weighting and introduced a Location-Aware Head (LAH) to reduce the pseudo-label noise in the unlabeled data. Kim et al. [30] introduced a simple yet effective data augmentation method, Mix/UnMix (MUM), to unmix feature tiles from the mixed image tiles under the semi-supervised object detection framework. Cai et al. [31] proposed a semi-supervised object detection method based on teacher–student models with strong–weak heads. The strong and weak heads of the teacher model solved the quality measurement problem of pseudo-label localization. Unbiased Teacher v2 [32] and Dense Learning (DSL) [33] tried to combine semi-supervised learning methods with anchor-free detectors. The former method removed the misleading impact on bounding box regression in pseudo-labels by estimating the corresponding uncertainty, while the latter method proposed an adaptive filtering strategy to assign dense pseudo-labels to each pixel. Additionallly, the latter introduced an integrated teacher model to improve the stability and quality of pseudo-labels.

Although pseudo-label-based semi-supervised object detection methods have led to the achievement of numerous successes in recent years, these approaches ignore the importance of regularization for semi-supervised detectors. These methods have a poor generalization capability, especially if there is a lack of labeled data. Meanwhile, in previous works, based on the detection results of the teacher model, a threshold was applied to each detection’s highest classification probabilities to filter out a subset of high-quality, reliable pseudo-labels. These pseudo-labels were employed in the student model’s training for the computation of classification loss. However, the classification probabilities of pseudo-labels can only reflect the classification results of the detected objects instead of their localization information; hence, unsupervised localization loss has often been ignored in previous works. To address these issues, based on the Unbiased Teacher method, we propose a semi-supervised object detection method with multi-scale regularization and bounding box re-prediction. The contributions of this paper are as follows:

(1)

A novel multi-scale regularization (MSR) strategy is proposed to constrain the Faster R-CNN and to generate the same prediction results for both the input images and their corresponding down-sampled ones, thus achieving better detection accuracy with the student model.

(2)

A novel bounding box re-prediction (BBRP) strategy is presented to re-predict the object’s bounding box, obtaining reliably located pseudo-labels of unlabeled data and thus improving the localization capability of the student model.

Experiments using the public MS COCO and Pascal VOC, which have been widely used in previous semi-supervised object detection works [17,22,23,24,25,26,30], show that the proposed method achieves a promising performance.

2. Materials and Methods

2.1. Overview of Our Proposed Method

The framework of our proposed semi-supervised object detection method based on multi-scale regularization and bounding box re-prediction is shown in Figure 1. Firstly, the dataset is divided into labeled data, ${\mathcal{X}}^{\mathcal{L}}=\left\{x_i^l{|}_{i=1}^{N_l}\right\}$, and unlabeled data, ${\mathcal{X}}^{\mathcal{U}}=\left\{x_i^u{|}_{i=1}^{N_u}\right\}$. $N_l$ and $N_u$ are the amounts of labeled and unlabeled data, respectively. $N_u\gg N_l$. The annotations of the labeled data are represented as ${\mathcal{Y}}^{\mathcal{L}}=\left\{y_i^l{|}_{i=1}^{N_l}\right\}$. Each annotation includes the center coordinates, width, height, and object classes of a bounding box. Then, based on the teacher–student framework, the teacher and student models both adopt the two-stage Faster R-CNN [6] as the object detector to predict the detection results. Specifically, in the first stage, the backbone network of the Faster R-CNN extracts the features of images and generates abundant region proposals for the foreground objects through the region proposal network (RPN). In the second stage, the features of all proposals are scaled to a fixed size using the region-of-interest (RoI) pooling operation, and then the RoI head is used to obtain the final labels, including the localization information and classification probabilities.

To alleviate the issue of false detection and duplicate detection, a multi-scale regularization strategy is integrated into the student Faster R-CNN to down-sample the strongly augmented images and constrain these images and the corresponding down-sampled ones so that both the RPN and RoI head of the student model produce consistent output. We employ $L_{scale}$ loss to constrain the student model in both the RPN and RoI head for accurate detection results.

On the other hand, for unlabeled data, based on the pseudo-labels predicted by the teacher model, our method uses thresholding technology to select reliably classified pseudo-labels, whose classification probabilities are higher than a threshold of ${\tau }_1$. Meanwhile, our method employs bounding box re-prediction to select reliably located pseudo-labels, whose location coordinates are steady. Using these reliably classified pseudo-labels and reliably located pseudo-labels, our method improves the student model’s generalization of unlabeled data with the constraint of the $L_{\mathrm{unsup}}$ loss, which includes the unsupervised classification loss and unsupervised localization loss from the RPN and RoI head in the student model.

Additionally, for the labeled data and their corresponding labels, we train the student model in a supervised way, with the constraint of the $L_{\sup }$ loss, which includes the supervised classification loss and supervised localization loss from the RPN and RoI head in the student model.

It should be noted that our method adopts three kinds of losses: supervised loss, $L_{\sup }$; unsupervised loss, $L_{\mathrm{unsup}}$; and multi-scale regularization loss $L_{\mathrm{scale}}$, which are all derived from both the RPN and RoI head of the student model. The student model uses the Gradient Descent method to update its parameters according to the total loss function. Different from the student model, the teacher model does not compute the loss function, and it only uses the EMA to slightly update its parameters according to those of the student model.

Moreover, during training, we first use the available labeled data to train and optimize our teacher model with the supervised loss. Then, we duplicate the trained weights from the teacher model and apply them to the student model. After that, we train the teacher–student framework using the three losses. Here, the teacher model takes the weakly augmented images as input to accurately predict the pseudo-labels, while the student model takes the strongly augmented images as input to be more robust. During the inference phase, we only use the teacher model to produce the final object detection results.

2.2. Multi-Scale Regularization

To improve the generalization of the student model, we propose multi-scale regularization loss for the RPN and RoI head of the detector. The pseudo-code is shown in Algorithm 1.

Multi-scale regularization is illustrated in Figure 2 and Figure 3. To increase the richness of the training samples, the Scale Jittering strategy [28] is applied to the labeled data and unlabeled data for data augmentation. Here, the images are randomly scaled within the range from −50% to 150%. These scaled images then enable the model to obtain different-sized features during training for better robustness.

Algorithm 1: Multi-scale regularization.

Input: Unlabeled Image $x^{\mathrm{u}}$; Output: Multi-Scale Regularization Loss $L_{\mathrm{scale}}^{\mathrm{rpn}}$, $L_{\mathrm{scale}}^{\mathrm{roihead}}$; 1: $x^{\mathrm{u}}$ = Scale Jittering($x^{\mathrm{u}}$); 2: $x_{\mathrm{d}}^{\mathrm{u}}$ = Down_Sample($x^{\mathrm{u}}$); 3: F = $\mathrm{Backbone}\_\mathrm{Student}$($x^{\mathrm{u}}$); 4: $F_{\mathrm{d}}$ = $\mathrm{Backbone}\_\mathrm{Student}$($x_{\mathrm{d}}^{\mathrm{u}}$); 5: $Proposals$, $\widehat{\mathit{OBJ}}$ = $\mathrm{RPN}\_\mathrm{Student}$(F); 6: $Proposal{s}_{\mathrm{d}}$ = Down_Sample($Proposals$); 7: _, ${\widehat{\mathit{OBJ}}}_{\mathrm{d}}$ = $\mathrm{RPN}\_\mathrm{Student}$($F_{\mathrm{d}}$); 8: ${\widehat{\mathit{OBJ}}}^{*}$ = Down_Sample($\widehat{\mathit{OBJ}}$); 9: Compute $L_{\mathrm{scale}}^{\mathrm{rpn}}$ according to Equation (1); 10: $\widehat{\mathit{CLS}}$, $\widehat{\mathit{REG}}$ = $\mathrm{RoI}\_\mathrm{Head}\_\mathrm{Student}$($x^{\mathrm{u}}$, F, $Proposals$); 11: ${\widehat{\mathit{CLS}}}_{\mathrm{d}}$, ${\widehat{\mathit{REG}}}_{\mathrm{d}}$ = $\mathrm{RoI}\_\mathrm{Head}\_\mathrm{Student}$($x_{\mathrm{d}}^{\mathrm{u}}$, $F_{\mathrm{d}}$, $Proposal{s}_{\mathrm{d}}$); 12: Compute $L_{\mathrm{scale}}^{\mathrm{roihead}}$ according to Equation (2); 13: return $L_{\mathrm{scale}}^{\mathrm{rpn}}$, $L_{\mathrm{scale}}^{\mathrm{roihead}}$.

Subsequently, we input the image $x^{\mathrm{u}}$ and the corresponding down-sampled one $x_{\mathrm{d}}^{\mathrm{u}}$ into the backbone network with a Feature Pyramid Network [35] (FPN) to extract feature maps F and $F^{\mathrm{d}}$ and generate region proposals using the RPN of the student model. After that, the objectness map $\widehat{\mathit{OBJ}}$ of $x^{\mathrm{u}}$ generated by the RPN is down-sampled to obtain ${\widehat{\mathit{OBJ}}}^{*}$, which has the same shape as that of the objectness map of $x_{\mathrm{d}}^{\mathrm{u}}$. The down-sample operation is called MaxPooling. Thus, the $L_{scale}$ loss for the RPN is computed as the Euclidean distance between the objectness map of $x^{\mathrm{u}}$ and that of $x_{\mathrm{d}}^{\mathrm{u}}$, as shown in Equation (1).

$$L_{\mathrm{scale}}^{\mathrm{rpn}}({\widehat{\mathit{OBJ}}}^{*},{\widehat{\mathit{OBJ}}}_{\mathrm{d}})=\frac{1}{m}\sum _{i=1}^m{∥{\widehat{\mathit{obj}}}^{*}\left[i\right]-{\widehat{\mathit{obj}}}_{\mathrm{d}}\left[i\right]∥}_2^2$$

(1)

where $\widehat{\mathit{obj}}\left[i\right]\in {\mathbb{R}}^{A\times W_{\mathrm{i}}\times H_{\mathrm{i}}}$ and ${\widehat{\mathit{obj}}}_{\mathrm{d}}\left[i\right]\in {\mathbb{R}}^{A\times \frac{W_{\mathrm{i}}}{2}\times \frac{H_{\mathrm{i}}}{2}}$ represent the i-th objectness map of $x^{\mathrm{u}}$ and $x_{\mathrm{d}}^{\mathrm{u}}$ generated by the RPN, respectively. ${\widehat{\mathit{obj}}}^{*}\left[i\right]\in {\mathbb{R}}^{A\times \frac{W_{\mathrm{i}}}{2}\times \frac{H_{\mathrm{i}}}{2}}$ is the spatially down-sampled version of $\widehat{\mathit{obj}}\left[i\right]$. A is the number of anchors in an image. $W_{\mathrm{i}}=\frac{W}{2^{(i-1)}}$ and $H_{\mathrm{i}}=\frac{H}{2^{(i-1)}}$ present the width and height of the i-th objectness map $\widehat{\mathit{obj}}\left[i\right]$. W and H are the width and height of the largest objectness map. m is the number of objectness maps (for one single image, the backbone network of the detector outputs m feature maps of different scales, and each feature map produces a corresponding objectness map; here, $m=5$).

Similarly, to make the detector’s prediction more robust, our method constrains the RoI head outputs of the student model on the feature map and the down-sampled ones to make them more consistent for the unlabeled data, as shown in Figure 3. Firstly, our student model employs the backbone and RPN to obtain the feature maps and region proposals of images $x^{\mathrm{u}}$ and $x_{\mathrm{d}}^{\mathrm{u}}$, respectively. Then, based on the feature maps and region proposals, RoI pooling converts the features inside the proposals into a fixed size. Finally, these converted features from the two feature maps are inputted into the RoI head to obtain the classification output and regression output. Here, $F_{\mathrm{d}}$ shares the same region proposals with F due to the down-sampling operation. The RoI head output (classification output or regression output) of the two feature maps (F and $F_{\mathrm{d}}$) have the same size, so the $L_{scale}$ loss for the RoI head can be computed according to Equation (2).

$$\begin{array}{c}{L}_{\mathrm{scale}}^{\mathrm{roihead}}(\widehat{\mathit{CLS}},{\widehat{\mathit{CLS}}}_{\mathrm{d}},\widehat{\mathit{REG}},{\widehat{\mathit{REG}}}_{\mathrm{d}})=\frac{1}{n}{\displaystyle \sum _{i=1}^n}{∥{\widehat{\mathit{cls}}}^i-{\widehat{\mathit{cls}}}_{\mathrm{d}}^i∥}_2^2+\frac{1}{n}{\displaystyle \sum _{i=1}^n}{∥{\widehat{\mathit{reg}}}^i-{\widehat{\mathit{reg}}}_{\mathrm{d}}^i∥}_2^2\end{array}$$

(2)

where ${\widehat{\mathit{cls}}}^i\in {\mathbb{R}}^C$ and ${\widehat{\mathit{reg}}}^i\in {\mathbb{R}}^4$ are the classification output (classification probabilities) and regression output (bounding box) of the i-th region proposal of image $x^{\mathrm{u}}$. ${\widehat{\mathit{cls}}}_{\mathrm{d}}^i\in {\mathbb{R}}^C$ and ${\widehat{\mathit{reg}}}_{\mathrm{d}}^i\in {\mathbb{R}}^4$ are the classification output and regression output of the i-th region proposal of image $x_{\mathrm{d}}^{\mathrm{u}}$. C is the number of object classes ($C=80$ for the COCO dataset). n is the number of region proposals, which we set to be 1000 for each image in our method. Since the regression outputs of images $x^{\mathrm{u}}$ and $x_{\mathrm{d}}^{\mathrm{u}}$ are of different scales, the parameters of each regressed bounding box are normalized using Equation (3).

$$\begin{array}{cccc}\hfill x^{\mathrm{reg}}& =(\widehat{x}-x_{\mathrm{a}})/w_{\mathrm{a}},\hfill & y^{\mathrm{reg}}& =(\widehat{y}-y_{\mathrm{a}})/h_{\mathrm{a}}\hfill \\ \hfill w^{\mathrm{reg}}& =log(\widehat{w}/w_{\mathrm{a}}),\hfill & h^{\mathrm{reg}}& =log(\widehat{h}/h_{\mathrm{a}})\hfill \\ \hfill x_{\mathrm{d}}^{\mathrm{reg}}& =({\widehat{x}}_{\mathrm{d}}-x_{\mathrm{d},\mathrm{a}})/w_{\mathrm{d},\mathrm{a}},\hfill & y_{\mathrm{d}}^{\mathrm{reg}}& =({\widehat{y}}_{\mathrm{d}}-y_{\mathrm{d},\mathrm{a}})/h_{\mathrm{d},\mathrm{a}}\hfill \\ \hfill w_{\mathrm{d}}^{\mathrm{reg}}& =log({\widehat{w}}_{\mathrm{d}}/w_{\mathrm{d},\mathrm{a}}),\hfill & h_{\mathrm{d}}^{\mathrm{reg}}& =log({\widehat{h}}_{\mathrm{d}}/h_{\mathrm{d},\mathrm{a}})\hfill \end{array}$$

(3)

where x, y, w, and h represent the center coordinates, width, and height of the region proposal. Variables $\widehat{x}$, $x_{\mathrm{a}}$, and $x^{\mathrm{reg}}$ represent the predicted box, anchor box, and normalized regressed box, respectively (as do y, w, and h). ${\widehat{\mathit{reg}}}^i=[x^{\mathrm{reg}},y^{\mathrm{reg}},w^{\mathrm{reg}},h^{\mathrm{reg}}]$ and ${\widehat{\mathit{reg}}}_{\mathrm{d}}^i=[x_{\mathrm{d}}^{\mathrm{reg}},y_{\mathrm{d}}^{\mathrm{reg}},w_{\mathrm{d}}^{\mathrm{reg}},h_{\mathrm{d}}^{\mathrm{reg}}]$.

Finally, the multi-scale regularization loss from the RPN and RoI head in the student model can be summarized with Equation (4).

$$L_{\mathrm{scale}}=L_{\mathrm{scale}}^{\mathrm{rpn}}+L_{\mathrm{scale}}^{\mathrm{roihead}}$$

(4)

It is worth noting that the loss functions $L_{\mathrm{scale}}^{\mathrm{rpn}}$ and $L_{\mathrm{scale}}^{\mathrm{roihead}}$ can be computed with one forward propagation based on the image $x^{\mathrm{u}}$ and the corresponding down-sampled one $x_{\mathrm{d}}^{\mathrm{u}}$. To make it clear, we illustrate the two loss functions separately.

2.3. Bounding Box Re-Prediction

Most of the semi-supervised methods set a threshold for the predicted classification score to obtain reliable pseudo-labels. However, these pseudo-labels only ensure that the classification output of the detection is reliable and cannot guarantee the reliability of the regression output, i.e., the bounding box, for detection. Therefore, we propose the bounding box re-prediction strategy to refine the bounding box of each pseudo-label initially predicted by the teacher model for better localization accuracy.

Specifically, the unlabeled image $x^u$ is input into the teacher model, and the RPN of the teacher model predicts the region proposals. Based on these region proposals, a classification and regression operation is performed for the RoI head of the teacher model to obtain the prediction results $B_0$ (including the classification scores and location coordinates). Then, we set a threshold ${\tau }_1$ for the classification score to filter out the reliably classified pseudo-labels for the unsupervised classification loss computation. To further obtain reliably located pseudo-labels, based on the initial prediction results $B_0$ from the teacher model, our method employs BBRP on the RoI head of the teacher model to re-predict the bounding box of each region proposal and to obtain the refined prediction results $B_{tmp}$. The pseudo-code of BBRP is shown in Algorithm 2. Thus, our method obtains reliably located pseudo-labels for unsupervised localization loss computation on unlabeled data.

Algorithm 2: Bounding box re-prediction.

Input: Feature map F, prediction results $B_0$; Output: Reliably located pseudo-labels $y_{\mathrm{loc}}^{\mathrm{u}}$; 1: $B_{tmp}$=RoI_Head_Teacher($\mathit{F}$, $B_0$); 2: $y_{\mathrm{loc}}^{\mathrm{u}}$=$\left\{\right\}$; 3: for b in $B_{tmp}$ do 4:     if $\mathit{b}.\mathrm{cls}\_\mathrm{prob} >$${\tau }_2$ then 5:         for $b_0$ in $B_0$ do 6:            $\mathrm{o}=\mathrm{IoU}(\mathit{b},{\mathit{b}}_0)$; 7:            if $\mathrm{o}>$ ${\tau }_3$ then 8:                $y_{\mathrm{loc}}^{\mathrm{u}}\leftarrow \mathrm{Append}\left(\mathit{b}\right)$; 9:            end if 10:         end for 11:     end if 12: end for 13: return $y_{\mathrm{loc}}^{\mathrm{u}}$.

2.4. Loss Function

The loss function of our proposed method is as follows:

$$L=L_{sup}+\alpha L_{\mathrm{unsup}}+\beta L_{\mathrm{scale}}$$

(5)

$$\begin{array}{c}{L}_{sup}=L_{\mathrm{cls}}^{\mathrm{rpn}}({\widehat{\mathcal{Y}}}_{\overline{\mathrm{cls}}}^{\mathrm{l}},{\mathcal{Y}}_{\overline{\mathrm{cls}}}^{\mathrm{l}})+L_{\mathrm{loc}}^{\mathrm{rpn}}({\widehat{\mathcal{Y}}}_{\mathrm{loc}}^{\mathrm{l}},{\mathcal{Y}}_{\mathrm{loc}}^{\mathrm{l}})+L_{\mathrm{cls}}^{\mathrm{roihead}}({\widehat{\mathcal{Y}}}_{\mathrm{cls}}^{\mathrm{l}},{\mathcal{Y}}_{\mathrm{cls}}^{\mathrm{l}})+L_{\mathrm{loc}}^{\mathrm{roihead}}({\widehat{\mathcal{Y}}}_{\mathrm{loc}}^{\mathrm{l}},{\mathcal{Y}}_{\mathrm{loc}}^{\mathrm{l}})\end{array}$$

(6)

$$\begin{array}{c}{L}_{\mathrm{unsup}}=L_{\mathrm{cls}}^{\mathrm{rpn}}({\widehat{\mathcal{Y}}}_{\overline{\mathrm{cls}}}^{\mathrm{u}},{\mathcal{Y}}_{\overline{\mathrm{cls}}}^{\mathrm{u}})+L_{\mathrm{loc}}^{\mathrm{rpn}}({\widehat{\mathcal{Y}}}_{\mathrm{loc}}^{\mathrm{u}},{\mathcal{Y}}_{\mathrm{loc}}^{\mathrm{u}})+L_{\mathrm{cls}}^{\mathrm{roihead}}({\widehat{\mathcal{Y}}}_{\mathrm{cls}}^{\mathrm{u}},{\mathcal{Y}}_{\mathrm{cls}}^{\mathrm{u}})+L_{\mathrm{loc}}^{\mathrm{roihead}}({\widehat{\mathcal{Y}}}_{\mathrm{loc}}^{\mathrm{u}},{\mathcal{Y}}_{\mathrm{loc}}^{\mathrm{u}})\end{array}$$

(7)

where $L_{\sup }$ is the supervised loss computed based on the labeled data ${\mathcal{X}}^{\mathcal{L}}$ and the corresponding label ${\mathcal{Y}}^{\mathcal{L}}$ (including ${\mathcal{Y}}_{\mathrm{cls}}^{\mathrm{l}}$ and ${\mathcal{Y}}_{\mathrm{loc}}^{\mathrm{l}}$). $L_{\sup }$ is the summation of the classification loss $L_{\mathrm{cls}}^{\mathrm{rpn}}$ and localization loss $L_{\mathrm{loc}}^{\mathrm{rpn}}$ generated by the RPN, and the classification loss $L_{\mathrm{cls}}^{\mathrm{roihead}}$ and localization loss $L_{\mathrm{loc}}^{\mathrm{roihead}}$ generated by the RoI head of the student model. $L_{\mathrm{unsup}}$ is the unsupervised loss computed based on the unlabeled data ${\mathcal{X}}^{\mathcal{U}}$ and its corresponding reliably classified pseudo-labels ${\mathcal{Y}}_{\mathrm{cls}}^{\mathrm{u}}$ and reliably located pseudo-labels ${\mathcal{Y}}_{\mathrm{loc}}^{\mathrm{u}}$. $L_{\mathrm{unsup}}$ has a similar computation procedure as $L_{\sup }$. Note that ${\mathcal{Y}}_{\overline{\mathrm{cls}}}^{\mathrm{l}}$ and ${\mathcal{Y}}_{\overline{\mathrm{cls}}}^{\mathrm{u}}$ include 0–1 labels for the background or foreground of the image. $L_{\mathrm{scale}}$ represents the multi-scale regularization loss, as defined in Equation (4). $\alpha$ and $\beta$ are the weighted coefficients of the loss function; here, $\alpha =2$ and $\beta =1$. The classification loss $L_{cls}$ ($L_{\mathrm{cls}}^{\mathrm{rpn}}$ or $L_{\mathrm{cls}}^{\mathrm{roihead}}$) is computed with Equation (8). The localization loss $L_{loc}$ ($L_{\mathrm{loc}}^{\mathrm{rpn}}$ or $L_{\mathrm{loc}}^{\mathrm{roihead}}$) is defined with Equation (10).

$$L_{cls}(\widehat{\mathcal{Y}},\mathcal{Y})=\frac{1}{N_{\mathrm{cls}}}\sum _{i=1}^{N_{\mathrm{cls}}}{l}_{cls}({\widehat{y}}_{\mathrm{i}},y_{\mathrm{i}})$$

(8)

$$l_{cls}(p,y)=-y{(1-p)}^{\gamma }log\left(p\right)-(1-y)p^{\gamma }log(1-p)$$

(9)

$$L_{loc}(\widehat{\mathcal{R}},\mathcal{R})=\frac{1}{N_{\mathrm{loc}}}\sum _{i=1}^{N_{\mathrm{loc}}}{l}_{loc}({\widehat{r}}_{\mathrm{i}},r_{\mathrm{i}})$$

(10)

$$l_{loc}(\widehat{r},r)=\left\{\begin{array}{c}0.5{(\widehat{r}-r)}^2,\mathrm{if}\left|\widehat{r}-r\right|<1\hfill \\ \left|\widehat{r}-r\right|-0.5,\mathrm{otherwise}\hfill \end{array}\right.$$

(11)

where $N_{\mathrm{cls}}$ indicates the number of ground-truth labels for the supervised loss or reliably classified pseudo-labels for the unsupervised loss. $N_{\mathrm{loc}}$ indicates the number of ground-truth bounding boxes for the supervised loss or reliably located pseudo-labels for the unsupervised loss. ${\widehat{y}}_{\mathrm{i}}$ is the highest classification probability predicted by the student model, and $y_{\mathrm{i}}$ is the corresponding ground truth. ${\widehat{r}}_{\mathrm{i}}$ is the region proposal predicted by the student model, and $r_{\mathrm{i}}$ is the corresponding ground truth. $\gamma =2.0$ in this paper.

2.5. Datasets

In our experiment, we validate our proposed method using the MS COCO [36] and Pascal VOC [37] datasets. Table 1 shows the number of images and categories in the datasets. The reasons for selecting these two datasets are as follows: (1) The MS COCO and Pascal VOC datasets consist of images depicting common objects of complex scenes. The COCO dataset is a large-scale dataset with over 100,000 images and 80 common categories, while the VOC dataset has a lot of images and 20 common categories. These datasets include a large number of samples and various categories of objects, as well as annotations (bounding boxes and categories), for each image. This is a convenient method for us to conduct effective experiments and analyses in different situations. (2) The MS COCO and Pascal VOC datasets are two baseline datasets that have been widely used in other semi-supervised object detection studies [17,22,23,24,25,26,30], and, thus, they will facilitate a fair comparison between our method and other state-of-the-art methods. For comparison, we follow the same settings used in previous works.

(1) In the COCO2017-Train dataset, 1%, 2%, 5%, and 10% images are randomly sampled as labeled data, and the rest of the images are used as unlabeled data. These labeled data and unlabeled data are used as the training data. The COCO2017-Val dataset is used as the test data.

(2) The VOC2007-Train and -Val datasets are used as labeled data and the VOC2012-Train and-Val datasets are used as unlabeled data. These labeled data and unlabeled data are used as the training data. The VOC2007-Test dataset is used as the test data.

Table 1. Detailed description of MS COCO and Pascal VOC datasets. The number of images and categories of each dataset are shown.

Dataset	Train	Val	Test	Total	Categories
VOC2007	2501	2510	4952	9963	20
VOC2012	5717	5832	-	11,549	20
COCO2017	118,287	5000	-	123,287	80

Table 1. Detailed description of MS COCO and Pascal VOC datasets. The number of images and categories of each dataset are shown.

Dataset	Train	Val	Test	Total	Categories
VOC2007	2501	2510	4952	9963	20
VOC2012	5717	5832	-	11,549	20
COCO2017	118,287	5000	-	123,287	80

3. Experiments

We conducted quantitative and qualitative experiments to evaluate the performance of our proposed semi-supervised object detection method.

3.1. Settings and Details

3.1.1. Evaluation Metrics

To quantitatively measure the performance of our semi-supervised object detection method, we adopted mAP50, mAP75, and mAP50:95 as evaluation metrics. The formula to determine the mean average precision (mAP) is as follows:

$$mAP=\underset{0}{\overset{1}{\int }}P\left(R\right)dR$$

(12)

$$P=\frac{TP}{TP+FP}$$

(13)

$$R=\frac{TP}{TP+FN}$$

(14)

where P is the precision, R is the recall, TP represents the number of positive samples predicted as positive, FP represents the number of negative samples predicted as positive, and FN represents the number of positive samples predicted as negative. Given an Intersection over Union (IoU = Intersection/Union) of two bounding boxes, we can draw a precision–recall (P-R) curve, and the area under the P-R curve is the mAP. So, mAP50 represents the mAP when the IoU is 0.5; mAP75 represents the mAP when the IoU is 0.75; and mAP50:95 represents the mean of the mAP values when the IoU is 0.5, 0.55, 0.6,…, 0.95.

3.1.2. Implementation Details

Our experiments were performed on Ubuntu 16.04 with Intel(R) Xeon(R) Gold 5218 CPU @ 2.30GHz and NVIDIA Tesla T4 GPU (×4). The proposed method was built on the Detectron2 [38] deep learning framework.

As for data augmentation, Horizontal Flipping is used as a weak augmentation method, while Color Jittering, Gaussian Blur, and Cutout [39] are applied as strong augmentation methods. The batch sizes of both the labeled data and unlabeled data are set as 4. The learning rate is 0.005, and the momentum is 0.9. The number of training iterations for the following ablation experiment is 120K. The thresholds are ${\tau }_1$ = 0.7, ${\tau }_2$ = 0.3, and ${\tau }_3$ = 0.95. The weighted coefficients $\alpha$ and $\beta$ of the loss function are set as 2 and 1, respectively. The student model uses the SGD optimizer and Gradient Descent method to update the parameters. The teacher model depends on the EMA to update its parameters based on the parameters of the student model, as shown in Equation (15).

$${\theta }_t^{\mathrm{T}}=\psi {\theta }_{t-1}^{\mathrm{T}}+(1-\psi ){\theta }_t^{\mathrm{S}}$$

(15)

where ${\theta }^{\mathrm{T}}$ is the teacher model’s parameter, ${\theta }^{\mathrm{S}}$ is the student model’s parameter, t is the current iteration, and $\psi$ is set as 0.9996.

We also conduct additional experiments to determine other hyper-parameters of our method. ${\tau }_1$ is set as 0.7, which is consistent with the settings of the Unbiased Teacher method [26]. ${\tau }_2$ and ${\tau }_3$ are the important hyper-parameters in the BBRP strategy.

Table 2. The performance of our method with different ${\tau }_2$ and ${\tau }_3$ values for BBRP. It can be seen that our method achieves the highest mAP when ${\tau }_2=0.3$ and ${\tau }_3=0.95$.

${\mathsf{\tau }}_2$	${\mathsf{\tau }}_3$	mAP50:95
0.2	0.98	21.90
	0.95	21.95
	0.9	21.82
	0.8	21.34
	0.7	20.67
0.3	0.98	22.21
	0.95	22.46
	0.9	22.13
	0.8	21.65
	0.7	20.89
0.4	0.98	21.70
	0.95	21.65
	0.9	21.51
	0.8	21.19
	0.7	20.38

shows the performance of our method with different values of ${\tau }_2$ and ${\tau }_3$. We can observe from Table 2 that the model achieves the highest mAP when ${\tau }_2=0.3$ and ${\tau }_3=0.95$.

Table 3. The performances of our method with different $\alpha$ and $\beta$ for the loss function. It can be seen that our method achieves the highest mAP when $\alpha =2.0$ and $\beta =1.0$.

$\mathsf{\alpha }$	$\mathsf{\beta }$	mAP50:95
1.0	1.0	21.12
	2.0	21.02
	3.0	20.78
2.0	1.0	22.46
	2.0	22.32
	3.0	21.95
3.0	1.0	Cannot Converge
	2.0	Cannot Converge
	3.0	Cannot Converge

shows the performance of our method with different values of $\alpha$ and $\beta$ for the loss function. We can observe that when $\alpha =3.0$, the training of the model cannot converge. Our method achieves the highest mAP when $\alpha =2.0$ and $\beta =1.0$.

During training, the mAP curve of our method evolves, as shown in Figure 4, using the COCO 1% labeled data. As a comparison, we also draw the mAP curve of the baseline semi-supervised method, Unbiased Teacher. We can see that both methods converge as the iterations increase.

3.2. Comparison with State-of-the-Art Methods

To verify the effectiveness of our method, we compared our method with other state-of-the-art semi-supervised object detection methods, as shown in

Table 4. Comparison with other state-of-the-art methods using the COCO dataset (to determine the metric mAP50:95 under different ratios of labeled data). Our method achieves the best results in all four situations.

	mAP50:95
Method	1%	2%	5%	10%
Supervised	9.05	12.70	18.47	23.86
CSD [17]	10.51	13.93	18.63	22.46
STAC [22]	13.97	18.25	24.38	28.64
Instant-Teaching [23]	18.05	22.45	26.75	30.40
Humble Teacher [24]	16.96	21.72	27.70	31.61
Robust Teacher [25]	17.91	21.88	25.81	28.81
Unbiased Teacher [26]	18.14	22.23	26.65	29.13
MUM [30]	21.88	24.84	28.52	31.87
Our Method	22.46	26.35	29.73	32.89

: consistency-based semi-supervised learning (CSD) [17], STAC [22], Instant-Teaching [23], Humble Teacher [24], Robust Teacher [25], Unbiased Teacher [26], and MUM [30]. The supervised method is a baseline that only uses labeled data for fully supervised training. The CSD method introduces consistency loss for an image and its horizontally flipped counterpart into semi-supervised learning. The STAC method adopts a self-training and augmentation-driven consistency regularization framework. The Instant-Teaching method co-rectifies the erroneous predictions of two structurally identical but independently trained models on the same image. The Humble Teacher method uses the EMA and soft-label mechanism to improve semi-supervised models. The Robust Teacher method introduces a new self-correcting pseudo-label module and pseudo-label-guided copy–paste technology to generate more reliable predictions for semi-supervised detection. The Unbiased Teacher method employs a real-time pseudo-label generation method. The MUM method presents an effective data augmentation method for semi-supervised models.

For a fair comparison, all methods use the Faster R-CNN with a Resnet-50 backbone [40] as the object detector. We can see from Table 4 that, compared with fully supervised training, when using COCO 1%, 2%, 5%, and 10% labeled data, our presented method improves mAP50:95 by 13.41%, 13.65%, 11.26%, and 9.03%, respectively. Compared with the Unbiased Teacher method, with the same four different proportions of labeled data, our proposed method improves mAP50:95 by 4.32%, 4.12%, 3.08%, and 3.76%. Our presented method outperforms the other semi-supervised methods under comparison in most scenarios. We can also see from

Table 5. Comparison with other state-of-the-art methods using the VOC dataset. Our method achieves the best results for the metrics mAP50 and mAP50:95.

Method	mAP50	mAP50:95
Supervised	76.70	43.60
CSD [17]	74.70	-
STAC [22]	77.45	44.64
Instant-Teaching [23]	79.20	50.00
Humble Teacher [24]	80.94	53.04
Robust Teacher [25]	80.24	53.47
Unbiased Teacher [26]	79.30	53.50
MUM [30]	80.04	52.31
Our Method	81.29	55.26

that, on the VOC dataset, our method outperforms other methods in terms of mAP50 and mAP50:95.

Due to the additional computational cost introduced by our innovative method compared to that of the traditional Unbiased Teacher method, we performed a comparative experiment to test the training speed of the model to evaluate its training efficiency. We found that the Unbiased Teacher model takes 0.347 s per iteration for training, while our model takes 0.471 s per iteration. Our model requires slightly more training time than the Unbiased Teacher model. The inference speed of a single image using our model or the Unbiased Teacher model is 0.116 s on a Tesla T4 GPU.

3.3. Ablation Study

To investigate the effectiveness of our proposed method’s components, we conducted the ablation experiments on partially labeled data, as shown in

Table 6. Ablation experiments to test multi-scale regularization strategy and bounding box re-prediction strategy. ✓ indicates that the corresponding component is used in our proposed method for the ablation experiment. Each component of our method is effective.

Aug	MSR (RPN)	MSR (RoI Head)	BBRP	mAP50	mAP75	mAP50:95
				33.50	16.65	17.71
✓				35.62	18.44	19.18
✓	✓			36.18	19.24	19.86
✓	✓	✓		37.02	20.02	20.74
✓	✓	✓	✓	37.21	21.44	22.46

. As an example, the experiment was performed using COCO 1% labeled data. Aug indicates the Scale Jittering strategy, MSR (RoI head) indicates the multi-scale regularization strategy for the RoI head, MSR (RPN) indicates the multi-scale regularization strategy for the RPN, and BBRP indicates the bounding box re-prediction strategy.

From Table 6, we can see that the Scale Jittering strategy can improve the detection accuracy of our method by 1.47% in mAP50:95. With the MSR strategy for the RPN, our method achieves a 0.68% improvement in mAP50:95. With the MSR strategy for the RoI head, our method achieves a 0.88% improvement in mAP50:95. In summary, with the MSR strategy, our method increases mAP50 by 1.40%, mAP75 by 1.58%, and mAP50:95 by 1.56%. With the BBRP strategy, our method obtains a 0.19% improvement in mAP50, 1.42% improvement in mAP75, and 1.72% improvement in mAP50:95. Finally, when combining the MSR strategy and BBRP strategy, our method increases by 3.71%, 4.79%, and 4.75% in mAP50, mAP75, and mAP50:95, respectively.

Up to now, the ablation experiment has proved the effectiveness of each component of our method. The reasons for this are as follows: (1) The multi-scale regularization strategy exploits the multi-scale information from the unlabeled data, thereby boosting the model’s detection performance of multi-scale objects and also improving the model’s robustness. (2) The bounding box re-prediction strategy selects reliably located pseudo-labels during training to further constrain the model for better localization accuracy when using unlabeled data.

3.4. Visualization

Figure 5 visualizes the prediction results of our method and those of the representative semi-supervised Unbiased Teacher method. As an example, the experiment is carried out with 1% labeled data. It can be seen that the Unbiased Teacher model suffers from false detection (the sports balls indicated by the red arrows in Figure 5a), missed detection (the elephant and persons indicated by the red arrows in Figure 5b,c), and duplicate detection (the flying person indicated by the red arrow in Figure 5c). These issues lead to unreliable pseudo-labels in the teacher model and affect the student model’s learning. In contrast, our method extracts the multi-scale features from the unlabeled data using the MSR strategy and robustly detects multi-scale objects in the images. Therefore, MSR can effectively solve these incorrect detection issues and improve the generalization capability of the model. On the other hand, as shown in Figure 5d, the Unbiased Teacher method only roughly locates the objects under some circumstances and ignores part of the object. However, our proposed method employs BBRP to select the reliably located pseudo-labels to improve the localization ability of our model. Therefore, our semi-supervised method can accurately detect the target objects.

4. Discussion

Semi-supervised object detection methods often encounter false detection and duplicate detection due to the limited amount of labeled data. During semi-supervised training, it is challenging for a poor-performance teacher model to provide high-quality pseudo-labels. Therefore, we introduce a multi-scale regularization strategy to boost the semi-supervised object detector’s performance. On the other hand, semi-supervised object detection methods typically use a thresholding method to filter pseudo-labels of unlabeled data and utilize them for the computation of unsupervised loss. However, these reliably classified pseudo-labels only produce accurate classification results and uncertain localization results. Hence, the localization information of unlabeled data is often ignored.

Our method is a semi-supervised object detection method based on multi-scale regularization and bounding box re-prediction. The multi-scale regularization strategy effectively alleviates the issue of false detection and duplicate detection by constraining the model to produce consistent predictions for images at different scales. This improvement notably enhances the robustness of the model. The bounding box re-prediction strategy improves the localization capability of the detector by using the RoI head of the Faster R-CNN to perform re-prediction on the detection results. It considers the bounding boxes with similar re-prediction results as reliably located pseudo-labels, which are then used for computing the unsupervised localization loss. This approach effectively enhances the detector’s localization ability. Our semi-supervised object detection framework can be applied in many scenarios, such as autonomous driving, industrial inspection, and agriculture automation.

Although the two proposed strategies effectively enhance the accuracy of semi-supervised object detectors, there are still some issues to be addressed:

(1)

Training efficiency. Our method introduces additional computational costs, which decreases the training efficiency to some extent.

(2)

Domain adaptation. When the model begins semi-supervised learning, the training of the model becomes unstable due to the distribution gap between the labeled data and unlabeled data. We will try domain adaptation to alleviate this issue.

(3)

Label assignment. Our method obtains reliably classified pseudo-labels using the thresholding method, which is a simple label assignment strategy. In the future, we will try other strategies to improve the performance of our method.

We will further improve our semi-supervised object detection method in our future work based on these perspectives.

5. Conclusions

Based on the teacher–student framework, this paper proposes a semi-supervised object detection method with multi-scale regularization and bounding box re-prediction. With a small amount of labeled data and a large amount of unlabeled data, our method trains a semi-supervised detection model to detect various objects in the images. Furthermore, to alleviate the detection issues suffered by the traditional semi-supervised object detection methods, our method proposes a multi-scale regularization strategy to effectively improve the generalization of the model. Additionally, our method presents a novel bounding box re-prediction strategy to select the reliably located pseudo-labels for the accurate location of objects in the unlabeled data. The experiments show that our semi-supervised method has a promising detection performance. Moreover, as a regularization strategy, multi-scale regularization can be applied to emerging technologies such as Transformer and its variants in contrastive language-image learning, human action recognition, semantic segmentation, and so on. While bounding box re-prediction can be combined with other novel semi-supervised object detection methods, such as semi-supervised vision Transformers and so on. In the future, we will mainly focus on the training efficiency of the method, domain adaptation between labeled data and unlabeled data, and label assignment to further improve our method.