Weakly Supervised Monocular Fisheye Camera Distance Estimation with Segmentation Constraints

Abstract

Monocular fisheye camera distance estimation is a crucial visual perception task for autonomous driving. Due to the practical challenges of acquiring precise depth annotations, existing self-supervised methods usually consist of a monocular distance model and an ego-motion predictor with the goal of minimizing a reconstruction matching loss. However, they suffer from inaccurate distance estimation in low-texture regions, especially road surfaces. In this paper, we introduce a weakly supervised learning strategy that incorporates semantic segmentation, instance segmentation, and optical flow as additional sources of supervision. In addition to the self-supervised reconstruction loss, we introduce a road surface flatness loss, an instance smoothness loss, and an optical flow loss to enhance the accuracy of distance estimation. We evaluate the proposed method on the WoodScape and SynWoodScape datasets, and it outperforms the self-supervised monocular baseline, FisheyeDistanceNet.

1. Introduction

Depth estimation is a vital problem in visual perception for autonomous driving. As distance measurement by LiDAR is sparse and costly [1], learning-based approaches have been developed to estimate distance from image sequences. However, most of the previous works [2,3] have focused on rectified images captured by pinhole cameras. Given the widespread adoption of surround-view fisheye cameras in modern vehicles, the importance of monocular depth estimation specifically tailored to fisheye imagery has grown substantially.

Supervised methods [4,5] rely on sparse LiDAR depth ground truth. In contrast, self-supervised methods [6,7,8] are based on the Structure-from-Motion (SfM) framework, which estimates distance and ego-motion from video sequence. Their loss function consists of an image reconstruction term and a depth smoothness term. The effectiveness of these approaches has been demonstrated on WoodScape dataset [9]. However, these methods have two limitations, as shown in Figure 1b. First, they crop the raw fisheye image and miss the distance estimation on image edges. Second, distance predictions on homogeneous regions are inaccurate.

In this paper, we propose a weakly supervised learning framework for monocular fisheye distance estimation. First, semantic and instance segmentation labels are adopted to penalize flatness of the road surface region and smoothness within objects. Second, to improve the estimation accuracy, optical flow results generated by a state-of-the-art flow network are used to guide the coordinate mapping between two images. Moreover, our method predicts all valid region of fisheye image without any information loss, as depicted in Figure 1c.

The remainder of this paper is structured as follows: Section 2 presents a review of related work. In Section 3, we revisit a self-supervised framework for monocular fisheye distance estimation. Section 4 details our proposed weakly supervised approach. Experimental results are provided in Section 5, and Section 6 concludes the paper.

2. Related Work

This section presents an overview of existing approaches for depth or distance estimation using monocular fisheye cameras. These methods are generally classified into self-supervised and supervised learning approaches, depending on the absence or presence of ground-truth depth labels.

Most of the current monocular distance estimation works are on the rectified KITTI [10] sequences. Structure-from-Motion (SfM) techniques [2,3] introduce self-supervised learning frameworks for estimating depth and ego-motion from pinhole camera video sequences with no manual labeling.

In the case of fisheye camera, Ravi Kumar et al. [7] developed a FisheyeDistanceNet within the SfM framework. Its training loss consists of an image reconstruction photometric error, an edge-aware distance smoothness term and a cross-sequence distance consistency term. It verified the validity of self-supervised method on WoodScape dataset. Afterwards, to improve the distance estimation accuracy of dynamic objects, thin structure, and homogeneous areas, SynDistNet [8] jointly learns semantic segmentation network and adopts the prediction to guide self-supervised distance estimation. Moreover, OmniDet [6] is another multi-task learning network designed for fisheye images, including depth estimation, semantic segmentation, motion segmentation, and object detection, and outperforms other single-task versions. In addition to the loss terms in FisheyeDistanceNet, OmniDet incorporate feature-metric losses from [11] to penalize the consistency of low-texture regions. However, these methods crop the raw fisheye image and miss the depth estimation of marginal areas. Moreover, Yan et al. [12] combined self-supervised depth estimation with knowledge distillation to handle the challenge of large homogeneous areas in indoor scenes. Zhao et al. [13] proposed a self-supervised method which replaces the pose network with real-scale pose information but relies on the ground-truth pose data from the KITTI-360 dataset [14].

In addition, Lee et al. [5] introduced a supervised fisheye depth estimation method, which represents distance in the slanted cylindrical layer to alleviate distortion of side camera image. Son et al. [4] presented a knowledge-distillation technique by using sparse LiDAR-depth ground truth for supervision. These supervised learning-based methods both rely on dense or sparse depth labels for training.

In this paper, we aim to improve the accuracy of self-supervised monocular distance estimation for fisheye images. Within our proposed weakly supervised learning framework, no ground-truth distance labels are used; instead, we incorporate semantic segmentation, instance segmentation, and optical flow labels as additional constraints.

3. Self-Supervised Distance Estimation Baseline

In this section, we review a self-supervised framework based on FisheyeDistanceNet [7] for monocular fisheye distance estimation. It is built within a self-supervised Structure-from-Motion (SfM) framework tailored for monocular fisheye imagery. It involves training two separate networks:

• A monocular depth predictor $f_{depth}:I_t\to {\widehat{D}}_t$ that generates a depth map ${\widehat{D}}_t(p)=f_{depth}(I_t(p))$ for each pixel p in the target frame $I_t$.
• A pose estimation network $f_{pose}:(I_t,I_{t\prime })\to T_{t\to t^{\prime }}$ predicting a 6-DoF camera transformation $T_{t\to t^{\prime }}\in SE(3)$ between the target frame $I_t$ and a reference frame $I_{t^{\prime }}$.

The loss function used for self-supervised learning comprises two components: a reconstruction loss ${\mathcal{L}}_r$, which calculates the photometric discrepancy between the reconstructed view ${\widehat{I}}_{t^{\prime }\to t}$ and the original target frame $I_t$, and an edge-aware smoothness loss ${\mathcal{L}}_{es}$ for estimated distance ${\widehat{D}}_t$. The total objective loss ${\mathcal{L}}_{base}$ is computed as the average across image batches, spatial locations, and scales:

$${\mathcal{L}}_{base}={\mathcal{L}}_r(I_t,{\widehat{I}}_{t^{\prime }\to t})+\beta {\mathcal{L}}_{es}({\widehat{D}}_t),$$

(1)

where $\beta$ is term weight factor.

Image reconstruction loss. Using the estimated distance ${\widehat{D}}_t$ at time t and the camera intrinsic parameters, pixel set $p_t$ in $I_t$ can be unreprojected to a 3D point cloud $P_t$, stated as

$$P_t={\mathsf{\Pi }}^{-1}(p_t,{\widehat{D}}_t)$$

(2)

where ${\mathsf{\Pi }}^{-1}$ denotes the unreprojection operation that transforms image coordinates into the camera’s 3D coordinate, as explained in [7]. Using the predicted pose transformation $T_{t\to t^{\prime }}$ from the pose estimation network, the point cloud under the camera coordinate at time $t^{\prime }$ is calculated as ${\widehat{P}}_{t^{\prime }}=T_{t\to t^{\prime }}{P}_t$. By projecting ${\widehat{P}}_{t^{\prime }}$ from camera to image coordinates ${\widehat{p}}_{t^{\prime }}$ at time $t^{\prime }$, a mapping from $p_t$ to ${\widehat{p}}_{t^{\prime }}$ is obtained as

$${\widehat{p}}_{t^{\prime }}=\mathsf{\Pi }(T_{t\to t^{\prime }}{\mathsf{\Pi }}^{-1}(p_t,{\widehat{D}}_t))$$

(3)

By warping the source frame $I_{t^{\prime }}$, we reconstruct an image ${\widehat{I}}_{t^{\prime }\to t}$.

The image reconstruction loss between the reconstructed image ${\widehat{I}}_{t^{\prime }\to t}$ and the target frame $I_t$ is computed by combining the L1 loss and the Structural Similarity Index (SSIM) [15], stated as

$${\mathcal{L}}_r(I_t,{\widehat{I}}_{t^{\prime }\to t})=\alpha {\displaystyle \frac{1-SSIM(I_t,{\widehat{I}}_{t^{\prime }\to t},{\mathcal{M}}_{t{t}^{\prime }})}{2}}+(1-\alpha )\parallel (I_t-{\widehat{I}}_{t^{\prime }\to t})\odot {\mathcal{M}}_{t{t}^{\prime }}\parallel$$

(4)

where $\alpha =0.85$, ${\mathcal{M}}_{t{t}^{\prime }}$ is the binary mask, and ⊙ denotes element-wise multiplication. ${\mathcal{M}}_{t{t}^{\prime }}$ is used to filter out invalid mapping regions and pixels that do not change their appearance between frames, which helps to remove effect of low-texture regions.

Additionally, large photometric cost between reconstructed and target images may arise in regions with brightness variations, occlusions and dynamic objects. To mitigate the impact of large photometric of some pixels, which are treated as outliers, errors larger than $95\%$ are clipped to zero gradient.

Edge-aware smoothness loss. To regularize distance for low-gradient regions, smoothness term is defined as

$${\mathcal{L}}_{es}({\widehat{D}}_t)=\left|{\nabla }_x{\widehat{D}}_t^{\ast }\right|e^{-\left|{\nabla }_x{I}_t\right|}+\left|{\nabla }_y{\widehat{D}}_t^{\ast }\right|e^{-\left|{\nabla }_y{I}_t\right|}$$

(5)

where ${\widehat{D}}_t^{\ast }$ denotes the mean normalization of $1/{\widehat{D}}_t$ to shrink distance.

4. Weakly Supervised Distance Estimation

In this section, we describe our weakly supervised method for distance estimation using monocular fisheye camera. We adopt SfM framework, which is the same with the self-supervised baseline. In contrast, our objective loss consists of a reconstruction term ${\mathcal{L}}_r$, which is defined the same with the baseline, a road surface flatness term ${\mathcal{L}}_f$, which is obtained using semantic labels, an instance smoothness term ${\mathcal{L}}_{is}$ for estimated distance ${\widehat{D}}_t$ using instance segmentation labels, and an optical flow error term ${\mathcal{L}}_{flo}$. All these terms are computed as the average across image batches, spatial locations, and scales. The total objective loss ${\mathcal{L}}_{total}$ is

$${\mathcal{L}}_{total}={\mathcal{L}}_r(I_t,{\widehat{I}}_{t^{\prime }\to t})+\beta {\mathcal{L}}_{is}({\widehat{D}}_t)+\gamma {\mathcal{L}}_f({\widehat{D}}_t)+\delta {\mathcal{L}}_{flo}({\widehat{D}}_t)$$

(6)

where $\beta$, $\gamma$, and $\delta$ are term weight factors. The pipeline of our method is shown in Figure 2.

4.1. Road Surface Flatness Loss

Road surface is a category that almost always appears in semantic segmentation labels of autonomous driving scene images and often has only a single instance. As road surface regions are low-texture in raw images, it is hard to accurately estimate distance for these pixels by optimizing image reconstruction loss. Moreover, as the depth of the road surface varies drastically from near to far, smoothness loss measuring gradient of estimated distance is not suitable. In order to regularize the distance of road surface pixels, we consider penalizing their corresponding 3D point cloud to a flat surface.

By Equation (2), a 3D point set is obtained from image coordinates and estimated distance ${\widehat{D}}_t$. A 3D point $X={(x,y,z)}^T$ is on a plane, if it satisfies $ax+by+cz=1$, where $n^p=(a,b,c)$ is plane parameter. Since $n^p$ is unknown, we first estimate them from the 3D point cloud before computing the point-to-plane distance errors for each 3D point. Given a set of 3D points whose labels are road surface, we can solve the plane parameter by solving a linear least-square problem,

$$n_t^p=\underset{n^p}{argmin}\sum _{p_t\in I_t\odot M_t^{road}}{(n^p{\mathsf{\Pi }}^{-1}(p_t,{\widehat{D}}_t)-1)}^2$$

(7)

where $M_t^{road}$ is the binary mask of road surface pixels, generated from semantic segmentation annotations. Thus, we define the road surface flatness loss to penalize road surface point cloud to a single plane by the L1 residual error,

$${\mathcal{L}}_f({\widehat{D}}_t,M_t^{road})=\sum _{p_t\in I_t\odot M_t^{road}}\left|n_t^p{\mathsf{\Pi }}^{-1}(p_t,{\widehat{D}}_t)-1\right|$$

(8)

Due to the correlation between $n_t^p$ and ${\widehat{D}}_t$, calculating gradients for ${\mathcal{L}}_f$ becomes numerically unstable. To address this, we detach $n_t^p$ from the computational graph during training, effectively stopping gradient back-propagation through this variable.

4.2. Instance Smoothness Loss

As depths of pixels from the same objects vary smoothly, we regularize distance for every instance regions separately. Given the instance segmentation label, we can first calculate an instance division map $M_t^{ins}$. If pixels $(x,y)$, $(x-1,y)$, and $(x,y-1)$ in $I_t$ are from the same instance, $M_t^{ins}(x,y)=1$; otherwise, $M_t^{ins}(x,y)=0$. Thus, we define the instance smoothness term as

$${\mathcal{L}}_{is}({\widehat{D}}_t,M_t^{ins})=(\left|{\nabla }_x{\widehat{D}}_t^{\ast }\right|+\left|{\nabla }_y{\widehat{D}}_t^{\ast }\right|)\odot M_t^{ins}$$

(9)

where ${\widehat{D}}_t^{\ast }$ is the mean-normalization of $1/{\widehat{D}}_t$.

4.3. Optical Flow Loss

To further improve the accuracy of distance estimation, state-of-the-art optical flow approaches can provide constraint for mapping from pixel set $p_t$ in $I_t$ to projecting coordinates ${\widehat{p}}_{t^{\prime }}$ at time $t^{\prime }$. Additionally, the optical flow constraint can compensate for the reconstruction term’s limitations in aligning dynamic objects. In our method, pretrained model FlowFormer [16] is adopted to obtain optical flow labels $fl{o}_{t\to t^{\prime }}$. Thus, the optical flow loss is defined by L1 loss as

$${\mathcal{L}}_{flo}({\widehat{D}}_t)=\parallel {\widehat{p}}_{t^{\prime }}-p_t-fl{o}_{t\to t^{\prime }}\parallel$$

(10)

where ${\widehat{p}}_{t^{\prime }}$ is calculated by Equation (3). Moreover, as the optical flow labels generated by existing model are sometimes not accurate in some low-texture regions, to mitigate the impact of error labels, L1 error larger than 3.0 for each pixel is clipped to zero gradient.

4.4. Implementation Details

Following previous works [6,8], training distance estimation along with semantic segmentation in a multi-task framework simultaneously improves the performance of a single self-supervised distance estimation network. The distance estimation network, pose estimation network, and semantic segmentation network are all based on an encoder–decoder architecture. ResNet18 [17] is chosen as the encoder in the baseline and our approach. The network details of distance estimation and pose estimation network are the same with FisheyeDistanceNet [7]. The semantic segmentation and the distance estimation network share the same encoder. Layers of semantic segmentation decoder are similar to corresponding ones in distance estimation decoder expect the predicting head.

We employ Adam [18] optimizer for training with a learning rate of ${10}^{-4}$. The model is trained for 100 epochs with a batch size of 24 on an RTX 3090 GPU, requiring approximately 10 h to complete training. The input resolution of the fisheye image is $480\times 320$ pixels. Different from previous works which crop the original fisheye image to remove regions of vehicle’s body, to preserve more valid regions, we use a binary mask obtained from semantic segmentation annotation. The mask is used to calculate ${\mathcal{M}}_{t{t}^{\prime }}$ in Equation (4).

5. The Experimental Verification

5.1. Datasets

WoodScape dataset [9]. The WoodScape dataset contains real-world 8234 images obtained with four surround-view fisheye cameras from driving scenes. It contains raw images of two consecutive frames, vehicle speed, and time interval, which can be used to predict ego-motion. In addition, it consists of instance and semantic segmentation annotations. However, depth estimation ground truth is not available. The dataset is split into a training set of 8029 images and a test set of 205 images.

SynWoodScape dataset [19]. The SynWoodScape dataset is a synthetic counterpart of the real-world WoodScape dataset, maintaining nearly identical semantic class distributions across urban driving scenarios. It provides 2000 pairs of synthetic fisheye images along with corresponding depth ground truth for four surrounding views.

5.2. Quantitative Evaluation

We adopt the depth estimation accuracy metrics provided in [20], which are used as a standard in some previous works [5,7]. The metrics contain the mean absolute relative error (Abs Rel), the root mean square relative error (Sq Rel), the root mean square error (RMSE), the root mean square log error (RMSE log), and the $\delta$-accuracy.

Since the WoodScape dataset contains 8234 real-world images but lacks depth ground truth, and the SynWoodScape dataset contains a smaller number of synthetic images with ground truth, we aim to evaluate the prediction accuracy of the depth estimation model while ensuring its generalization ability across different datasets. Therefore, in the quantitative evaluation experiments, we merge the two datasets for training and testing. The training set consists of 8029 images from the WoodScape dataset and 1800 randomly selected images from the SynWoodScape dataset. The testing set comprises the remaining 200 images from the SynWoodScape dataset, which include depth ground truth.

Since the self-supervised models [6,8,21] for fisheye images lack open-source implementations, our comparative analysis uses FisheyeDistanceNet [7] as the primary baseline, based on the authors’ released code. Additionally, we present the results of the proposed method with certain loss terms removed. To ensure fairness, the network architecture and training details for each method are identical, except for the differences in the loss functions. The depth estimation evaluation metrics are presented in

Table 1. Quantitative results on SynWoodScape dataset. Note: The arrows ↓ and ↑ indicate that smaller values are better and larger values are better, respectively. The bold values indicate the optimal results.

Method	Abs Rel↓	Sq Rel↓	RMSE↓	RMSE log↓	$\mathit{\delta }\mathbf{<}\mathbf{1.25}\mathbf{\uparrow }$	$\mathit{\delta }\mathbf{<}{\mathbf{1.25}}^{\mathbf{2}}\mathbf{\uparrow }$	$\mathit{\delta }\mathbf{<}{\mathbf{1.25}}^{\mathbf{3}}\mathbf{\uparrow }$
FisheyeDistanceNet [7]	0.289	7.696	9.464	0.330	0.830	0.915	0.950
Ours(${\mathcal{L}}_r+{\mathcal{L}}_{is}$)	0.132	2.032	9.430	0.263	0.830	0.920	0.959
Ours(${\mathcal{L}}_r+{\mathcal{L}}_f$)	0.146	2.426	9.797	0.274	0.814	0.910	0.954
Ours(${\mathcal{L}}_r+{\mathcal{L}}_{is}+{\mathcal{L}}_f$)	0.139	2.096	9.458	0.264	0.815	0.914	0.958
Ours(${\mathcal{L}}_r+{\mathcal{L}}_{is}+{\mathcal{L}}_f+{\mathcal{L}}_{flo}$)	0.128	1.995	9.473	0.256	0.831	0.920	0.959

. Our proposed instance smoothness term achieves smaller depth estimation errors compared to FisheyeDistanceNet’s edge-aware smoothness term. Comparing the second and fourth rows in Table 1, the addition of the road surface flatness term slightly reduces depth estimation accuracy. This performance trade-off likely stems from the fact that real-world road surfaces are not perfectly planar. However, the flatness constraint remains essential to prevent significant distance estimation errors along the road surface, ensuring reasonable depth prediction consistency for road areas. Furthermore, the addition of the optical flow term further improves the accuracy of depth estimation.

5.3. Qualitative Evaluation

In qualitative evaluation, we compare our method with FisheyeDistanceNet, as well as our method with certain loss term removed. In Figure 3, all models are trained and evaluated on the WoodScape dataset. The figure presents experimental results from four real-world scenes, shown from left to right. In the first and fourth examples, when the fisheye camera is mounted at the front and rear of the vehicle, FisheyeDistanceNet tends to incorrectly predict nearby ground regions as being at a farther distance. In contrast, our method effectively corrects this issue by jointly leveraging the instance smoothness loss and road surface flatness loss. Comparing Figure 3b,c, our instance smoothness loss achieves smoother depth estimations within each instance compared to the edge-aware smoothness loss used in FisheyeDistanceNet. Furthermore, as shown in Figure 3e,f, the optical flow loss further refines the depth estimation results.

In Figure 4, both our method and FisheyeDistanceNet are trained on the combined WoodScape and SynWoodScape datasets and evaluated on the SynWoodScape test set. Our method produces depth estimation results that are closer to the ground truth compared to FisheyeDistanceNet. However, it still shows limited accuracy on some fine-scale objects. The limited training data results in inadequate representation of fine-scale objects that infrequently occur in road environments, consequently causing degraded prediction performance for these under-represented categories.

6. Conclusions

In this paper, a weakly supervised learning framework for monocular fisheye distance estimation is proposed. First, to ensure accurate depth estimation in low-texture road surface regions, a planar constraint is introduced on the 3D point cloud of the ground area. Second, instance segmentation labels are used to enforce smoothness in depth estimation within each instance. Third, optical flow fields generated by a state-of-the-art optical flow estimation model are incorporated to further enhance image registration accuracy. By integrating these constraints into the loss function, our method achieves more accurate depth estimation compared to self-supervised approaches on both real-world and synthetic datasets. However, depth estimation accuracy for fine-grained objects still has room for improvement, particularly for low-frequency categories that lack sufficient training samples. Future work will address this through targeted dataset expansion to better represent these under-represented objects.