Milepost-to-Vehicle Monocular Depth Estimation with Boundary Calibration and Geometric Optimization

Abstract

Milepost-assisted positioning estimates the distance between a vehicle-mounted camera and a milepost as a reference position for autonomous driving. However, the accuracy of monocular metric depth estimation is compromised by camera installation angle, milepost inclination, and image occlusions. To solve the problems, this paper proposes a two-stage monocular metric depth estimation with boundary calibration and geometric optimization. In the first stage, the method detects a milepost in one frame of a video and computes a metric depth map of the milepost region by a monocular depth estimation model. In the second stage, in order to mitigate the effects of road surface undulation and occlusion, we propose geometric optimization with road plane fitting and a multi-frame fusion strategy. An experiment using pairwise images and depth measurement demonstrates that the proposed method exceeds other state-of-the-art methods with an absolute relative error of 0.055 and root mean square error of 3.421.

1. Introduction

Mileposts provide the mileage information between a current location and a road starting point. Milepost-assisted localization technology estimates the distance between a vehicle and a milepost, offering reliable mileage references for constructing and updating high-precision maps in autonomous driving systems. For instance, global navigation satellite system (GNSS) signals may be obstructed or disrupted in a mountainous area, leading to inaccurate navigation positioning. In such scenarios, the integration of a milepost coordinate and the relative positions between the milepost and a vehicle can guarantee localization accuracy.

Milepost-assisted positioning technologies can be categorized into three directions: the matching between map and road features, the multi-sensor fusion, and the global position system (GPS) and image fusion. The first direction aligns high-precision maps with road features (e.g., lane markings and traffic signs) to assist positioning [1,2,3], offering high accuracy and requiring minimal hardware resources. However, the high cost of constructing and updating high-precision maps limits their application, especially in areas with indistinct road features.

The second direction integrates position data from multiple sources, including GPS, inertial navigation systems (INS), vision sensors, and odometry, achieving millimeter-level positioning accuracy [4,5]. Though this direction provides superior precision, it heavily relies on high-performance hardware. Specifically, the reliability of the direction decreases in cases of sensor failure or extreme terrains.

The final direction utilizes onboard cameras to capture video or images, extracting the location information of mileposts through object detection, which are then combined with GPS coordinates for precise positioning [6,7]. However, there is a distance deviation between the onboard camera and the milepost due to their non-parallel alignment in the third step in Figure 1. The existing solutions primarily involve adjusting the camera installation angle for compensation. The uncertainty of the location of the mileposts and the fixed camera angle may lead to the omission of the targets. Furthermore, onboard cameras need to capture road features ahead, and adjusting the angle for milepost detection may weaken the detection of other objects, reducing coverage and overall task efficiency.

To address the problem of depth estimation in milepost-assisted positioning, this paper proposes a two-stage monocular metric depth estimation method with boundary calibration and geometric optimization. In the first stage, an unsupervised monocular depth estimation network is first employed to generate a relative depth map. A scale recovery module based on a lower field-of-view road boundary is then introduced during the post-processing phase to convert the relative depth into a metric depth map. Subsequently, the milepost region is identified using an object detection algorithm, and its average metric depth is extracted as the initial estimate of the camera-to-milepost distance. In the second stage, a road plane is fitted using SVD and RANSAC, and the milepost depth is projected along the road longitudinal direction to correct errors caused by camera installation angle and milepost inclination. Finally, a multi-frame geometric fusion module aligns and fuses depth information from adjacent frames to obtain a final depth result. An experiment with 352 real-world samples is used to demonstrate the effectiveness of the proposed method. The main contributions of this study can be summarized as follows:

(a) The proposed monocular metric depth estimation method achieves superior performance compared to existing metric depth prediction-based and relative depth scaling-based approaches, while maintaining stable results under various real-world conditions such as different distance ranges, illumination, and road surfaces.

(b) By leveraging the lower field-of-view road boundary, the proposed scale recovery module effectively resolves the inherent scale ambiguity in monocular depth estimation. In the first stage, a scale recovery module based on the lower field-of-view road boundary effectively resolves the inherent scale ambiguity in monocular depth estimation, enabling accurate metric depth recovery.

(c) Road-plane-based geometric optimization mitigates depth estimation errors caused by non-parallel camera installation and milepost inclination, while multi-frame geometric fusion enhances robustness against occlusions and illumination variations.

2. Related Works

Research on monocular depth estimation has achieved remarkable progress in recent years, with numerous approaches proposed to enhance depth accuracy and robustness under complex scenarios. At the same time, how to recover a reliable metric scale from inherently relative depth predictions remains a critical challenge. In this section, we first review representative approaches to monocular depth estimation, and then summarize scale recovery algorithms that are particularly relevant to our work.

2.1. Monocular Depth Estimation

Monocular depth estimation predicts the depth information of an object in a single image. In recent years, deep learning methods have achieved significant success in this field.

Supervised learning-based methods rely on large-scale training datasets with ground-truth depth labels for model optimization. Eigen et al. [8] were the first to employ a convolutional neural network (CNN) architecture for depth estimation; this method pioneered the use of neural networks in this field. Subsequently, numerous studies have significantly improved depth estimation accuracy and detail preservation by enhancing network architectures [9,10,11] or introducing novel optimization strategies [12,13]. However, these methods are highly dependent on large-scale, high-quality depth-annotated datasets, which are costly and labor-intensive to acquire.

Semi-supervised learning-based methods train models by leveraging a small amount of labeled data along with a large amount of unlabeled data. It can reduce dependence on depth-annotated datasets. These methods typically incorporate 3D LiDAR data [14], semantic information [15,16], stereo disparity maps [17] or optical flows [18] as auxiliary supervision signals to improve depth estimation accuracy. However, semi-supervised learning-based methods involve complex loss function designs and hyperparameter tuning, which increase the difficulty of model training while also imposing higher computational resource requirements.

Unsupervised learning-based monocular depth estimation methods require only an image sequence as input during training. It is beneficial for eliminating the need for ground-truth depth labels. Early approaches can be categorized into image pair-based training [19] and image sequence-based training [20]. The former reconstructs images using disparity maps, employing reconstruction error as the supervision signal, while the latter jointly estimates camera pose and depth information to address the scale ambiguity problem. Subsequent studies have improved depth prediction accuracy and robustness in complex scenes by optimizing network architectures [21,22]. Additionally, recurrent depth estimation models [23] and continual learning frameworks [24] have further enhanced the adaptability and generalization capability in dynamic environments. By leveraging weather-related augmentations together with a pseudo-supervised loss for depth and pose, Robust-Depth effectively addressed the challenging conditions such as nighttime, fog, and rain [25]. However, unsupervised learning methods often feature complex network structures, leading to increased inference time and higher memory consumption.

2.2. Monocular Depth Estimation Scale Recovery Algorithms

Monocular metric depth estimation can be understood as a problem of depth ranking and depth scale recovery [26]. Due to the lack of multi-view information and scene geometric constraints, monocular cameras provide only relative depth estimates, leading to the scale ambiguity problem. Depth scale recovery refers to the process of converting the relative depth obtained from relative depth into metric depth.

Currently, depth scale recovery methods can be broadly categorized into two main approaches. The first approach involves directly predicting metric depth using monocular depth estimation models. Zero-shot methods [27,28,29,30,31] train models on a source dataset and directly apply them to a target dataset without requiring specific camera intrinsics or prior scene knowledge. These methods can output metric depth values for arbitrary unknown environments. They rely on strong geometric consistency across domains, making them sensitive to domain shifts. Weakly supervised methods incorporate non-depth information (e.g., GPS [32], velocity [33], surface normal [34], LiDAR [35], scene semantics [36] and virtual-SfM signals [37] as auxiliary supervision signals to refine the metric depth scale. MonoPP leverages monocular video and known camera mounting geometry using planar-parallax reconstruction to achieve self-supervised metric-scaled depth predictions [38]. While these methods reduce reliance on dense depth labels, they require high-quality sensor data, making them heavily dependent on hardware capabilities. Hybrid supervision methods combine synthetic and real-world data, using style transfer techniques [39,40] to enhance model adaptability. PatchRefiner refines high-resolution metric depth using a tile-based architecture and a detail-and-scale disentangling loss, achieving strong performance across synthetic and real-world datasets [41]. However, the distribution gaps between synthetic and real-world data lead to limited generalization performance, reducing the effectiveness of models in real-world scenarios.

The second method estimates relative depth using a monocular depth estimation model. Then they calculate a scale factor between the relative and metric depth values to achieve scale recovery. Zhang et al. [42] utilized pose data from navigation sensors and applied a multi-view geometric model to compute the scale factor between relative and metric depth. Masoumian et al. [43] employed curve fitting techniques to establish a mapping between relative depth values and known ground-truth depth values, optimizing the scale factor using least squares minimization. However, this method requires extensive real-world depth data and assumes a uniform and diverse data distribution. Dana et al. [44] proposed a field-of-view correction method, adjusting the source dataset’s FoV to match the target dataset, and then computing a depth correction scale factor using the ground-truth depth from the source dataset. This method is effective for FoV discrepancies but struggles with more complex geometric domain shifts. Zhu et al. [45] decomposed metric depth estimation into scene scale prediction and relative depth estimation, improving scale recovery accuracy. McCraith et al. [46] and Xue et al. [47] utilized known camera height and estimated road plane distances to compute the scale factor. Yang et al. [48] further introduced a frame buffering algorithm, using adjacent frames to optimize ground point sets, reducing the impact of noisy points in single-frame images on depth correction. Although this method mitigates scale bias, it still relies on ground-truth depth data to compute the scale factor. To address these limitations, we propose a multi-frame depth scale recovery method based on boundary calibration and geometric fusion. The proposed method eliminates the need for extensive real-world depth data, complex model training, and hardware-dependent constraints, providing a more practical and efficient solution for monocular metric depth estimation.

3. Methodology

3.1. Overview

This paper proposes a two-stage monocular metric depth estimation with boundary calibration and geometric optimization to estimate the distance between a vehicle and a milepost. The procedure can be summarized in the following steps, as shown in Figure 2.

In Stage 1, a boundary-calibrated monocular metric depth estimation model is proposed to generate a metric depth map. This stage consists of Section III (B), III (C), and III (D). The model first utilizes an unsupervised monocular depth estimation network to predict a relative depth map from the input image. The monocular depth estimation network will be introduced in Section III (B). Subsequently, a scale recovery module based on a lower field-of-view road boundary is introduced as a post-processing step. The road lower boundary is a line of pixels at the bottom of the image, such as the red one in Figure 3 The average depth along this boundary is used to compute a global scale factor. The global scale factor transforms the relative depth map into the metric depth map. The detailed implementation of the scale recovery process is described in Section III (C). Finally, a YOLOv11 object detection algorithm [49] is applied to locate the milepost region, from which the average metric depth value is extracted as the initial estimate of the distance between the vehicle and the milepost. The detailed procedure for milepost detection and image coordinate extraction based on YOLOv11 is presented in Section III (D).

In Stage 2, to further optimize the initial metric depth, a geometric optimization process is introduced. This stage consists of Section III (E) and III (F). First, a road longitudinal direction vector is computed by a road plane fitting method based on value decomposition (SVD) and random sample consensus (RANSC). The road longitudinal direction vector is used to optimize the metric depth map from Stage 1, avoiding the negative effects of the camera installation angle and milepost inclination. Thus, an optimized metric depth map is obtained. The details of the road plane fitting model are described in Section III (E). Considering that the milepost region in certain frames may be occluded or unclear, which could lead to inaccurate depth estimation, a multi-frame fusion strategy is applied. The strategy first computes the frame-by-frame pose transformation matrices by the pose estimation. The pose transformation matrices then convert the optimized metric depth maps of adjacent frames from Section III (E) into the coordinate system of the current frame if the occluded and unobvious exist. Next, the metric depth maps in the same coordinate system are integrated by a weighted averaging method. Finally, the optimized metric depth of the milepost is output as the distance between the vehicle and the milepost. The multi-frame fusion models are described in Section III (F).

3.2. YOLOv11-Based Milepost Detection and Positioning

YOLOv11 [49] is the latest generation of object detection algorithm released by Ultralytics in September 2024. Its overall architecture consists of three main components: the backbone, the neck, and the detection head. The backbone is responsible for extracting image features and mainly includes the Conv module, C3K2 module, SPPF module, and C2PSA module. The detection head performs the final object detection tasks. It supports higher input resolutions with minimal latency and provides stronger small-object detection capability, making it well-suited for distant mileposts under varying illumination and occlusion. In this study, an improved version of the YOLO v11 algorithm is adopted to automatically detect and localize milepost targets in road scene images.

To construct a reliable dataset for YOLO v11 training, a large number of road front-view images were collected from highways across multiple regions, including Shandong and Jiangsu provinces, using a multifunctional detection vehicle. Among these, images containing visible milepost targets were manually selected. Each selected image was annotated with bounding boxes to label the position of the milepost, resulting in a labeled dataset for supervised learning. A total of 2547 annotated images were prepared and split into training, validation, and test sets in a ratio of 7:2:1, yielding 1783 training images, 509 validation images, and 244 test images. This dataset serves as the foundation for training and evaluating the YOLOv11 model, ensuring its robustness and generalization under diverse road scenarios.

The trained YOLOv11 model is applied to the input image sequence in a frame-by-frame manner to automatically identify the presence of milepost targets and output their bounding box coordinates in the image plane. These detection results are used to locate the corresponding milepost regions in the monocular metric depth maps. By mapping the bounding box coordinates onto the depth map, the depth values within the detected regions can be effectively extracted. Subsequently, the average depth value within the region is computed to provide an initial estimate of the distance between the camera and the milepost.

3.3. Unsupervised Learning Monocular Relative Depth Estimation Model

In this section, the milepost region is converted into a relative depth map using the model, where the relative depth provides a depth reference for metric depth optimization. Figure 4 presents the network architecture of the unsupervised learning monocular relative depth estimation model, which integrates a vision transformer-large (ViT-L) as the encoder and a dense prediction transformer (DPT) as the decoder.

The ViT-L encoder consists of 24 transformer layers, learning the global contextual information related to scene structure, object boundaries, depth relationships, and spatial coherence effectively. The encoder extracts multi-scale features from the 6th, 12th, 18th, and 24th transformer layers, capturing a hierarchy from high-resolution edge details at shallow layers to low-resolution semantic features at deeper layers. These multi-scale features are directly passed to the decoder through skip connections, ensuring semantic consistency while refining depth estimation details. The DPT decoder further processes these features to reconstruct a high-quality relative depth map, maintaining both fine-grained depth details and global scene understanding.

The DPT decoder restores the feature resolution through progressive upsampling. The decoder fuses the current feature map with the corresponding encoder feature map, ensuring effective utilization of multi-level information. The decoder improves the clarity of depth map details by incorporating low-level edge information. At the same time, it preserves semantic completeness by integrating high-level global features. Finally, a fully connected convolutional layer maps the high-dimensional feature representation into a single-channel depth map. Each pixel value in this map corresponds to the relative depth of the point in the scene.

The model training process consists of two stages: teacher model training and student model training. In the teacher model training stage, the model is trained by a dataset of 595 K high-quality synthetic images, covering a wide range of scenes and complex layouts while providing precise depth supervision. Multiple loss functions are employed to optimize depth prediction, ensuring robustness across different conditions. These loss functions are derived from the methods described in [50]. The scale-shift invariant loss $L_{ssi}$ is used to maintain consistent in-depth predictions under varying scales and shifts. The loss function can be calculated as

$$L_{ssi}={\displaystyle \frac{1}{N}}{\sum }_{i=1}^N\Vert log\widehat{d_i}-log{d}_i{\Vert }^2-{\displaystyle \frac{1}{N}}{\left({\sum }_{i=1}^N\left(log\widehat{d_i}-log{d}_i\right)\right)}^2,$$

(1)

where $N$ denotes the number of pixels used for loss computation, $\widehat{d_i}$ is the predicted depth values and $d_i$ represents the ground truth depth values.

The gradient matching loss $L_{gm}$ is employed to enhance the sharpness of depth boundaries. The loss function can be calculated as

$$L_{gm}={\displaystyle \frac{1}{N}}{\sum }_{i=1}^N\Vert {\nabla }_x\widehat{d_i}-{\nabla }_x{d}_i{\Vert }^2+\Vert {\nabla }_y\widehat{d_i}-{\nabla }_y{d}_i{\Vert }^2,$$

(2)

where ${\nabla }_x$ and ${\nabla }_y$ represent the gradient computed in the horizontal and vertical depth variation.

In the student model training stage, the teacher model is first used to generate pseudo-depth labels for 62 M unlabeled real-world images. These high-quality pseudo labels are then leveraged to train the student model, further enhancing its generalization capability in real-world scenarios. To ensure that the feature representations of the student model remain aligned with those of the teacher model, a feature alignment loss $L_{align}$ is introduced during training. The loss function can be calculated as

$$L_{align}={\displaystyle \frac{1}{N}}{\sum }_{i=1}^N\Vert F_s\left(x_i\right)-F_t{\left(x_i\right)\Vert }^2,$$

(3)

where $F_s\left(x_i\right)$ and $F_t\left(x_i\right)$ represent the feature representations of the student model and teacher model at pixel $x_i$.

3.4. Scale Recovery Module Based on Lower Field-of-View Road Boundary

The relative depth maps in Section III (B) require a reference for metric depth conversion. The proposed depth calibration model uses the distance between the camera and the boundary of the road surface at the bottom boundary as the reference since the distance is fixed and can be pre-measured once the camera is fixed in a vehicle, as shown in the red dashed line of Figure 3. The relative depth maps in Section III (B) can be recovered into the metric depth map using the known depth information at the field-of-view boundary, without requiring additional calibration procedures.

The depth calibration model first obtains the road surface points in the low boundary region of the camera view field. A U-Net network is adopted to obtain the surface points in the input frame. Let the semantic segmentation result of the input frame be denoted as $S\left(u,v\right)$. If the prediction class of a pixel is road surface, the set of road surface pixels $p_r$ as

$$p_r=\left\{\left(u,v\right)|S\left(u,v\right)=l_r\right\},$$

(4)

where $\left(u,v\right)$ represents the 2D coordinates of a pixel, $l_r$ represents the road class label.

The road surface pixels in the low boundary region of the camera view can be refined by selecting pixels from the last row of the frame. The set of road pixels in the bottommost row $p_b$ can be calculated as

$$p_b=\left\{\left(u,v\right)\in p_r|v\in \left(\left.H-1,H\right]\right.\right\},$$

(5)

where $H$ is the frame height. Equation (5) selects only those road pixels whose vertical coordinate $v$ lies in the last row of the frame, ensuring that only the pixels at the lower boundary of the camera’s field of view are considered.

The depth value set the field-of-view boundary road pixels $D_b$ for the FoV boundary road pixels is defined based on the selected boundary pixel set $p_b$ as

$$D_b=\left\{D\left(u,v\right)|\left(u,v\right)\in p_b\right\},$$

(6)

where $D\left(u,v\right)$ represents the relative depth value at the pixel $\left(u,v\right)$.

The median depth of the field-of-view boundary road is computed to obtain the preliminary estimation of the road surface $D_{r,est}$ as

$$D_{r,est}=median\left(D_b\right).$$

(7)

Subsequently, a scaling relationship between the true depth and the estimated depth is established by conducting a real-world measurement of the true metric depth between the onboard camera and the field-of-view boundary road region. The depth scale factor $f$ is defined as

$$f=D_{r,tru}/D_{r,est},$$

(8)

where $D_{r,tru}$ represents the measured metric depth of the road at the field of view boundary.

Using the depth scale factor calibration formula, all relative depth values in the depth map are initially converted into metric depth values $D_{ab}\left(u,v\right)$ as

$$D_{ab}\left(u,v\right)=f\cdot D_{rel}\left(u,v\right),$$

(9)

where $D_{rel}\left(u,v\right)$ represents the relative depth at the pixel $\left(u,v\right)$ in the relative depth map.

3.5. Road Plane Fitting Method Based on SVD and RANSAC Optimization

The depth estimation in Equation (9) may deviate from the actual distance owing to the effects of the camera installation angle and milepost inclination, as shown in Figure 5. Since the road plane serves as a crucial geometric reference for the vehicle-camera relationship, its plane normal vector can be transformed into a longitudinal directional vector along the road, providing geometric constraints between the vehicle and the milepost. However, the road surface point cloud used for plane fitting contains a significant amount of noise points. This study incorporates SVD and RANSAC for iterative optimization, filtering out outliers, and refining both the plane normal vector and the longitudinal directional vector in Figure 6. This optimization provides a reliable geometric reference for subsequent depth projection and scale recovery.

First, by using the intrinsic parameters of the camera and the depth map, the set of road pixels obtained in Section III (C) is projected into 3D space. The road pixel set is transformed into a 3D point cloud $\left\{P_i^r\right\}$ as

$$P_i^r=K^{-1}\cdot D_{ab}\left(u,v\right)\cdot \left[\begin{array}{c}u\\ v\\ 1\end{array}\right], \forall \left(u,v\right)\in p_r,$$

(10)

where $K$ is the camera intrinsic matrix.

Before performing road plane fitting, the number of 3D road surface points is first evaluated. If the number of road surface points is less than 3, it indicates that “No Valid Road Information Exists”, making plane fitting infeasible. If the number of points is greater than or equal to 3, the road plane fitting process proceeds. Next, the initial parameters for the RANSAC algorithm are defined, including the inlier threshold $\epsilon$. A point is considered an inlier if its distance to the candidate plane is less than $\epsilon$. Additionally, the maximum number of iterations $N$ for RANSAC and the inlier ratio threshold $r_{in}$ for early termination conditions are specified.

In each iteration, $k$ points are randomly selected from the 3D point cloud $\left\{P_i^r\right\}$ to form the sampling point set $P_{sam}=\left\{P_{i_1}^r,P_{i_2}^r,\dots ,P_{i_k}^r\right\}$. The centroid of the sampling point set c is then computed as

$$c={\displaystyle \frac{1}{k}}{\displaystyle \sum }_{j=1}^k{P}_{i_j}^r,$$

(11)

where $k$ is the number of points sampled in each iteration, $P_{i_j}^r$ denotes the $j$-th of selected 3D point from the road point cloud.

After obtaining the centroid of the sampling point set, the offset matrix $A$ is computed and subjected to SVD as

$$A={\left\{P_{i_j}^r-c\right\}}_{j=1}^k$$

(12)

$$A=U\sum V^T$$

(13)

with the plane normal vector $n$ as

$$n=V_{:,-1}/\Vert V_{:,-1}\Vert ,$$

(14)

where $U$ is an $N\times N$ matrix whose column vectors are orthogonal, $\sum$ is an $N\times 3$ diagonal matrix, with the singular values arranged in descending order, $V$ is a $3\times 3$ matrix whose column vectors are the right singular vectors of $A$. The last column vector of $V$, corresponding to the smallest singular value, represents the normal vector direction of n the fitted road plane.

Then the distance between the three-dimensional point set of the road and the candidate road plane is calculated as

$$D_i=\left|n\cdot \left(P_i^r-c\right)\right|/\Vert n\Vert .$$

(15)

For each point $P_i^r$, if the condition $D_i<\epsilon$ is satisfied, the point is classified as an inlier, and all inliers are recorded in the inlier set $P_{in}=\left\{P_i^r|D_i<\epsilon \right\}$. During each iteration, the inlier ratio is computed as

$$r_{cur}=P_{in}/P_i^{road}.$$

(16)

If $r_{cur}>r_{in}$ or the maximum number of iterations $N$ is reached, the iteration stops, and the optimized road plane equation and optimized normal vector $n_{opt}$ are obtained. To further refine the depth estimation, a rotation matrix is introduced to transform the optimized road plane normal vector $n_{opt},$ into a direction vector along the road $s_r$. Specifically, $R$ is constructed to align the estimated normal vector with the global road coordinate system, ensuring that the recovered depth values are geometrically consistent with the actual road surface. The transformation is expressed as

$$s_r=R\times n_{opt},$$

(17)

where $R$ is the rotation matrix aligning the estimated normal vector with the road direction; and $n_{opt}$ is the optimized normal vector of the road plane.

The longitudinal direction vector along the road $s_r$ serves as the optimized depth projection direction, providing a stable geometric foundation for single-frame and multi-frame fused depth estimation. After obtaining the optimized projection direction vector $s_r$, the semantic segmentation model trained in Section III (B) is used to infer the image and extract the milepost pixel set $p_m$ with category label $l_m$. The set of milepost pixels is projected into the three-dimensional space to obtain the three-dimensional point cloud of the milepost $\left\{P_i^m\right\}$ as

$$P_i^m=K^{-1}\cdot D_{ab}\left(u,v\right)\cdot \left[\begin{array}{c}u\\ v\\ 1\end{array}\right],\forall \left(u,v\right)\in p_m,$$

(18)

The point set of the milepost is projected along the vector $s_r$ and the average of the projected point set is taken as the depth after optimizing the current frame. The metric depth estimation between the vehicle and the milepost is obtained as $D_{f\_sin}$ as

$$D_{f\_sin}={\displaystyle \sum }_{i=1}^W{s}_r\cdot P_i^m/W,$$

(19)

where $W$ is the number of points in the milepost set.

3.6. Dynamic Single-Frame Computation and Multi-Frame Fusion Optimization

Due to occlusions and camera perspective limitations, the milepost point set in a single-frame image may not accurately reflect the true depth of the milepost. Directly computing the projection along the road longitudinal normal vector based on the milepost point set from a single frame as the optimized metric depth between the vehicle and the milepost may introduce errors. To address this issue, we propose a multi-frame fusion-based milepost point set optimization method, which integrates information from the current frame and the adjacent previous and next frames to compensate for missing milepost information caused by occlusions or viewpoint limitations in the current frame.

In the multi-frame fusion depth estimation process, to ensure the effectiveness of the fusion, it is necessary to verify that the current frame and its adjacent frames both contain valid milepost information. For frames $t-1$, $t$ and $t+1$, the semantic segmentation model is used to extract the milepost encodings $p_{m,pre}$, $p_{m,cur}$ and $p_{m,nex}$. To determine whether a frame contains valid milepost information, the total number of mileposts encoding pixels $p_m$ is compared with the threshold as

$$Valid\left(p_m\right)=\left\{\begin{array}{c}True, if\left|p_m\right|\ge \tau \\ False, otherwise\end{array}\right.,$$

(20)

where $Valid\left(·\right)$ is a Boolean function that determines whether the frame contains valid milepost encoding information. $\tau$ is a predefined threshold for the minimum required milepost encoding pixels to consider the frame as valid. Frames meeting this criterion are selected as valid frames for depth estimation.

To ensure the reliability of depth estimation, it is necessary to verify that the current frame and its neighboring frames simultaneously contain valid milepost mask information. This condition is satisfied when the validity function for all three frames returns True, which is expressed as

$$Valid\left(p_{m,pre}\right)\wedge Valid\left(p_{m,cur}\right)\wedge Valid\left(p_{m,nex}\right),$$

(21)

If the above conditions are not met, the system falls back to a single-frame computation mode. If the conditions are met, the multi-frame fusion mode is used to compute the metric depth between the vehicle and the milepost. To obtain the pose transformation matrix between the adjacent frames and the current frame, the oriented fast and rotated brief algorithm is used to detect key points in the images and generate descriptors. After detecting key points, the brute-force matcher is applied to match descriptors between adjacent frames, selecting the optimal matching pairs based on matching distance. Once the matched key points are obtained, the pose transformation matrices between adjacent frames are estimated. These matrices are then used to align the milepost point sets from the previous and next frames with the current frame’s coordinate system, as

$$\begin{array}{c}{P}_{pre\_cur}=\left\{T_{pre\_cur}\cdot P|PϵP_{m,pre}\right\}\\ P_{nex\_cur}=\left\{T_{nex\_cur}\cdot P|PϵP_{m,nex}\right\}\end{array},$$

(22)

where $P_{pre\_cur}$ and $P_{nex\_cur}$ represent the transformed milepost point set from the previous and next frame to the current frame’s coordinate system, respectively. $T_{pre\_cur}$ and $T_{nex\_cur}$ are the estimated transformation matrices from the previous and next frame to the current frame, respectively. $P_{m,pre}$ and $P_{m,nex}$ denote the sets of milepost points extracted from the previous and next frames, respectively.

According to Equation (19), the projected depth values along the depth projection direction vector are computed for the transformed milepost point sets from different frames. The projected depth values $D_{cur}$, $D_{pre\_cur}$ and $D_{nex\_cur}$ are fused based on milepost occlusion, image quality, and other factors. The final multi-frame optimized metric depth estimation $D_{f\_mul}$ between the vehicle and the milepost is computed as

$$D_{f\_mul}={\displaystyle \frac{\sum w_i\cdot D_i}{w_i}},i\in \left\{\mathit{pre\_cur}, cur,\mathit{nex\_cur}\right\},$$

(23)

where $D_{cur}$ is the projected depth value from the current frame. $D_{pre\_cur}$ and $D_{nex\_cur}$ are the projected depth values from the previous and next frame, aligned with the current frame. $w_{cur}$, $w_{pre\_cur}$ and $w_{nex\_cur}$ are the weight coefficients for the current, previous, and next frames, respectively.

The pseudocode for the dynamic single-frame computation and multi-frame fusion depth scale recovery is as Algorithm 1.

Algorithm 1. Geometric optimization for monocular metric depth estimation (GO-MMDE)

Input:  $I_{pre}, I_{cur}, I_{nex}$: Input images     K: Camera intrinsic matrix     $D_{r,tru}$: True depth of lower boundary road points Output: Df: Estimated metric depth of milepost

4. Results and Discussion

4.1. Implementation Details

Figure 7 shows the process of data collection in the experiment. A milepost was custom-made and placed at a designated location on the roadside according to standard regulations [51]. A high-precision handheld laser rangefinder was used for distance measurement with a resolution of $\pm 3 \mathrm{mm}+5\times {10}^{-5}\times \varphi$, where $\varphi$ represents the measured distance in the unit of mm. The measurements include the distance between the camera to the low boundary of the road surface and the distance between the camera and the milepost.

A total of 352 images were collected from 19 distinct real-world scenarios. These scenarios include a variety of illumination conditions and road conditions. Images from the same scenario are grouped together, and within each group, the images are sorted in ascending order of distance.

The programming and execution of all experiments were carried out using Python 3.10. For model development, training, and evaluation, we employed the PyTorch 2.2.0 framework with CUDA 12.1 for GPU acceleration. The training process was executed on an NVIDIA RTX 3080 GPU with 12 GB memory, running on Ubuntu 22.04 LTS.

The performance of the proposed method was evaluated using five indicators: absolute relative error, square relative error, root mean square error (RMSE), log-scale RMSE, and the ratio conditions between the predicted and ground truth values under three different tolerance levels (1.25, 1.25², and 1.25³) as

$$Abs Rel={\displaystyle \frac{1}{N}}{\displaystyle \sum }_{i=1}^N{\displaystyle \frac{\left|D_i^{pred}-D_i^{true}\right|}{D_i^{true}}}$$

(24)

$$Sq Rel={\displaystyle \frac{1}{N}}{\displaystyle \sum }_{i=1}^N{\left({\displaystyle \frac{D_i^{pred}-D_i^{true}}{D_i^{true}}}\right)}^2$$

(25)

$$RMSE=\sqrt{{\displaystyle \frac{1}{N}}{\displaystyle \sum }_{i=1}^N{\left(D_i^{pred}-D_i^{true}\right)}^2}$$

(26)

$$RMSE log=\sqrt{{\displaystyle \frac{1}{N}}{\displaystyle \sum }_{i=1}^N{\left(log\left(D_i^{pred}\right)-log\left(D_i^{true}\right)\right)}^2}$$

(27)

$$\delta =max\left({\displaystyle \frac{D_i^{pred}}{D_i^{true}}},{\displaystyle \frac{D_i^{true}}{D_i^{pred}}}\right)$$

(28)

where $Abs Rel$ measures the relative deviation of the predicted values from the ground truth. $Sq Rel$ reflects larger errors between the predicted and true values. $RMSE$ and $RMSE log$ which quantify the overall error level of the estimation. $\delta <1.25$ (strict condition), $\delta <{1.25}^2$ (loose condition) and $\delta <{1.25}^3$ (looser condition) to assess the stability of the method across different error tolerance ranges.

4.2. Comparison with State-of-the-Arts

Table 1. Results of the comparison study.

Method	Lower Is Better ↓					Higher Is Better ↑
	$\mathit{A}\mathit{b}\mathit{s} \mathit{R}\mathit{e}\mathit{l}$	$\mathit{S}\mathit{q} \mathit{R}\mathit{e}\mathit{l}$	$\mathit{R}\mathit{M}\mathit{S}\mathit{E}$	$\mathit{R}\mathit{M}\mathit{S}\mathit{E} \mathit{l}\mathit{o}\mathit{g}$	Latency	$\mathit{\delta }<1.25$	$\mathit{\delta }<{1.25}^2$	$\mathit{\delta }<{1.25}^3$	FPS
DA V2 [50]	0.212	0.050	5.962	0.257	36	0.449	1.000	1.000	28
ZoeDepth-K [28]	0.216	0.071	7.351	0.330	45	0.463	0.662	1.000	22
Depth Pro [29]	0.238	0.067	8.046	0.318	50	0.344	0.875	0.986	20
Masoumian [43]	0.129	0.042	8.570	0.167	59	0.881	0.960	0.997	17
Dana [44]	0.195	0.042	5.684	0.234	42	0.534	1.000	1.000	24
Ours	0.055	0.008	3.421	0.080	30	0.972	1.000	1.000	33

Bold numbers indicate the best performance, and underlined numbers represent the second-best performance.

presents the overall performance of the depth estimation methods. The Depth Anything V2 (DA V2) [50] is used as the baseline model. To ensure the robustness of the reported improvements, statistical significance testing was conducted by repeating experiments with different random seeds and applying paired t-tests on the resulting error distributions. The results confirmed that the proposed method consistently outperforms the baseline with significance at the 0.05 level.

The metrics of $\delta <1.25$, $\delta <{1.25}^2$ and $\delta <{1.25}^3$ are commonly used to evaluate the depth estimation accuracy and reliability. The proposed method achieves the $\delta <1.25$ of 0.972, demonstrating that it accurately estimates the depth under various real-world conditions. Under $\delta <{1.25}^2$ and $\delta <{1.25}^3$ conditions, the proposed method achieves an accuracy of 1.000, indicating that all predictions meet the required criteria under moderate and relaxed conditions. This also verifies the accuracy and reliability of the proposed method. This further confirms the robustness and reliability of the proposed method when evaluated across different levels of permissible error, ranging from strict accuracy requirements to relaxed thresholds. Figure 8 shows the error percentage trend of six metric depth estimation methods in 19 groups (G1–G19). The zoomed-in section further reveals the differences between our method and the Dana method. The Dana method exhibits significant prediction instability, manifested by severe fluctuations in error and large-scale deviations. From Figure 9, it can be observed that our method exhibits the lowest and most stable error surface at different distances and in different scenarios, further verifying the effectiveness of the proposed two-stage estimation strategy in improving robustness and generalization capability.

Figure 8 and Figure 9 present some failure cases from the proposed methods with high errors. Figure 10 presents two visual explanations of such failures. In Scene 1, the depth estimation is affected by interference from nearby vehicles, causing some target regions’ depth values to shift toward the depth range of adjacent vehicles. Although multi-frame weighted fusion can mitigate the impact of occluded areas on the final depth estimation, its optimization effect remains inferior to that of depth estimation scale recovery in non-interfered scenarios. In Scene 2, the overall frame contrast decreases in a low-light environment, making it difficult to distinguish the features of the distant milepost. This results in information loss in the target region affects the accuracy of depth estimation. In such cases, the effectiveness of weighted fusion significantly declines if successive multi-frame depth information exhibits large deviations or missing values. For Scene 1, temporal smoothing with longer multi-frame fusion and visibility-aware weighting could further alleviate the influence of occlusions from nearby vehicles. For Scene 2, applying low-light enhancement techniques such as gamma correction or exposure normalization can improve contrast and preserve target features under dark conditions.

4.3. Ablation Study

An ablation study is conducted to verify the contribution of the key modules in the proposed method, as shown in

Table 2. Results of the ablation study.

Method	Lower Is Better ↓				Higher Is Better ↑
	$\mathit{A}\mathit{b}\mathit{s} \mathit{R}\mathit{e}\mathit{l}$	$\mathit{S}\mathit{q} \mathit{R}\mathit{e}\mathit{l}$	$\mathit{R}\mathit{M}\mathit{S}\mathit{E}$	$\mathit{R}\mathit{M}\mathit{S}\mathit{E} \mathit{l}\mathit{o}\mathit{g}$	$\mathit{\delta }<1.25$	$\mathit{\delta }<{1.25}^2$	$\mathit{\delta }<{1.25}^3$
No Depth Cal	0.216	0.071	7.351	0.330	0.463	0.662	1.000
No Plane Fit	0.102	0.018	3.622	0.139	0.875	1.000	1.000
No Mul-Fra	0.081	0.013	4.224	0.108	0.960	1.000	1.000
Ours	0.055	0.008	3.421	0.080	0.972	1.000	1.000

Bold numbers indicate the best performance, and underlined numbers represent the second-best performance.

. “No Depth Cal” represents the removal of the scale recovery module based on a lower field-of-view road boundary. “No Plane Fit” indicates the removal of the road plane fitting method based on SVD and RANSAC optimization. “No Mul-Fra” corresponds to the removal of the multi-frame fusion optimization module. “Ours” refers to the full proposed method.

In Table 2, after removing the scale recovery module based on the lower field-of-view road boundary, the $Abs Rel$ increases from 0.055 to 0.216, nearly quadrupling the error; $Sq Rel$ increases from 0.008 to 0.071, expanding by nearly nine times; $RMSE$ increases from 3.421 to 7.351; and $\delta <1.25$ decreases from 0.972 to 0.463, while $\delta <{1.25}^2$ decreases from 1.000 to 0.662. The main reason is that the depth estimation entirely relies on the relative depth maps after removing the scale recovery module, leading to the low performance of converting relative depth values into metric ones. Furthermore, the absence of this module amplifies the impact of the camera field of view, scene geometry, and uneven target distribution, leading to increased depth estimation errors, especially in scenarios where depth estimation heavily depends on boundary points. The resulting error accumulation effect becomes more pronounced.

After removing the module of road plane fitting, the $Abs Rel$ increases from 0.055 to 0.102, $Sq Rel$ increases from 0.008 to 0.018, $RMSE$ increases from 3.421 to 3.622, and $\delta <1.25$ decreases from 0.972 to 0.875. This phenomenon derives from the fact that the module prevents an effective geometric model from being established for the road point set, leading to an increase in depth estimation directional errors. However, since the role of this module mainly focuses on optimizing the geometric relationship of the road plane, its overall impact on global error is relatively minor. Therefore, its influence on depth scale recovery is comparatively moderate.

After removing the module of the multi-frame fusion optimization, the $RMSE$ increases from 3.421 to 4.224, while $Abs Rel$, $Sq Rel$, and $\delta$ shows no significant changes. This is because, after removing this module, depth estimation relies entirely on single-frame information, leading to reduced robustness in complex scenarios (such as dynamic occlusions or illumination variations). Additionally, $\delta <1.25$ value drops significantly, while $\delta <{1.25}^2$ and $\delta <{1.25}^3$, which represent more relaxed conditions, remain relatively stable. This indicates that extreme cases (e.g., occlusions) are relatively rare in the collected dataset, and single-frame estimation can still meet basic accuracy requirements in general scenarios. However, under extreme conditions such as occlusions and noise, multi-frame fusion optimization significantly reduces depth scale recovery errors, improving estimation accuracy and stability.

As shown in Figure 11 and Figure 12, removing the scale recovery module based on lower field-of-view road boundary tjs the error percentage, indicating that this module plays a key role in resolving scale ambiguity in monocular metric depth estimation and confirms it as a core component for ensuring the accuracy of depth recovery. The road plane fitting method based on SVD and RANSAC optimization enhances geometric consistency by filtering out noisy points and refining the overall structure. Due to the limited occlusion in the selected images, the multi-frame fusion module has a relatively limited impact on the overall error. it proves highly effective in more complex conditions involving dynamic occlusions or illumination changes, significantly improving robustness. Overall, the complete method outperforms variants that remove any submodule in all scenarios, further verifying the importance of collaborative work between modules. By leveraging the complementary advantages of each component, the proposed method achieves high-precision and highly robust metric depth estimation between the vehicle and the milepost.

4.4. Stability Study

(1) Distance segment performance evaluation study

To further evaluate the performance of the proposed method across different distance ranges, a distance-segmented error evaluation method is designed. Based on the true distance of the target, the data is divided into three segments: short-range (0–15 m), mid-range (15–30 m), and long-range (above 30 m). This allows for an analysis of the method’s adaptability and limitations under different distance conditions.

Table 3. Performance evaluation of the method across different distance segments.

Method	Lower Is Better ↓				Higher Is Better ↑
	$\mathit{A}\mathit{b}\mathit{s} \mathit{R}\mathit{e}\mathit{l}$	$\mathit{S}\mathit{q} \mathit{R}\mathit{e}\mathit{l}$	$\mathit{R}\mathit{M}\mathit{S}\mathit{E}$	$\mathit{R}\mathit{M}\mathit{S}\mathit{E} \mathit{l}\mathit{o}\mathit{g}$	$\mathit{\delta }<1.25$	$\mathit{\delta }<{1.25}^2$	$\mathit{\delta }<{1.25}^3$
short-range	0.046	0.003	0.667	0.058	1.000	1.000	1.000
mid-range	0.038	0.002	1.070	0.049	1.000	1.000	1.000
long-range	0.079	0.017	5.489	0.113	0.923	1.000	1.000

Bold numbers indicate the best performance, and underlined numbers represent the second-best performance.

presents the performance of the proposed method across different distance segments, where the mid-range segment exhibits the best performance. Except for $RMSE$, all other indicators outperform those in the short-range and long-range segments. This may be because, in the mid-range segment, the pixel size and depth estimation range are in an optimal balance, with less occlusion and noise interference, making depth estimation more likely to converge to the ground truth. In the short-range segment, although $RMSE$ is better than in the mid-range segment; the other six indicators are slightly worse. This may be due to mileposts occupying a larger pixel proportion in close-range images, making depth estimation more sensitive to boundary pixels, which often carry higher noise and uncertainty, thus affecting overall performance. In the long-range segment, the method’s performance declines, likely due to the sparsity of target information, reduced resolution, and increased noise interference at the edges of the field of view, making depth estimation more challenging.

(2) Illumination condition performance evaluation study

To evaluate the performance of the proposed method under different illumination conditions, image datasets were selected from three typical lighting scenarios: strong illumination (daytime), normal illumination (dusk), and low illumination (nighttime). These datasets contain 166 frames, 118 frames, and 68 frames, respectively, allowing for an analysis of the method’s adaptability and limitations across different illumination conditions.

Table 4. Performance evaluation of the method under different illumination conditions.

Method	Lower Is Better ↓				Higher Is Better ↑
	$\mathit{A}\mathit{b}\mathit{s} \mathit{R}\mathit{e}\mathit{l}$	$\mathit{S}\mathit{q} \mathit{R}\mathit{e}\mathit{l}$	$\mathit{R}\mathit{M}\mathit{S}\mathit{E}$	$\mathit{R}\mathit{M}\mathit{S}\mathit{E} \mathit{l}\mathit{o}\mathit{g}$	$\mathit{\delta }<1.25$	$\mathit{\delta }<{1.25}^2$	$\mathit{\delta }<{1.25}^3$
Strong	0.052	0.010	3.891	0.087	0.958	1.000	1.000
Normal	0.044	0.003	1.618	0.055	0.992	1.000	1.000
Low	0.082	0.010	4.368	0.097	0.971	1.000	1.000

Bold numbers indicate the best performance, and underlined numbers represent the second-best performance.

presents the performance of the proposed method under different illumination conditions. Under normal illumination at dusk, the method achieves the highest accuracy and the lowest error. This may be because the moderate lighting and high image quality ensure that the milepost features remain clear and well-defined, improving depth scale recovery accuracy. In Figure 13, under strong illumination conditions, the performance of the proposed method slightly declines, possibly due to surface reflections on the milepost or the presence of shadows caused by strong sunlight, leading to overexposure or uneven brightness in certain areas, which affects depth scale recovery accuracy. However, even in this scenario, the method achieves an $Abs Rel$ of 0.052 and a $RMSE$ of 3.891, indicating that despite the reflection effects, the proposed method maintains high accuracy. Under low-light nighttime conditions, the $Abs Rel$ increases to 0.082, and the $RMSE$ rises to 4.368, indicating a further decline in performance. This may be due to insufficient illumination, which degrades image quality and reduces the visibility of target features, thereby affecting the stability of the algorithm. The last column of Figure 13 shows the images under three illumination conditions (strong, normal, and low illumination), along with the regression fit between the predicted distance and the actual distance, and visualized analysis using edge distribution histograms. The results showed that under normal illumination conditions, the fitting accuracy was the highest, with an R² of 0.97; Under strong and weak light conditions, R² is 0.89 and 0.88, respectively, although slightly decreased, it still maintains a high estimation accuracy. This result indicates that normal illumination is beneficial for the accuracy of depth estimation, while different illumination conditions can have an impact on estimation performance, especially in situations where strong illumination or low illumination imaging is unclear. However, the proposed method still exhibits good robustness and reliability in illumination scenarios.

(3) Road condition performance evaluation study

To evaluate the adaptability and robustness of the proposed method under different road conditions, experiments were conducted on two typical road types: flat roads and curved roads. This analysis aims to investigate the impact of road geometry characteristics on the performance of the proposed method.

Table 5. Performance evaluation of the method across different road conditions.

Method	Lower Is Better ↓				Higher Is Better ↑
	$\mathit{A}\mathit{b}\mathit{s} \mathit{R}\mathit{e}\mathit{l}$	$\mathit{S}\mathit{q} \mathit{R}\mathit{e}\mathit{l}$	$\mathit{R}\mathit{M}\mathit{S}\mathit{E}$	$\mathit{R}\mathit{M}\mathit{S}\mathit{E} \mathit{l}\mathit{o}\mathit{g}$	$\mathit{\delta }<1.25$	$\mathit{\delta }<{1.25}^2$	$\mathit{\delta }<{1.25}^3$
Flat road	0.053	0.008	3.277	0.080	0.970	1.000	1.000
Curved road	0.062	0.008	3.827	0.082	0.957	1.000	1.000

Bold numbers indicate the best performance.

presents the performance results of the proposed method under different road conditions. In flat road scenarios, metrics such as $Abs Rel$ and $RMSE$ show outstanding performance, with errors distributed more compactly and estimation results closer to the ground truth. In curved road scenarios, error metrics show a slight increase, which may be due to the simpler geometric characteristics of flat roads, where lane lines and target object edges are more stable, providing sufficient geometric and texture features for depth estimation scale recovery. In contrast, the complex geometric structure and dynamic perspective changes in curved road scenarios introduce depth uncertainty and viewpoint distortion, leading to increased estimation errors. However, despite these challenges, the overall performance remains satisfactory, demonstrating high accuracy and robustness in depth estimation scale recovery.

5. Conclusions

This paper has proposed a two-stage monocular metric depth estimation with boundary calibration and geometric optimization, aiming to address the unreliable and inaccurate estimation problem due to the camera installation angle, the milepost inclination, and occlusions. An experiment using 352 samples with real images and depth measurement demonstrates the effectiveness of the proposed method. The following conclusions can be drawn.

(1) The proposed method outperforms other state-of-the-art approaches in monocular metric depth estimation, achieving an absolute relative error of 0.055 and a threshold accuracy of 0.972, demonstrating its effectiveness in providing a reliable and accurate depth estimation solution across various real-world conditions.

(2) The improvement of the proposed method is attributed to depth calibration using field-of-view boundary road points and multi-frame fusion optimization. Scale recovery module effectively eliminates scale ambiguity, reducing the Abs Rel error from 0.216 to 0.055. Road-plane-based geometric optimization addresses depth estimation inaccuracies resulting from non-parallel camera installation and milepost inclination, while multi-frame fusion enhances robustness by mitigating the effects of occlusions and camera perspective limitations, improving RMSE from 4.224 to 3.421.

(3) The proposed method exhibits high stability in monocular metric depth estimation, maintaining consistent accuracy across varying environments, including different distances, illuminations, and road conditions. For example, it achieves Abs Rel = 0.082 under low illumination. Across distances, performance is nearly perfect at short range (δ < 1.25 = 1.000) and still reliable at long range (Abs Rel = 0.079). With high accuracy under loose error tolerance levels, it demonstrates robustness and reliability, ensuring stable depth estimation in real-world applications.

(4) The future work should focus on two aspects. First, the proposed method should incorporate dynamic object tracking and occlusion compensation strategies to further improve the robustness of the proposed method on dynamic occlusion scenarios. Second, future studies should integrate high-resolution sensors and multi-modal data fusion strategies to supplement geometric and spatial information for long-distance targets.