InfraredStereo3D: Breaking Night Vision Limits with Perspective Projection Positional Encoding and Groundbreaking Infrared Dataset

Abstract

In fields such as military reconnaissance, forest fire prevention, and autonomous driving at night, there is an urgent need for high-precision three-dimensional reconstruction in low-light or night environments. The acquisition of remote sensing data by RGB cameras relies on external light, resulting in a significant decline in image quality and making it difficult to meet the task requirements. The method based on lidar has poor imaging effects in rainy and foggy weather, close-range scenes, and scenarios requiring thermal imaging data. In contrast, infrared cameras can effectively overcome this challenge because their imaging mechanisms are different from those of RGB cameras and lidar. However, the research on three-dimensional scene reconstruction of infrared images is relatively immature, especially in the field of infrared binocular stereo matching. There are two main challenges given this situation: first, there is a lack of a dataset specifically for infrared binocular stereo matching; second, the lack of texture information in infrared images causes a limit in the extension of the RGB method to the infrared reconstruction problem. To solve these problems, this study begins with the construction of an infrared binocular stereo matching dataset and then proposes an innovative perspective projection positional encoding-based transformer method to complete the infrared binocular stereo matching task. In this paper, a stereo matching network combined with transformer and cost volume is constructed. The existing work in the positional encoding of the transformer usually uses a parallel projection model to simplify the calculation. Our method is based on the actual perspective projection model so that each pixel is associated with a different projection ray. It effectively solves the problem of feature extraction and matching caused by insufficient texture information in infrared images and significantly improves matching accuracy. We conducted experiments based on the infrared binocular stereo matching dataset proposed in this paper. Experiments demonstrated the effectiveness of the proposed method.

1. Introduction

In the current era of the vigorous development of computer vision and robotics technology, accurate three-dimensional (3D) scene reconstruction has become a crucial cornerstone for research and application in the fields of virtual reality (VR), robot navigation, and remote sensing [1,2,3]. To compute depth, compared with the monocular camera-based method [4,5,6], the binocular camera-based method is more accurate. The reason for this is that the intrinsic and extrinsic parameters of the camera are calibrated accurately [7,8,9,10,11,12]. However, traditional binocular stereo matching methods based on visible light face many challenges in practical applications [7,8,9,10,11,12], such as severe interference from external environmental factors like complex illumination conditions and harsh weather, which greatly limit their effectiveness in all-weather scenarios. Compared with the 3D reconstruction method of lidar, the infrared-based method has better imaging effects in rainy and foggy weather, close-range scenes, and scenes requiring thermal imaging data [13,14,15,16,17,18,19,20].

To address the problem of poor-quality remote sensing data acquired from RGB cameras and lidar in special environments, we can use infrared cameras to capture images without relying on external light. There are two main reasons for this situation: first, there is a lack of a dataset specifically for infrared binocular stereo matching [21,22,23]; second, the lack of texture information in infrared images causes a limit in the extension of the RGB method to the infrared reconstruction problem. To solve these problems, this study constructed an infrared binocular stereo-matching dataset. By mapping the 3D lidar point cloud data to the 2D coordinate system of infrared cameras, high-precision 3D spatial information is provided for infrared images. However, there are significant differences between lidar point clouds and infrared images. The former are sparsely and unevenly distributed, while the latter feature a dense and continuous pixel structure. To solve this problem, we designed a joint calibration board of infrared cameras and lidar for high-precision sensor alignment and optimized the depth map generation algorithm [24,25,26,27,28,29]. Finally, the dataset was successfully constructed.

As shown in Figure 1, to solve the problem of an unsatisfactory effect after the traditional RGB method is transferred, this research proposes an innovative perspective projection positional encoding-based transformer method to complete the infrared binocular stereo matching task. In this method, a stereo matching network combined with the transformer and cost volume is constructed. Based on the negative inverse Gaussian distribution theory, we innovatively designed the fusion of these aspects. As an important part of the transformer, positional encoding is critical to model performance. The existing work extensively uses the parallel projection model, which simplifies the calculation as an approximation but has some limitations. In contrast, we use a perspective projection model in the positional encoding part so that each pixel is associated with a different projection ray. Because this positional encoding module improves the encoding precision, the method proposed in this paper can capture the geometric features of each pixel in the image more accurately [30]. It effectively solves the problem of feature extraction and matching caused by insufficient texture information in infrared images and significantly improves the matching accuracy.

In summary, our contributions are as follows:

•

An infrared binocular stereo matching dataset is constructed;

•

A binocular stereo matching network combined with the transformer and cost volume is created to complete global feature extraction and local feature extraction tasks, respectively;

•

The perspective projection model is embedded in the attention mechanism positional encoding part of the transformer network instead of the unrealistic parallel projection model.

We conducted experiments based on the infrared binocular stereo matching dataset proposed in this paper. Experiments demonstrated the effectiveness of the proposed method.

2. Related Works

First of all, we introduce the relevant stereo matching datasets. Then, we present the related work on 3D reconstruction from infrared images. Finally, the stereo matching network is introduced.

2.1. Datasets

In the 3D reconstruction task, datasets play a crucial role in providing the necessary data support for the quantitative evaluation of the model. For the time being, there is no mature infrared binocular dataset, so the datasets related to RGB are mainly introduced here. The generation of these datasets can be divided into two main categories: reality acquisition and annotation [31,32,33,34,35,36,37,38,39] and computer graphics-based data [40].

Scharstein et al. proposed the Middlebury stereo matching dataset, which pioneered the quantitative evaluation of stereo matching algorithms based on the dataset [31]. Subsequently, a series of Middlebury datasets based on this work were successively released [32,33,34,35]. Geiger et al. [21,22,23] proposed the KITTI dataset, which is a very famous stereoscopic vision evaluation dataset. The KITTI dataset mainly collects outdoor scene data. In addition to stereoscopic matching tasks, it can also be applied to autonomous driving tasks and includes dynamic scenes. Allan et al. [36] proposed the SCARED dataset. This dataset focuses on the depth estimation of endoscopic images. Shi et al. [37] proposed the stereo waterdrop removal dataset. This dataset is specifically designed to study the removal of raindrop effects. Knapitsch et al. [38] proposed ETH3D dataset. The dataset is a gray-scale stereo dataset containing indoor and outdoor scenes. Mayer et al. [39] proposed a scene flow dataset. This dataset is a virtual stereo dataset generated via software rendering, including disparity ground truth, optical flow, and scene flow data. At present, the binocular stereo dataset obtained by the infrared camera is still blank and cannot provide effective reference data for the quantitative evaluation of the infrared stereo matching algorithm.

2.2. 3D Reconstruction from Infrared Images

Lin et al. [13] used a tilt photogrammetry system composed of four infrared cameras and four RGB cameras to collect data. The Structure from Motion (SFM) method is used to generate an infrared point cloud and RGB point cloud for registration to realize thermal property mapping in large-scale regions. Ham et al. [14] proposed a method of 3D modeling with the help of RGB images and thermal images. This method uses SFM and Multi-View Stereo (MVS) to generate point clouds and then automatically stacks the point clouds according to the calibration parameters to complete 3D reconstruction. Vidas et al. [15] developed an imaging system composed of a thermal imager, distance sensor, and color camera. The system generated a 3D model through Simultaneous Localization and Mapping (SLAM) technology. Vidas et al. [16] used the handheld sensor system [15] and employed a risk-averse neighborhood weighting mechanism to reduce mismatching problems caused by an inaccurate temperature. Yang et al. [17] proposed a 3D infrared modeling method for data acquisition using a system composed of two smartphones and an infrared camera. In this method, the RGB 3D model is constructed first, and then the infrared image is fused with the 3D model to generate a 3D thermal model. Hoegner et al. [18] used Nistér’s five-point algorithm and image triplets to perform automatic Texture Mapping of infrared images on a given 3D building model. Hoegner et al. [19] proposed two mobile thermal imaging methods. The first step is to register the thermal image sequence with a given 3D model. The second step is to register the point cloud generated using high-resolution RGB images with an infrared point cloud to generate a 3D model with thermal texture. Cho et al. [20] pointed out that existing research on the 3D reconstruction of infrared images mainly focuses on two-dimensional infrared image data processing. This is mainly realized by mapping infrared image texture on the 3D model generated using mature technology. It is impossible to conduct an accurate quantitative evaluation of its performance because it is mainly evaluated using qualitative indicators such as the visualization effect. Therefore, in order to measure and optimize the performance of infrared imaging processing technology more accurately, future research requires exploring more quantitative evaluation methods.

2.3. Stereo Matching

Stereo matching is conducted using both traditional methods and deep learning methods. Due to the limited space, we mainly review the deep learning methods here. The deep learning methods are mainly divided into two categories. One is the stereo matching algorithm based on cost volume [40,41,42,43,44,45]. The other is the stereo matching algorithm based on the transformer [10,11,12,46].

Regarding cost volume-based methods, Kendall et al. [40] proposed the GC-Net. To better utilize context information, they creatively proposed using 3D convolution to obtain the cost volume and regress the disparity map by matching the cost volume. To better combine context information, Chang et al. [41] proposed the PSM-Net. They introduced a pyramid pooling module to aggregate multi-scale information and obtain the cost volume. To better utilize the contextual information, the cost volume is replicated by 3D CNN for stereo matching. Guo et al. [42] proposed GWCNet based on PSMNet. This method uses group-wise correlation to generate a cost volume and provide a better similarity measure for the network. Zhang et al. [43] proposed the GA-Net based on GC-Net and traditional SGM. The algorithm consists of two neural network layers (semi-global aggregation and locally guided aggregation), which capture the cost dependence of the local image and the whole image, respectively. Xu et al. [44] proposed AANet, which used the sparse point-based intra-scale cost aggregation method instead of 3D convolution for intra-scale cost aggregation to improve the operation speed. Shen et al. [45] proposed a multi-scale cost volume fusion module and an efficient warping volumetric-based disparity refinement module. The stereo matching performance of multi-resolution images is improved, and the difficulty of network search is reduced.

Regarding the tansformer-based method, Li et al. [10] proposed the STTR, which is a method of binocular disparity estimation using a transformer. It uses a transformer to replace traditional cost volume construction. By making full use of contextual information, the occlusion problem is solved effectively, and the limitation of the disparity range is extended. Ding et al. [11] proposed a feature matching transformer depth network architecture, applying the transformer to MVS for the first time. Guo et al. [12] proposed a transformer stereo matching estimation model based on a context enhancement path module. The model performs well in feature extraction in areas missing texture, mirror, and transparency information. Xu et al. [46] proposed a unified model to simultaneously solve three tasks: optical flow estimation, rectified stereo matching, and unrectified stereo depth estimation. The model uses a transformer to realize cross-task migration via cross-attention mechanism learning.

In summary, image-based stereo matching has been thoroughly studied. However, as far as we know, there is no mature binocular stereo matching method based on infrared images.

3. Infrared Depth Dataset

As pointed out in [35], the research progress of stereo vision in the field of computer vision is largely driven by datasets. By calculating metrics such as EPE and D1 using stereo-matching datasets, we can quantitatively measure the algorithm’s performance, thereby providing a clear optimization direction. The dataset constructed in this paper consists of 374 sets of binocular infrared image sequences after stereo correction and corresponding disparity maps of remaining images. When constructing the dataset, we referred to the commonly used stereo-matching datasets, including KITTI 2012 [21], KITTI 2015 [23], and Middlebury 2014 [35]. Referring to the production method of the KITTI2012 dataset in [21], the purpose of using lidar is to provide precise depth values and offer ground truth annotations for the construction of infrared stereo datasets. Compared with these datasets, our infrared stereo dataset is sizable. Its scale is on par with the KITTI 2012 and KITTI 2015 datasets and far exceeds that of the Middlebury 2014 dataset. We collected infrared data on the campus and residential areas. The data depict the library, teaching buildings, cars, and so on. Campus and residential areas include different background elements, such as trees, roads, and fences. These elements introduce rich texture features, lighting conditions, and object permutations and combinations, enhancing the dataset’s ability to represent complex real-world scenes. The raw data described in this paper can be accessed from https://github.com/n1227363905/Infrared-Binocular-Dataset (accessed on 6 June 2025).

3.1. Data Acquisition

3.1.1. Sensor Setup

Our data acquisition platform is shown in Figure 2.

•

Two long-wave infrared cameras with a resolution of $640\times 480$ (Huajingkang K26E4) operate in a wavelength range and pixel spacing of 8–14 um and 17 um, respectively.

•

DJI Livox Avia Lidar: wavelength is 950 nm, frequency is 10 Hz, ray divergence angle is ${0.28}^{°}\left(\mathrm{vertical}\right)\times {0.03}^{°}\left(\mathrm{horizontal}\right)$, built-in IMU (Model: BMI088), 480,000 points/s in double echo mode, ${77.2}^{°}\left(\mathrm{vertical}\right)\times {70.4}^{°}\left(\mathrm{horizontal}\right)$ of view in the non-repeat scan mode, and the detection distance is 450 m.

•

Two laptops with Intel i7 processors running Ubuntu 18.046 and Windows 10 systems, respectively.

3.1.2. Data Acquisition

In this study, two laptops equipped with Intel Core i7 processors were used as terminal devices for data acquisition. Among them, the acquisition of lidar point cloud data was carried out on a laptop installed with the Ubuntu 18.046 LTS operating system. The data were acquired by starting the Robot Operating System (ROS) and using the command line interface. Meanwhile, the acquisition of infrared binocular image data was completed on another laptop running the Windows 10 operating system through the graphical interface provided by the Huajingkang manufacturer. Since the timestamp synchronization function between the systems cannot be automated, this study adopted the method of manually aligning the timestamps to synchronize the acquisition of lidar point cloud and infrared image data.

3.2. Equipment Calibration

3.2.1. Calibration of Infrared Binocular Cameras

Due to the thermal radiation imaging of infrared cameras, the temperature distribution on the surface of objects can be converted into infrared images. Based on this principle, this study designs an infrared calibration plate suitable for natural light environments. This calibration plate is made of infrared radiation materials with different infrared radiation reflectivities. Compared with traditional thermal radiation materials, this new type of calibration plate does not require additional heating materials to distinguish different areas of the checkerboard. In the highly reflective thermal radiation area, the calibration plate can emit infrared rays. Meanwhile, in the low-reflection thermal radiation area, it cannot emit infrared rays. This design effectively avoids the problem of inaccurate corner detection caused by heat conduction in the corner area of self-heating materials, thereby improving the accuracy and reliability of infrared image acquisition.

Using the binocular camera acquisition platform, the previously designed infrared calibration plate was photographed under natural light conditions, and a total of 50 groups of infrared binocular image data were collected. On this basis, the calibration work of the intrinsic and extrinsic parameters of the binocular infrared cameras was completed by minimizing the reprojection error using Zhang’s calibration method [24]. The internal and external parameters of the infrared binocular camera were calculated using the MATLAB R2021b stereo camera calibrator tool. This process not only improves the accuracy of calibration but also provides reliable camera model parameters for subsequent image processing and analysis.

3.2.2. Calibration Between Infrared Camera and Lidar

For the purpose of precisely estimating the relative positional relationship between the infrared camera and the lidar, this study utilizes the approach of manually selecting feature points to extract the essential corresponding points. By default, the left-eye camera is selected as the reference camera for the depth map. As mentioned above, we already obtained the intrinsic parameter matrix and distortion coefficients of the left-eye infrared camera. On this basis, we solved the PnP problem by minimizing the reprojection error using the Levenberg–Marquardt algorithm to facilitate subsequent data processing and analysis [25,26,27,28]. The external parameters of the lidar and infrared camera were calculated using cv::solvePnP(), and the calibration accuracy was evaluated by projecting onto the image plane with the help of the cv::projectPoints. This method can provide us with accurate camera pose information, thereby realizing the effective fusion of infrared camera and lidar data.

3.3. Infrared Image Processing

3.3.1. Distortion Removal

Distortion refers to the phenomenon that during the camera imaging process, the pixel points on the imaging plane deviate from the strict camera imaging model in terms of geometric distribution. First, the actual position of each pixel in the distorted image is calculated according to the intrinsic parameter matrix and distortion coefficient of the camera and then using the cv.undistort() to map these pixels back to the undistorted image to eliminate the influence of the distortion. The effect before and after distortion is shown in Figure 3.

3.3.2. Stereo Rectification

According to the binocular camera calibration results, although the two cameras are fixed on parallel guide rails, the optical axes of these two cameras are not completely parallel. Therefore, a simple distortion removal treatment cannot achieve the ideal parallel view effect. So, it is necessary to perform stereo rectification on the left and right eye images according to the intrinsic and extrinsic parameters of the binocular camera to obtain parallel views. First, the binocular correction transformation matrix is calculated using cv.stereoRectify(), and then cv.initUndistortRectifyMap() is used to generate the mapping table; finally, the cv.remap() is used to complete the polar line correction. This operation can reduce the complexity of searching corresponding points in stereo matching. The stereo correction effect diagram is shown in Figure 4.

3.4. Generation of Disparity Map

3.4.1. Generation of Lidar Point Cloud Map

Due to the narrow field of view of the lidar, the point cloud density in resolution is lower than that of the infrared camera and is affected by the divergence of the lidar ray. To label as many 3D coordinate objects as possible in the field of view of the infrared camera, it is necessary to process the multi-frame lidar point cloud to generate high-quality point cloud images. The generated point cloud image is shown in Figure 5.

In this paper, we adopt the FAST-LIO proposed by Zhang [47]. It is a laser inertial odometry method based on tightly coupled iterative Kalman filtering and is used to generate the point cloud map. The data source is the laser SLAM bag data packet collected by DJI Livox Avia Lidar. In the FAST-LIO algorithm, the initial location of lidar point cloud acquisition is set as the origin of the world coordinate system of the point cloud map. Since the lidar has built-in IMU, after effective state initialization, the point cloud map drift generated by the collected data through FAST-LIO processing is less than 0.05%.

3.4.2. Generation of Depth Map

To obtain the corresponding depth map from the infrared camera image, we need to obtain the projection matrix of the lidar point cloud to the left eye camera plane [29]. The projection matrix is calculated using the calibration results of external parameters between the lidar and infrared camera and the stereo rectification matrix. The 3D coordinates of the lidar point cloud map data are projected into the image coordinate system of the left-eye camera to obtain the corresponding depth value of each pixel.

When mapping the 3D data of the lidar point cloud to the 2D coordinates of the infrared camera, the lidar point cloud data can provide high-precision three-dimensional spatial information for the infrared image. In the depth map generation stage, the three-dimensional points of the lidar point cloud are projected onto the two-dimensional image plane through cv.projectPoints() to achieve the conversion from the point cloud to the depth map. The projection transformation can be expressed as

$$\left[\begin{array}{c}\mu \\ \upsilon \\ 1\end{array}\right]=\left[\begin{array}{ccc}{f}_x& 0& c_x\\ 0& f_y& c_y\\ 0& 0& 0\end{array}\right]\left[\begin{array}{cccc}{r}_{11}& r_{12}& r_{13}& t_1\\ r_{21}& r_{22}& r_{23}& t_2\\ r_{31}& r_{32}& r_{33}& t_3\end{array}\right]\left[\begin{array}{c}x\\ y\\ z\\ 1\end{array}\right].$$

(1)

where $(x,y,z)$ and $(u,v)$ represent lidar point cloud coordinates and infrared camera image pixel coordinates, respectively, and $(t_1,t_2,t_3)$ represent the translation from the origin of the lidar coordinate system to the origin of the infrared camera coordinate system.

The projected depth diagram is shown in Figure 6. However, the characteristics of lidar point cloud and infrared image data are quite different. The laser point cloud data are sparse and unevenly distributed. The infrared image data are relatively dense and continuous. Therefore, mapping lidar point cloud data to the pixel positions of dense images requires necessary interpolation, which affects the accurate acquisition of infrared image depth values. In this process, we need to explain the following three points.

•

In the real scene, the object scanned by laser has an occlusion phenomenon. When multiple lidar points cover the same pixel, we select the point with the smaller depth value as the corresponding depth value of the pixel.

•

The depth of the surface of an object in a real scene is usually relatively continuous in a local area. So, when the point cloud cannot cover all the pixels, we fill in the void locations with an average depth of 8 pixels to obtain a more complete depth map.

•

The semi-solid lidar used in this paper has a detection blind area when it is less than 3 m.

3.4.3. Generation of Disparity Map

In stereo matching related tasks, it is necessary to further convert the depth map into a disparity map for the quantitative evaluation of stereo matching algorithms. The generation of disparity maps is based on the implementation of the disparity calculation in Equation (2) using the NumPy library, and the ground truth of the disparity map is saved through cv.imwrite(). The disparity map is shown in Figure 7. According to the binocular stereo vision principle shown in Figure 8, the disparity value can be expressed as

$$disp={\displaystyle \frac{f\ast b}{z}}.$$

(2)

Among them, f represents the focal length of the camera, b represents the baseline length, and z represents the depth.

4. Proposed Method

We proposed a new architecture, named ICVT (Integration of Cost Volume and Transformer). This method solves the stereo-matching problem of infrared images by integrating local feature extraction (Cost Volume) and global context modeling (Transformer). Furthermore, we introduced the Perspective Positional Encoding (PPE) module on the basis of ICVT to form the ICVT-PPE method in order to enhance the depth perception ability. In Section 4.1, we introduce the infrared binocular stereo matching network proposed in this paper by combining the transformer and cost volume. In Section 4.2, we introduce the transformer module because this paper focuses on innovation in the transformer positional encoding section. In Section 4.3, we introduce the transformer perspective projection positional encoding design, which allows each pixel to be associated with a different projected ray.

4.1. Stereo Matching Network Combined with Transformer and Cost Volume

Figure 1 shows the transformer network architecture based on infrared binocular vision stereo matching. This architecture is mainly divided into three core modules: the cost volume feature extraction module, the transformer feature extraction module, and the stereo matching estimation module. The features extracted by the first two modules are fused by the third module. This method can improve the performance of the infrared binocular stereo matching network significantly by combining the cost volume with the transformer. The main reason for this is that the cost volume is used to extract local features, and the transformer is used to capture global features. Since this paper focuses on innovation in the transformer section, we specifically introduce the transformer module in Section 4.2.

4.2. Transformer Network Architecture

The transformer module proposed by the infrared binocular stereo-matching network is based on self-attention and cross-attention and uses them alternately for feature extraction and matching. Self-attention is used to model the feature relationship along the polar line direction within a single image and extract the context information. Cross-attention calculates the feature relationship along the polar line direction of the left and right images for feature matching of the left and right images, preparing the matching for the best transmission. This scheme of alternating the use of self-attention and cross-attention can utilize the image context information to interact features with the left and right images and continuously update the features. The self-attention and cross-attention proposed in this paper both adopt the multi-head attention mechanism. The multi-head attention mechanism divides the input features into multiple heads according to their dimensions and calculates the attention. Assuming the dimension of the input features is d and the number of heads is n, then the dimension of each head is $d_i=d/n$. Each head performs a linear transformation on the input feature $\left(F\right)$, respectively, to generate the query vector $\left(Q_i\right)$, key vector $\left(K_i\right)$, and value vector $\left(V_i\right)$, which are expressed as

$$Q_i=F{W}_i^Q+b_i^Q$$

(3)

$$K_i=F{W}_i^K+b_i^K$$

(4)

$$V_i=F{W}_i^V+b_i^V.$$

(5)

Among them, the weight matrices are $W_i^Q$, $W_i^K$, and $W_i^V$∈${\mathbb{R}}^{d_i\times d_i}$, and the bias vectors are $b_i^Q$, $b_i^K$, and $b_i^V$∈${\mathbb{R}}^{d_i}$. It is worth noting that the parallel projection model is widely used in the attention calculation of the transformer module in stereo matching networks, which simplifies the calculation but has certain limitations. We use the projection vector to represent the perspective projection model, which is connected to $V_i$ in the above formula through a convolutional layer and embedded into the attention calculation. Make each pixel associated with a different projected ray. The result of attention calculation can be expressed as

$$a_i=softmax\left({\displaystyle \frac{Q_i{K}_i}{\sqrt{d}}}\right)·conv\left(V_i\oplus v\right).$$

(6)

Among them, d is the scaling factors introduced to avoid the vanishing gradient caused by excessive dot product, and v∈${\mathbb{R}}^{d_i}$ represents the perspective projection vector (for a specific introduction, see Section 4.3). ⊕ represents the splicing operation. Calculate the attention respectively, and finally concatenate them together. The output vector V0 can be expressed as

$$V_0=W_{0}contact\left(a_1,a_2\cdots ,a_n\right)+b_0.$$

(7)

Among them, the weight matrix is $W_0$∈${\mathbb{R}}^{d_i}$, and the bias vector is $W_0$∈${\mathbb{R}}^{d_i}$. After adding the output vector to the original feature F to form a residual connection, the final feature is obtained and expressed as

$$F=F+V_0.$$

(8)

In the self-attention layer, $Q_i$, $K_i$ and $V_i$ originate from the same image. In the cross-attention layer, $Q_i$ comes from the source image, and $K_i$ and $V_i$ come from the target image, enabling information interaction between different images on the left and right.

4.3. Perspective Projection for Transformer Positional Encoding

In the design of the stereo matching deep learning network, the traditional method often adopts the parallel projection model. The parallel projection model assumes that there is no difference in the image of the object along the ray direction at a long distance. This model cannot effectively capture the imaging characteristics of near objects appearing large and far objects appearing small in perspective projection, limiting the network’s accurate estimation of the scene’s depth information. The proposed transformer module cleverly simulates the perspective projection model through the principal point coordinates and focal length information of the camera and imparts it into the positional encoding of the attention mechanism. According to the perspective projection imaging principle, the 3D point coordinates in the camera coordinate system are converted to the image pixel coordinate system by using the intrinsic parameter matrix of the camera, which can be expressed as

$$\left[\begin{array}{ccc}{f}_x& 0& 0\\ 0& f_y& 0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}{X}_c\\ Y_c\\ Z_c\end{array}\right]=Z_c\left[\begin{array}{c}u-c_x\\ v-c_y\\ 1\end{array}\right].$$

(9)

Among them, $\left(\begin{array}{ccc}{X}_c,& Y_c,& Z_c\end{array}\right)$ represent the coordinates of 3D points in the camera coordinate system, and $(u,v)$ represent the coordinates of corresponding points in the image pixel coordinate system. Using Equation (9), we can derive the following equation:

$$Z_c\left[\begin{array}{c}{\displaystyle \frac{u-c_x}{f_x}}\\ {\displaystyle \frac{v-c_y}{f_y}}\\ 1\end{array}\right]=\left[\begin{array}{c}{X}_c\\ Y_c\\ Z_c\end{array}\right].$$

(10)

It can be seen that when the focal length is 1 and the origin is on the optical axis, the corresponding normalized pixel coordinates can be represented by

$$\left[\begin{array}{c}{u}^{*}\\ v^{*}\\ 1\end{array}\right]=\left[\begin{array}{c}{\displaystyle \frac{u-c_x}{f_x}}\\ {\displaystyle \frac{v-c_y}{f_y}}\\ 1\end{array}\right].$$

(11)

When the infrared camera pixels used in this article are square, $f_x$ and $f_y$ are equal. We uniformly represent it as f. In the normalized coordinate system, z is 1. Given the coordinates of the principal point and the focal length f, the direction components $v_x$, $v_y$, and $v_z$ represented by each pixel in the perspective projection model can be calculated using the pixel coordinates $(u,x)$. Specifically, in the normalized coordinate system, the u and v coordinates, respectively, represent $v_x$, $v_y$, and the value of $v_z$ is 1. Therefore, the perspective projection vector can be expressed as

$$v=contact\left(v_x,v_y,v_z\right).$$

(12)

We embedded the normalized planar pixel coordinates of Equation (11) into the positional encoding part of the attention mechanism to simulate the imaging process of the perspective projection camera [30]. This positional encoding allows each pixel to be associated with a different projected ray, thus capturing the geometric features of each pixel in the image more precisely. It can effectively solve the problem of feature extraction and matching caused by insufficient texture information in infrared images and significantly improve the matching accuracy.

5. Experiments

5.1. Experiments Setup

Evaluation metrics. To accurately measure the performance of the network in disparity estimation, we chose three evaluation metrics: the endpoint error (EPE), the percentage of outliers for 3-pixel disparity (D1-3px) [10], and the percentage of mismatched points for 1-pixel error (BMP-1px) [31]. Through the comprehensive evaluation of these metrics, the actual effect of the proposed network in processing disparity estimation can be objectively reflected.

Usage of datasets. Our infrared 3D dataset consists of 374 stereo image pairs with marked disparity maps. In this study, to avoid the risk of overfitting and truly reflect the network’s actual performance, we divided 90% of the dataset into the training set and 10% into the test set.

Implementation details. During the training process, the network goes through a total of 200 rounds of training. The learning rate of transformer-based part is initialized to 0.0002. The learning rate of the cost-volume-based part is initialized to 0.002, and the learning rate of the backbone network is initialized to 0.0002. The experiment was conducted on the Ubuntu 18.046 operating system workstation equipped with NVIDIA GeForce RTX 4090 GPU, and the configuration environment was Python 3.8.19, CUDA 11.7, and PyTorch 2.0.0. Through this well-designed network architecture and optimized training strategy, the stereo matching task can achieve significant performance improvement and provide a new solution for the stereo matching field of infrared binocular stereo vision.

5.2. Comparison with Baselines

Methods for comparison. We denote our method that fuses the cost volume and transformer by ICVT-PPE. ICVT represents the integration of the cost volume and transformer method. PPE represents the perspective positional encoding module. Given that there is no mature method suitable for infrared stereo matching, we treat STTR [10] and PCWNet [45], which are originally designed for RGB images, as the baselines for comparison. We compare our method with these baselines as follows.

The experimental results are presented as shown in

Table 1. The experimental results.

Model	EPE	D1-3px	BMP-1px
PCWNet [45]	2.799	0.242	0.5123
STTR [10]	1.4125	0.1181	0.231
ICVT (Ours)	1.309	0.118	0.2157
ICVT-PPE (Ours)	1.25	0.1076	0.1918

. Compared with PCWNet, ICVT-PPE has an improvement of 55.3% in EPE (from 2.799 to 1.25), 55.5% in D1-3px (from 0.242 to 0.1076), and 62.6% in BMP-1px (from 0.5123 to 0.1918). Compared with STTR, ICVT-PPE has an improvement of 11.5% in EPE (from 1.4125 to 1.25), 8.9% in D1-3px (from 0.1181 to 0.1076), and 17.0% in BMP-1px (from 0.231 to 0.1918).

As shown in the experimental results of Table 1, the infrared stereo matching scheme is deeply integrated with the transformer, and the cost volume demonstrates significant performance advantages. This is specifically reflected in the outstanding performance in three dimensions: global feature capture, local feature extraction, and dual-module collaborative optimization.

Global feature capture. In the following section, we analyze these results. For one thing, the transformer module shows a strong global feature extraction capability in stereo matching tasks. Through the attention mechanism, it can fully capture the correlation information between different regions of the image when processing stereo image data so as to build a comprehensive contextual understanding. This global information helps to identify the feature correspondence across a large spatial range in stereo image pairs, avoiding missing important matching clues due to local visual field limitation.

Local feature extraction. The cost volume module focuses on the capture and analysis of local features. It can accurately process the detailed information in the local area of the image and provide accurate basic data for feature matching. Specifically, when processing infrared images, the cost volume can effectively focus on local features such as the edges and textures of the target object. By determining the similarities and differences of these local features in stereo image pairs, the foundation is laid for accurate disparity calculation.

Dual-module collaborative optimization. As shown in Figure 9, when combined with the cost volume, the transformer provides complementary advantages. The global information provided by the transformer guides the cost volume to better encode the overall direction in the local feature extraction process so that it does not fall into the local optimal solution. Specifically, the fusion method based on the cost volume and transformer has obvious advantages in the face of complex infrared scenes (such as multiple overlapping objects or irregularly shaped objects in the image). Therefore, the above method can improve the adaptability of the whole stereo matching network to complex scenes. At the same time, rich local features extracted from the cost volume also provide more solid detail support for the construction of global features in the transformer, making global features not only a vague overall impression but also a comprehensive expression of rich local details.

Moreover, our designed transformer’s perspective-based location encoding module can further improve the performance of the proposed method. A detailed discussion about the effectiveness of PPE is provided in the next subsection.

5.3. Ablation Study

To verify the effectiveness of the PPE module, we conducted an ablation study. The quantitative results are provided in Table 1. We compared ICVT-PPE with ICVT. Compared to ICVT, ICVT-PPE had an improvement of 4.5% in EPE (from 1.309 to 1.25), 8.8% in D1-3px (from 0.118 to 0.1076), and 11.1% in BMP-1px (from 0.2157 to 0.1918). The specific comparison effect diagram is presented in Figure 10. By means of the positional encoding part, ICVT-PPE effectively enhances the disparity estimation performance of the infrared binocular stereo matching network. In the next part, we analyze in detail how to improve the disparity estimation accuracy of infrared binocular images after embedding the attention mechanism in the perspective projection model.

Differentiated processing of projected light. The existing work in the positional encoding of the transformer usually uses a parallel projection model to simplify the calculation. The PPE module is based on the actual perspective projection model so that each pixel is associated with a different projected ray.

3D spatial position modeling. The PPE module based on perspective projection position coding adopts optical geometric principles to construct a depth-aware correlation relationship between pixels. This module describes the coordinate transformation process of the points on the object surface projected onto the imaging plane. Therefore, we achieved high-precision spatial perception in the depth dimension by modeling the mapping relationship between the object’s spatial coordinates and the image’s pixel coordinates.

Feature matching optimization. During the attention calculation process, the PPE module can flexibly adjust the weights and matching strategies of features based on depth differences. For objects of different depths, their pixel features are assigned different weights. Therefore, the feature matching step can significantly improve the accuracy of feature matching and the precision of depth estimation.

As shown in Figure 10, when applied to infrared images with poor texture information, this method can effectively solve the difficulty of feature extraction and matching caused by limited texture information. The disparity estimation performance of the infrared binocular stereo matching network is significantly improved.

6. Conclusions

In this paper, an infrared stereo benchmark is constructed using a lidar point cloud. The construction of this dataset includes tasks such as calibrating extrinsic parameters between lidar and infrared camera using the PnP method, mapping lidar coordinates to the image coordinate system, and converting depth maps to disparity maps. The dataset provides the possibility of a quantitative evaluation of stereo matching tasks in infrared binocular images.

In addition, this research proposes an innovative perspective projection positional encoding-based transformer method to complete the infrared binocular stereo matching task. In this method, a stereo matching network combined with the transformer and cost volume is constructed. The existing work extensively uses the parallel projection model, which simplifies the calculation as an approximation but has some limitations. In contrast, we use a perspective projection model in the positional encoding part so that each pixel is associated with a different projection ray. This effectively solves the problem of feature extraction and matching caused by insufficient texture information in infrared images and significantly improves the matching accuracy. We conducted experiments based on the infrared binocular stereo matching dataset proposed in this paper. Experiments demonstrated the effectiveness of the proposed method.