Abstract
We propose a feature enhancement method for depth images that applies convolutions over residuals and parameters obtained from dominant plane. The proposed method can obtain the initial features of depth images that are less sensitive to surface orientation and more representative of intrinsic geometric properties. Specifically, the features are obtained through the plane-based convolution that performs operation on residuals with respect to the dominant plane within a local patch of the depth image. For each patch, a dominant plane is fitted to the corresponding depth pixel values using a least-squares method. Then, convolutional operations are performed on plane residuals computed between the original depth values and the corresponding depth values on the dominant plane. In addition, standard convolution is applied to the dominant plane parameters to capture local variations and spatial consistency of surface orientation. A plane-based convolution module incorporating these convolutions is attached to the initial layer of the existing feature extractor in parallel to supplementarily obtain surface geometric features. Experiment results demonstrate that the proposed method consistently achieves performance gains on both segmentation and classification tasks.
1. Introduction
Depth images whose pixels store the distances to a capture device provide 3D structural information that can be utilized for various tasks such as volume measurement and 3D pose estimation [1,2,3]. Similar to color images, depth images can also be processed using Convolution Neural Network (CNN) or Vision Transformer (ViT) [4] architectures to extract and utilize their spatial correlations as features. Several studies have attempted to improve vision task performance by additionally incorporating depth-image features extracted through existing CNN- or ViT-based networks originally designed for color images [5,6,7,8]. However, such approaches suffer from inherent limitations because depth images have representational characteristics that differ from color images. For color images, vision tasks exploit either patterns formed by local pixel distributions in CNN or global relationships among patch-level representations formed by embedding pixel distributions in ViT. In contrast, depth images are characterized by significant variations in the pixel distribution across scenes with identical semantic content, depending on camera viewpoint or distance. Instead, relative pixel variations in depth images serve as more critical cues for scene understanding because they reflect geometric structures and object boundaries. To effectively improve vision task performance on depth images, it is appropriate to consider either transforming depth images into representations aligned with network architectures designed for color images or introducing operations that explicitly incorporate the geometric characteristics of depth images.
Several studies have explored the transformation of depth images for applying color image–based networks [9,10,11]. These methods enhance the feature representation of depth images in color image–based networks by transforming depth images using intrinsic three-dimensional geometric information. Such approaches can effectively improve vision tasks by fine-tuning networks pre-trained on large-scale color image datasets. However, these studies rely on restricting depth images to a three-channel representation compatible with existing networks, which imposes limitations on information representation. Therefore, the rich geometric information inherent in depth images may not be fully utilized. As an alternative approach, specialized convolution operations [12,13,14,15,16] have been proposed to capture 3D structural features from depth images. Inspired by the insight that relative depth variations between adjacent pixels convey richer spatial information than absolute depth values, depth-aware convolution methods have been proposed, such as those based on relative depth differences and those incorporating local outlier masking. Although these methods effectively address pixel distribution variations due to distance changes, pixel distribution variations induced by viewpoint changes remain insufficiently considered. Furthermore, these methods are limited in their ability to exploit existing network architectures that are pre-trained to learn abstract feature representations from large-scale color image datasets since they require architectural modifications.
In this paper, we propose a feature enhancement method for depth images using a plane-based convolution module that performs convolution over plane residuals and plane parameters. The plane-based convolution module enhances initial feature representations of depth images by effectively extracting features of local 3D surface structures. Specifically, the proposed module fits a dominant plane from pixel-wise depth values within each local patch. Convolution is then applied to the residuals between the original depth values and the corresponding depths derived from the dominant plane. Since plane-based residuals remain largely consistent under changes in the viewpoint and distance, the proposed method extracts the features that capture more discriminative intrinsic surface characteristics. In addition, the surface normal vector estimated during the dominant plane fitting are also processed through convolution and fused with the features derived from the plane-based residuals. The proposed module is attached in parallel to the stem layer, the initial layer of the feature extractor, to enhance low-level feature representations. By incorporating the proposed module, performance on vision tasks using either depth-only or RGB-D images is improved. Since the proposed method requires minimal modifications to the network architecture, it can effectively leverage pre-trained weights.
The main contributions of the proposed method are as follows.
- The plane-residual convolution tailored for depth images enables the extraction of intrinsic features of 3D surfaces that are less sensitive to changes in the distance and viewpoint.
- The proposed module can be integrated into the initial layer of an existing feature extractor to incorporate 3D spatial information into low-level feature representations.
- The proposed module integration enables stable and efficient fine-tuning during training while preserving the original architecture of the feature extractor.
2. Related Works
Although single-channel depth images represent the geometric structure of a scene through depth values, directly feeding them into CNNs or ViTs makes stable feature learning difficult due to the nonlinear distribution of depth values and the implicit representation of geometric cues. In addition, single-channel inputs limit representational capacity along the channel dimension, and performance degradation may occur due to statistical discrepancies from RGB images used for pretraining. To address these challenges, prior works have attempted to convert single-channel depth images into multi-channel representations, explicitly separating and encoding geometric cues and expanding representational capacity by leveraging the channel dimension.
Gupta et al. [9] proposed the HHA representation, which extracts disparity, height above ground, and angle with gravity for each pixel from a depth image. While the transformed depth map provides geometrically rich features, this approach suffers from high computational cost and pipeline complexity because ground plane estimation is performed based on principal component analysis (PCA). Caglayan et al. [10] estimated local surface normal vectors from depth images and converted them into a three-channel representation using the three components of each normal vector, enabling CNN-based models to effectively exploit geometric information of continuous 3D surfaces. However, this transformation focuses on geometric orientation information while losing distance-based structural information. Gopalapillai [11] proposed a transformation that expands a depth image into a three-channel representation consisting of the original depth values and the results of applying Gaussian filters with different kernel sizes. This method extends depth representations to multi-scale feature representations; however, the resulting feature maps do not include geometric cues such as surface orientation. Since these depth image transformation methods primarily focus on converting depth images into three-channel formats for compatibility with pretrained color-image-based networks, part of the inherent 3D structural information in depth images is inevitably lost. In contrast, the proposed method preserves the depth image without explicit channel conversion and directly extracts features related to surface residuals and surface orientation in the early layers, thereby effectively retaining 3D structural cues embedded in depth images.
Meanwhile, to effectively exploit the 3D geometric information in depth images, several approaches have been proposed to extend convolution operations or to adapt convolutional weights or receptive fields based on depth information. Wang [12] introduced a weighting term based on depth similarity between the center pixel and its neighboring pixels in convolution operations, suppressing contributions from pixels with large depth differences and enhancing those from pixels belonging to the same geometric structure without adding extra learnable parameters. Similarly, Yang [13] modified convolution operations by computing responses based on pixel-wise differences between the center pixel and neighboring pixels, rather than directly aggregating pixel values within the receptive field, thereby effectively emphasizing local geometric variations and boundary information in depth images. Chu [14] proposed a convolution operation that adjusts the size of the receptive field to be inversely proportional to the depth value of the center pixel, enabling the capture of local regions with consistent physical size in 3D space. Wu et al. [15] proposed a method to obtain viewpoint-robust features by computing sampling offsets from surface normal vectors of locally estimated planes, instead of using a fixed grid-shaped receptive field. These depth-specific convolution operations enhance features related to 3D structures that are difficult to capture with standard convolutions. However, since these methods replace conventional convolutions, they are limited in their ability to leverage pretrained networks. In contrast, the proposed method can utilize pretrained network weights by simply attaching an additional plane-based convolution module.
3. Proposed Method
3.1. Plane-Based Convolution Module
In this paper, we propose the plane-based convolution module to obtain depth-based 3D structural features. The proposed module takes a H × W × 1 depth image as input. Within each local patch, the pixel-wise depth values are normalized by subtracting the depth value of the patch central pixel. Then, a dominant plane is fitted based on the depth values within the patch. The depth values within the patch are reconstructed based on the parameters of the corresponding plane. Convolution is applied to the residuals between the reconstructed and original depth values to extract features that are less sensitive to surface orientation. In parallel, standard convolution is applied to the plane parameters to capture local variations and spatial consistency of the surface orientation. Finally, the two feature maps are fused to enhance the initial features with additional 3D structural information extracted by the proposed method. Figure 1a,b illustrate the overall pipeline of the proposed module and the flow of plane residual convolution, respectively. In Figure 1, h × w × f and k are the dimensions of the output feature map and a kernel size, respectively. The convolutions factors in the plane-based convolution module such as kernel size, stride, and padding are set to ensure that the output dimensions match the input dimensions of the subsequent layer.
Figure 1.
Flow of the proposed method: (a) Plane-based convolution module; (b) Plane residual convolution.
3.1.1. Dominant Plane Fitting Using Outlier Correction
For each k × k patch involved in a convolution, the depth value at the patch center is subtracted from the depth values. Then, a plane fitting is performed as described in Appendix A. The plane fitting assumes a single plane within a local area. When the area contains multiple objects or object–background boundaries, some pixels become outliers with respect to the actual dominant plane. Consequently, the dominant plane fitting may estimate a spurious plane as shown in Figure 2. To suppress these outliers, pixel-wise confidences with respect to the dominant plane are estimated using fully connected layers followed by a sigmoid activation. Then, the pixel values are weighted by the estimated confidences. The outlier correction is described as follows:
where Z is the depth values within the patch with the central depth value subtracted, is the adjusted depth values, is a sigmoid function, and is a fully connected operation. Subsequently, is substituted into (A9) to estimate the parameters of the dominant plane.
Figure 2.
Example of spurious plane fitting due to presence of an outlier depth value.
3.1.2. Plane-Based Convolution
For each k × k local patch, the estimated plane parameters a and b are substituted into (A6) to compute the depth values of the corresponding pixels reconstructed from the fitted plane as follows:
Subsequently, the residuals is calculated as the difference between and as follows:
This operation aligns the depth values within the local patch with respect to the normal direction of the dominant plane and removes the component along that direction. As a result, R emphasizes the remaining geometric structures after eliminating the planar component. This representation enables the extraction of surface-intrinsic geometric features that are less sensitive to changes in surface orientation by compensating for the depth variations induced by surface orientation and removing orientation-dependent bias.
Pixels with large residuals that are computed with respect to the dominant plane can be interpreted as not following the dominant local structure represented by the corresponding patch. However, the residuals may be excessively large due to the large dynamic range of pixel values in depth images. When such large residuals are directly used as inputs to convolution operations, the contribution of the corresponding pixels can become disproportionately large, thereby degrading the stability of feature extraction. To suppress the excessive contribution of large residuals in convolution, a softsign function is employed to smoothly clamp the residual values as follows:
where s is a learnable scalar value that controls the suppression range of the residuals.
For all patches in the image, the dominant plane parameters are estimated via (A8), and then the corresponding residuals are computed according to (3) and (4), followed by a local convolution operation that produce a plane-residual feature map in . In addition, a standard convolution is applied to the plane parameter map in to obtain a plane-gradient feature map in that represents local surface variations. Batch normalization is applied separately to the plane feature map and the surface feature map to normalize their distinct feature distributions. The two normalized feature maps are fused using such a 1 × 1 convolution and then an activation function is applied to produce a plane feature map in .
3.2. Integration of Plane-Based Convolution Module into Existing Feature Extractor
To enhance initial features of depth images, the proposed plane-based convolution module is attached in parallel to the stem layer, the entry point of an existing CNN-based feature extractor as shown in Figure 3a. The original stem-layer branch extracts pixel-wise features, such as boundaries and gradients of depth values, while the plane-based convolution branch captures plane-aware features from the corresponding plane residuals and plane parameters. To match the spatial resolution of the original stem layer, patches are extracted using the same kernel size, stride, and padding as in the stem layer. Figure 3b illustrates a structure in which the proposed module is attached in parallel to the stem layer of MobileNetV3. Since most CNN-based feature extractors such as MobileNet [16], employ odd-sized patches in their stem layers, the dominant plane fitting by (A9) can be applied directly. In contrast, it is difficult to apply (A9) to stem layers with even-sized patches, as in ConvNeXt [17]. In this case, the depth value at the center, which lies at non-integer coordinates, is obtained by bilinear interpolation of the four pixel values at the neighboring integer coordinates, and then (A8) is applied to the dominant plane fitting. Subsequently, the initial features extracted from the two branches are fused and fed into the following layers.
Figure 3.
Structure of feature extractor with dual-branch stem layers: (a) overall structure; (b) applied to MobileNetV3.
4. Experiment Results and Discussion
4.1. Dataset
To verify the improvement in feature extraction performance achieved by the proposed method, we conducted experiments on a single-object classification task on depth and RGB-D images. A Large Dataset of Object Scans [18] was used for single-object classification experiments. This dataset provides RGB-D images of individual objects captured under various environmental conditions. The images of this dataset were captured using PrimeSense Carmine manufactured by PrimeSense (Tel Aviv, Israel) [19]. The dataset includes RGB-D images captured in real-world indoor and outdoor environments. Both the color and depth images have a resolution of 640 × 480. Spatial registration is applied in advance to the corresponding color and depth images. In the depth images, each pixel value represents the distance from the sensor measured in millimeters. As described in Table 1, we sampled the 28 most frequent classes from the full dataset to construct an classification experiment dataset consisting of 244,816 image pairs. Of these, 80% (195,852 image pairs) were used for training the network, while the remaining 20% (48,964 image pairs) were used for validation. We preprocessed the depth images by correcting missing pixels with a 5 × 5 mean filter and then converting pixel values to real-valued representations through division by 1000. Figure 4 shows the samples of the classification experiment dataset.
Table 1.
Class distribution of classification experiment dataset.
Figure 4.
Samples of classification experiment dataset.
We also conducted experiments on a semantic segmentation task on RGB-D images. NYUDepth v2 [20] and SUN RGB-D [21] were utilized for the experiments for semantic segmentation. Both NYUDepth v2 and SUN RGB-D are datasets collected in indoor environments. NYUDepth v2 offers a limited number of samples with preprocessed depth maps in which missing regions are filled. In contrast, SUN RGB-D comprises a larger-scale dataset captured using diverse depth sensors and covers more diverse indoor scenes, while providing raw depth maps that retain missing pixels without preprocessing. The images in NYUDepth v2 were captured by Kinect v1 [22] manufactured by Microsoft (Redmond, WA, USA). SUN-RGBD was collected from a variety of sensors across the dataset, including Microsoft Kinect v1 and Kinect v2 (Redmond, WA, USA), Intel RealSense F200 (Santa Clara, CA, USA), and Asus Xtion Pro Live (Taipei, Taiwan) [23]. NYUDepth v2 and SUN-RGBD consists of 1449 image pairs with 795 for training and 654 for testing and 10,335 pairs with 5285 for training and 5050 for testing, respectively. NYUDepth v2 and SUN-RGBD include 40 and 37 categories, respectively. For both NYU Depth v2 and SUN-RGBD, the color and depth images were provided with spatial registration applied. We applied the same preprocessing method used for the classification experiment dataset to the segmentation experiment datasets. Figure 5 shows the samples of the segmentation experiment datasets including the corresponding color images, depth images, and color-coded segmentation labels for multiple scenes.
Figure 5.
Samples of segmentation experiment datasets: (a) NYUDepth v2; (b) SUN-RGBD.
4.2. Experiment for Single-Object Classification
To assess the performance improvement of the proposed module for the object classification task, we employed a network consisting of a feature extractor and a classification head composed of a fully connected layer. We adopted MobileNetV3 [16], ConvNeXt [17], and CoAtNet [24] as baseline feature extractors. The baseline feature extractors were initialized with weights pre-trained on ImageNet-1k [25]. We compared the performance of the baseline feature extractors with the proposed module attached to their stem layers against those without the module. A 1 × 1 convolution was employed as the fusion method to combine the features extracted from the original stem layer and the proposed module. The depth image input to the original stem layer was replicated along the channel dimension to form a three-channel input, thereby ensuring compatibility with the original stem layer designed for color images. The images were resized to 224 × 224 to match the input resolution of the baseline feature extractor. The network was trained for 100 epochs using a cross-entropy loss function, the Adam optimizer and a learning rate of 5 × 10−5. For RGB-D inputs, separate stem layers are employed for the color and depth images as shown in Figure 6. Although the two stem layers share the same network architecture, their weights are not shared during training. The initial features extracted from each stem layer are fused through a 1 × 1 convolution and then forwarded to the subsequent layers of the baseline feature extractor.
Figure 6.
Network architecture for object classification for RGB-D images.
Table 2 shows a comparison of single-object classification performances on depth-only images. In Table 2, ‘Ours’ and ‘Baseline’ indicate the classification networks with and without the proposed module, respectively. Compared to the baseline models without the proposed module, the proposed module improves the F1 score by an average of 0.0015 with only a negligible increase in computational cost. Specifically, the F1 score increases by 0.0013, 0.0016, and 0.0016 for MobileNetV3, ConvNeXt, and CoAtNet, respectively. This improvement can be attributed to the enhancement of the initial feature representation through the acquisition of three-dimensional features that are less sensitive to distance and viewpoint.
Table 2.
Comparison of Single-object classification performance on depth-only images.
Figure 7 shows plane fitting results for 3 × 3 patches in a depth image where the displayed values are scaled. The patches A and B correspond to regions on non-planar surfaces and the patch C corresponds to a plane with an outlier pixel. The nonlinearly distributed pixels or the outlier pixels caused the residuals from the plane fitting without the outlier correction to excessively activate pixels that slightly deviate from the dominant plane. In contrast, the residuals from the plane fitting with the proposed outlier correction suppress such activations effectively.
Figure 7.
Examples of plane fitting with and without outlier correction.
Figure 8 shows feature representations extracted by the original stem layer and the proposed module of MobileNet v3. In the original stem layer, strong activations are observed on non-frontal surfaces. In contrast, such gradient-based responses are effectively reduced by the proposed module. This suppression of gradient-based responses enables the network to focus more on the intrinsic three-dimensional surface characteristics of objects rather than on surface orientation variations induced by changes in the viewpoint.
Figure 8.
Visualization of features extracted by stem layer: (a) depth input image; (b) extracted by original stem layer; (c) extracted by the proposed module.
Table 3 shows an ablation study on branch configuration of the proposed module in MobileNet v3. The proposed module that includes only the plane residual convolution branch achieves performance comparable to that of the module with all branches. In contrast, the module with only the plane parameter convolution branch also improves the F1 score by 0.0007 compared to the baseline model, but the improvement is relatively limited. This indicates that the plane parameter convolution branch contributes marginally to the overall performance improvement.
Table 3.
Ablation study on branch configuration of the proposed module in MobileNet v3.
Table 4 presents an ablation study on the MobileNet model incorporating the proposed module, investigating the effects of the outlier correction and residual clamping. When outlier correction and residual clamping are individually removed, the F1 Score decreases by 0.0002 and 0.0005, respectively. Removing both components leads to the F1 Score of approximately 0.0002. These results indicate that the proposed plane residual convolution effectively mitigates the influence of depth outliers within each patch through depth correction during the dominant plane fitting, while residual clamping limits the impact of excessively large residuals during convolution.
Table 4.
Ablation study on outlier correction and residual clamping.
In Table 5, the proposed residual clamping method based on the softsign function was evaluated against alternative formulations in which this function in (4) was substituted with either a clipping method that truncates out-of-range residuals or the tanh function that exhibits similar continuous saturation behavior. Both the clipping method and the tanh function yield lower performance than the case without the residual clamping. The performance degradation is caused by the gradient converging to zero in regions with large input magnitudes, making it difficult to effectively update the parameter s in (4), which controls the residual limiting range. In contrast, the softsign function preserves meaningful gradients even for large-magnitude inputs, which enables more stable updates of the parameter s.
Table 5.
Classification accuracy with different clamping methods.
Figure 9 presents Grad-CAM++ [26] visualizations of the regions attended by each network during classification with depth-only images, where warmer colors indicate regions that contribute more strongly to the model prediction. MobileNet generally demonstrates a more uniform focus over object regions. ConvNeXt attends to salient object regions for large objects, whereas relatively higher activations are observed in surrounding background regions for smaller objects. The networks attached with the proposed module exhibit increased attention directed toward object surface regions with reduced background activations compared to their respective baseline models.
Figure 9.
Heat map visualization for depth-only images.
Table 6 shows a comparison of single-object classification performances on RGB-D images. MobileNet, ConvNeXt, and CoAtNet with the proposed module achieved F1 Score improvements of 0.0018, 0.0019, and 0.0012, respectively, for RGB-D images. This improvement is comparable to that observed with depth-only images.
Table 6.
Comparison of Single-object classification performance on RGB-D images.
Figure 10 shows Grad-CAM++ visualizations of the regions attended by each network during the classification with RGB-D images. Compared to their respective baseline models, the networks equipped with the proposed module suppress background-related attention present in the original networks, while focusing more on object surface regions.
Figure 10.
Heat map visualization for RGB-D images.
4.3. Experiment for Semantic Segmentation
To evaluate the segmentation performance of the proposed module, an experiment was conducted based on HDBFormer [27] which is a semantic segmentation framework for RGB-D images. HDBFormer employs a color image encoder that combines Swin Transformer [28] and CNN architectures, along with a CNN-based depth image encoder. The proposed module was attached into the first layer of the depth image encoder. The segmentation performance was evaluated using mean intersection over union (mIoU) and pixel accuracy (PA).
Table 7 presents a comparison of segmentation performances on RGB-D images. On the NYUDepth v2 dataset, the proposed module improves mIoU and pixel accuracy (PA) by 0.3 percentage points, respectively. These results suggest that the proposed method is effective not only for classification but also for segmentation tasks. On the SUN RGB-D dataset, the proposed module achieves improvements of 0.1 percentage points in both mIoU and PA. Although the performance gains on SUN RGB-D which consists of data acquired from diverse sensors are relatively limited, the proposed module demonstrated consistent performance improvements. This indicates that the proposed module is capable of learning feature representations that are less sensitive to domain discrepancies across sensors.
Table 7.
Comparison of semantic segmentation performance on RGB-D images.
5. Discussion
Several previous works share a similar motivation with the proposed method by modifying standard convolutional operations to better align with the 3D geometric structure of depth images. SurfConv [14] adjusts the scale of the receptive field according to depth values and Depth-aware CNN [12] suppresses the contributions of pixels belonging to different surfaces within the receptive field based on depth similarity. These methods have proposed point-wise depth-specialized convolutions, while the proposed method performs surface-based convolution. Depth-adapted CNN [15] shares a similar motivation with the proposed approach in terms of surface-centric feature extraction. However, while Depth-adapted CNN acquires surface-orientation-aware receptive fields through deformed kernels, the proposed method suppresses the influence of surface orientation by decomposing depth values into planar and residual components. Yang et al. [13] introduce an approach that convolves residuals with respect to pixel-wise reference values, which is conceptually related to the proposed method. In contrast, the proposed method convolves residuals defined with respect to the principal plane, thereby further enhancing features that capture the underlying 3D geometric structure.
The experimental results demonstrated that the proposed method improved performance across different tasks by enhancing the initial feature representations of depth images. Although the observed performance gains are limited in magnitude, this outcome is expected given that the proposed method focuses solely on modifying the stem layer at the very early stage of the network. Importantly, even such a lightweight modification results in consistent improvements across multiple tasks, highlighting the effectiveness of enhancing early-stage depth feature representations. In particular, the experimental results on the SUN-RGBD dataset which consists of data acquired from different sensors suggest that the proposed method generalizes well across different camera intrinsic parameters and noise conditions. The results indicates that the effectiveness of the proposed module can be generally applied to different CNN-based and CNN–ViT hybrid network architectures [17,24,29] without requiring extensive architectural modifications or additional computational complexity.
The proposed plane fitting and residual computation are performed directly within the convolution operation over the receptive field, without introducing separate patch partitioning or additional preprocessing steps. Therefore, the integration of the proposed module introduces a small increase in computational cost. As a result, the proposed module is applicable not only to standard architectures but also to lightweight models intended for edge-device deployment without a noticeable increase in inference latency.
Nonetheless, the ablation study indicates that the contribution of surface orientation is relatively limited. This limitation may stem from the use of simple convolution over normal vectors of the planes, which does not explicitly model directional similarity between vectors. To mitigate this issue, future work could explore operations such as inner or outer products that more effectively characterize directional relationships among vectors.
To obtain high-quality 3D geometric features using the proposed method, complete depth images are required. In the presence of missing pixels or regions in depth images, the performance of the proposed method may degrade significantly. In our experiments, the preprocessing was applied to fill in missing depth pixels prior to network training and evaluation. However, there remains room for further improvement in the quality of the completed depth values. To further improve performance, an effective depth completion method should be incorporated to fill in missing depth pixels.
6. Conclusions
In this paper, we proposed the module that enhances the representation of initial depth features by performing convolution operations on plane-based residuals and plane parameters. Within each patch, a dominant plane was estimated from depth pixel values. Convolution was applied to the residuals between the estimated plane and the original depth pixels. In addition, a separate convolution operation was performed on the plane parameters estimated for each patch. The feature maps extracted from these two operations were fused and forwarded to subsequent layers. The proposed module was attached in parallel to the stem layer of the existing feature extractor to integrate 3D spatial structural information into the initial features and to encourage the network to attend more on actual objects in depth images. The overall experimental results demonstrated that the proposed module improved the F1 score for RGB-D image object classification by an average of 0.0016 and increased the mIoU for semantic segmentation by an average of 0.2 percentage points. Since the proposed module can be seamlessly integrated with the existing feature extractors pre-trained on large-scale RGB datasets, it can be optimized even with a relatively small amount of depth data. The vision task network with the proposed module is expected to be applicable to a wide range of 3D vision applications including 3D pose estimation, robotic vision, and autonomous driving.
Author Contributions
Conceptualization, D.-s.L. and S.-k.K.; software, D.-s.L.; writing—original draft preparation, D.-s.L. and S.-k.K.; supervision, S.-k.K. All authors have read and agreed to the published version of the manuscript.
Funding
This research was partly supported by the Innovative Human Resource Development for Local Intellectualization program through the Institute of Information & Communications Technology Planning & Evaluation (IITP), grant funded by the Korean government (MSIT) (IITP-2026-RS-2020-II201791, 100%) and by the BB21plus funded by Busan Metropolitan City and Busan Techno Park.
Institutional Review Board Statement
Not applicable.
Informed Consent Statement
Not applicable.
Data Availability Statement
The original contributions presented in this study are included in the article. Further inquiries can be directed to the corresponding author.
Conflicts of Interest
The authors declare no conflicts of interest.
Appendix A
Appendix A describes the plane fitting method based on the least squares method. In a depth image, 2D image coordinates (x, y) can be transformed into 3D space coordinates (X, Y, Z) using its depth value z as follows [30]:
where f is the focal length, which is an intrinsic parameter of the camera. If a local region lies on a plane, then a 3D coordinates (Xp, Yp, Zp) corresponding to a pixel (xp, yp) within the region satisfy the following plane equation:
Substitution of (A1) into (A2) yields the following equation:
After translating the origin of Equation (A3) to the 3D point corresponding to (x0, y0), the following equation is obtained:
When (xp, yp) is (x0, y0), zp is equal to z0. Thus, must be zero to satisfy (A4). For pixels within a sufficiently small local area, the depth variation can be approximated as linear. Therefore, and can be approximated by and , respectively. These approximations are applied to simplify (A4) as follows:
where a and b are and , respectively, and Δxp, Δyp, and Δzp are , , and , respectively. In (A5), a and b which are the plane parameters reflect the corresponding surface orientation. By removing the sensor-specific term f from (A4), no additional camera-dependent constants are required for the plane fitting.
For an arbitrary local area, let X, Y, and Z denote the matrices of Δx, Δy, and the corresponding Δz, respectively. Then, the following matrix equation is obtained:
where X is , Y is , and Z is , respectively, and nk is the number of the pixels within the local area. By applying the least-squares method to (A6), a dominant plane can be fitted to the set of 3D points constructed from the pixels in the local region and their corresponding depth values, from which the plane parameters a and b can be estimated as follows:
If (x0, y0) is chosen such that the relative coordinates (Δxp, Δyp) for are symmetric, then the parameters a and b are calculated as follows.
When the local area is a k × k square patch with odd k, the sums of Δx2 and Δy2 are both equal to . Therefore, (A8) can be expressed in a matrix equation as follows:
where is an element-wise multiplication and sum(⋅) denotes the scalar value obtained by summing all elements of its matrix.
References
- Kim, J.H.; Lee, D.S.; Kwon, S.K. Volume estimation method for irregular object using RGB-D deep learning. Electronics 2025, 14, 919. [Google Scholar] [CrossRef] [Scilit]
- Gao, Q.; Ju, Z.; Chen, Y.; Wang, Q.; Chi, C. An efficient RGB-D hand gesture detection framework for dexterous robot hand-arm teleoperation system. IEEE Trans. Hum. Mach. Syst. 2023, 53, 13–23. [Google Scholar] [CrossRef] [Scilit]
- Huang, J.; Yu, H.; Yu, K.T.; Navab, N.; Ilic, S.; Busam, B. MatchU: Matching unseen objects for 6D pose estimation from RGB-D images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 16–22 June 2024. [Google Scholar]
- Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. An image is worth 16 × 16 words: Transformers for image recognition at scale. In Proceedings of the International Conference on Learning Representations, Online, 3–7 May 2021. [Google Scholar]
- Dong, S.; Feng, Y.; Yang, Q.; Huang, Y.; Liu, D.; Fan, H. Efficient multimodal semantic segmentation via dual-prompt learning. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, Abu Dhabi, United Arab Emirates, 14–18 October 2024. [Google Scholar]
- Wang, Y.; Chen, X.; Cao, L.; Huang, W.; Sun, F.; Wang, Y. Multimodal token fusion for vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA, 21–24 June 2022. [Google Scholar]
- Yin, B.; Zhang, X.; Li, Z.; Liu, L.; Cheng, M.M.; Hou, Q. DFormer: Rethinking RGB-D representation learning for semantic segmentation. In Proceedings of the International Conference on Learning Representations, Vienna, Austria, 7–11 May 2024. [Google Scholar]
- Zhang, J.; Liu, H.; Yang, K.; Hu, X.; Liu, R.; Stiefelhagen, R. CMX: Cross-modal fusion for RGB-X semantic segmentation with transformers. IEEE Trans. Intell. Transp. Syst. 2023, 24, 14679–14694. [Google Scholar] [CrossRef] [Scilit]
- Gupta, S.; Girshick, R.; Arbelaez, P.; Malik, J. Learning rich features from RGB-D images for object detection and segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA, 23–28 June 2014. [Google Scholar]
- Çağlayan, A.; İmamoğlu, N.; Can, A.B.; Nakamura, R. When CNNs meet random RNNs: Towards multi-level analysis for RGB-D object and scene recognition. Comput. Vis. Image Underst. 2022, 221, 103373. [Google Scholar] [CrossRef] [Scilit]
- Gopalapillai, R.; Gupta, D.; Zakariah, M.; Alotaibi, Y.A. Convolution-based encoding of depth images for transfer learning in RGB-D scene classification. Sensors 2021, 21, 7950. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Wang, W.; Neumann, U. Depth-aware CNN for RGB-D segmentation. In Proceedings of the European Conference on Computer Vision, Munich, Germany, 8–14 September 2018. [Google Scholar]
- Yang, J.; Bai, L.; Sun, Y.; Tian, C.; Mao, M.; Wang, G. Pixel difference convolutional network for RGB-D semantic segmentation. IEEE Trans. Circuits Syst. Video Technol. 2024, 34, 1481–1492. [Google Scholar] [CrossRef] [Scilit]
- Chu, R.; Sun, Y.; Li, Y.; Liu, Z.; Wei, Y. SurfConv: Bridging 3D and 2D convolution for RGB-D images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–22 June 2018. [Google Scholar]
- Wu, Z.; Allibert, G.; Stolz, C.; Ma, C.; Demonceaux, C. Depth-adapted CNN for RGB-D cameras. In Proceedings of the Asian Conference on Computer Vision, Online, 30 November–4 December 2020. [Google Scholar]
- Howard, A.; Sandler, M.; Chu, G.; Chen, L.C.; Chen, B.; Tan, M.; Wang, W.; Zhu, Y.; Pang, R.; Vasudevan, V.; et al. Searching for MobileNetV3. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea, 27 October–2 November 2019. [Google Scholar]
- Liu, Z.; Mao, H.; Wu, C.Y.; Feichtenhofer, C.; Darrell, T.; Xie, S. A ConvNet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA, 19–24 June 2022. [Google Scholar]
- Choi, S.; Zhou, Q.Y.; Miller, S.; Koltun, V. ScanNet: Richly-annotated 3D reconstructions of indoor scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017. [Google Scholar]
- PrimeSense. Carmine 1.09 RGB-D Sensor. Available online: http://xtionprolive.com/primesense-carmine-1.09-3d-sensor-motion-camera/ (accessed on 18 December 2025).
- Silberman, N.; Kohli, P.; Hoiem, D.; Fergus, R. Indoor segmentation and support inference from RGBD images. In Proceedings of the European Conference on Computer Vision, Florence, Italy, 7–13 October 2012. [Google Scholar]
- Song, S.; Lichtenberg, S.; Xiao, J. SUN RGB-D: A RGB-D scene understanding benchmark suite. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 7–12 June 2015. [Google Scholar]
- Kinect for Windows Sensor Components and Specifications. Available online: https://learn.microsoft.com/en-us/previous-versions/windows/kinect-1.8/jj131033(v=ieb.10) (accessed on 15 January 2026).
- 3D Camera Survey. Available online: https://rosindustrial.org/news/2016/1/13/3d-camera-survey (accessed on 15 January 2026).
- Dai, Z.; Liu, H.; Le, Q.V.; Tan, M. CoAtNet: Marrying convolution and attention for all data sizes. In Proceedings of the Advances in Neural Information Processing Systems, Online, 6–14 December 2021. [Google Scholar]
- Deng, J.; Dong, W.; Socher, R.; Li, L.J.; Li, K.; Li, F.F. ImageNet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA, 20–25 June 2009. [Google Scholar]
- Chattopadhay, A.; Sarkar, A.; Howlader, P.; Balasubramanian, V.N. Grad-CAM++: Generalized gradient-based visual explanations for deep convolutional networks. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision, Lake Tahoe, NV, USA, 12–15 March 2018. [Google Scholar]
- Wei, S.; Zhou, Z.; Lu, Z.; Yuan, Z.; Su, B. HDBFormer: Efficient RGB-D semantic segmentation with a heterogeneous dual-branch framework. IEEE Signal Process. Lett. 2025, 32, 91–95. [Google Scholar] [CrossRef] [Scilit]
- Liu, Z.; Lin, Y.; Cao, Y.; Hu, H.; Wei, Y.; Zhang, Z.; Lin, S.; Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada, 11–17 October 2021. [Google Scholar]
- Wu, H.; Xiao, B.; Codella, N.; Liu, M.; Dai, X.; Yuan, L.; Zhang, L. CvT: Introducing convolutions into vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada, 11–17 October 2021. [Google Scholar]
- Hartley, R.; Zisserman, A. Multiple View Geometry in Computer Vision, 2nd ed.; Cambridge University Press: New York, NY, USA, 2000; pp. 153–177. [Google Scholar]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2026 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license.









