Fast Intra-Coding Unit Partitioning for 3D-HEVC Depth Maps via Hierarchical Feature Fusion

Abstract

As a new generation 3D video coding standard, 3D-HEVC offers highly efficient compression. However, its recursive quadtree partitioning mechanism and frequent rate-distortion optimization (RDO) computations lead to a significant increase in coding complexity. Particularly, intra-frame coding in depth maps, which incorporates tools like depth modeling modes (DMMs), substantially prolongs the decision-making process for coding unit (CU) partitioning, becoming a critical bottleneck in compression encoding time. To address this issue, this paper proposes a fast CU partitioning framework based on hierarchical feature fusion convolutional neural networks (HFF-CNNs). It aims to significantly accelerate the overall encoding process while ensuring excellent encoding quality by optimizing depth map CU partitioning decisions. This framework synergistically captures CU’s global structure and local details through multi-scale feature extraction and channel attention mechanisms (SE module). It introduces the wavelet energy ratio designed for quantifying the texture complexity of depth map CU and the quantization parameter (QP) that reflects the encoding quality as external features, enhancing the dynamic perception ability of the model from different dimensions. Ultimately, it outputs depth-corresponding partitioning predictions through three fully connected layers, strictly adhering to HEVC’s quad-tree recursive segmentation mechanism. Experimental results demonstrate that, across eight standard test sequences, the proposed method achieves an average encoding time reduction of 48.43%, significantly lowering intra-frame encoding complexity with a BDBR increment of only 0.35%. The model exhibits outstanding lightweight characteristics with minimal inference time overhead. Compared with the representative methods under comparison, this method achieves a better balance between cross-resolution adaptability and computational efficiency, providing a feasible optimization path for real-time 3D-HEVC applications.

1. Introduction

With the advancement of 3D vision technology, 3D video has garnered significant attention due to its immersive and realistic visual effects. Stereoscopic displays based on planar stereo imaging have become a major research focus [1]. However, the increase in the number of viewpoints, while enhancing the stereoscopic effect, leads to an exponential growth in data volume, posing significant challenges for video storage and transmission [2].

To address these challenges, the Joint Collaborative Team on 3D Video Coding (JCT-3V) [3], established by the ITU’s VCEG and ISO’s MPEG, has developed various 3D video compression standards. The evolution of coding technology has progressed from the H.264-based Multiview Video Coding (MVC) extension standard in 2005 [4] to the MVC+D extension standard in 2013 [5], the 3D-AVC standard at the end of 2013 [6], and the MV-HEVC and 3D-HEVC standards developed between 2014 and 2015 [7,8]. Among these, 3D-HEVC not only efficiently compresses texture images but also introduces depth map coding techniques, leveraging the correlation between texture and depth to achieve high-efficiency 3D video compression.

To enhance the efficiency and robustness of depth map coding, 3D-HEVC introduces several new coding tools, such as Depth Modeling Modes (DMMs), Depth Intrinsic Segmentation (DIS), Segment-Based Depth Coding (SDC), and View Synthesis Optimization (VSO) [9,10]. However, the recursive quadtree partitioning of Coding Units (CUs) and the exhaustive prediction methods significantly increase encoding time and computational complexity. Currently, the complexity of 3D-HEVC severely limits its further development. To facilitate the widespread adoption of 3D-HEVC, it is imperative to reduce computational complexity while ensuring negligible coding performance loss. This is crucial for improving practical application efficiency and advancing 3D video technology.

In the exploration of improving 3D-HEVC depth map intra coding, researchers have focused on two key directions: refining CU partitioning and optimizing prediction mode selection. These efforts aim to reduce encoding complexity while maintaining coding quality. In recent years, proposed CU partitioning optimization strategies can be broadly categorized into three types. Statistical decision-based methods: Zou et al. [11] leveraged Bayesian statistical decision theory to compute conditional probabilities and risk costs for splitting versus retaining CUs, providing a rigorous mathematical criterion for early termination of partitioning. In contrast, Wang et al. [12] treat Bayes’ theorem as a classifier, predicting the optimal partition depth through feature patterns learned from data to simplify the decision process. Omran et al. [13] innovatively applied the traditional image processing technique of connected component labeling (CCL) to the CU partitioning problem in depth maps. This method leverages the inherent structural properties of depth maps to achieve highly accurate segmentation in low-bitrate applications. Machine learning-based approaches: Hamout and Elyousfi [14] proposed an optimized CU segmentation method based on the AutoMerge Possibility Clustering (AM-PCM) algorithm. By extracting amplitude-simplified centroid (ASMCV) and variance from depth maps as features, dynamic thresholds were derived through clustering analysis to filter CU sizes requiring evaluation, thereby reducing redundant computations. Subsequently, they proposed a joint optimization method combining clustering and edge detection [15], precisely identifying and reducing redundant computations by dynamically clustering depth map features with edge detection techniques. Su et al. [16] employed Gradient Boosting Machines (GBMs), a powerful ensemble learning algorithm, using Average Local Variance (ALV) as a feature to learn adaptive segmentation thresholds for CUs at different depths. Li et al. [17] transformed CU segmentation into a clustering problem via unsupervised learning, proposing an adjustable early decision scheme; Wang [18] applied IoT data to accelerate 3D-HEVC encoding by extracting CU segmentation-related information through data analysis to optimize decision efficiency. Deep learning-based approaches: Liu et al. [19] proposed a fast deep intra-frame coding method using a deep edge classification network, classifying depth map edge features via CNN to significantly reduce coding complexity; Omran et al. [20] introduced a fast segmentation algorithm based on multi-depth convolutional neural networks (MD-CNNs), training specialized depth map models integrated into the encoder and leveraging inter-view dependencies to lower complexity.

Optimization of intra prediction mode selection is also an important direction to improve 3D-HEVC coding efficiency. Hamout et al. [21] employed a self-organizing map (SOM) network combined with manually selected efficient features to construct a model for skipping redundant intra modes in depth maps; Zhang et al. [22] proposed a global context aggregation intra prediction network, optimizing intra prediction modes for depth video coding by aggregating global and local features; Zhang et al. [23] proposed an intra prediction scheme based on depth region segmentation, using deep networks to learn depth map region features to achieve refined mode classification; Lee et al. [24] proposed a fast depth intra mode decision method based on intra prediction cost and probability, adaptively skipping redundant HEVC prediction modes and DMM by utilizing prediction cost and probability characteristics, effectively reducing coding complexity; Song et al. [25] proposed a content-adaptive mode decision method, dynamically adjusting strategies according to video content characteristics to balance complexity and coding quality; Chen et al. [26] simplified redundant calculations of intra prediction modes and depth modeling modes by analyzing the total sum of squares and variance of prediction unit boundaries, and dynamically adjusting decision thresholds in combination with rate-distortion costs; Huo et al. [27] proposed a fast rate-distortion optimization method for depth maps, introducing a cumulative check mechanism in RD cost calculation, terminating the current mode calculation when the cumulative cost reaches the existing minimum value, and simplifying the selection process.

Although existing methods have achieved significant success in reducing the intra-frame encoding complexity of 3D-HEVC depth maps, there remains room for improvement in the direction of CU segmentation for depth maps: statistical decision-based approaches rely on manual thresholds and heuristic rules, failing to adaptively capture global-local relationships within CUs and struggling to handle complex content; machine learning-based approaches, while capable of data-driven learning of feature-partition relationships, are constrained by feature engineering quality and model capacity, making it difficult to capture the highly nonlinear structure of depth maps.

In recent years, deep learning has been widely applied in video coding, yet significant shortcomings persist in 3D-HEVC depth map coding. For instance, Liu et al. [19] employed single-scale feature extraction, while [20] utilized a multi-channel architecture but failed to effectively fuse global and local features, resulting in poor segmentation accuracy in texture-complex regions. Furthermore, existing methods lack a quantization parameter awareness mechanism, making it challenging to balance computational efficiency and coding performance across resolution scenarios. To address these issues, this paper proposes the Hierarchical Feature Fusion Convolutional Neural Network (HFF-CNN). Experimental results demonstrate that the proposed method achieves a 48.43% encoding time reduction across eight standard test sequences with an incremental BDBR of only 0.35%. The core contributions of this paper are as follows:

• Construct external features: Designing a wavelet energy ratio for quantifying the texture complexity of depth map CUs and combining it with QP to jointly serve as external features. These are fused with network-extracted features, thereby enhancing the model’s perception of texture structure and coding quality;
• Propose the HFF-CNN: Hierarchically processes input CU features and employs a multi-scale feature fusion strategy. Combines with a channel attention mechanism (SE module) to achieve adaptive enhancement of key features, followed by integration with external features to enable accurate predictions.

The structure of this paper is arranged as follows: Section 2 analyzes the coding structure of 3D-HEVC, focusing on the complexity of depth map coding and the CU partitioning process for depth maps to identify the encoding complexity issues and lay the foundation for subsequent research. Section 3 details the proposed algorithm, including the setting of external features, model design, and overall process, to construct a complete framework for fast intra coding of 3D-HEVC depth maps. Section 4 summarizes the experimental results. Finally, Section 5 provides a conclusion to this study and outlines directions for future research.

2. Observations and Analysis

2.1. Complexity Analysis of 3D-HEVC Depth Map Coding

As a derivation of the HEVC framework, 3D-HEVC provides an effective solution for three-dimensional video coding. By eliminating redundancy between viewpoints, it significantly improves the compression efficiency of multi-viewpoint video and depth maps. Within the 3D-HEVC encoding framework, multi-viewpoint texture and depth maps require processing. Figure 1 illustrates its encoding structure (from Viewpoint 0 to Viewpoint N-1, where N is the total number of viewpoints). These images capture perspectives at the same moment but from different spatial positions. Consequently, viewpoints exhibit both high correlation and variations in parallax and occlusion due to perspective differences. The encoding process advances through two data streams: red arrows convey cross-viewpoint information, leveraging viewpoint similarity to reduce redundancy, while black arrows handle intra-viewpoint encoding data transfer between texture and depth maps. All encoded data are ultimately multiplexed into a 3D bitstream.

Depth maps play a critical role in 3D-HEVC encoding. They convey the key depth information of objects in the scene, which is essential for synthesizing high-quality 3D views. In applications such as virtual reality (VR) and augmented reality (AR), accurate depth map encoding serves as the foundation for providing realistic 3D visual experiences. For example, in a VR environment, depth map information can precisely describe the spatial relationships between objects, enabling users to perceive realistic spatial depth. Different from natural scene videos, the pixel values in depth maps directly represent the depth of objects, featuring unique value ranges and distribution patterns. The edge information in depth maps is crucial for determining the shape and position of objects, and any blurring or distortion of these edges will severely degrade the visual quality of synthesized views.

In terms of encoding time, depth map encoding presents a significant time-consuming issue. As shown in Figure 2, we statistically analyzed the encoding time distribution of four groups of video sequences at two resolutions: Balloons [28], Newspaper [29] at 1024 × 768, and Shark [30], and Poznan_Hall2 [31] at 1920 × 1088. The results show that the time required for depth map encoding far exceeds that of texture map encoding. Data indicate that depth map encoding accounts for an average of 78.42% of the total encoding time, while texture map encoding only accounts for 21.58%, with the former taking 3.6 times longer than the latter. This is because, in addition to the conventional intra prediction techniques used for texture map encoding, depth map encoding introduces new technologies such as VSO, DMM, and SDC. These technologies enhance encoding efficiency but significantly increase computational complexity. Therefore, how to effectively reduce the computational load of depth map encoding while maintaining encoding quality has become a critical issue that needs urgent resolution.

2.2. Complexity Analysis of CU Partitioning for 3D-HEVC Depth Maps

The high encoding complexity of H.265/HEVC [32] stems from its recursive quadtree-based CU partitioning mechanism. 3D-HEVC inherits this structure, where video frames are initially divided into 64 × 64 Coding Tree Units (CTUs) corresponding to depth level 0, as shown in Figure 3. In subsequent encoding stages, these CTUs are further subdivided according to the quadtree structure to form smaller blocks. As the partitioning depth increases, the granularity of CUs becomes finer. Lower-depth CUs typically represent large-scale uniform regions, while higher-depth CUs usually involve more complex detailed regions. Although this layered partitioning approach effectively improves encoding accuracy, it also leads to a substantial increase in computational load during the encoding process—especially in depth map encoding, where the complexity of CUs sharply increases with higher depth levels.

The quadtree partitioning of CTU adopts a hierarchical decision-making mechanism, which is divided into two stages: top-down RD-cost calculation and bottom-up partitioning decision. In the top-down stage, CUs at each level (from depth 0 to 3) are traversed in the order of increasing depth, and RD-cost is calculated layer by layer; in the bottom-up stage, optimal partitioning is determined through cross-layer comparison: if the RD-cost of a CU at depth n is higher than the sum of its four child partitions at the sub-layer (n = 0, 1, 2), partitioning is enforced; otherwise, the CU is retained. For a 64 × 64 CTU, the full traversal mode requires 85 CU partitioning validations, 11,935 SATD (Sum of Absolute Transformed Differences) cost calculations, and 2623 RD cost calculations to obtain the final partitioning result. This process balances distortion and bitrate cost through the RDO algorithm, where the RD-cost calculation formula is as follows:

$$R{D}_{cost}=SS{E}_{luma}+{\omega }_{chroma}\times SS{E}_{chroma}+{\lambda }_{mode}\times R_{mode}$$

(1)

Here, ${\omega }_{chroma}$ represents the chroma weighting parameter, while ${\lambda }_{mode}$ denotes the Lagrange multiplier. Specifically, the objective of RDO is to determine the optimal coding mode by evaluating the total cost of each possible partitioning. Although this method excels in compression efficiency, its computational complexity remains high, particularly in depth map encoding, where the recursive partitioning process significantly increases the computational burden.

To quantitatively analyze the depth distribution of CUs in depth maps, we used the reference software HTM16.3 [33] in All-Intra mode [34], encoded the JCT-3V standard test sequences [35] (including Balloons, Newspaper, Shark, and Poznan_Hall2) with multiple QP combinations (25–34, 30–39, 35–42, 40–45), and statistically analyzed the depth map CU partitioning results.

Table 1. Distribution of CU depth levels in depth maps.

Sequence	QP	Depth 0 (%)	Depth 1 (%)	Depth 2 (%)	Depth 3 (%)
Balloons	34	33.1	30.9	23.8	12.2
	39	44.3	36.7	15.1	3.9
	42	57.4	34.6	6.7	1.3
	45	73.6	23.9	2.1	0.4
Newspaper	34	12.5	29.1	32.8	25.6
	39	25.1	40.4	23.8	10.7
	42	41.7	39.8	14.8	3.7
	45	61.4	31.4	6.4	0.8
Shark	34	30.4	35.4	19.7	14.5
	39	53.0	30.2	11.3	5.5
	42	68.2	23.8	6.3	1.7
	45	80.6	16.4	2.8	0.2
Poznan_Hall2	34	71.4	20.9	5.8	1.9
	39	83.4	13.3	2.8	0.5
	42	91.0	7.6	1.2	0.2
	45	95.8	3.8	0.4	0.0
Average		57.7	26.1	11.0	5.2

clearly shows the distribution of depth map CUs across different depths after encoding. Observation reveals that CUs at depths 0 and 1 dominate in most cases, with their proportion significantly increasing at high QP or in static scenes. Conversely, divisions at depths 2 and 3 concentrate in high-motion regions or complex edge scenes at low QP, exhibiting overall lower proportions. This result indicates that traditional exhaustive coding modes involve substantial redundant computations. By identifying the depth level of the current CU and skipping unnecessary CU size calculations, approximately two-thirds of the pattern search and RDO computations can be effectively avoided. This significantly reduces computational complexity, providing a practical optimization path for real-time 3D video coding systems.

3. The Proposed Algorithm

3.1. Construction of CU Complexity Features Based on Wavelet Energy Ratio and Quantization Parameter

Although convolutional neural networks can effectively extract features, their limited perceptual domain makes it challenging to comprehensively support complex CU partitioning predictions. To further improve the partitioning prediction accuracy of the HFF-CNN model, we introduce two key external features: the wavelet energy ratio that reflects the texture complexity of depth map CUs, and the quantization parameter that reflects coding quality. These two features respectively provide supplementary information from their respective dimensions; fused with the depth features extracted by the model, they jointly enhance the understanding of CU characteristics to achieve more accurate partitioning prediction. In the proposed model, these two features are added as external features and participate in training together.

In traditional image processing, the wavelet energy ratio is often used as an analytical tool for distinguishing between simple and detailed regions. In contrast, targeting the depth map CU partitioning task, this study designs a dedicated wavelet energy ratio calculation scheme and innovatively converts it into a quantitative feature that drives intelligent decision-making. To accurately capture the texture information of CU blocks from the macro to the micro level, we adopt the 2D Discrete Wavelet Transform (2D-DWT) to perform two-level decomposition on CU blocks, and select the Daubechies-2 wavelet basis from the Daubechies family (whose low-pass and high-pass filters have predefined fixed structures with a length of 4) to construct a filter bank, so as to balance computational complexity and detail extraction capability. Subsequently, for the sub-bands obtained after decomposition, we calculate the ratio of the total energy of high-frequency sub-bands to the total energy of all sub-bands, and finally obtain the wavelet energy ratio feature for quantifying the texture complexity of depth map CUs. Figure 4 illustrates the detailed two-level decomposition process, where $A\in \left(L{L}_1,L{H}_1,H{L}_1,H{H}_1,L{L}_2,L{H}_2,H{L}_2,H{H}_2\right)$, and $A\left(i,j\right)$ denotes the coefficient of each sub-band at position $\left(i,j\right)$, which is generated by convolving with the selected filter and combining with downsampling and used for subsequent energy calculation.

First, conduct the first layer decomposition: For an input CU block of size 64 × 64, first apply the low-pass filter $h_{low}$ and the high-pass filter $h_{high}$ to each row of pixels in the horizontal direction. When the filters slide along each row, multiply them with the corresponding pixels and sum up the results to generate low-frequency (smooth) and high-frequency (detail) information in the row direction. Then, through sub-sampling at every other point, the column dimension is halved from 64 to 32, yielding two intermediate matrices of size 64 × 32, $H_{low}$ (horizontal low-frequency) and $H_{high}$ (horizontal high-frequency). Immediately afterwards, repeat the same filtering and sub-sampling operations on each column of these two intermediate matrices in the vertical direction. The row dimension is halved to 32, and finally four sub-bands of size 32 × 32 are generated: the low-frequency sub-band $L{L}_1$ (vertical low-pass + horizontal low-pass), $L{H}_1$ which captures horizontal details (vertical low-pass + horizontal high-pass), $H{L}_1$ which captures vertical details (vertical high-pass + horizontal low-pass), and $H{H}_1$ which captures diagonal details (vertical high-pass + horizontal high-pass).

Subsequently, perform the second-level decomposition on the low-frequency sub-band $L{L}_1$ (32 × 32) obtained in the first level. The process is the same as that of the first level: first, perform horizontal filtering and downsampling to get to 32 × 16, then perform vertical filtering and downsampling to reach 16 × 16, resulting in the low-frequency sub-band $L{L}_2$ and three high-frequency sub-bands $L{H}_2$, $H{L}_2$, and $H{H}_2$ of the second level. The energy of each sub-band is calculated by the sum of squares of all coefficient values $A\left(i,j\right)$ within the sub-band, with the following formula:

$$E\left(A\right)=\sum _{i=1}^{W_A}\sum _{j=1}^{H_A}{\left|A\left(i,j\right)\right|}^2$$

(2)

Here, $W_A$ and $H_A$ represent the width and height of sub-band A, respectively. The total energy of the high-frequency sub-bands ($E_{high}$) and the overall energy across all sub-bands ($E_{total}$) are calculated as follows:

$$E_{high}={\sum }_{k\in \left\{L{H}_1,H{L}_1,H{H}_1,L{H}_2,H{L}_2,H{H}_2\right\}}{E}_k$$

(3)

$$E_{total}=E\left({\mathit{LL}}_1\right)+E\left({\mathit{LL}}_2\right)+E_{high}$$

(4)

To quantify the texture complexity of CU, the ratio of the total energy of the extracted high-frequency sub-bands to the total energy of all sub-bands is calculated to obtain the wavelet energy ratio $R_{wavelet}$, defined as follows:

$$R_{wavelet}={\displaystyle \frac{E_{high}}{E_{total}}}$$

(5)

After this feature is introduced into the proposed HFF-CNN model, it can effectively enhance the model’s segmentation sensitivity in complex texture regions and reduce the risk of over-segmentation in flat regions.

Another external feature is the quantization parameter, which serves as a crucial metric for assessing encoding quality, directly influencing the compression rate and partitioning strategy of CU blocks. Normalization not only mitigates biases caused by different QP scales but also enhances the model’s sensitivity to the impact of encoding quality. In this study, the QP value is normalized to the range [0, 1] using the following formula:

$${\mathit{QP}}_{norm}={\displaystyle \frac{QP-{\mathit{QP}}_{min}}{{\mathit{QP}}_{max}-{\mathit{QP}}_{min}}}$$

(6)

Here, $QP\in \left[0,51\right]\cap \mathbb{Z}$, $Q{P}_{min}$, and $Q{P}_{max}$ represent the minimum and maximum values within the dataset, respectively. Through normalization, the values are standardized to a uniform scale, enabling the model to more reliably learn the impact of the quantization parameter on partitioning decisions. This process provides comprehensive data support for partitioning prediction.

3.2. CU Partitioning Prediction Based on HFF-CNN

To efficiently predict the partitioning of CUs in depth maps, this model incorporates a framework that integrates multi-scale feature extraction, squeeze and excitation (SE) module enhancement, and external feature fusion, as illustrated in Figure 5. The model takes a 64 × 64 CU block as input, processes it through multiple channels and feature fusion operations, and ultimately outputs the partitioning probabilities for both the CU and its sub-blocks.

First, the frames to be encoded of the depth video are cropped into 64 × 64 CTUs and input into the network after preprocessing. Among the three input channels, one remains at the original 64 × 64 size, while the other two are reduced to 32 × 32 and 16 × 16 respectively through average pooling. This multi-scale design aims to better adapt to the hierarchical structure of CU partitioning, enabling the model to extract features of coding units hierarchically from global to local levels.

Each scale of feature maps undergoes three convolutional layers, gradually extracting high-level features and compressing the feature map size to focus on important regions. The first layer uses 4 × 4 convolutional kernels with a stride of 4, outputting 16 feature maps, and the size of the feature maps is reduced to one-fourth of the input size; the next two layers use 2 × 2 convolutional kernels with a stride of 2, outputting 24 and 32 feature maps respectively. Each channel focuses on features of different levels: the 64 × 64 feature map extracts global low-level information, the 32 × 32 feature map focuses on medium-scale local information, and the 16 × 16 feature map concentrates on high-level abstract structures. A ReLU activation function is applied after each convolutional layer to enhance nonlinear expression capability, and its mathematical expression is as follows:

$$R_m\left(C{U}_n\right)=\left\{\begin{array}{c}C{U}_n,m=0\hfill \\ max\left(0,W_m{R}_{m-1}\left(C{U}_n\right)+B_m\right),1\le m\le M\hfill \end{array}\right.$$

(7)

Here, $C{U}_n$ represents the n-th input CU block, while $W_m$ and $B_m$ denote the convolutional weights and bias term of the m-th layer, respectively.

The joint feature maps generated by the convolutional layers are fed into the SE module for feature enhancement, where Squeeze and Excitation operations are performed sequentially. During the Squeeze operation, global average pooling is applied to the output feature map from the convolutional layer (with dimensions $H\times W\times C$, where H and W represent the height and width of the feature map, respectively, and C denotes the number of channels). Specifically, global average pooling computes the spatial average over H and W, resulting in a 1 × 1 × C vector. This operation effectively compresses spatial information and extracts global features for each channel, providing essential global context for the subsequent Excitation operation. The calculation formula is as follows:

$$Z_c={\displaystyle \frac{1}{H\times W}}\sum _{i=1}^H\sum _{j=1}^W{x}_c\left(i,j\right)$$

(8)

Here, $Z_c$ represents the globally aggregated feature vector, and $x_c\left(i,j\right)$ (i = 1, 2, …, H; j = 1, 2, …, W; c = 1, 2, …, C) denotes the pixel values of the input feature map. After obtaining the results from the Squeeze operation, the Excitation operation is performed. This process involves two fully connected (FC) layers. The first FC layer compresses the 1 × 1 × C vector to a smaller dimension (with a reduction ratio of 4), and the compressed vector is passed through a ReLU activation function to introduce nonlinearity, thereby enhancing the network’s representational capability. Subsequently, the second FC layer restores the compressed vector to its original dimension of 1 × 1 × C. Finally, a Sigmoid activation function is applied to constrain the output values within the range of from 0 to 1, generating channel-wise attention weights $s_c$:

$$s_c=\sigma \left(W_2\delta \left(W_1{z}_c\right)\right)$$

(9)

Here, $\delta \left(x\right)=max\left(0,x\right)$ represents the ReLU activation function, while $\sigma \left(x\right)={\displaystyle \frac{1}{1+e^{-x}}}$ denotes the Sigmoid function. Next, the channel-wise attention weights are multiplied element-wise with the original feature map to achieve feature reweighting:

$$x_c^{\prime }\left(i,j\right)=s_c·x_c\left(i,j\right)$$

(10)

Here, $x_c^{\prime }\left(i,j\right)$ represents the weighted pixel value. This process adaptively adjusts the weights of different channels to enhance important features while suppressing redundant ones, thereby optimizing feature representation. Through this weighting mechanism, the model can more precisely focus on critical features, ultimately improving overall performance.

The features weighted by the SE module are fed into the fully connected (FC) layers. Each FC layer progressively extracts features at varying hierarchical levels. FC1-1 and FC1-2, with fewer neurons, extract the most critical information to generate Output 1. This output corresponds to the probability value of whether the 64 × 64 CU block input to the model will be further partitioned. FC2-1 and FC2-2, containing more neurons, extract deeper features to produce Output 2. Output 2 corresponds to the respective partition probabilities of the four 32 × 32 sub-CUs generated if the parent 64 × 64 CU is partitioned. FC3-1 and FC3-2, possessing the highest neuron count, process the finest-grained features for Output 3. The values of Output 3 respectively denote the partitioning probabilities of the resulting 16 × 16 sub-CUs when the corresponding 32 × 32 sub-CU is partitioned. At the feature extraction stage of each hierarchy (i.e., after the operations of FC1-1, FC2-1, and FC3-1), the wavelet energy ratio and the normalized QP value are combined with the output of the first fully connected layer of the hierarchy to additionally provide the wavelet energy ratio reflecting the texture complexity and the normalized QP information reflecting the coding quality requirement for the partitioning decision of the corresponding hierarchy. In the subsequent fully connected operations, the fused feature vector is further abstracted and refined through the weighted calculation of neurons, so that the model can make more reasonable partitioning decisions. Finally, each output layer outputs the corresponding partitioning probability value through the sigmoid activation function. The partitioning decision process is recursive. By comparing with the threshold of 0.5, if the output probability value is greater than 0.5, the corresponding CU continues to be partitioned to the next depth size; if it does not exceed 0.5, the partitioning of the corresponding CU stops. According to the outputs of the three fully connected layers, the complete partitioning decision for the input 64 × 64 CU can be realized.

Since the binary cross-entropy (BCE) loss function is commonly used in binary classification tasks and the proposed model in this study is essentially a binary classification problem, the binary cross-entropy loss function is employed during model training. This function evaluates each predicted partition probability against its corresponding ground truth label. It is defined as follows:

$$L=-{\displaystyle \frac{1}{N}}\sum _{n=1}^N\left(l_n\times ln{Y}_n+\left(1-l_n\right)\times ln\left(1-Y_n\right)\right)$$

(11)

Here, $l_n$ represents the ground truth partition label for the n-th sample, and $Y_n$ denotes the predicted probability generated by the model. The loss function effectively optimizes the classification performance of the model.

3.3. Overall Algorithm Flow

In practical application scenarios, this study applies the proposed HFF-CNN to the HTM encoder to replace the original depth map CU partitioning method, with the overall encoding process shown in Figure 6. During the depth map CU partitioning stage, the system invokes the trained HFF-CNN model to predict the partitioning mode of the CU. Leveraging the collaborative effects of multi-scale feature extraction, channel attention mechanism, and external auxiliary features (wavelet energy ratio and normalized quantization parameter), the HFF-CNN model accurately analyzes the characteristics of the current CU and outputs the partitioning probabilities for corresponding hierarchical levels. After obtaining the prediction results, a pre-set threshold is used to determine whether to continue partitioning the CU. Once CU partitioning is completed, subsequent encoding operations proceed strictly according to the established rules of the HTM encoder until the final bitstream is generated, completing the entire encoding process and achieving efficient encoding of 3D-HEVC depth maps.

4. Experimental Results and Analyses

4.1. Dataset Construction and Experimental Setup

We selected six sequences from the JCT-3V standard test sequences [35] (1024 × 768 resolution: Outdoor, Lovebird, BookArrival; 1920 × 1088 resolution: Dancer, Cafe, CarPark), whose content covers various types including static backgrounds, indoor and outdoor scenes, fine details, and complex motions. Based on the reference software HTM16.3 and its built-in configuration file encoder_intra_main.cfg, we encoded the depth maps under different QP values (34, 39, 42, 45) to extract the corresponding 64 × 64 CU partitioning results, which will be used as ground truth labels to construct the dataset. To avoid overlap between training and testing data, training and testing frames were completely separated by at least 50 frames.

The experiments were conducted on a computer equipped with an AMD Ryzen 9 8900X3D CPU (manufactured by Advanced Micro Devices, Inc. (AMD), Santa Clara, CA, USA), 32 GB RAM, and the Windows 11 64-bit operating system. Both the training of the proposed HFF-CNN model and the encoding tests for the proposed method and the reference software (HTM16.3) were performed in the same environment. The CPU only impacts absolute encoding time. Subsequent experiments yielded relative metrics unaffected by CPU model or performance, with consistent stability across variations, thereby enhancing the generalizability of conclusions. The CNN implementation relies on functions from the TensorFlow library and accelerates training by activating the GPU (used exclusively during the model training phase).

Table 2. Experimental environment.

	Hardware
CPU		AMD Ryzen 9 8900X3D
GPU		RTX 4070Ti Super
RAM		32GB
OS		windows 11 64bits
	Software
Reference software		HM 16.18
Configuration file		encoder_intra_main.cfg
QP(depth)		34, 39, 42, 45

summarizes the hardware and software environment used in the experiments. During the model training process of this study, we set a series of optimized hyperparameters to ensure the efficiency and stability of the depth map CU partitioning task. Specifically, the batch size was set to 64, and the initial learning rate was set to 0.005, which was decayed by a factor of 5 after every 50 epochs to guide the model to converge more smoothly. The training was conducted for a maximum of 120 epochs, and an early stopping mechanism was adopted. The training would be terminated if there was no improvement in the validation loss for 15 consecutive epochs. The Adam optimizer was selected to adapt to the characteristics of the multi-label classification task. The binary cross-entropy loss function was used, which was consistent with the design of the model’s output layer to ensure the rationality of the predicted probabilities. Regarding the training computational cost of the HFF-CNN model, in the experimental environment described, training with a dataset containing four QP values and corresponding depth map CTU partitions, the actual time taken for the model to complete the entire training process is approximately 12 h. This one-time training investment is entirely acceptable given the significant acceleration benefits the model brings during the encoding phase.

4.2. Analysis of Experimental Results

To evaluate the effectiveness of the proposed method, we conducted experiments on the 3D-HEVC reference software HTM16.3. The trained model was integrated into HTM16.3, and tests were performed using eight standard test sequences officially provided by JCT-3V [35], with resolutions covering 1024 × 768 and 1920 × 1088.

Table 3. JCT-3V standard test sequences and related information.

Sequence	Resolution	Frames	Frame Rate	3-Views Input	Scene Characteristics
Kendo	1024 × 768	300	30	1-3-5	Multiple overlapping objects
Balloons		300	30	1-3-5	High-dynamic motion
Newspaper		300	30	2-4-6	Static text, dynamic person
GT_Fly	1920 × 1088	250	25	9-5-1	CG city geometry
Shark		300	30	1-5-9	Complex biological contours
Poznan_Hall2		200	25	7-6-5	Large smooth areas
Poznan_Street		250	25	5-4-3	Complex street dynamics
Undo_Dancer		250	25	1-5-9	Rapid complex motion

provides further details on the test video sequences. Since this study primarily focuses on reducing intra-frame encoding complexity, experiments employed the All Intra [34] mode. The configuration file used was baseCfg_3view+depth_AllIntra.cfg (included in the HTM16.3 software package), with four sets of QP values applied for texture and depth encoding: (25–34, 30–39, 35–42, 40–45). The encoding time savings (TS) represents the reduction in intra encoding time compared to the reference software HTM16.3, serving as a metric for the extent of encoding complexity reduction. TS is calculated as follows:

$$TS={\displaystyle \frac{1}{4}}\sum _{i=1}^4{\displaystyle \frac{T_{HTM}^{Q{P}_i}-T_{pro}^{Q{P}_i}}{T_{HTM}^{Q{P}_i}}}$$

(12)

$T_{pro}$ represents the encoding time of the proposed algorithm, and $T_{HTM}$ refers to the encoding time of the reference model HTM16.3. Additionally, BDBR (Bjøntegaard Delta Bit Rate) [36] is used as a standard metric to evaluate the coding efficiency gains achieved by different methods while maintaining a consistent target quality.

Table 4. Comparison of encoding effects between the proposed method and the original HTM16.3.

Sequence	Resolution	BD-PSNR (db)	Proposed (Without ${\mathit{R}}_{\mathit{wavelet}}$)		Proposed
			BDBR (%)	TS (%)	BDBR (%)	TS (%)
Kendo	1024 × 768	−0.01	0.15	44.32	0.13	46.68
Balloons		−0.02	0.29	46.81	0.27	49.33
Newspaper		−0.03	0.38	44.12	0.36	46.97
Average (1024 × 768)		−0.02	0.27	45.08	0.25	47.66
GT_Fly	1920 × 1088	−0.02	0.42	46.52	0.39	48.23
Shark		−0.01	0.61	49.53	0.58	52.23
Poznan_Hall2		−0.02	0.46	43.71	0.44	45.93
Poznan_Street		−0.03	0.40	44.92	0.41	46.57
Undo_Dancer		−0.02	0.22	49.14	0.21	51.49
Average (1920 × 1088)		−0.02	0.42	46.76	0.41	48.89
Average (overall)		−0.02	0.37	46.13	0.35	48.43

fully presents the experimental results of the proposed method. Verified through ablation experiments, on eight groups of standard test sequences, the model without integrating the wavelet energy ratio feature designed by us achieved an average coding time saving of 46.13% compared with HTM16.3. The complete HFF-CNN framework fused with this feature achieved more excellent comprehensive performance: it not only realized an average coding time saving of 48.43%, but also incurred extremely low coding performance loss (with only a 0.35% increase in BDBR and a 0.02 dB decrease in BD-PSNR). The experimental results validate the effectiveness of the proposed method from two aspects: on the one hand, the 2.30% performance improvement directly proves the effective contribution of the designed wavelet energy ratio feature. Through linear convolution operations with low computational cost, this feature significantly enhances the model’s accuracy in judging the partitioning of low-texture regions, and avoids the exponential growth of RDO computation caused by excessive CU partitioning due to misjudgment. On the other hand, the final results show that the proposed HFF-CNN successfully achieves an optimized balance between coding efficiency and quality by virtue of hierarchical feature fusion, channel attention mechanism, and the integration of external features.

We conducted a comprehensive evaluation of the computational complexity and actual inference overhead of the HFF-CNN model: Using the ptflops tool, we measured that processing a single 64 × 64 CU requires only 1.15 MFLOPs (including external feature extraction and the entire network forwarding process). In actual encoding, the average encoding duration for the eight sequences is 3114.1 s, while the average inference time of the model is 10.9 s, accounting for only 0.35% of the total encoding time. The lightweight design of the model ensures that the inference overhead it introduces is significantly offset by the 48.43% encoding acceleration. With negligible impact on coding quality, this fully demonstrates the deployment feasibility and practical value of the proposed method in a pure CPU environment.

4.3. Comparison with Other Algorithms

Table 5. Comparison of encoding effects between the proposed method and other methods under HTM16.3.

Sequence	[24]		[14]		[15]		Proposed
	BDBR (%)	TS (%)	BDBR (%)	TS (%)	BDBR (%)	TS (%)	BDBR (%)	TS (%)
Kendo	0.6	39.5	0.17	35.2	0.43	44.6	0.13	46.68
Balloons	1.4	39.0	0.12	32.9	0.27	38.7	0.27	49.33
Newspaper	1.4	38.0	0.08	32.3	0.20	39.8	0.36	46.97
GT_Fly	0.3	38.3	0.08	35.0	0.15	55.7	0.39	48.23
Shark	0.5	39.4	0.26	44.0	0.14	42.6	0.58	52.23
Poznan_Hall2	0.6	41.4	0.39	51.6	0.40	52.7	0.44	45.93
Poznan_Street	0.4	39.9	0.26	41.6	0.62	52.6	0.41	46.57
Undo_Dancer	0.2	38.6	0.29	49.3	0.44	49.9	0.21	51.49
Average	0.68	39.26	0.21	40.2	0.32	45.8	0.35	48.43

presents the comprehensive comparison results of intra coding optimization methods for 3D-HEVC. Different research teams have developed unique acceleration strategies via differentiated designs for coding unit decision mechanisms and prediction mode selection. The technical evolution path of the Hamout team is particularly clear: the early proposed optimization strategy [14] achieved a 40.2% time acceleration with a 0.21% BDBR increment, laying the foundation for intra coding efficiency optimization; the subsequent improved method based on clustering algorithm and edge detection [15] further balanced performance, increasing the time acceleration rate to 45.8%. Although the BDBR increment slightly increased to 0.32%, the overall performance is better, reflecting the progress of the team’s technical iteration. In addition, the method proposed by Lee et al. [24] based on intra prediction cost and probability achieved a 39.26% time saving with a 0.68% BDBR increment, verifying the feasibility of prediction cost-driven optimization. The method proposed in this paper ranks the best in the TS metric with 48.43%. Although the BDBR increment (0.35%) is slightly higher than that of [14] (0.21%), the difference is only 0.14 percentage points, which is completely within the acceptable range of coding efficiency loss in practical applications; in terms of time efficiency, it has achieved significant improvement compared with existing comparison methods. This characteristic of significantly improving computational efficiency within acceptable efficiency loss makes it more suitable for scenarios with strict timeliness requirements such as real-time 3D video transmission.

A deeper analysis of the impact of different resolutions on coding time savings reveals that, under different resolutions, there are significant differences in the time savings performance of various methods, and the advantages of the proposed method in this paper are particularly prominent. As shown in Figure 7, among the first three sequences with 1024 × 768 resolution, the TS of the early method [14] by the Hamout team is only 33.47%, the subsequent proposed [15] increases it to 41.03%, and that of Lee et al. [24] is 38.83%, all lower than the 47.66% of the proposed method in this paper. Among the latter five high-resolution sequences with 1920 × 1088 resolution, the TS of [14] increases to 44.30%, and [15] further reaches 50.70%, but the cross-resolution performance fluctuation of [15] is as high as 9.67 percentage points, with poor stability. In contrast, the proposed method in this paper achieves a TS of 48.89% at high resolution, with a cross-resolution fluctuation of only 1.23 percentage points. It not only maintains efficient acceleration but also achieves excellent stability, fully verifying the algorithm’s advantages in adaptive processing of video features at different spatial scales, and providing more reliable technical support for the flexible application of 3D-HEVC in multi-resolution scenarios.

4.4. Comparison of Synthesized Views

To comprehensively evaluate our method, we selected two representative sequences from video sequences with resolutions of 1024 × 768 and 1920 × 1088, respectively. In Balloons, the camera moves uniformly in a parallel direction, with significant dynamics of foreground characters, and the depth map has both complex boundaries and local flat regions; in Newspaper, the camera remains fixed, with slight movement of foreground objects and completely static background, and the depth map shows fine boundary details and some gentle regions; in Shark, the camera follows the movement of objects, with diverse movement transformations of foreground objects, and the depth map has a small amount of regular boundary information and more flat regions. GT_Fly contains a large number of buildings; as the camera moves slowly forward, the background buildings change from far to near, and its depth map contains a large amount of complex boundary information and concentrated flat regions. According to the definition of virtual viewpoints in 3D-HEVC, the position between adjacent viewpoints is parameterized as a continuous value in [0.0,1.0]. In this experiment, the geometric center position n = 0.5 (the 50% position between View0 and View1) is selected as the core observation point. Meanwhile, considering that the QP combination of 35–42 in mainstream streaming media application scenarios is exactly the critical point sensitive to subjective distortion, this QP group is used for testing. As shown in

Table 6. Comparison of synthesized views.

Sequence	Resolution	Original Reference Frame	Synthesized Views (HTM16.3)	Synthesized Views (Proposed)
Balloons	1024 × 768
Newspaper	1024 × 768
Shark	1920 × 1088
GT_Fly	1920 × 1088

, through juxtaposed comparison of the original reference synthesized frames, HTM16.3 benchmark synthesized frames, and the synthesized frames of the proposed method, it is found that there is no visual perception degradation in sensitive regions such as the contours of moving characters in Balloons and the text edges in Newspaper.

To quantitatively verify the above conclusions,

Table 7. Perceptual quality assessment of synthesized video sequences.

Sequence	SSIM		VMAF
	HTM16.3	Proposed	HTM16.3	Proposed
Balloons	0.948	0.945	92.7	91.9
Newspaper	0.962	0.960	94.3	93.8
Shark	0.935	0.931	91.5	90.6
GT_Fly	0.956	0.953	93.8	93.1
Average	0.950	0.947	93.1	92.4

presents the objective perceptual quality evaluation results of the aforementioned sequences under the same conditions. The average SSIM and VMAF values of the synthesized videos generated by the proposed method are 0.947 and 92.4, respectively, which are extremely close to the synthesis results of HTM16.3. These objective measurement results are consistent with the minimal coding loss indicated by the only 0.35% BDBR increase, indicating that the proposed method effectively maintains the visual fidelity of synthesized views while achieving coding acceleration.

5. Conclusions

This paper addresses the high computational complexity of 3D-HEVC depth map intra-coding by proposing a fast CU partitioning framework based on a Hierarchical Feature Fusion Convolutional Neural Network (HFF-CNN). The core contribution lies in the collaborative design of multi-scale feature extraction and a channel attention mechanism (SE module), effectively capturing both global structural features and local details of coding units. By fusing the designed wavelet energy ratio and normalized quantization parameters, the model significantly enhances the robustness of CU partitioning prediction. Experimental results demonstrate that the proposed method reduces encoding time by an average of 48.43% across multiple resolution scenarios while maintaining a negligible BDBR increase of 0.35%, achieving an efficient balance between computational efficiency and coding quality. Additionally, the model’s cross-resolution adaptability validates its generalization capabilities under diverse video contents and encoding parameters, offering a versatile technical foundation for practical deployment of 3D video coding systems.

Future work will focus on enhancing the efficiency and scene adaptability of 3D-HEVC depth map coding: On one hand, we will explore joint optimization strategies with intra-frame prediction mode selection, reducing redundant computations during encoding through feature sharing and decision coordination. On the other hand, we will deepen lightweight model design by integrating dynamic network compression techniques to lower computational resource demands without compromising encoding quality. Furthermore, we will expand experimental validation by adapting the proposed method to Random Access (RA) and Low Delay (LD) encoding configurations. By fine-tuning external features to accommodate diverse scenario requirements, we will validate the method’s adaptability across multiple applications, providing a more comprehensive technical pathway for widespread real-time deployment of 3D-HEVC.