A Graph-Enhanced Self-Supervised Framework for 3D Tooth Segmentation Using Contrastive Masked Autoencoders: An In Silico Study

Abstract

With the advancement of 3D digital dentistry, accurate 3D tooth segmentation has become increasingly important in orthodontics and computer-aided diagnosis. However, existing supervised approaches heavily rely on exhaustive face-wise annotations and often exhibit limited generalization across complex clinical meshes. Although self-supervised learning offers a promising alternative to alleviate annotation costs, current paradigms remain challenged by sensitivity to data augmentations, suboptimal representation learning in pure masking schemes, and the complex structural characteristics of dental geometry. To address these limitations, we propose Dental-CMAE, a graph-enhanced hierarchical Contrastive masked AutoEncoder framework tailored for 3D tooth segmentation. The framework incorporates a dual-branch masking strategy that leverages graph-based structural priors to generate distinct corrupted views while preserving intrinsic mesh topology, thereby facilitating robust reconstruction. This is integrated with a feature-level contrastive objective designed to enforce semantic consistency between co-masked regions, which enhances representation discriminability without the requirement for negative sample queues. Additionally, the architecture utilizes a hierarchical multi-scale attention mechanism that partitions feature channels into parallel streams, enabling the simultaneous capture of fine-grained morphological variations and the overarching global dental arch context. Extensive experiments demonstrate that our Dental-CMAE consistently outperforms state-of-the-art fully supervised and self-supervised methods across multiple evaluation metrics. Specifically, our framework achieves an Overall Accuracy (OA) of 95.57%, a mean Intersection-over-Union (mIoU) of 88.14%, and a mean Accuracy (mAcc) of 90.85%. Supported by these quantitative findings, our method validates its effectiveness for robust 3D tooth segmentation, highlighting its strong potential to alleviate annotation bottlenecks and improve the reliability of automated 3D digital dental workflows.

1. Introduction

The widespread adoption of digital technologies is fundamentally reshaping modern dentistry workflows [1,2]. In particular, the routine use of intraoral scanners enables the direct acquisition of high-precision 3D surface mesh models [3]. Compared to conventional dental impressions, intraoral scanning provides a non-invasive means to accurately preserve the complex geometry and spatial structure of teeth and gingiva. These high-fidelity digital models form the foundation of computer-aided design and manufacturing (CAD/CAM) systems in dentistry [4], supporting advanced clinical applications such as orthodontic simulation, clear aligner design, implant planning, and customized restoration [5], as well as the fabrication of 3D-printable dental restorations [6]. A critical prerequisite for enabling these digital workflows is precise 3D tooth segmentation, which involves accurately delineating individual teeth and assigning semantic labels from continuous and irregular tooth–gingiva mesh surfaces. The quality of this segmentation directly impacts the reliability of downstream analysis and treatment planning.

Traditional approaches based on hand-crafted features or geometric heuristics [7,8,9] often struggle to model the complex topological structures inherent in real-world intraoral scans. To address the growing demand for automated processing, deep learning methods have been extensively explored, achieving significant progress through improved feature extraction and task-specific architectural designs [10]. However, most existing approaches are fully supervised and rely heavily on limited annotated datasets, which restricts their scalability and generalization to unseen clinical cases. Although increasing data diversity can partially alleviate this issue, the labor-intensive and costly process of dense face-wise annotation on high-resolution meshes makes large-scale labeling impractical. Given the abundance of unlabeled intraoral scan data in clinical practice, self-supervised learning (SSL) has emerged as a promising paradigm for improving model generalization. SSL has demonstrated remarkable success across domains such as natural language processing [11,12], computer vision [13,14,15], and 3D representation learning [16,17,18]. In particular, the Masked Autoencoder (MAE) paradigm has shown strong capability in learning robust representations by reconstructing masked inputs from partial observations [19,20], providing an effective solution to mitigate annotation scarcity in 3D dental analysis. Recently, pioneering studies have begun tailoring SSL paradigms to intraoral scans. Contrastive approaches [21,22] attempt to learn representations by establishing point or region-level correspondences across augmented dental meshes, while the generative DentalMAE framework [23] adapts the standard MAE paradigm to reconstruct masked dental patches.

Despite these initial explorations, existing dental SSL frameworks still exhibit certain limitations. First, pure MAE primarily emphasizes local reconstruction, which may limit its ability to capture high-level semantic representations. A common solution is to incorporate contrastive learning (CL); however, naïvely combining CL with MAE can lead to performance degradation rather than improvement [24]. This is largely due to the reliance of conventional CL methods on strong data augmentations (e.g., random rotations), which significantly perturb geometric structures and introduce conflicts with the reconstruction objective of MAE. In addition, standard transformer architectures rely heavily on global self-attention, which lacks explicit modeling of fine-grained local geometric variations, making it difficult to simultaneously capture detailed tooth morphology and the overall dental arch structure.

To address these challenges, our work is motivated by two concrete premises. First, replacing destructive spatial augmentations with a dual-branch graph masking strategy can generate distinct corrupted views for robust contrastive learning while preserving the intrinsic mesh topology. Second, standard global attention often dilutes fine-grained details; thus, explicitly decoupling local boundary extraction from global spatial structure modeling can effectively alleviate semantic ambiguity in crowded dental regions. Guided by these premises, we propose Dental-CMAE, a Hierarchical Local–Global Attention-Guided Contrastive Masked Autoencoder framework tailored for 3D tooth segmentation.

In summary, our Dental-CMAE is a novel self-supervised framework that advances 3D pre-training for dental applications. The main contributions are:

• We propose the first dual-branch masked autoencoding framework named Dental-CMAE for 3D tooth segmentation that effectively integrates contrastive learning to enhance representation quality.
• We introduce a graph-enhanced masking and encoding scheme with hierarchical multi-scale attention to jointly model local morphology and global structure.
• Extensive experiments on public datasets validate that our Dental-CMAE consistently outperforms state-of-the-art methods across multiple metrics, including Overall Accuracy (OA), mean Accuracy (mAcc), and mean Intersection-over-Union (mIoU).

2. Related Work

2.1. Self-Supervised Learning on Point Clouds

Self-supervised learning (SSL) for point clouds aims to learn generalizable 3D representations by designing pretext tasks without relying on manual annotations. While executing these pretext tasks entails intensive computational and memory overheads during pre-training, this trade-off is increasingly acceptable given modern hardware advancements. Existing approaches can be broadly categorized into two major paradigms: contrastive learning and generative masked modeling.

In earlier studies, contrastive learning (CL) dominated 3D representation learning. For example, PointContrast [16] learns invariant features by maximizing point-level consistency between a point cloud and its augmented views. To alleviate the dependence on extensive data augmentation and negative sample mining, Info3D [25] maximizes mutual information between the input and its geometric transformations to improve global representation learning. Cross-modal contrastive learning has also gained increasing attention. CrossPoint [26] aligns 3D point clouds with 2D images to exploit mature 2D visual representations, while PointCLIP [27] leverages multi-view projections to bridge 3D geometry and pre-trained vision–language models. For mesh data, SSL-MeshCNN [28] introduces a contrastive pre-training framework tailored for 3D mesh segmentation, where positive and negative pairs are constructed via mesh-specific augmentations such as anisotropic scaling, vertex perturbation, and edge flipping within a SimCLR [29] paradigm. Despite these advances, CL-based methods still face notable challenges, including reliance on carefully designed proxy tasks and memory banks, as well as the risk of disrupting geometric and topological consistency. Developing robust contrastive strategies that preserve structural integrity remains an open problem.

Inspired by the success of BERT [12] and MAE [14] in natural language processing and computer vision, generative masked modeling has recently emerged as a dominant paradigm for 3D representation learning. Point-BERT [20] introduces discrete tokenization to enable masked point modeling, while Point-MAE [19] and its hierarchical extension Point-M2AE [30] adopt Transformer-based architectures to reconstruct masked point patches. MaskPoint [31] reformulates the pretext task as a binary classification problem that distinguishes noise from object points. More recently, hybrid methods such as Point-CMAE [24] and ReCon [32] attempt to combine generative and contrastive objectives for enhanced representation learning. In parallel, graph-based masked modeling has shown strong potential. GraphMAE [33] introduces feature masking on graphs by preserving structural topology in the encoder while masking node features, and employs a re-masking strategy in the decoder to capture higher-order semantic dependencies. Extending this idea to 3D geometry, MGM-AE [34] constructs face-level graphs for mesh data and learns expressive representations by reconstructing masked geometric regions through graph attention, effectively preserving the intrinsic structural properties of the mesh.

2.2. 3D Dental Segmentation

Traditional tooth segmentation methods primarily rely on geometric heuristics, including curvature-based analysis [7,35,36], harmonic field methods [8], and contour-driven strategies [9]. While these approaches establish important foundations, they often require substantial manual intervention and exhibit limited robustness and scalability in real clinical scenarios.

With the advent of deep learning, research has shifted toward data-driven approaches. Early methods project 3D dental meshes onto 2D parameterized domains to enable the use of convolutional neural networks (CNNs) for automated segmentation [37,38,39]. Subsequently, point-based frameworks [40] improve segmentation performance through strategies such as non-uniform resampling and adversarial training to enhance semantic consistency.

More recently, mesh-based methods have gained increasing attention due to their ability to naturally preserve geometric and topological structures. MeshSegNet [41] incorporates multi-scale graph constraints along with graph-cut refinement, while TSGCNet [42] employs a dual-stream graph convolutional network to extract complementary features from spatial coordinates and surface normals. Building upon these advances, Transformer-based architectures such as TSegFormer [43] and SGTNet [44] further model long-range dependencies while maintaining fine-grained local details, enabling more precise boundary delineation.

Motivated by the success of self-supervised learning (SSL) in 3D vision, recent works have begun to explore SSL for intraoral scan analysis. He et al. [21] introduce unsupervised pre-training for 3D tooth segmentation by establishing point-level correspondences across augmented mesh views. STSNet [22] extends this paradigm with region-level contrastive learning to capture multi-scale representations spanning local geometry and regional morphology. More recently, DentalMAE [23] adapts the Masked Autoencoder (MAE) framework to dental meshes by partitioning remeshed inputs into face tokens and reconstructing latent representations, achieving improved performance under limited annotation settings.

Despite these advances, existing SSL-based approaches still face fundamental limitations. Contrastive learning methods typically depend on strong geometric augmentations, which can disrupt the structural consistency of dental meshes and introduce semantic ambiguity. In contrast, purely generative approaches such as DentalMAE mainly focus on local reconstruction within a single view, lacking sufficient semantic supervision. This over-reliance on reconstruction objectives often results in suboptimal representation learning, limiting their ability to capture discriminative and globally consistent geometric features.

3. Method

3.1. Overview

The overall architecture of our Dental-CMAE is illustrated in Figure 1, presenting a unified pipeline from raw mesh processing to multi-task representation learning. The process begins with a dual-branch graph masking strategy, where input 3D dental meshes are first remeshed and partitioned into patches to ensure topological regularity. A graph structure is then constructed via an adjacency matrix to explicitly encode dental topology, upon which two independent random masking operations are applied to node features. This design produces two distinct corrupted views while preserving graph connectivity, thereby facilitating subsequent cross-view consistency learning.

The resulting masked sequences are fed into a graph-enhanced encoder–decoder architecture, which serves as the core module for geometric representation learning. The graph-constrained encoder, implemented with shared weights, extracts fine-grained morphological features from partially observed inputs by restricting attention to local graph neighborhoods. To recover missing geometry, the encoded features are processed by a hierarchical multi-scale decoder that decomposes feature channels into parallel streams, enabling simultaneous modeling of local details and global structural dependencies. The decoded representations are further projected through reconstruction heads to predict 3D coordinates.

Finally, the model is optimized using a joint objective that combines reconstruction and consistency losses. This multi-task formulation encourages the learning of robust and discriminative representations by enforcing both geometric fidelity and cross-view feature alignment. As a result, our Dental-CMAE provides a strong initialization for downstream 3D tooth segmentation, achieving high precision and improved generalization in challenging scenarios such as dental crowding.

3.2. Graph-Enhanced Dual-Branch Masking Strategy

3.2.1. Data Preprocessing and Graph Construction

To address the irregular face distribution and inconsistent patch sizes in raw dental meshes, we first perform remeshing using the MAPS [45] algorithm. This process involves mesh simplification, patch expansion, and vertex mapping, resulting in a uniform manifold with N faces. Following Point-BERT [20], the remeshed surface is then partitioned into n patches using Farthest Point Sampling (FPS) and K-Nearest Neighbors (KNN) grouping.

To facilitate structure-aware feature learning, we further construct a graph $G=(V,E)$ based on an n-ring adjacency matrix A. Here, V denotes the set of patch tokens, including masked tokens that serve as spatial placeholders, while E encodes the local topological relationships among neighboring patches. This graph formulation provides the structural basis for graph-constrained attention in the encoder as well as the hierarchical multi-scale attention mechanism in the decoder.

For each face j within the patch $p_i$, we extract its center coordinates $c_j\in {\mathbb{R}}^3$, normal vector $n_j\in {\mathbb{R}}^3$, circumradius $r_j\in \mathbb{R}$, and normalized direction vectors from the center to its three vertices $v_{j,1},v_{j,2},v_{j,3}\in {\mathbb{R}}^3$. These geometric attributes are concatenated to form the initial geometric feature $f_j$:

$$f_j=c_j\Vert n_j\Vert r_j\Vert v_{j,1}\Vert v_{j,2}\Vert v_{j,3}$$

(1)

where ‖ denotes the vector concatenation operation. Following the processing paradigm of MeshMAE [46], the faces within each patch $p_i$ are first arranged according to a predefined topological order. Their corresponding geometric features are then concatenated to form a high-dimensional, patch-level representation $F_i=[f_{i,1}\Vert f_{i,2}\Vert \cdots \Vert f_{i,K}]$, where K denotes the number of faces in the patch. This holistic feature is subsequently projected into a latent embedding space via a Multi-Layer Perceptron (MLP) to obtain the initial patch representation. Finally, the patch centroid $C_i$ is incorporated as Positional Encoding (PE), yielding the initial encoder input $h_i^0$:

$$h_i^0=\mathrm{MLP}\left(F_i\right)+\mathrm{PE}\left(C_i\right)$$

(2)

3.2.2. Dual-Branch Masking Strategy

In conventional contrastive learning frameworks [15,29], distinct views are typically generated via aggressive spatial augmentations. However, for 3D geometric data, such transformations inevitably distort coordinate structures, increasing the difficulty of masked reconstruction and hindering the learning of consistent geometric representations. To address this issue, we provide a dual-branch masking strategy that replaces explicit data augmentation with feature-level corruption. Specifically, two independent masking operations are applied to the same input, producing distinct corrupted views without altering the underlying geometry. This design not only enlarges the diversity of latent representations but also naturally introduces co-masked regions, which serve as the basis for contrastive consistency learning.

Formally, given the initialized patch token set T and a masking ratio r, we perform independent random sampling to generate two mask index sets $m_1$ and $m_2$, with $m_1\ne m_2$. Following GraphMAE [33], the selected patches are not removed; instead, their features are replaced to preserve the mesh topology during masking. For each branch $k\in \{1,2\}$, the features of patches indexed by $m_k$ are substituted with a shared learnable mask token $T_m$, while the remaining patches retain their original features, forming the visible token set $T_{v,k}$. This process yields two masked input sequences that share the same graph adjacency matrix but differ in their visible geometric context. These sequences are then fed into the graph-enhanced encoder for subsequent representation learning.

3.3. Graph-Enhanced Encoder–Decoder Structure

3.3.1. Graph-Constrained Encoder

Unlike conventional Masked Autoencoders (MAEs), which discard masked patches during encoding, we retain them as spatial placeholders within the input sequence, following GraphMAE [33]. To extract robust local geometric priors from incomplete inputs, the encoder $f_{\theta }$ adopts a graph-constrained attention mechanism guided by the adjacency matrix A. Specifically, during attention computation, each patch token $h_i$ attends only to its neighboring nodes in the graph G, thereby restricting information flow to local regions.

This locality constraint encourages the encoder to focus on modeling fine-grained geometric continuity and morphological relationships among adjacent patches, while suppressing interference from irrelevant long-range dependencies at early stages of pre-training. By incorporating such a structural inductive bias, the encoder learns more discriminative and topology-aware representations, providing a solid foundation for accurate reconstruction and downstream segmentation.

3.3.2. Hierarchical Multi-Scale Decoder

The decoder $g_{\varphi }$ aims to reconstruct missing spatial coordinates from the latent representations produced by the encoder. To achieve accurate reconstruction, the model must simultaneously capture fine-grained morphological details (e.g., interdental gaps) and the global structure of the dental arch. To this end, we introduce a hierarchical multi-scale attention mechanism.

As illustrated in Figure 2, the decoder input feature sequence $C\in {\mathbb{R}}^{N\times D}$ is partitioned along the channel dimension into two complementary subspaces, $C_1$ and $C_2$, each with dimensionality $D/2$:

$$C=[C_1\Vert C_2],C_1,C_2\in {\mathbb{R}}^{N\times {\displaystyle \frac{D}{2}}},$$

where ‖ denotes the concatenation operator. This channel-wise decomposition enables parallel attention branches to model geometric features at different scales.

Specifically, the local branch operates on $C_1$ to capture fine-grained surface variations. Similar to the encoder, it employs graph-constrained attention to produce the local representation $C_{\mathrm{local}}$. In parallel, the global branch processes $C_2$ using standard self-attention to model long-range dependencies across the dental arch, yielding the global representation $C_{\mathrm{global}}$.

To effectively integrate information from the local and global branches, we adopt an adaptive feature fusion strategy inspired by [47]. This design dynamically balances the contributions of fine-grained local details and global structural context, conditioned on their respective feature representations.

Formally, the local features $C_{\mathrm{local}}\in {\mathbb{R}}^{N\times D/2}$ and global features $C_{\mathrm{global}}\in {\mathbb{R}}^{N\times D/2}$ are first projected into a unified semantic space via independent linear transformations:

$$C_l^{\prime }={\mathrm{MLP}}_l\left(C_{local}\right),C_g^{\prime }={\mathrm{MLP}}_g\left(C_{global}\right).$$

(3)

The transformed features are then concatenated, and a weighting factor $\omega$ is dynamically generated via a sigmoid function to adaptively balance the contributions of the two branches at each patch location:

$$\omega =\sigma \left(\left[\mathrm{Concat}(C_l^{\prime },C_g^{\prime })\right]\right),$$

(4)

where $\sigma (·)$ denotes the Sigmoid function. The final fused representation $C_{fuse}$ is obtained through a weighted sum of the local and global branches:

$$C_{fuse}=\omega ·C_l^{\prime }+(1-\omega )·C_g^{\prime }.$$

(5)

To fully exploit the structural information from the dual-branch masking, we use two independent decoders instead of a shared one. Since each branch masks different parts of the mesh, separate decoders prevent the reconstruction tasks from interfering with each other. This design also forces the shared encoder to extract more essential geometric features that can satisfy both reconstruction tasks simultaneously, which makes the learned representations more robust for segmentation.

3.4. Joint Optimization Loss

To learn representations that capture both fine-grained geometry and global semantic consistency, we define a joint objective that combines generative reconstruction with discriminative contrastive learning. The training process is jointly optimized by two complementary loss functions, promoting accurate geometric recovery and robust cross-view feature alignment.

3.4.1. Reconstruction Loss

The generative objective of our Dental-CMAE is to reconstruct the 3D geometry of masked regions. The hierarchical multi-scale decoder first produces latent representations for masked tokens, which are subsequently passed to reconstruction heads composed of lightweight MLPs that project high-dimensional features back to the 3D coordinate space.

Following the point-level reconstruction paradigm [19], each reconstruction head predicts the relative 3D coordinates $(x,y,z)$ of vertices within a patch $p_i$, defined with respect to its centroid $C_i$. For each masked branch $k\in \{1,2\}$, the model outputs a set of predicted coordinates $P_{r,k}$. The discrepancy between the predictions and the ground-truth relative coordinates $G_r$ is measured using the ${\mathcal{l}}_2$ Chamfer Distance (CD). We adopt CD rather than strict point-wise regression metrics (e.g., Mean Squared Error) because the reconstructed vertices within a patch essentially constitute an unordered point set, making explicit one-to-one point correspondences computationally prohibitive. CD elegantly circumvents this by evaluating the bidirectional spatial discrepancy: the first component penalizes isolated predicted outliers, while the second ensures the ground-truth geometric structure is adequately covered. Formally, for a given predicted coordinate set $P_r$ and the ground truth $G_r$, the loss is computed as:

$${\mathcal{L}}_{CD}(P_r,G_r)={\displaystyle \frac{1}{|P_r|}}\sum _{p\in P_r}\underset{g\in G_r}{min}{\parallel p-g\parallel }_2+{\displaystyle \frac{1}{|G_r|}}\sum _{g\in G_r}\underset{p\in P_r}{min}{\parallel g-p\parallel }_2$$

(6)

The total reconstruction loss ${\mathcal{L}}_{re}$ is computed as the sum of the Chamfer Distances across both branches to ensure that the weight-shared encoder captures universal geometric priors:

$${\mathcal{L}}_{re}={\mathcal{L}}_{CD}(P_{r,1},G_r)+{\mathcal{L}}_{CD}(P_{r,2},G_r)$$

(7)

3.4.2. Contrastive Consistency Loss

Beyond geometric reconstruction, enforcing cross-view semantic consistency is crucial for learning robust representations. To this end, we focus on patches that are simultaneously masked in both branches, defined as the shared masked region $M=m_1\cap m_2$. For each patch in M, the corresponding latent features $h_1$ and $h_2$ are extracted from the outputs of the two decoders.

As the token order is disrupted during masking, the features are first restored to their original spatial arrangement, yielding $h_1^{\prime }$ and $h_2^{\prime }$. We then compute their similarity using cosine similarity, denoted as $\mathcal{M}(·,·)$, to quantify cross-view feature alignment within the shared region. This consistency constraint encourages the model to learn discriminative and stable representations that remain invariant across different masking patterns. The resulting consistency loss ${\mathcal{L}}_{\mathrm{cons}}$ is defined as:

$${\mathcal{L}}_{cons}={\displaystyle \frac{1}{\left|M\right|}}\sum _{i\in M}(1-\mathcal{M}(h_{1,i}^{\prime },h_{2,i}^{\prime }))$$

(8)

where $\left|M\right|$ denotes the number of co-masked patches in the shared region M.

The final pre-training objective ${\mathcal{L}}_{total}$ is formulated as a weighted combination of the generative reconstruction loss and the discriminative contrastive consistency loss:

$${\mathcal{L}}_{total}={\mathcal{L}}_{re}+\lambda {\mathcal{L}}_{con}$$

(9)

where $\lambda$ is a balancing hyper-parameter that controls the relative contribution of the contrastive signal.

4. Experiments

4.1. Settings

4.1.1. Datasets and Pre-Processing

For self-supervised pre-training, we utilize two datasets: Teeth3DSeg [48], which contains 1800 3D intraoral scans from 900 patients (with separate upper and lower jaws), and a large-scale unlabeled dataset introduced by Liu et al. [49], comprising 6000 intraoral scans.

To address the non-uniform mesh density commonly observed in raw scans, which complicates graph construction and masking, all meshes are standardized via a unified remeshing procedure. Specifically, each mesh model is normalized to $N=16,000$ faces to ensure topological regularity. The mesh is then partitioned into 512 patches, with each patch containing 32 faces.

During pre-training, we apply lightweight data augmentations, including random rotations along the vertical axis within $[-{15}^{°},{15}^{°}]$ and isotropic scaling in the range of $[0.95,1.05]$.

For downstream fine-tuning, we adopt the 3D-IOSSeg dataset [50], which consists of 220 high-quality annotated dental meshes. The pre-processing pipeline is kept consistent with the pre-training stage to reduce domain discrepancy.

4.1.2. Implementation Details

Our Dental-CMAE is implemented in PyTorch and trained on a single NVIDIA RTX 3090 GPU. The software environment includes Python 3.8, PyTorch 1.12.0, and CUDA 11.6.

Regarding the network architecture, the graph-enhanced shared encoder consists of 10 transformer blocks, while the independent hierarchical decoders comprise 6 blocks each. The feature embedding dimension is set to $D=768$, utilizing a multi-head attention mechanism with 6 parallel heads. Based on empirical optimization, the masking ratio is set to $r=60\%$ during the final pre-training phase, and the contrastive loss balancing weight is configured as $\lambda =0.1$.

During self-supervised pre-training, the model is trained for a maximum of 300 epochs using the AdamW optimizer with a weight decay of 0.05. The initial learning rate is set to $5\times {10}^{-4}$ and follows a cosine annealing schedule. To ensure optimal convergence and avoid unnecessary computational overhead, we employ an early stopping criterion, halting the training process if the validation reconstruction loss does not improve for 30 consecutive epochs.

For downstream 3D tooth segmentation, we adopt the same optimizer and training configuration to ensure consistent and stable fine-tuning.

4.1.3. Metrics

To comprehensively assess the performance of our method on the 3D tooth segmentation task, we adopt three standard metrics widely used in 3D shape analysis: Overall Accuracy (OA), mean Intersection-over-Union (mIoU), and mean Accuracy (mAcc). These metrics are computed at the face level, providing a fine-grained evaluation of the model’s ability to correctly segment different dental regions.

OA measures the proportion of correctly classified mesh faces over the entire model. mAcc computes the average per-class accuracy, reflecting balanced performance across dental categories. mIoU evaluates the average overlap between predicted segments and ground-truth annotations across all classes, serving as a robust indicator of segmentation quality. Together, these metrics provide a comprehensive evaluation of both segmentation accuracy and structural consistency. The formal definitions are given as follows:

$$OA={\displaystyle \frac{{\sum }_{i=0}^L{m}_{ii}}{{\sum }_{i=0}^L{\sum }_{j=0}^L{m}_{ij}}}$$

(10)

$$mAcc={\displaystyle \frac{1}{L+1}}\sum _{i=0}^L{\displaystyle \frac{m_{ii}}{{\sum }_{j=0}^L{m}_{ij}}}$$

(11)

$$mIoU={\displaystyle \frac{1}{L+1}}\sum _{i=0}^L{\displaystyle \frac{S(g_i\cap p_i)}{S(g_i\cup p_i)}}$$

(12)

In the above formulations, $m_{ij}$ denotes the number of mesh faces whose ground-truth label is i but are predicted as j, while $m_{ii}$ corresponds to the number of correctly classified faces for class i. L indicates the total number of segmentation classes. Furthermore, $g_i$ and $p_i$ are the ground-truth and predicted regions for the i-th class, respectively, and $S(g_i\cap p_i)$ and $S(g_i\cup p_i)$ represent the intersection and union areas (measured by the number of faces) between them.

In the above formulations, $m_{ij}$ denotes the number of mesh faces whose ground-truth label is i but are predicted as j, while $m_{ii}$ corresponds to the number of correctly classified faces for class i. Here, L indicates the total number of specific tooth segmentation classes; by additionally designating index 0 as the background category (i.e., gingiva), the overall evaluation accounts for $L+1$ semantic classes. Furthermore, $g_i$ and $p_i$ are the ground-truth and predicted regions for the i-th class, respectively, and $S(g_i\cap p_i)$ and $S(g_i\cup p_i)$ represent the intersection and union areas (measured by the number of faces) between them.

4.2. Quantitative Comparison

We compare our Dental-CMAE with several state-of-the-art 3D segmentation methods, including five fully supervised approaches (PointNet++ [51], DGCNN [52], MeshSegNet [41], TSGCNet [53], and SGTNet [44]) and three self-supervised methods (STSNet [22], Point-BERT [20], and Point-MAE [19]). For fair comparison, all self-supervised methods are pre-trained on Teeth3DSeg and fine-tuned on 3D-IOSSeg under the same experimental settings as our Dental-CMAE.

As reported in

Table 1. Performance comparison of different methods for 3D tooth segmentation. Bold values indicate the best performance.

Category	Method	OA (%)	mIoU (%)	mAcc (%)
Full Supervision	PointNet++	85.11	75.34	78.24
	DGCNN	91.68	83.52	85.51
	MeshSegNet	92.23	81.86	83.86
	TSGCNet	93.89	86.00	88.57
	SGTNet	92.75	84.97	86.35
Self-Supervision	Point-BERT	91.54	83.93	86.17
	Point-MAE	93.10	86.15	89.69
	STSNet	94.58	86.36	88.72
	Dental-CMAE (Ours)	95.57	88.14	90.85

, our Dental-CMAE consistently outperforms all competing methods across all evaluation metrics, including both supervised and self-supervised baselines, demonstrating its effectiveness for 3D tooth segmentation.

Within the fully supervised setting, our Dental-CMAE achieves substantial performance gains over PointNet++, improving OA and mIoU by 10.46% and 12.80%, respectively. It also surpasses DGCNN and MeshSegNet with mIoU improvements of 4.62% and 6.28%. Notably, even when compared with strong supervised baselines such as TSGCNet and SGTNet, our method maintains a clear advantage, yielding mIoU gains of 2.14% and 3.17%.

In the self-supervised setting, our Dental-CMAE consistently demonstrates superior performance. Compared with Point-BERT, it improves OA and mIoU by 4.03% and 4.21%, respectively. When compared to Point-MAE, which also adopts a masked modeling paradigm, our method achieves mIoU, OA, and mAcc gains of 1.99%, 2.47%, and 1.16%. Furthermore, our Dental-CMAE outperforms STSNet by 0.99% in OA and 1.78% in mIoU.

4.3. Visual Comparison

Figure 3 presents a visual comparison of segmentation results, further supporting the quantitative findings. As highlighted by the red dashed circles, baseline methods struggle in geometrically complex regions, particularly under severe crowding, overlap, and ambiguous boundaries. Fully supervised methods such as PointNet++ and DGCNN often produce fragmented predictions and misclassify adjacent teeth, leading to noticeable boundary leakage. This vulnerability arises because general point cloud networks primarily rely on raw 3D spatial coordinates as their sole inputs. By inherently ignoring the topological connectivity of the dental mesh and lacking explicit guidance from higher-order geometric features , these methods suffer from severe semantic confusion across ambiguous boundaries. Even specialized architectures like MeshSegNet and TSGCNet exhibit boundary blurring in dense anterior regions. Similarly, self-supervised baselines, including Point-BERT, Point-MAE, and STSNet, tend to miss fine-grained topological details, resulting in inaccurate delineation between neighboring tooth structures. While domain-specific supervised models incorporate richer local features, they still fundamentally lack the robust, generalized semantic priors established through large-scale pre-training, restricting their adaptability to highly varied clinical cases. Conversely, standard self-supervised frameworks often fail to respect the intrinsic manifold structure of dental meshes during their data perturbation processes, leading to suboptimal topological awareness.

In contrast, our Dental-CMAE produces clean and well-defined segmentation boundaries that closely match the ground truth. These results validate that our graph-constrained architecture effectively prevents semantic confusion across boundaries by respecting mesh topology, while the pre-trained semantic priors enhance robustness in complex scenarios. By effectively modeling both fine-grained geometric variations and global dental arch structure, our method significantly reduces semantic ambiguity. As shown in the visual results, our Dental-CMAE demonstrates strong robustness in separating individual teeth, even in highly crowded scenarios, outperforming both advanced supervised models such as SGTNet and other self-supervised approaches.

4.4. Ablation Study

4.4.1. Impact of Dual-Branch Masking and Decoder Design

To assess the effectiveness of our dual-branch masking strategy and the role of decoder design, we compare a single-branch baseline with two dual-branch variants. The single-branch model follows a standard masked modeling pipeline and serves as the reference. For the dual-branch setting, we consider two configurations: (i) dual branches with independent decoders, and (ii) dual branches with a shared decoder. This comparison aims to isolate the impact of multi-view masking and decoder parameterization on representation learning. The results are reported in

Table 2. Ablation study of the dual-branch masking strategy. Bold values indicate the best performance.

Model Structure	OA (%)	mIoU (%)	mAcc (%)
Single-branch (Baseline)	93.41	86.15	87.69
Dual-branch w/ Shared Decoder	94.32	87.26	90.04
Dual-branch w/ Separate Decoder	95.57	88.14	90.85

.

As shown in the table, both dual-branch variants consistently outperform the single-branch baseline across all metrics, demonstrating that dual-view masking provides more informative supervisory signals and facilitates the learning of richer representations. Moreover, the configuration with independent decoders achieves the best performance, indicating that decoupled decoding branches encourage the encoder to learn more discriminative and generalized features to accommodate diverse masking patterns.

4.4.2. Impact of Hierarchical Multi-Scale Attention

We evaluate the impact of feature interaction range on the decoder’s ability to reconstruct fine-grained geometric details. We compare two architectural variants: a baseline model with standard multi-head self-attention (MHSA), which models global cross-patch dependencies across the entire feature space, and the proposed hierarchical multi-scale attention mechanism.

Specifically, our design adopts a dimension-splitting strategy, where half of the feature channels ($D/2$) are allocated to global attention for capturing long-range spatial relationships, while the remaining channels are restricted to local interactions guided by the face adjacency graph.

As reported in

Table 3. Ablation study on the impact of the hierarchical multi-scale attention mechanism. MSA: Standard Multi-Head Self-Attention; HMSA: Hierarchical Multi-Scale Attention. Bold values indicate the best performance.

Model Structure	OA (%)	mIoU (%)	mAcc (%)
MSA	94.84	86.92	90.15
HMSA	95.57	88.14	90.85

, incorporating the hierarchical multi-scale attention consistently improves performance, yielding gains of 0.73%, 1.22%, and 0.70% in OA, mIoU, and mAcc, respectively. We attribute these improvements to the alleviation of attention dilution commonly observed in standard MHSA. Although global attention effectively models long-range dependencies, it tends to overlook subtle local geometric structures—such as interdental boundaries—due to the dispersion of attention weights across the entire mesh.

In contrast, the proposed mechanism enforces localized attention within $D/2$ channels via graph-based constraints, enabling the model to capture fine-grained morphological details, while the remaining channels preserve global structural context. This hierarchical design facilitates a balanced integration of local and global information, allowing the decoder to reconstruct more accurate and structurally consistent tooth boundaries.

4.4.3. Impact of Masking Ratio

To investigate the effect of masking ratio on 3D tooth segmentation and identify an appropriate level of pre-training difficulty, we evaluate a range of masking ratios from 30% to 70%. The corresponding performance variations are illustrated in Figure 4 and

Table 4. Performance comparison across different masking ratios in the pre-training phase. All metrics are evaluated after fine-tuning on the 3D-IOSSeg dataset. Bold values indicate the best performance.

Masking Ratio r	OA (%)	mIoU (%)	mAcc (%)
30%	93.65	85.92	88.84
40%	94.28	86.74	89.65
50%	94.88	87.82	90.06
60%	95.57	88.14	90.85
70%	95.02	87.55	90.31

.

As shown in the results, model performance follows a clear parabolic trend with respect to the masking ratio, peaking at 60%. At relatively low masking ratios (e.g., 30–40%), a large proportion of visible faces makes the reconstruction task overly simple, limiting the model’s ability to learn rich geometric and semantic representations and thus leading to inferior fine-tuning performance.

When the masking ratio increases to 60%, the reconstruction task becomes sufficiently challenging, encouraging the model to better exploit the hierarchical multi-scale attention mechanism to capture structural dependencies between sparse visible patches and masked regions. This results in more robust and discriminative feature representations.

However, further increasing the masking ratio to 70% leads to a noticeable performance drop. In this case, the limited number of visible patches restricts the local branch of the hierarchical multi-scale attention from accessing adequate neighborhood information. As a result, the model lacks sufficient geometric context for accurate reconstruction, ultimately degrading feature quality and segmentation accuracy.

5. Discussion

Although self-supervised learning offers a promising alternative for 3D representation learning, directly applying masked autoencoders or conventional contrastive frameworks to intraoral meshes often yields limited improvements. Traditional contrastive methods rely heavily on aggressive geometric augmentations that can disrupt the fragile topological structures of teeth.Basic masking methods typically focus on local reconstruction, which fails to provide enough guidance from the global dental structure. To address these challenges, our Dental-CMAE introduces a dual-branch graph masking strategy that leverages structural priors and is integrated into a contrastive learning paradigm. Specifically, by performing masking operations directly on the graph nodes—corrupting node features while leaving the spatial coordinates and graph connectivity intact—the framework is designed to preserve intrinsic geometric properties without altering the original mesh structures. This approach enables the model to capture discriminative representations without the need for negative sample queues or extensive manual annotations, establishing a reliable basis for downstream segmentation tasks.

Beyond achieving state-of-the-art quantitative metrics, the comparative results of Dental-CMAE provide valuable insights into the broader limitations of current 3D dental segmentation paradigms. As observed in the visual comparisons, standard networks like PointNet++ and DGCNN struggle with severe semantic confusion, highlighting the fundamental inadequacy of relying solely on raw spatial coordinates without explicit topological constraints. More importantly, our results expose the inherent methodological bottlenecks of domain-specific architectures. While specialized fully supervised models like MeshSegNet, TSGCNet, and SGTNet enhance local feature extraction, they remain fundamentally constrained by limited annotated datasets, restricting their generalizability across highly varied clinical cases. Our framework effectively overcomes this limitation by leveraging self-supervised pre-training to establish robust, generalized semantic priors. Furthermore, compared to other self-supervised baselines—including point-based masked models (Point-BERT, Point-MAE) and contrastive approaches (STSNet)—our method addresses a critical flaw: conventional data perturbation and pure masking schemes often fail to respect the intrinsic manifold structure of dental meshes. By employing a topology-preserving dual-branch masking strategy, Dental-CMAE successfully avoids topological distortion, ensuring highly discriminative and structurally aware feature learning.

Furthermore, the results demonstrate that a masking ratio of 60% yields the optimal pre-training performance for the task of 3D tooth segmentation. Interestingly, this optimal ratio is notably lower than the extremely high masking ratios (e.g., 70–80%) typically required for general 3D point cloud MAEs. This discrepancy can be logically attributed to the complex and densely packed nature of dental mesh patches. A 60% ratio achieves a crucial balance between the difficulty of the pretext reconstruction task and the preservation of sufficient local geometric context. Ultimately, this configuration provides a robust self-supervised representation, enabling the model to achieve high segmentation accuracy while reducing the need for exhaustive manual face-wise labels.

Despite these promising results, certain limitations of the current study must be acknowledged. First, the dual-branch masking strategy inherently introduces notable computational and memory overheads during the pre-training phase compared to standard single-branch generative models. Second, the model’s generalization to extremely complex clinical scenarios—such as severe geometric incompleteness or extreme dental crowding—requires further validation, as these cases remain underrepresented in the current datasets. Future work will focus on optimizing the dual-branch pre-training architecture to reduce computational costs, and incorporating more diverse clinical datasets to improve the model’s reliability when processing defective or highly complex intraoral scans.

6. Conclusions

In this work, we proposed Dental-CMAE, a graph-enhanced hierarchical contrastive masked autoencoder framework tailored for 3D tooth segmentation. By integrating a topology-preserving dual-branch masking strategy with a hierarchical multi-scale attention mechanism, Dental-CMAE effectively addresses the limitations of existing pre-trained models in capturing both fine-grained geometric variations and global dental arch topology. Extensive experiments validate that our approach significantly outperforms state-of-the-art fully supervised and self-supervised methods, demonstrating remarkable robustness in complex clinical scenarios such as severe dental crowding. Despite these promising results, certain limitations of the current study must be acknowledged. First, the dual-branch masking strategy inherently introduces notable computational and memory overheads during the pre-training phase compared to standard single-branch generative models. Second, the model’s generalization to extremely complex clinical scenarios—such as severe geometric incompleteness or extreme dental crowding—requires further validation, as these cases remain underrepresented in the current datasets. Future work will focus on optimizing the dual-branch pre-training architecture to reduce computational costs, and incorporating more diverse clinical datasets to improve the model’s reliability when processing defective or highly complex intraoral scans.