Anatomical Plausibility in Deformable Image Registration Using Bayesian Optimization for Brain MRI Analysis

Abstract

Deformable image registration plays a crucial role in medical imaging by aligning anatomical structures across multiple datasets, which is essential for accurate diagnosis and treatment planning. However, existing deep learning-based deformable registration models often face challenges in ensuring anatomical plausibility, leading to unnatural deformations in critical brain structures. This paper proposes a novel framework that uses Bayesian optimization to address these challenges, focusing on registering 3D point clouds that represent brain structures. Our method uses probabilistic modeling to optimize non-rigid transformations, providing smooth and interpretable deformations that align with anatomical constraints. The proposed framework is validated using MRI data from patients diagnosed with hypoxic-ischemic encephalopathy (HIE) due to perinatal asphyxia. These datasets include brain scans taken at multiple time points, enabling the modeling of structural changes over time. By incorporating Bayesian optimization, we enhance the accuracy of the registration process while maintaining anatomical fidelity. Our results demonstrate that the approach provides interpretable, anatomically plausible deformations, outperforming conventional methods in terms of accuracy and reliability. This work offers an improved tool for brain MRI analysis, aiding healthcare professionals in better understanding disease progression and guiding therapeutic interventions.

1. Introduction

Image registration is a fundamental technique in medical imaging that plays a critical role in clinical applications such as image-guided radiation therapy, image-guided surgery, and minimally invasive treatments [1,2]. The primary goal of image registration is to determine an optimal transformation that aligns images from different datasets, which is essential when critical information is distributed across multiple images, such as those captured at different time points or from distinct modalities [3,4]. Accurate registration is crucial for integrating relevant information, enhancing clinical decision-making, and ultimately improving patient outcomes [5,6].

Typically, image registration involves defining one image as the “fixed” image while another, the “moving” image, undergoes transformation with the fixed image as a reference [6,7]. Over the years, registration techniques have evolved significantly, from classical intensity-based methods to advanced machine learning-driven approaches [4,8]. Recently, deep learning-based methods have shown particular promise, achieving efficient and accurate registration through complex transformation models, often using convolutional neural networks (CNNs) to capture intricate patterns in an unsupervised manner [9].

Within this context, deformable registration has emerged as a key approach to aligning anatomical structures that undergo substantial flexibility and variation. Unlike rigid transformations, deformable registration allows for nonlinear adjustments that adapt to local tissue variations, making it especially valuable for imaging complex organs in dynamic scenarios, such as the brain or lungs [8,10]. Techniques such as B-spline free-form deformations (FFDs) [8] and diffeomorphic methods, including large displacement diffeomorphic metric mapping (LDDMM), are extensively used for such tasks, ensuring the preservation of anatomical topology during alignment [11]. However, deformable models must manage flexibility carefully to prevent unnatural deformations in sensitive anatomical regions; this balance is typically achieved through regularization terms that guide the transformations.

A significant recent advancement is the use of 3D point clouds—data representations that capture anatomical structures’ geometric shapes and spatial configurations instead of conventional pixel-based methods in deformable registration. Point clouds, composed of discrete points in 3D space, enable a highly detailed and precise representation of structures, making them particularly suitable for applications requiring anatomical fidelity, such as brain MRI analysis [12]. This shift allows deformable registration approaches to capture complex structural details, yielding a more accurate alignment of 3D anatomical structures and facilitating in-depth analyses of brain regions.

Despite these advancements, several challenges in deformable registration persist. High computational costs, the limited interpretability of deformation fields, and a propensity for anatomically implausible transformations in critical brain regions remain problematic [13,14]. Additionally, the effectiveness of deep learning-based methods is often limited by the availability of large training datasets, which can be scarce in medical imaging contexts [15]. Moreover, local pixel-based loss functions, commonly employed in these models, may overlook the global context of medical images, leading to the potential misalignment or distortion of critical structures.

To address these challenges, we propose a novel probabilistic framework utilizing Conformal Bayesian Optimization (CBO) for the deformable registration of 3D point clouds representing brain structures. CBO builds upon Bayesian optimization (BO), introducing conformal prediction techniques that enable calibrated uncertainty estimates throughout the optimization process [16,17]. This probabilistic approach enhances the registration process by delivering reliable uncertainty quantifications and interpretable deformation models, making it well suited for high-precision tasks in MRI analysis.

Furthermore, CBO’s capacity to incorporate prior anatomical knowledge into its optimization framework allows for enhanced alignment in complex cases where traditional optimization techniques may produce unrealistic deformations or converge on suboptimal local minima. The conformal component in CBO adjusts the confidence intervals around model predictions, facilitating a balance between speed and accuracy, especially in high-dimensional, noisy datasets. This method thus ensures that transformations respect anatomical constraints, yielding clinically relevant outcomes for brain MRI analyses.

In this paper, we present a deformable point cloud registration framework that employs CBO to align the 3D point clouds of brain structures with anatomical plausibility. Our primary contribution is a robust and interpretable model for point cloud alignment that leverages CBO to achieve anatomically accurate deformations in sensitive regions. This framework is evaluated on MRI data from patients diagnosed with hypoxic-ischemic encephalopathy (HIE) due to perinatal asphyxia, underscoring its effectiveness in aligning brain structures across time points. By offering an improved tool for MRI analysis, this method aims to aid healthcare professionals in monitoring disease progression and refining therapeutic interventions.

2. Materials and Methods

2.1. Datasets

This study uses two datasets to evaluate our proposed method for anatomically plausible nonrigid shape-matching via Bayesian optimization.

The first dataset is the TOSCA dataset [18,19], which contains high-resolution three-dimensional nonrigid shapes. The dataset consists of 80 objects with varying poses and shapes, including eleven cats, nine dogs, three wolves, eight horses, six centaurs, four gorillas, twelve female figures, and two male figures, each in 7 and 20 poses.

The second dataset, referred to as the Brain Asphyxia Dataset, comprises Magnetic Resonance Imaging (MRI) scans from patients with perinatal asphyxia acquired during early childhood at a medical center in Colombia. We processed the MRI images using the infant FreeSurfer framework [20] to obtain 3D point cloud representations of the neuroanatomical structures present. Specifically, we segmented 20 different neuroanatomical regions relevant to perinatal asphyxia [21,22]. Each subject’s anatomy is thus represented by a collection of m ($m=20$) point clouds $S=P_1,P_2,\dots ,P_m$, where each point cloud represents a specific brain structure. Each point cloud $P_i$ consists of $n_i$ points, with each point ${\mathbf{p}}_j\in {\mathbb{R}}^3$ defined by its coordinates ${\mathbf{p}}_j=(x_j,y_j,z_j)$ [23].

We tested our approach on 20 different point clouds from the Tosca dataset. We applied random rigid transformations to evaluate the model’s ability to recover rigid transformations from point clouds containing significant variability. For the Brain Asphyxia Dataset, we tested our model using MRI scans acquired at different times from the same patient (e.g., at birth and one year later) to evaluate clinical outcomes. Each patient has 20 different neuroanatomical structures at different ages, including the left and right white matter, caudate nucleus, putamen, and thalamus.

2.2. Deformable Registration of Point Clouds

Let us consider two point clouds: the fixed point cloud $\mathcal{P}={\mathbf{p}}_1,{\mathbf{p}}_2,\cdots ,{\mathbf{p}}_n$ and the moving point cloud $\mathcal{Q}={\mathbf{q}}_1,{\mathbf{q}}_2,\cdots ,{\mathbf{q}}_m$, where ${\mathbf{p}}_i,{\mathbf{q}}_j\in {\mathbb{R}}^3$. The goal is to find a transformation $\mathcal{T}:{\mathbb{R}}^3\to {\mathbb{R}}^3$ that aligns $\mathcal{Q}$ to $\mathcal{P}$ by minimizing the alignment error.

In the case of nonrigid registration, a global transformation is insufficient to recover point correspondences accurately due to the deformable nature of the shapes. Therefore, we approximate the transformation function locally by grouping points into c clusters, where neighboring points are likely to undergo similar transformations. The deformation of the moving point cloud is modeled as follows:

$$\mathbf{q}i=\sum _{j=1}^c{w}_{ij}{\mathcal{T}}_j\left(\begin{array}{c}{\mathbf{p}}_i\mathbf{1}\end{array}\right),\forall {\mathbf{p}}_i\in \mathcal{P},$$

(1)

where ${\mathcal{T}}_j\in {\mathbb{R}}^{3\times 4}$ represents the affine transformation matrix for cluster j, and $w_{ij}$ is a weight indicating the influence of cluster j on point ${\mathbf{p}}_i$. The homogeneous coordinate (factor 1) is used to facilitate affine transformations.

The choice of the number of clusters c is crucial; if $c=n$, where n is the number of points, the problem becomes mathematically and computationally intractable. Conversely, a very low c may not capture the complexity of the deformation. We typically select a c such that $c<n$ is sufficient to model the deformation accurately.

To determine the clusters, we use an initial set of correspondences, possibly obtained through a matching strategy or feature descriptors, to group each point ${\mathbf{p}}_i$ with the key points ${\mathbf{n}}_1,\cdots ,{\mathbf{n}}_k$. We define the surface representation of $\mathcal{P}$ as $\mathcal{D}$ and of $\mathcal{Q}$ as $S$. We then partition the surfaces into the patches ${\mathcal{D}}_1,\cdots ,{\mathcal{D}}_k\subset \mathcal{D}$ and $S_1,\cdots ,S_k\subset S$ based on geodesic distances from the key points.

Each patch ${\mathcal{D}}_i$ and $S_i$ is further divided into r neighboring rings, ${\mathcal{D}}_i^1,\cdots ,{\mathcal{D}}_i^r$ and $S_i^1,\cdots ,S{i}^r$, respectively, based on geodesic distances. The entire surface can thus be represented as $\mathcal{D}=\bigcup {i=1}^k{\mathcal{D}}_i$, and each patch as $\mathcal{D}i=\bigcup {j=1}^r{\mathcal{D}}_i^j$. The same applies to $S$.

We estimate the deformation hierarchically using different geometric levels: local, intermediate, and global. At the local level, we estimate the transformation ${\mathcal{T}}_{ij}$ for each neighboring ring ${\mathcal{D}}_i^j$. At the intermediate level, we estimate the transformation ${\mathcal{T}}_i$ for each patch ${\mathcal{D}}_i$ based on the deformed neighboring rings. Finally, at the global level, we estimate the overall transformation $\mathcal{T}$ based on the deformed patches.

The estimation of the transformation parameters is performed iteratively, updating one level at a time while keeping the others fixed, as summarized in Algorithm  1.

Algorithm 1 Estimation of transformation parameters

Require: Initial correspondences between $\mathcal{P}$ and $\mathcal{Q}$ Require: Geodesic surface partitions ${\mathcal{D}}_i,S_i$ 1: while Convergence criterion not met do 2:     for each patch i do 3:         for each neighboring ring j do 4:            Compute local transformation ${\mathcal{T}}_{ij}$ by minimizing local energy ${\mathcal{E}}_l$ 5:            Update positions of vertices in ${\mathcal{D}}_i^j$ 6:         end for 7:         Compute intermediate transformation ${\mathcal{T}}_i$ by minimizing intermediate energy ${\mathcal{E}}_i$ 8:         Update positions of vertices in ${\mathcal{D}}_i$ 9:      end for 10:     Compute global transformation $\mathcal{T}$ by minimizing global energy ${\mathcal{E}}_g$ 11:     Update positions of all vertices in $\mathcal{D}$ 12: end while

Energy Model

The total energy function ${\mathcal{E}}_{\mathrm{total}}$ guiding the deformation is composed of three terms:

$${\mathcal{E}}_{\mathrm{total}}={\mathcal{E}}_l+{\mathcal{E}}_i+{\mathcal{E}}_g,$$

(2)

where ${\mathcal{E}}_l$, ${\mathcal{E}}_i$, and ${\mathcal{E}}_g$ are the local, intermediate, and global energy terms, respectively. The core idea is to combine local and global geometric information with an intermediate representation to ensure accurate and smooth deformations.

Local Energy ${\mathcal{E}}_l$

Local energy operates at the finest level, focusing on small neighboring regions (rings) around each key point. For the i-th patch, we define the j-th neighboring ring ${\mathcal{D}}_i^j$ as the set of vertices at the geodesic distance j from the key point ${\eta }_i$. The goal is to find the local transformation ${\mathcal{T}}_{ij}$ for each ring that aligns ${\mathcal{D}}_i^j$ with the corresponding ring $S_i^j$ in the moving point cloud.

We formulate this as a weighted least squares problem:

$$\mathcal{E}l=\sum {i=1}^k\sum _{j=1}^r{\xi }_{ij}\sum _{\mathbf{p}\in \mathcal{D}{i}^j}{\left|\mathcal{T}ij\mathbf{p}-{\mathbf{q}}_{\mathrm{corr}\left(\mathbf{p}\right)}\right|}^2,$$

(3)

where ${\xi }_{ij}$ is an influencing factor based on the surface area or other criteria, and ${\mathbf{q}}_{\mathrm{corr}\left(\mathbf{p}\right)}$ is the corresponding point in $S_i^j$. The local transformations are assumed to be locally rigid and independent.

Intermediate Energy ${\mathcal{E}}_i$

We aim to integrate the deformed neighboring rings at the intermediate level to obtain a coherent transformation for each patch ${\mathcal{D}}_i$. We consider the deformed vertices from the local transformations and model the problem as fitting a Gaussian Mixture Model (GMM), where the deformed vertices $\tilde{\mathcal{V}}{\mathcal{D}}_i$ are the centroids. The corresponding vertices in $S_i$ are observations.

We minimize the negative log-likelihood

$$\mathcal{E}i=-\sum {i=1}^{k}lnp\left({\mathcal{V}}_{Si}\mid \tilde{\mathcal{V}}{\mathcal{D}}_i,{\mathsf{\Theta }}_i\right),$$

(4)

where ${\mathsf{\Theta }}_i$ represents the GMM parameters for patch i.

Global Energy ${\mathcal{E}}_g$

Global energy corrects any residual misalignments on a global scale. It refines the overall pose of the moving point cloud by finding a global rigid transformation $\mathcal{T}$ that minimizes

$$\mathcal{E}g=\sum {i=1}^n{ϵ}_i{\left|\mathcal{T}{\tilde{\mathbf{p}}}_i-{\mathbf{q}}_i\right|}^2,$$

(5)

where ${\tilde{\mathbf{p}}}_i$ are the deformed points after the intermediate step and ${ϵ}_i$ is a weighting factor that may depend on the distance between corresponding points.

2.3. Anatomical Plausibility

While the deformation field provides a mechanism for aligning point clouds, ensuring that the resulting transformations adhere to anatomical constraints is vital for clinical validity. Deformable techniques can yield highly flexible mappings, but without constraints, they risk generating unrealistic or biologically implausible results.

To address this, we incorporate a regularization term $\mathcal{R}\left(\mathcal{T}\right)$ into the cost function to enforce anatomical plausibility:

$$\mathcal{L}(\mathcal{P},\mathcal{Q},\mathcal{T})=\mathcal{M}(\mathcal{Q}\circ \mathcal{T},\mathcal{P})+{\lambda }_r\mathcal{R}\left(\mathcal{T}\right),$$

(6)

where $\mathcal{M}$ is the similarity metric between the deformed moving point cloud and the fixed point cloud, ${\lambda }_r$ is a regularization parameter, and $\mathcal{R}\left(\mathcal{T}\right)$ quantifies the smoothness and plausibility of the deformation field.

2.3.1. Regularization Terms

We consider several regularization terms commonly used in deformable registration.

L2 Norm Regularization

This penalizes large gradients in the deformation field:

$$\mathcal{R}\left(\mathcal{T}\right)={\int }_{\mathsf{\Omega }}{\left|\nabla \mathcal{T}\right|}_2^2,d\mathbf{x},$$

(7)

where $\nabla \mathcal{T}$ is the gradient of the deformation field.

Total Variation (TV) Regularization

This encourages piecewise smoothness while preserving edges:

$$\mathcal{R}\left(\mathcal{T}\right)={\int }_{\mathsf{\Omega }}\left({\left|\nabla u\right|}_2+{\left|\nabla v\right|}_2+{\left|\nabla w\right|}_2\right),d\mathbf{x},$$

(8)

where $\mathcal{T}=(u,v,w)$.

Bending Energy Regularization

This penalizes curvature in the deformation field to maintain smoothness:

$$\mathcal{R}\left(\mathcal{T}\right)={\int }_{\mathsf{\Omega }}\left({\left(\mathsf{\Delta }u\right)}^2+{\left(\mathsf{\Delta }v\right)}^2+{\left(\mathsf{\Delta }w\right)}^2\right),d\mathbf{x},$$

(9)

where $\mathsf{\Delta }$ is the Laplacian operator.

Smoothness Constraints

These enforce smoothness by penalizing differences between neighboring deformation vectors:

$$\mathcal{R}\left(\mathcal{T}\right)=\sum _{\mathrm{neighbors}(\mathbf{x},{\mathbf{x}}^{\prime })}{\left|\mathcal{T}\left(\mathbf{x}\right)-\mathcal{T}\left({\mathbf{x}}^{\prime }\right)\right|}^2.$$

(10)

2.3.2. Similarity Metric $\mathcal{M}$

We use the Hausdorff distance as the similarity metric, which is defined as follows:

$$\begin{array}{c}\hfill d_H(\mathcal{P},\mathcal{Q})=max\left\{\underset{\mathbf{p}\in \mathcal{P}}{sup}\underset{\mathbf{q}\in \mathcal{Q}}{inf}|\mathbf{p}-\mathbf{q}|,;\underset{\mathbf{q}\in \mathcal{Q}}{sup}\underset{\mathbf{p}\in \mathcal{P}}{inf}|\mathbf{q}-\mathbf{p}|\right\}.\end{array}$$

(11)

This metric is sensitive to outliers and captures the maximum deviation between the point clouds, minimizing the largest errors.

2.4. Conformal Bayesian Optimization with Gaussian Process Priors

To optimize the hyperparameters $\theta$ controlling the deformation model, we employ Bayesian optimization with Gaussian Process (GP) priors. The objective is to minimize the cost function $f\left(\theta \right)=\mathcal{L}(\mathcal{P},\mathcal{Q},\mathcal{T}(\theta \left)\right)$ over the bounded domain $\mathsf{\Theta }$.

We model $f\left(\theta \right)$ as a sample from a GP prior, specified by the mean function $m\left(\theta \right)$ (often zero) and the covariance function $k(\theta ,{\theta }^{\prime })$. The GP provides a probabilistic model of the function, allowing us to predict $f\left(\theta \right)$ at unobserved points.

The acquisition function $a\left(\theta \right)$ guides the selection of the next hyperparameters to evaluate by balancing exploration and exploitation. We use Conformal Bayesian Optimization [16], which provides robustness against model misspecification and covariate shift, offering coverage guarantees.

Table 1. Hyperparameters for deformable point cloud registration.

Hyperparameter	Symbol	Range
Number of clusters	c	$[2,20]$
Regularization weight	${\lambda }_r$	$[0.001,1.0]$
Number of neighboring rings	r	$[1,5]$
Influence factors for local energy	${\xi }_{ij}$	$[0.1,10]$
Threshold for global energy correction	d	$[0.01,1.0]$

presents the hyperparameters optimized within the proposed Bayesian framework for deformable point cloud registration. Each hyperparameter plays a crucial role in balancing the accuracy and computational feasibility of the registration process.

• Number of clusters (c): This parameter determines the granularity of local deformations. A higher number of clusters allows for finer transformations; however, excessively high values may lead to overfitting or increase computational complexity. The range $[2,20]$ ensures adaptability without compromising efficiency.
• Regularization weight (${\lambda }_r$): This controls the trade-off between alignment precision and deformation smoothness. Higher values promote smoother transformations, which are crucial for maintaining anatomical plausibility, particularly in sensitive brain structures.
• The number of neighboring rings (r): This parameter defines the local neighborhood around each point and influences how adjacent points contribute to the transformation. Smaller values encourage localized deformations, while higher values capture broader contextual relationships.
• Influence factor (${\xi }_{ij}$): This modulates the importance of local energy contributions from each neighboring ring. This ensures that the algorithm appropriately balances the impact of different regions on the final deformation, which is essential for aligning structures with varying geometric properties.
• Threshold for global energy correction (d): This parameter determines when global transformations, such as translation and rotation, are applied. A lower threshold triggers more frequent global adjustments, ensuring accurate alignment across large-scale transformations.

By integrating anatomical plausibility through regularization and optimizing hyperparameters via Conformal Bayesian Optimization, our method aims to achieve accurate deformable registrations of point clouds.

Figure 1 shows the proposed framework for the deformable registration of point clouds, using Bayesian optimization to enhance their alignment. The process starts with two input point clouds: a fixed cloud $\mathcal{P}$ and a moving cloud $\mathcal{Q}$. The transformation function $\mathcal{T}$ estimates the non-rigid transformations needed to align $\mathcal{Q}$ with $\mathcal{P}$. In addition, the alignment is driven by a cost function, $\mathcal{L}(\mathcal{P},\mathcal{Q},\mathcal{T})$, combining geometric distance with a regularization term to ensure smooth and anatomically plausible deformations. Bayesian optimization iteratively adjusts the transformation parameters to refine the alignment and optimize the number of initial correspondences. Finally, the output is the deformed point cloud $\mathcal{Q}\circ \widehat{\mathcal{T}}$, which is the transformed version of the moving cloud. This approach balances local flexibility and global alignment, ensuring adherence to anatomical constraints. Its modular design and probabilistic strategy make it ideal for medical imaging, where precise and interpretable alignments are essential for clinical decision-making.

3. Results

3.1. Tosca Database

To validate the deformable registration model, the performance of the model was mainly analyzed using synthetic databases, such as the Tosca database. In this context, a series of experiments were initiated that allowed us to identify how the cost function weights the deformations applied to the point clouds. Figure 2 shows an example of the initial correspondence process applied to the point clouds to calculate the deformation $\mathcal{T}$. The plots on the right side of each subfigure illustrate the initial correspondences between the fixed (red) and moving (blue) point clouds. Close alignment between the red and blue points indicates successful registration, while more significant gaps suggest areas with alignment discrepancies.

In our first experiment, we applied the proposed deformable registration method to the Tosca dataset, which consists of non-rigid 3D models of various subjects, such as horses and cats. Figure 2 illustrates the registration process between a fixed point cloud (red) and a moving point cloud (blue). The right side of each subfigure presents the point correspondences established during the initial registration phase. Figure 2a shows the deformable registration of a horse model, where both the fixed and moving point clouds exhibit significant variations in pose. The corresponding points identified between the clouds (shown on the right) highlight the complexity of aligning the limbs and head, which exhibit distinct positional changes. However, the method’s robustness is evident as it can initialize correspondences even when significant structural differences exist between poses. Figure 2b presents a similar process for a cat model, which also underwent non-rigid deformations. The initial correspondences are visualized, indicating a robust alignment of the major body components, such as the torso and limbs. The method’s versatility is demonstrated as it effectively handles variations in pose, with the established correspondences maintaining consistency despite local deformations. The results show that even in highly non-rigid cases, such as those seen in the Tosca dataset, our method successfully aligns the point clouds in an anatomically plausible manner, outperforming conventional rigid alignment techniques.

To validate the plausibility of this approach in the Tosca database, we analyzed the behavior of the deformable model for point clouds with variations in pose (Figure 2a) and for point clouds representing different quadrupeds (Figure 2b). Simultaneously, we examined the cost function in Figure 3. Here, “lower cost” refers to the value of the cost function $\mathcal{L}(\mathcal{P},\mathcal{Q},\mathcal{T})$, which combines geometric distance with a regularization term to ensure smooth and anatomically plausible deformations. A lower cost indicates a better alignment with minimal deformation, achieving a more accurate and natural registration. For both experiments, we observed that point clouds from the same subjects generally result in a lower cost, reflecting more effective alignment than those from different subjects.

The results shown in Figure 4a,b illustrate the effectiveness and limitations of our deformable registration approach that uses Bayesian optimization. By using a cost function that combines the Hausdorff distance with a regularization term, we achieve a quantifiable measure of alignment quality, where lower values indicate more accurate alignment and anatomically plausible deformations. Figure 4a shows that the method successfully aligns point clouds with significant pose variations, demonstrating its robustness in handling non-rigid deformations. This capability is particularly useful in medical imaging scenarios where anatomical structures exhibit significant but plausible variations over time or due to patient movement. However, as shown in Figure 4b, this method encounters challenges when applied to point clouds with fundamentally different anatomies, resulting in anatomically implausible registrations. This highlights the critical importance of incorporating anatomical constraints into the registration process. This method’s ability to preserve structural consistency in plausible cases underscores its potential for applications in clinical settings, particularly when tracking changes in brain structures over time or within a population.

The results from Figure 3 provide a quantitative analysis of the registration loss, comparing alignments between point clouds from the same subject in different poses versus point clouds from different subjects. The lower loss values observed in the same group indicate that the deformable registration framework, enhanced by Bayesian optimization, performs well in cases where anatomical consistency is preserved, as demonstrated in Figure 2. The use of the Hausdorff distance combined with regularization effectively captures the quality of the alignment, showing minimal deformation and high anatomical plausibility when the same subject is being registered across varying poses. In contrast, the higher loss values in the different-shapes group, as seen in Figure 4b, reflect the model’s limitations when aligning point clouds with fundamentally different anatomies. These higher loss values indicate that the method struggles to maintain anatomical fidelity when confronted with subjects that deviate significantly in shape and structure, leading to implausible deformations. The results further emphasize the importance of incorporating anatomical constraints and highlight that while the method excels in non-rigid but plausible scenarios, it requires further refinement to handle more complex cases involving diverse anatomies. Compared to well-known approaches in deformable registration, our framework demonstrates distinct advantages in handling anatomical variability. While methods like VoxelMorph [13] leverage convolutional neural networks (CNNs) for efficient and unsupervised image registration, their reliance on pixel-based transformations often leads to limitations in preserving anatomical structure, particularly in cases with high variability or intricate anatomical details. Large Deformation Diffeomorphic Metric Mapping (LDDMM) [11], a widely used technique in medical imaging due to its diffeomorphic properties, excels at handling complex deformations but is computationally intensive, making it less practical for clinical scenarios that require rapid processing. DeepReg [24] integrates deep learning with regularization to improve anatomical accuracy, but its performance can be sensitive to initial conditions, and it may converge to suboptimal solutions when faced with significant shape variations, such as those encountered in neuroanatomical structures affected by perinatal asphyxia.

Our proposed framework, driven by Conformal Bayesian Optimization (CBO), addresses these limitations by introducing probabilistic modeling that dynamically adapts regularization to anatomical structures, ensuring both smooth and plausible deformations.

3.2. Brain Asphyxia Database

In the case of Brain Asphyxia, two sets of experiments were conducted: one focused on registering the point clouds of brain structures from healthy patients and another on registering point clouds from patients diagnosed with perinatal asphyxia. In the case of healthy patients, the brain structures, while exhibiting non-rigid changes, tend to maintain volumetric consistency across time points. This consistency means that although slight variations in contours may exist, critical features such as size and shape are preserved, as illustrated in Figure 5. These deformations are typically mild, which makes the registration task less challenging. One potential reason for this uniformity is that the point clouds were extracted using probabilistic atlases, which ensure a high level of alignment. On the other hand, brain structures from patients with perinatal asphyxia exhibit a significant degree of variability in both shape and size. This is due to the severe anatomical changes caused by the condition. This increased variability leads to a more complex and challenging registration task, where the model struggles to converge, resulting in a higher cost function, as shown in Figure 6. A comparative analysis of different regularization terms—such as L2 Norm, Total Variation, Bending Energy, and Smoothness Constraints—reveals their distinct impacts on deformable registration performance. For healthy patients, where deformations are relatively small, the L2 Norm effectively maintains the overall smoothness and alignment accuracy of the registration. It imposes a penalty on large deformations, ensuring that the anatomical consistency of brain structures is preserved.

For patients with perinatal asphyxia, where significant deformations are present, the use of more advanced regularization terms like Total Variation and Bending Energy becomes crucial. The Bending Energy regularization helps to maintain the smoothness of large deformations, ensuring anatomically plausible transformations by minimizing sharp or unnatural changes. Total Variation, on the other hand, balances smoothness with the ability to preserve sharp edges or discontinuities in the deformation, which can be critical in pathological cases where certain brain regions undergo more pronounced changes.

Figure 7 provides a comparative analysis of the loss function $\mathcal{L}$ across four different regularization methods—L2 norm, Total Variation (TV), Bending Energy (BE), and Smoothness Constraints (SCs). The comparison is presented for two patient groups: healthy individuals and patients diagnosed with perinatal asphyxia. This analysis highlights how each regularization method influences the deformation field in brain structure registration, emphasizing the variability between patient groups. The Smoothness Constraints regularization assists in controlling local deformations, ensuring that small neighborhoods within the brain’s structure undergo consistent transformations without introducing unrealistic distortions. However, its effectiveness diminishes when the variability in the shape and size of the structures is extreme, as seen in perinatal asphyxia cases. This variability explains the more significant performance variance observed for asphyxia patients when using SCs, as illustrated in the figure. These findings further underscore the adaptability of different regularization terms to varying levels of deformation complexity. Standard regularization methods, such as the L2 Norm and Smoothness Constraints, perform well for healthy patients, ensuring smooth transformations and accurate alignment. However, for patients with perinatal asphyxia, more advanced methods like Total Variation and Bending Energy become essential due to their ability to handle complex anatomical alterations. This adaptability highlights the need for case-specific approaches in medical imaging to ensure accurate and anatomically plausible brain structure registrations across diverse clinical scenarios.

Figure 5 and Figure 6 show the deformable registration of point clouds representing brain structures from the Brain Asphyxia database. Figure 5 shows the registration process in healthy patients, focusing on two brain structures: the left white matter and the left thalamus. These examples highlight the algorithm’s ability to align brain structures with minimal deformation, ensuring smooth transformations and accurate alignment. In contrast, Figure 6 presents the difference in registration between the brain structures of healthy patients (red) and those affected by perinatal asphyxia (blue). The comparison reveals a higher degree of structural variability in the asphyxia group, requiring more complex deformation fields to achieve alignment. Despite this complexity, the framework successfully aligns the brain structures, showcasing the robustness of this approach. The benefit of incorporating probabilistic optimization through Conformal Bayesian Optimization (CBO) is evident in these results. CBO enables the framework to iteratively learn plausible shape deformations, balancing local flexibility with global alignment. The probabilistic nature of CBO ensures that transformations adhere to anatomical constraints, reducing the risk of producing unrealistic deformations. This is particularly valuable in cases with significant anatomical variability, such as perinatal asphyxia, where traditional optimization techniques might struggle to maintain anatomical plausibility. The framework’s modular design allows it to adapt to the specific anatomical characteristics of each patient group. In healthy patients, simpler deformations suffice, while in the asphyxia group, the optimization dynamically adjusts the deformation parameters to account for more substantial changes in brain structures. These results demonstrate the effectiveness of the proposed approach in addressing varying levels of complexity, ensuring both accuracy and clinical relevance in brain structure registration.

Figure 8 compares the registration costs of aligning brain structures from healthy patients and those affected by perinatal asphyxia. The cost metric combines the Hausdorff distance and regularization terms, reflecting alignment precision and deformation smoothness. As depicted, the registration cost is consistently higher for the asphyxia group, indicating the increased complexity associated with aligning brain structures that exhibit significant anatomical variability. The elevated costs observed in the asphyxia group reflect the more significant structural deformations required to achieve alignment. Brain structures affected by perinatal asphyxia often exhibit altered shapes and sizes, making the registration task more challenging. In contrast, healthy patients display more consistent anatomical features across different scans, lowering registration costs. This contrast highlights the importance of advanced optimization methods, such as Conformal Bayesian Optimization (CBO), in adapting deformation models to varying anatomical conditions. The results further demonstrate the benefit of incorporating regularization into the loss function. While achieving accurate alignment is crucial, the regularization term ensures that the deformations remain anatomically plausible. The balance between alignment precision and regularization promotes meaningful transformations for both patient groups, minimizing the risk of overfitting or producing unrealistic deformations. These findings emphasize the need for adaptive frameworks in medical imaging to address the complexities inherent in pathological conditions like perinatal asphyxia.

4. Conclusions

This study presents a novel framework for the deformable registration of 3D point clouds representing brain structures, leveraging Conformal Bayesian Optimization (CBO) to enhance alignment precision and maintain anatomical plausibility. The proposed method uses probabilistic modeling to optimize non-rigid transformations, addressing critical challenges in aligning brain structures with significant anatomical variability. Through the combination of the Hausdorff distance and regularization terms, the framework ensures smooth deformations that respect anatomical constraints, reducing the risk of implausible transformations. The experiments conducted on both the Tosca and Brain Asphyxia databases demonstrate the robustness and versatility of this approach. Results from the Tosca database highlight the model’s ability to handle substantial non-rigid deformations, achieving accurate alignments even with significant pose variations. Furthermore, analyzing brain structures from the Brain Asphyxia database underscores the importance of adaptive optimization strategies. The use of CBO enabled the framework to accommodate the anatomical complexities presented by pathological conditions such as perinatal asphyxia, ensuring that transformations remained clinically relevant. Comparative evaluations of different regularization methods revealed the necessity of case-specific configurations to optimize performance. While standard regularization techniques proved sufficient for aligning healthy brain structures, more advanced methods, such as Bending Energy and Total Variation, were critical for managing the more complex deformations observed in asphyxia cases. The modular nature of the proposed framework allows it to adjust to varying anatomical conditions dynamically, enhancing its applicability across diverse clinical scenarios. In addition, integrating CBO into the deformable registration process offers several advantages, including improved alignment accuracy, reduced computational complexity, and better control over deformation smoothness. This framework addresses the key limitations of existing methods by providing an interpretable and adaptable approach that balances alignment precision with anatomical plausibility. The findings of this study demonstrate the potential of CBO-enhanced registration models to support clinical decision-making, particularly in scenarios that require the tracking of anatomical changes over time or across patient populations. Future work will explore the extension of this approach to other imaging modalities and further refine the optimization framework to better address cross-subject anatomical variability.

In future work, we plan to integrate elastic transformations controlled by Bayesian latent variable frameworks into our model to enhance the reliability and robustness of the registration process. Bayesian latent variables will enable probabilistic representations of deformation fields, better capturing global and local anatomical variability. Elastic transformations will offer greater adaptability in handling complex deformations, especially in pathological cases, while maintaining anatomical plausibility. We also aim to optimize the computational cost of these models, ensuring efficient training and uncertainty management. This approach will support more reliable alignments across diverse imaging modalities, promoting better clinical outcomes.