^{*}

This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution license (http://creativecommons.org/licenses/by/3.0/).

This paper proposes a novel global-to-local nonrigid brain MR image registration to compensate for the brain shift and the unmatchable outliers caused by the tumor resection. The mutual information between the corresponding salient structures, which are enhanced by the joint saliency map (JSM), is maximized to achieve a global rigid registration of the two images. Being detected and clustered at the paired contiguous matching areas in the globally registered images, the paired pools of DoG keypoints in combination with the JSM provide a useful cluster-to-cluster correspondence to guide the local control-point correspondence detection and the outlier keypoint rejection. Lastly, a quasi-inverse consistent deformation is smoothly approximated to locally register brain images through the mapping the clustered control points by compact support radial basis functions. The 2D implementation of the method can model the brain shift in brain tumor resection MR images, though the theory holds for the 3D case.

Image registration is an important enabling technology for neuronavigation [_{R}_{F}

The key challenge for the nonrigid registration of pre- and intra-operative MR images is to compensate for the local large tissue distortion caused by the tumor resection. The local large tissue deformations with irregular shapes violate the usual assumption of smoothness of the deformation fields. An additional challenge exists when the unmatchable outlier features (

To reject the outliers, many intensity-based registration approaches are proposed including M-estimator [

Recently, landmark-based registration methods using local invariant features, such as salient region features [

Although the results of these above methods clearly demonstrate the power of local invariant feature-based nonrigid deformations, the desired landmark-based registration algorithm should establish robust control point correspondence to accurately model the complex local deformation around the tumor resection region. To find robust point correspondence, some approaches are proposed including soft correspondence detections [

To make the rigid registration more accurate and robust for matching brain tumor resection images, we use joint saliency map (JSM), first proposed in our previous work [

The proposed algorithm has been applied to the nonrigid registration of 2D brain tumor resection MR images. Experimental results show that, compared to other classic nonrigid registration methods, our proposed method can provide better robustness and higher accuracy for the registration of brain tumor resection images. The rest of this paper is organized as follows. Section 2 briefly summarizes the definition of JSM and its application in global rigid registration, which are the same as in [

Since 1995 [_{R}_{F}_{R}_{F}_{i}p_{R}, I_{F}_{r,f}p_{f}p_{r}p

MI-based registration methods take advantage of the fact that properly registered images usually correspond to compactly-clustered joint histograms [

Noting that many techniques have defined the saliency of image to enhance the image pixels we are interested in, _{l}_{u}_{∈}_{N}_{v}_{l}_{l}^{2}, where _{v}_{l}_{l}_{l}_{l}

In the second step, a principal axis analysis of regional saliency distribution assigns the regional saliency vector to each pixel based on the inertia matrix:
_{mn}_{x}^{m}_{y}^{n}S_{x}, g_{y}_{10}_{00}, _{01}_{00}), _{mn}^{m}y^{n}S

Given the two RSVs s_{r}_{f}_{r}_{f}_{r}_{f}_{r}_{f}

A JSM value near one suggests that the underlying pixel pair comes from the corresponding spatial structures. Contrarily, a JSM value near zero indicates that the pixel pair originates from either the outliers or a homogeneous region. To speed up the rigid registration procedure without reducing accuracy, the registration only uses the salient pixel pairs with large saliency values. The pixel with a small saliency value below a fixed threshold value (10 percent of the maximum saliency value) is assigned zero JSM value directly. Generally, the JSM would primarily respond to the high-gradient common edges in both images if a high threshold is chosen to exclude more pixels from estimating the JSM. However, as in the

The accurate rigid registration can be iteratively achieved by maximizing the MI between the corresponding salient structures that are highlighted by the JSM. The JSM is updated with the transformation _{f}

Given the globally registered images and the affiliated JSM (see

Ideas related to clustering based control point setup was first suggested by Chui

In order to divide the keypoints into groups, we make use of the EM algorithm [_{i}_{1},…, _{K}, θ_{1},…, _{K}_{i}_{i}_{i}_{i}

The first step in applying the EM algorithm to the Gaussian mixture problem is to initialize _{1},…, _{K}_{1},…,Σ_{K}_{j}, θ^{old}_{j}^{old}

The above update equations are iteratively computed until the log likelihood

We have thus far not discussed how to choose _{1},…, _{K}, θ_{1},…_{K}_{j}

(2) detect peaks and valleys. A point _{i}_{i}_{max} − _{min})_{threshold}_{max}; (4) reject each keypoint which is far from any peaks obtained from step 3 under the distance threshold (1-2 times _{threshold}_{0}, _{1},…, _{l}_{0}, _{1},…, _{l}_{i}

Given a set of keypoints (_{1}, _{2},…, _{N}_{1},_{2},…,_{K}_{i}_{i}_{1},_{2},…,_{K}

The results of the EM clustering method are probabilistic. This means that every keypoint belongs to all clusters, but each assignment of a keypoint to a cluster has a different probability. Given the resultant parameters (_{1},…, _{K}, θ_{1},…, _{K}_{i}_{i}_{k}p_{k}_{i}_{k}_{i}

Note that if one cluster pair of tumor regions suffers from the large structural mismatches with local large deformation and outliers, this cluster pair will bring together many keypoints and small average JSM value _{k}_{k}_{i}_{k}_{i}_{k}_{k}_{threshold}_{1} = 0.266 could be automatically detected from the nine clusters. _{x}^{2}_{x}^{2} + (_{y}^{2}_{y}^{2}] < 1 with

In general, correspondences are found by choosing the two points with the optimum similarity measure (such as mutual information and cross correlation) between the points' surrounding regions, but this template matching method can give unsatisfactory correspondence due to the large number of keypoints detected in each image and the region around each keypoint being too small to include sufficient information. To enhance the confidence of template matching-based correspondence detection, we use two steps to find the robust correspondence:

Choose the significant keypoints for irregular control point setup. The SIFT keypoint detector has assigned a location and a scale to each stable DoG keypoint. The scale defines the saliency measure of each keypoint such that the keypoint with a large scale could be identified at the same location in the noisy pre- and intra-operative MR brain images. Based on the above consideration, a keypoint with the largest scale measure within a neighborhood could be saved as the significant keypoint in a cluster. In

Use the cluster-to-cluster correspondence and inverse consistent correspondence calculation to enhance the confidence of MI-based template matching. In detail, we use a local MI (see the details in the following section) to match a reference point _{i}_{k}^{i}_{i}^{i}_{i}^{i}_{i}^{i}_{i}_{i}

In recent years, increased attention has been paid to the local similarity measure of small image regions for local nonrigid registration [

Although we have initialized correspondence detection in the context of local large deformation and outliers, it may not yet lead to 100% accurate matching of DoG control points, a few of the corresponding pairs are likely to be incorrect. We adopt an approximating [_{1} : _{i}_{i}_{2} : _{i}_{i}_{i}_{i}_{1} is the RBF forward transformation,

In this work we have used the Wendland's function as RBFs [_{1}, Δ_{2})(Δ_{1} and Δ_{2} are the displacement from the floating control point to the reference control point in

Generally, an interpolation transformation function ^{2} → R^{2} for 2D images based on control points must fulfill the constraints _{i}_{i}, i_{l}^{2} → R is solved for each coordinate _{l}_{i}_{l,i}_{i}_{i}_{i}_{l}_{l,}_{1},…_{l,N}^{T}_{l}_{l,}_{1},…, _{l,N}^{T}_{ij}_{i}_{j}_{i}

Nonrigid registration is started by the correspondence detection and the local deformation computation for the tumor resection area, subsequently modified by the surrounding clusters in the ascending order of _{k}

Specifically, in the initial registration iteration, the search radius _{k}_{k}

We preliminarily evaluated our algorithm on 5 pairs of pre- and intra-operative (or post-operative) MR T1 images that are corresponding slices of rigidly transformed 3D MR datasets. We first remove the skull near to the tumor resection area to prevent from introducing the deformation of rigid skull. We use 18 × 18 bins for 2D DoG keypoint joint histogram, which is exponentially smoothed with the exponential smoothing factor

In contrast to four state-of-the-art intensity-based registration methods, including B-Spline with correlation ratio [

The BCR and BMI registration are implemented at two pass with the different transformation options (B-Spline degree for all axes: 1, 2; B-Spline control points for all axes: 8, 16; gradient descent minimize step size: 1.0, 0.5; gradient descent minimize maximum search steps: 10, 10) and the different iteration options (the convergence limit of minimum change rate for one iteration: 0.1, 0.01; maximum number of iterations: 10, 10) [

The DEM and DIF registration are conducted with a maximum step length of 2 pixels, 1.0 standard deviations of the Gaussian smoothing, a maximum number of 200 iterations and 0.001 intensity difference threshold. Treating each image as a set of iso-intensity contours and assuming the same anatomical point having the same intensity level in both images, the DEM and DIF easily distort the data to some extent, which may introduce strange artifacts similar to pieces of small mosaic patterns in the deformed pre-operative images (

In the two cases of local small structure distortion, all five methods get good results (see

We have presented a new hybrid nonrigid registration of brain MRI images to model tumor resection-induced brain shift. While SIFT keypoint detector was designed under the assumption of linear changes in intensity, the DoG keypoint detected by the SIFT detector can be effective in robustly matching intra- and pre-operative MR image pairs taken under substantially different illumination condition due to the spatially-varying intensity inhomogeneity and large intra-operative noise. Nevertheless, the keypoint detection (and the keypoint description) could not sufficient [

The main drawback of our approach is that a number of parameters should be set. By applying more experiments on the brain tumor resection images, the future study must be carried out determining the optimal parameters and comparing the JSM reconstruction methodologies with discussion their robustness. An interesting future development is to design a robust deformation invariant keypoint detector [

The authors would like to thank Simon K.Warfield for allowing the use of the images, all reviewers for their useful comments and Andrea Vedaldi for the open-source implementation of the SIFT algorithm. The authors thank the open source implementation of diffeomorphic demons and the open source ITK project. The authors also give thanks to Wang Yikang, Zhuang Tiange, Zhu Yisheng and Xu Yongjiang for their help to our work. This work was supported in part by the NSFC (Grant 60872102), the National Basic Research Program of China (Grant 2010CB834303), the Science Foundation of Shanghai Municipal Science & Technology Commission (Grant 04JC14060), Shanghai Municipal Health Bureau (Grant 2008115) and the small animal imaging project (Grant 06-545).

(a)–(b) Intra- and pre-operative MR images with tumor resection. (c)–(d) Gradient value profiles of the lines in (a)–(b), which are marked as dashed lines. (e) JSM value profiles of the lines in (a)–(b). (f) Joint histogram dispersion with two clotted clusters (dark red in pseudo color). (g) The JSM-weighted joint histogram with smoothed compact clusters for (a)–(b). (h) JSM for the two images in (a)–(b) with low JSM values at the tumor resection area. (i) The intra-operative MR image and the circle marked DoG keypoints. (j) The pre-operative image and resultant keypoint clustering with circle marked floating keypoints and cross marked reference keypoints. Different colors mean different clusters.

(a)–(b) and (e)–(f) Clustered significant keypoints in the two images. (c)–(d) and (g)–(h) Boundary significant keypoints around tumor resection regions with circles defining the scale measures of the points.

(a)–(b) Intra- and pre-operative MR images. (c) JKC. (d) Displacement vector field with the vector orientation and the variations of the vector color (the color scale encodes the norm of the displacement vector, in pixels). (e) BMI. (f) BMI deformation image. (g) BCR. (h) BCR deformation image. (i) DEM. (j) DEM deformation image (the color scale encodes the norm of the displacement vector, in pixels). (k) DIF. (l) DIF deformation image.

(a)–(b) Intra- and pre-operative MR images. (c) JKC. (d) Displacement vector field. (e) BMI. (f) BMI deformation image. (g) BCR. (h) BCR deformation image. (i) DEM. (j) DEM deformation image. (k) DIF. (l) DIF deformation image.

(a)–(b) Corresponding landmarks for intra- and pre-operative MR images at

(a)–(b) The pre- and post-operative brain tumor images. (c)–(d) Significant keypoints selected from the different clusters. (e) JSM between the two images. (f) Fused image using check pattern after registration. (g) Warped mesh after registration. (h) Displacement vector field.

(a)–(b) The MR-T1 and MR proton density weighted images. (c)–(d) Significant keypoints selected from the different clusters. (e) JSM between the two images. (f) Fused image using check pattern after registration. (g) Warped mesh after registration. (h) Displacement vector field.

CR and NMI values for different registration methods (The smaller values usually mean the better registration results).

cases | CR | NMI | ||||||||
---|---|---|---|---|---|---|---|---|---|---|

| ||||||||||

JKC | BCR | BMI | DEM | DIF | JKC | BCR | BMI | DEM | DIF | |

1 | 0.0613 | 0.0553 | 0.1068 | 1.0000 | 0.9998 | 0.8619 | 0.8595 | 0.8713 | 0.9845 | 0.9846 |

2 | 0.0889 | 0.0743 | 0.2167 | 0.0974 | 0.0349 | 0.8607 | 0.8590 | 0.8775 | 0.8495 | 0.8195 |

3 | 0.0695 | 0.0704 | 0.2673 | 0.9354 | 0.8917 | 0.8421 | 0.8611 | 0.8697 | 0.9783 | 0.9624 |

4 | 0.0637 | 0.0974 | 0.4181 | 0.0590 | 0.0649 | 0.8745 | 0.8239 | 0.8580 | 0.7880 | 0.7925 |

5 | 0.0749 | 0.0858 | 0.3177 | 0.0582 | 0.0638 | 0.8657 | 0.8471 | 0.8693 | 0.7919 | 0.8142 |