A 3D Image Registration Method for Laparoscopic Liver Surgery Navigation

Abstract

At present, laparoscopic augmented reality (AR) navigation has been applied to minimally invasive abdominal surgery, which can help doctors to see the location of blood vessels and tumors in organs, so as to perform precise surgery operations. Image registration is the process of optimally mapping one or more images to the target image, and it is also the core of laparoscopic AR navigation. The key is how to shorten the registration time and optimize the registration accuracy. We have studied the three-dimensional (3D) image registration technology in laparoscopic liver surgery navigation and proposed a new registration method combining rough registration and fine registration. First, the adaptive fireworks algorithm (AFWA) is applied to rough registration, and then the optimized iterative closest point (ICP) algorithm is applied to fine registration. We proposed a method that is validated by the computed tomography (CT) dataset 3D-IRCADb-01. Experimental results show that our method is superior to other registration methods based on stochastic optimization algorithms in terms of registration time and accuracy.

1. Introduction

During the operation, the information that doctors can obtain through laparoscopy is very limited. They can only obtain the image information of a part of the surface area, and cannot obtain the information inside the organs, which rely heavily on preoperative imaging [1]. In this case, doctors can only rely on their own experience to judge the location of the internal lesions, which has high requirements for doctors and may cause the wrong location of the lesions [2,3]. In 1986, Roberts et al. [4] and Kelly et al. [5] performed AR-assisted surgery in neurosurgery. Since then, with the development of AR applications in auxiliary surgery, AR surgery navigation can accurately match the preoperative anatomical structure information with the intraoperative information, and then present it to the doctor, which has been applied in neurosurgery and orthopedic surgery [6]. The image guidance function of laparoscopic AR navigation has also made much progress in hepatectomy and nephrectomy [7,8,9]. The realization methods of laparoscopic AR navigation mainly include medical image processing, graphic image rendering, image registration, and display technology [10]. The main challenge is the speed and accuracy of 3D image registration [11]. Laparoscopic images and preoperative CT images were obtained from different imaging devices. Due to the different imaging modes, they belong to multi-modal registration. In laparoscopic AR navigation, the speed and accuracy of registration are critical to the impact of surgery [12].

In this paper, the 3D image registration method of laparoscopic AR liver surgery navigation is studied. The registration process involves preoperative point cloud reconstruction, intraoperative point cloud reconstruction, and related registration methods. As the imaging principles of preoperative images and intraoperative laparoscopic images are different, and there is no same standard to match it [13], after studying the multi-modal image registration method, the 3D–3D point cloud registration method is selected. Here we only list the most relevant work. A binocular vision camera can provide doctors with images similar to laparoscopy, which can be used for surface reconstruction by matching features between images [14,15,16]. In this study, we used a binocular vision camera to obtain intraoperative information. The novelty of the 3D image registration method proposed in this paper is that a combination of rough registration and fine registration is used for multi-modal liver image registration. The rough registration uses the AFWA with adaptive amplitude, which replaces the amplitude operator in the enhanced fireworks algorithm, and the fine alignment uses the ICP algorithm improved by the k-dimensional tree (KD-tree). Our goal is to achieve fast and more accurate 3D image registration for laparoscopic AR liver surgery navigation. In particular, the main work of this study includes the following:

(1)

A 3D reconstruction of the segmented preoperative CT images using the Marching Cubes algorithm on the VTK platform, and the 3D point cloud was generated after obtaining the 3D model of the liver;

(2)

The laparoscopic (binocular vision camera) image was processed, and the 3D point cloud of the intraoperative liver image was generated;

(3)

A two-step combined registration method through rough registration and fine registration is introduced. First, AFWA is applied to rough registration, and then the optimized ICP is applied to fine registration, which solves the problem that the ICP algorithm will fall into local extreme values during the iterative process;

(4)

The registration method we proposed and other registration methods based on stochastic optimization algorithms are jointly tested in experiments. From the point cloud registration results, our method is better in terms of computation time and registration accuracy.

2. Background

Surgery navigation is to accurately overlay the patient’s preoperative or intraoperative images and the patient′s anatomical structure to assist the doctor in accurately locating the lesion, thereby making the operation more precise and safer. Image registration in surgery navigation is the process of optimally mapping one or more images to the target image, and it is also the core of laparoscopic AR navigation. As shown in Figure 1, using the surgery navigation system, doctors can see the AR overlay image in the virtual reality glasses or display of laparoscopy, which seems to build a map for surgery, so that doctors can accurately find the location of lesions.

3. Related Work

Literature [17,18] reported the use of electromagnetic tracking to achieve the registration technology of preoperative CT and ultrasound imaging. Literature [19,20,21,22] has reported different registration techniques for image guidance in liver surgery. Fusaglia et al. [23] proposed a new registration method for liver surgery, which can register the intraoperative real-time reconstruction image with the preoperative image. Tam et al. [24] conducted a comprehensive investigation on rigid and non-rigid registration methods. In the process of 3D image registration, the ICP algorithm provides high precision and robustness and is widely used. Segal et al. [25] optimized the ICP algorithm and obtained higher robustness. Bentley et al. [26] proposed a KD-tree data structure, which provides a new space search idea. Liu et al. [27] used a KD-tree optimization algorithm to improve the original ICP, and the test results showed that the stability and registration speed were improved. It is worth noting that the application of ICP in 3D image registration also has certain drawbacks, such as a certain probability of falling into a local optimum. Li et al. [28] introduced an AFWA with high performance, and the experimental results proved that AFWA has high performance and does not take much time. Shi et al. [29] proposed a 3D point cloud registration method based on AFWA and ICP, which was verified by 3D point cloud registration of the physical model of the statue. The experimental results show that this method shows good calculation speed and accuracy, and can be applied in the field of cultural relics restoration. Chen et al. [30] proposed a new medical image registration method, which uses the fireworks algorithm to improve the coral reefs optimization algorithm for medical image registration. Through experimental tests, the method has a fast convergence speed and a significant improvement in computational performance. Zhang et al. [31] evaluated the LARN system they developed for application in liver surgery navigation. Through comparative analysis, the LARN system can help doctors to identify important anatomical structures during liver surgery, thus reducing surgery injuries. Pelanis et al. [32] tested and evaluated a liver surgery navigation system that provides an AR overlay on the laparoscopic camera view during laparoscopic liver surgery. The system can help doctors solve the difficulties associated with liver surgery, and thus perform safer liver surgery.

4. Materials and Methods

4.1. CT Data Preprocess

We used data from the publicly available 3D-IRCADb-01 dataset, which is provided by https://www.ircad.fr/research/3d-ircadb-01 (accessed on 3 January 2022). The CT dataset of three patients was selected, one of whom was a female patient, born in 1987, with a liver tumor located in the fifth zone. The CT voxel size is 0.78 mm × 0.78 mm × 1.6 mm, the pixels are 512 × 512 × 172, the average intensity of the liver in CT is 84, and the liver size is 20.1 cm × 16.9 cm × 15.7 cm. We use 3D Slicer as a tool for image segmentation. We import the patient′s CT data into the 3D Slicer, use the segmentation module to segment the CT images, and extract the target area.

4.2. Preoperative Liver Point Cloud Generation

The 3D reconstruction of medical images has been extensively researched and is becoming increasingly mature [33], and it has contributed to the diagnosis of the patient′s condition and 3D model printing. We choose to use the Marching Cubes algorithm in the VTK platform to perform a 3D reconstruction of the segmented CT images. The reconstructed models of liver, gallbladder, hepatic vena cava and portal vein, and liver tumor are shown in Figure 2a–d. After setting the transparency, these models are placed according to the original 3D space position, as shown in Figure 2e. At the same time, import the reconstructed model into the MeshLab software to generate a surface point cloud, which is shown in Figure 2f. The point cloud includes 7760 points.

4.3. Intraoperative Liver Point Cloud Generation

4.3.1. Calibration of Binocular Vision Camera

The MER-130-30UM binocular vision camera was used to simulate a stereo laparoscope. The installation and fixing of the binocular vision camera are shown in Figure 3a. Both cameras of the binocular vision camera are placed horizontally and fixed on the same reference plane, while the Y coordinates of the cameras must be horizontally aligned. The chessboard on the cardboard is placed in different positions such as far and near, up and down, left and right to take 20 pairs of images. We imported 20 pairs of chessboard images into Matlab (R2019a, America), and used the stereo vision calibration toolbox to obtain various parameters of the binocular vision camera through calculation. At the same time, 20 pairs of image feature points were matched, respectively, and the matching result of one pair of images is shown in Figure 3b.

4.3.2. Image Acquisition and Image Processing

We used the processed liver CT data to obtain a 3D printed model to simulate the real liver. The calibrated binocular vision camera was used to photograph the liver model as shown in Figure 3c. The obtained images were corrected to remove distortions using the rectifyStereoImages function in Matlab. Figure 3d shows the result of using the Laplacian to sharpen the image.

4.3.3. Point Cloud Generation

The disparity image was generated using the SGM algorithm in Matlab, as shown in Figure 3e. The filtering operation is performed after reconstructing the liver model point cloud, and the final obtained point cloud is shown in Figure 3f. Since the computation time in the registration process is directly related to the number of points in the point cloud, it should be considered how to reasonably reduce the number of points in the point cloud. We select representative points in the point cloud through the filtering method to filter out unnecessary points and noise points [34]. The KD-tree algorithm is used to find the spatially neighboring point set of the point cloud, and to solve the average distance between the point cloud and the spatially neighboring point set, and the average and standard deviation of the global distance are calculated. After this, the points outside the range of the mean distance ± standard deviation are removed to obtain the filtered point cloud containing 1830 points.

4.4. Two-Step Combined Registration Method through Rough Registration and Fine Registration

As both binocular vision imaging and CT images can be regarded as 3D data, the 3D–3D registration method is used here. The registration of the 3D point cloud is used as the basis for the registration of CT images and binocular vision imaging, so as to register the obtained preoperative point cloud and intraoperative point cloud.

4.4.1. Rough Registration Process Based on AFWA

Before performing AFWA, it is necessary to determine the dataset, establish the fitness function, and determine the optimization goal. In the initial setting, the KD-tree can be used cleverly to determine the closest point. The preoperative model point cloud is stored in a KD-tree structure, and the K-nearest neighbor algorithm is used to search for the nearest neighbors of all points in the intraoperative point cloud in the KD-tree, and establish corresponding points. The point cloud generated from the intraoperative image was set as the target and set as P, while the point cloud generated from the preoperative model was set as the reference and set as Q. As the nearest neighbor point set of point cloud P, $\mathrm{q}$ can be obtained by searching in point cloud Q. For the sake of unity and convenience, we use p point set as a shorthand for point cloud P, and the fitness function is established

$$f(R,T)=\frac{1}{n}{{\displaystyle \sum _{i=1}^n\Vert {\mathrm{q}}_i-\left(R\times {\mathrm{p}}_i+T\right)\Vert }}^2=\min$$

(1)

Among them, R is the rotation variable and T is the translation variable, including 3 rotation variables and 3 translation variables. Where $n$ represents the number of points in the target p point set. After that, AWFA is used to realize rough registration. It is worth noting that adaptive explosion radius is the core mechanism of AWFA. In addition, in the AWFA, the fitness function is established to calculate the fitness value of each spark, so as to produce different numbers of sparks at different explosion radii. Figure 4a shows the rough registration process based on AFWA.

4.4.2. Fine Registration Process

The main purpose of fine registration is to correct the previously obtained registration results and obtain more accurate registration results. Therefore, after the rough registration, we use the ICP based on KD-tree optimization to correct the result obtained by the rough registration. At the same time, the rotation variable R and translation variable T obtained in the rough registration process are extracted as the optimization parameters of the transformation operation, so that the p point set is transformed as follows

$${\mathrm{p}}^{\prime }=\mathrm{p}\times R+T$$

(2)

where ${\mathrm{p}}^{\prime }$ is the new target point set after the transformation calculation. The fine registration steps for the proposed design using optimized ICP are:

(1)

Input the calculated target point set ${\mathrm{p}}^{\prime }$ and the original point set Q together. At this time, the KD-tree structure is used to store the point set Q. Then the focus is to search the closest neighbor point set ${\mathrm{q}}^{\prime }$ of ${\mathrm{p}}^{\prime }$, which is implemented by the nearest neighbor algorithm, and then set the iteration number k (the initial value is k = 1).

(2)

Calculate the rotation variable $R_k$ and translation variable $T_k$ from ${\mathrm{p}}^{\prime }$ to ${\mathrm{q}}^{\prime }$. Here, the quaternion calculation method is used and the value of Equation (3) should be minimized.

$${{\displaystyle \sum _{i=1}^n\Vert {\mathrm{q}}_i-\left(R_k\times {\mathrm{p}}_i^{\prime }+T_k\right)\Vert }}^2$$

(3)

Use the solved $R_k$ and $T_k$ to transform the ${\mathrm{p}}^{\prime }$ to obtain a new target point set ${\mathrm{p}}^{″}$, which is calculated as follows:

$${\mathrm{p}}^{″}=R_k\times {\mathrm{p}}^{\prime }+T_k.$$

(4)

(3)

Calculate the average distance $d_{k+1}$ between point set ${\mathrm{p}}^{″}$ and point set ${\mathrm{q}}^{\prime }$

$$d_{k+1}=\frac{1}{n}{{\displaystyle \sum _{i=1}^n\Vert {\mathrm{q}}_i^{\prime }-\left(R_k\times {\mathrm{p}}_i^{″}+T_k\right)\Vert }}^2.$$

(5)

Determine whether the convergence condition $\Vert d_{k+1}-d_k\Vert <\epsilon$ is satisfied, where $\epsilon$ is the minimum iteration accuracy, and $d_k$ is the average distance of the previous generation. If it is not satisfied, the point set ${\mathrm{p}}^{″}$ is used as the new initial target point set ${\mathrm{p}}^{\prime }$, and let k = k + 1, repeat steps 1–3 until the iteration condition is satisfied.

(4)

According to the obtained rotation variable $R_k$ and translation variable $T_k$, the p point set is transformed, and finally, the final registration result is obtained together with the reference point cloud Q.

Figure 4b shows the process of the two-step combined registration method based on rough registration and fine registration.

5. Experiments and Validation

The comparative experiment we designed was implemented with Matlab. All the four experiments were completed on an Intel Core i5-4210m 2.6 GHz/8 GB and NVIDIA geforce GTX 850 computer. At present, in the research of 3D image registration methods, rough registration based on stochastic optimization algorithms is a popular method, and the genetic algorithm (GA) and particle swarm optimization (PSO) algorithms are mainly used [35]. We designed four experiments to verify the performance of our registration method, mainly from the aspects of registration accuracy and speed for comparison and verification. The first experiment is to use the registration method we introduced for registration. In the other two experiments, we used GA or PSO to replace the AFWA in the rough registration. The fine registration process remains unchanged, and the fine registration step based on optimized ICP is still used. In the last experiment, rough registration with AFWA and fine registration with ICP. In these four experiments, the input data are all the same. Figure 5 shows the point cloud registration results of the four experiments.

In point cloud rough registration, Figure 6 shows the relationship between the number of iterations and the fitness of the three stochastic optimization algorithms.

In the experiment, the four registration methods were tested.

Table 1. In the experiment, the overall computation time and accuracy of the four registration methods are compared.

		GA + Improved ICP	PSO + Improved ICP	AFWA + ICP	Ours
Dataset 1	Registration time (s)	0.709	0.814	16.186	0.606
	Accuracy (mm)	0.0208	0.0019	0.0018	0.0018
Dataset 2	Registration time (s)	0.768	0.861	17.548	0.657
	Accuracy (mm)	0.0346	0.0027	0.0022	0.0022
Dataset 3	Registration time (s)	0.849	0.953	18.658	0.726
	Accuracy (mm)	0.0253	0.0023	0.0019	0.0019

shows the overall computation time and accuracy comparison of the four methods.

6. Conclusions and Discussion

This study analyzes the 3D image registration technology in laparoscopic liver surgery navigation. The most important thing is to introduce a two-step combined registration method of rough registration and fine registration, which can quickly and accurately superimpose the preoperative liver model on the laparoscopic image. We segmented and reconstructed the preoperative CT images to obtain the 3D model and point cloud of the liver. During the surgery, we built the intraoperative 3D surface model point cloud, which was then registration. Using SLAM to track the camera motion can finally realize AR visualization. These works are applicable to laparoscopic liver tumor resection, while for open surgeries, there are easier and more accurate ways to reconstruct the 3D model of the liver, such as using an Intel RealSense RGB-D camera. Comparing our registration method with other registration methods based on a stochastic optimization algorithm, from the analysis in Figure 6, our method converges very fast and can converge in about 12 generations, which is obviously better than the other two registration methods based on the stochastic optimization algorithm. The registration accuracy of our proposed registration method in three tests was 0.0018 mm, 0.0022 mm, and 0.0019 mm, respectively, which also demonstrates the good robustness of the method. As can be seen from Table 1, our proposed registration method is clearly better than other registration methods in terms of computation time and registration accuracy. It is worth noting that our registration method has better performance for searching corresponding points, reduces a lot of iterative calculations in fine registration, and can also overcome the defect that ICP has the possibility of falling into a local optimum when it is applied. During laparoscopic liver tumor resection, the liver will be deformed to some extent due to the patient′s breathing or the collision of the surgical instruments with the liver. If this deformation is to be compensated, one can consider building a deformable biomechanical model of the liver, deformation guidance of the model by the surface motion of the liver, and a non-uniform deformation field. However, our proposed registration method compensates for liver deformation by taking into account improved calculation time and registration accuracy for initial registration and multiple intraoperative updates of the registration. This approach relies on a hybrid operating room with a real-time instrument to provide real-time images intraoperatively. During the surgery, the first registration is performed first, followed by multiple intraoperative registration updates. The CT image provided intraoperatively is used as the reference image and the image provided by the laparoscopic camera is used as the target image, thus performing multiple registration updates. After the first registration, the liver is deformed to varying degrees compared to its initial state due to a number of effects. Intraoperative registration updates can compensate for this deformation, and it still works even if the liver has undergone a large deformation.

Our proposed 3D image registration method will have a beneficial effect on surgery navigation systems, especially it will improve the registration accuracy and speed of surgery navigation systems. It is predictable that the improved surgery navigation system will help doctors quickly locate the lesion while bringing a good user experience to the doctor, so as to perform more accurate and safer surgery. Before our proposed method is applied in a real surgery setting, consideration should also be given to how to eliminate the effects caused by electrocautery during the procedure, such as smoke and liver bleeding. In future research, the better real-time registration of 3D images should be achieved through the improvement of computer hardware technology and the search for higher performance and faster 3D image registration methods, so as to provide a strong technical foundation for precision medicine and clinical application.