Virtual and Real Bidirectional Driving System for the Synchronization of Manipulations in Robotic Joint Surgeries

Abstract

Surgical robots are increasingly important in orthopedic surgeries to assist or replace surgeons in completing operations. During joint surgeries, the patient’s joint needs to be adjusted several times by the surgeon. Therefore, the virtual model, built on the preoperative medical images, cannot match the actual variation of the patient’s joint during the surgery. Conventional virtual reality techniques cannot fully satisfy the requirements of the joint surgeries. This paper proposes a real and virtual bidirectional driving method to synchronize the manipulations in both the real operation site and the virtual scene. The dynamic digital twin of the patient’s joint is obtained by decoupling the joint and dynamically updating its pose via the intraoperative measurements. During surgery, the surgeon can intuitively monitor the real-time position of the patient and the surgical tool through the system and can also manipulate the surgical robot in the virtual scene. In addition, the system can provide visual guidance to the surgeon when the patient’s joint is adjusted. A prototype system is developed for orthopedic surgeries. Proof-of-concept joint surgery demo is carried out to verify the effectiveness of the proposed method. Experimental results show that the proposed system can synchronize the manipulations in both the real operation site and the virtual scene, thus realizing the bidirectional driving.

1. Introduction

1.1. Background and Significance

Surgical robots are increasingly utilized to assist or replace surgeons in completing surgery operations [1,2,3]. Their high-precision operation capability and reliability are very attractive in surgeries [4,5]. In clinics, each surgeon needs to be trained for years so as to acquire qualifications. On the contrary, robots do not have to be trained for such a long time, and the program of one robot can be easily transferred to another robot. The most representative surgical robot is the Da Vinci surgical robot, which can be used in a variety of surgeries [6,7,8]. For spinal surgery robots, the Mazor X from Medtronic has improved flexibility and working range, and can position individual vertebral segments [9,10]. The ROSA robot from MedTech is used for spinal surgery, and it can adjust the position of the robot according to the patient’s breathing [11]. For the knee joint surgery, the representative one is the ROBODOC surgical robot developed in the 1990s [12]. It can complete the surgical operation according to the planned robot trajectory. The CASPER surgical robot can assist surgeons in the positioning and installation of the prostheses during joint surgery [13]. The MAKO [14,15] surgical robotic system uses haptics to guide the surgery, and defines the surgical field based on the preoperative medical images, ensuring that the cutting does not cross the defined boundaries.

Medical images are important for surgical robots as the surgery planning and intraoperative registration all depend on the medical images of the patient. Digital twin is the virtual analogy of the physical object, which can be used to simulate [16], analyze [17], and monitor the physical object [18]. It can be used in surgeries so as to facilitate the surgeons in preoperative planning or intraoperative surgery [19]. For instance, digital twin-based systems were developed to accurately predict the diameter of the tumor [20], the adverse side effects caused by the deep brain stimulation in the subthalamic nucleus [21], and the length of stay in patients undergoing appendectomy [22]. Virtual reality (VR) and augmented reality (AR) are popular technologies that facilitate preoperative planning and surgical training [23]. VR technology is mainly used for surgical planning. For example, in dental surgery, the surgical planning can be finished after the mandible and maxilla are reconstructed from the preoperatively scanned CT images [24].

In orthopedic surgeries, the virtual model of the patient’s bone can be reconstructed from the preoperatively obtained computed tomography (CT) images. However, the CT images are static, which cannot fully satisfy the requirements of the joint surgeries. For a single bone, a single model is enough to describe its position and orientation. However, for joints, the two adjacent bones can rotate around the joint center. The locomotion of one bone often drives the locomotion of the adjacent bones. Therefore, the single model reconstructed from the static CT images cannot match the joints well if the surgeon adjusts the patient’s joint during surgery. For instance, in total knee arthroplasty (TKA), the surgeon will move the patient’s knee joint back and forth at different stages of the surgery so as to facilitate the bone cutting. The relative relationship between the femur and tibia will change accordingly. Therefore, the virtual model constructed from the CT images cannot reflect the actual variations of the poses of the patient’s joints. As the surgery plan for the surgical robot is determined preoperatively, the misalignments between the preoperative images and the intraoperative poses of the joints are fatal to medical robots. It is desired that the preoperatively defined surgical plan can be transferred and mapped to the actual pose of the patient’s joint, similar to an experienced surgeon.

In robotic surgeries, there exist two parallel spaces: the real operation site and the corresponding digital twin. Manipulations can be made in either space, and it is desired that the other space can be synchronized accordingly. The mappings between the real objects and the virtual models can be briefly classified into the following two categories: real to virtual mapping and virtual to real mapping. In real to virtual mapping, different sensing and imaging techniques are utilized to dynamically monitor the variation of the real objects in the operation site [25,26,27,28]. Intraoperative measurements are important in the updating of the virtual models [29,30,31]. Different methods have been proposed to finish the real to virtual mapping. For instance, P. Haigron et al. used the visual information to update the angioscope position in the virtual scene [32]. S. Moccia proposed an anatomical structure classification and image labeling method driven by multispectral imaging data [33]. S. Sefati et al. proposed a data-driven learning method for remote position estimation of the robots [34]. Neural network was also used to synchronize the virtual model [35,36]. In virtual to real mapping, the manipulations in the virtual scene can be used to drive the real surgical robot or provide guidance to the surgeon, e.g., AR [37] and the tele-surgical system [38]. J. Guo et al. used deep learning algorithms to drive the catheter to the designated position during vascular interventional surgery [39]. Y. Li et al. proposed a sensorless grip strength estimation method based on a dynamic model [40].

The above research studies are unidirectional mapping in nature, which is not adequate for the fully intelligent joint surgeries. This paper proposes a virtual and real bidirectional driving system for the synchronization of manipulations in robotic joint surgeries. As shown in Figure 1, the overall system includes the real operating room (detailed information provided in Section 2.1) and its digital twin in a virtual scene. During surgery, an optical tracking system (OTS) is utilized to dynamically monitor the surgical tool and the patient’s joint. When the surgeon adjusts the patient’s joint or manipulates the devices in the operating room, the virtual models will be updated according to the measurements of the OTS, thus enabling the real to virtual mapping. The surgeon can intuitively see the specific position of the patient and surgical tool through the visual feedback of the system. In the virtual to real mapping, the surgeon can manipulate the virtual surgical tool in the virtual scene. This virtual manipulation or preoperative trajectories can be sent to the operation robot’s controller such that the operation robot can reproduce the surgeon’s manipulation. Further, during the intraoperative adjustment of the patient’s joint, the preoperatively planned desired poses of the joint can be reprojected to the surgeon as a visual guidance. Different from the unidirectional mappings, both the manipulations in the real operation site and the virtual scene can be synchronized.

1.2. Aims and Contributions

This paper mainly focuses on the synchronization between the real operation site and its digital twin. The contributions can be summarized as follows: (1) This paper presents a novel strategy to obtain the dynamic digital twin of the patient’s joint from the operatively obtained CT images by decoupling the patient’s joint and using intraoperative position feedback. (2) For joint surgeries, the adjustments and locomotion of the joint can be dynamically updated in the virtual scene such that the preoperative surgery plan can always match the dynamic variations of the joint during surgery. (3) The preoperative surgery plan can be dynamically modified according to the dynamic digital twin of the patient’s joint. The updated plan and the manipulations on the virtual models can then be re-projected to the operation robot and the surgeon, enabling highly intuitive and efficient human–computer interactions.

This research work provides a new solution to the establishment of the dynamic digital twin of the patient’s joint, which can dynamically transfer the preoperative planning to the intraoperative pose of the joint when the joint experiences adjustments or locomotion. In addition, the synchronization between the real operating site to its digital twin provides a powerful human–computer interaction for the surgeon. These are very important to increase the efficiency of the surgical robots in joint surgeries.

2. Materials and Methods

2.1. Universal Robotic Surgery System Orthopedic Surgeries

This paper proposes a dual-robot system for orthopedic surgeries. Different from the conventional surgical robot system, an additional navigation robot is introduced to dynamically adjust the pose of the OTS during surgery, i.e., active navigation. The accuracy of the OTS is influenced by the observation pose of the OTS to its targets. For the multiple targets utilized in orthopedic surgeries, it is important to find the optimal pose of the OTS so as to guarantee the navigation accuracy of all the targets. In addition, the surgeon, the staff, and the medical instruments might block the sight of the OTS if it is stationary. With the help of the navigation robot, the OTS can be adjusted before the occlusion occurs, thus guaranteeing consistent navigation throughout the surgery.

The proposed robotic surgery system mainly includes a navigation module, a surgical operation module, and a bidirectional driving module. The navigation module includes an OTS mounted on the flange of an active navigation robot. The surgical operation module includes the surgical tools mounted on the flange of an operation robot. The bidirectional driving module mainly includes a large screen and a host PC for data and image processing and transmitting.

As shown in Figure 2, the universal robotic surgery system prototype is developed for knee surgeries. Two 7-degree-of-freedom (7-DOF) robots (Model Panda from Franka Emika) are used as the operation and navigation robots. A medical oscillating saw (Model JT-II from BAIDE) is used as the surgical tool. The OTS (Model Polaris Vega from NDI) is used to construct the navigation module. The bidirectional driving module is developed using C++ and VTK (The Visualization Toolkit) library on the Windows platform.

In the navigation module, the OTS is used to define the coordinate system of the surgical site, and the navigation robot is used to adjust the pose of the OTS when necessary. In the surgical operation module, different surgical tools can be mounted on the flange of the operation robot for different surgeries. Preoperative calibration is necessary to calibrate the parameters of the overall system, e.g., the transformation relationships among the robot and its flange, the tracked tool, and the tip of the surgical tool. Thus, the surgical tool can be precisely positioned to the predefined location. In the bidirectional driving module, a host PC dynamically monitors the changes of the operating room via the navigation module, and a large screen is used to display the digital twin of the operation site. When the position and orientation of an object in the operating room change, the virtual scene can be automatically updated. During surgery, if the virtual patient or the virtual surgical tool is manipulated by the operator, this information can also be sent to the surgeon or the operation robot such that the patient and surgical tool in the operating room can follow the virtual models.

During robotic surgery, tracked tools are fixed on the patient’s joint, the surgical tool, etc. The OTS is used to obtain the pose information of the tracked tools in the OTS coordinate system {OT}. A global coordinate system {W} is defined as the base coordinate system of the operating room and the virtual scene. The coordinate system of the preoperative CT image is defined as {CT}. The coordinate systems of the tracked tools in the images are defined as {JTn}. The base coordinate system of the operation robot is defined as {Base}, and its flange coordinate system is defined as {Flange}, and the tracked tool coordinate system on the robot is defined as {RT}. The coordinate system of the surgical tool is defined as {Tool}, and its origin locates at the tip.

2.2. Bidirectional Driving Method

2.2.1. Intraoperative Registration

Registration is a necessary step to derive the positions and orientations of the patient’s joint and the surgical tool via the measurements of the tracked tools. Both the marker-based and marker-free registrations can be implemented for the registration.

Marker-based registration can provide a high-precision registration, which is beneficial to improving the accuracy of the overall system. However, the implant of the markers introduces extra injury to the patient’s bones. The marker-based registration proceeds as follows: First, a data set containing images of both the patient’s joint and the tracked tool is obtained. The marker balls on the tracked tool are visible in both the CT images and the OTS, whose center points are used for the registration. The transformation matrix (TM) between {CT} and {JT_n} can be expressed as ${}_{JTn}^{CT}$T. ${}_{JTn}^W$T is known, and thus the TM between {CT} and {W} can be derived using

$${}_{CT}{}^{W}T={}_{J{T}_{\mathrm{n}}}{}^{W}T\cdot {}_{J{T}_n}{}^{CT}T{}^{-1}={}_{J{T}_{\mathrm{n}}}{}^{W}T\cdot {}_{CT}{}^{J{T}_{\mathrm{n}}}T$$

(1)

In order to obtain a dynamic digital twin of the patient’s joint, the joint is decoupled to separate bones. Each bone is loaded into the virtual scene separately, and the above registration is repeated. Therefore, the digital twin of the patient’s joint can be dynamically updated according to the measurements of the tracked tools fixed to the bones of the patient’s joint.

If marker-based registration is not applicable, marker-free registration can be implemented instead. The advantage of marker-free registration is that no markers are implanted into the patient’s bone during the CT scanning, whereas the registration accuracy is slightly lower than marker-based registration. In marker-free registration, the point cloud of the patient’s joint is obtained from the CT images. During surgery, after the joint is exposed, a probe of the OTS can be used to obtain the point cloud of the bone surface. These two sets of point clouds can be utilized for registration. The relationship of {CT} and {W} can be calculated using the N-point registration method. After registration, the joints in the CT images can be aligned to the operation site.

Similarly, the relationship between the tracked tool and the tip of the surgical tool needs to be calibrated and registered. This TM can be calculated using

$${}_{Tool}{}^{Base}T={}_{Flange}{}^{Base}T\cdot {}_{RT}{}^{Flange}T\cdot {}_{Tool}{}^{RT}T$$

(2)

where ${}_{Flange}^{Base}$T is the TM between the robot base and the flange, ${}_{RT}^{Flange}$T is the TM between the flange and the tracked tool, and ${}_{Tool}^{RT}$T is the TM between the tip of the surgical tool and the tracked tool.${}_{Flange}^{Base}$T is available in the operation robot’s firmware, ${}_{RT}^{Flange}$T can be calibrated using the classic eye-to-hand calibration algorithm [41], and ${}_{Tool}^{RT}$T can be measured using the probe of the OTS.

In addition, according to the above equation, ${}_{RT}^{Base}$T can also be calculated. Since the OTS locates the tracked tool at the end of the robot, ${}_{RT}^W$T can be calculated by the world coordinate system and the tracked tool at the flange of the robot. The representation of the robot base in the world coordinate system is obtained.

$${}_{Base}{}^{W}T={}_{RT}{}^{W}T\cdot {}_{Base}{}^{RT}T={}_{RT}{}^{W}T\cdot {}_{RT}{}^{Base}T{}^{-1}$$

(3)

According to the known TMs, the representation of the tool coordinate system in the world coordinate system can be obtained using

$${}_{Tool}{}^{W}T={}_{Base}{}^{W}T\cdot {}_{Tool}{}^{Base}T$$

(4)

Based on the above TMs, the virtual objects can be dynamically updated via the measurements from the OTS. Therefore, the connections between the real and virtual are established.

The effectiveness of the proposed method is tested on the developed robotic surgery system prototype. For TKA, based on our previous work [42], the registration of the phantom’s femoral and tibia proceeds as follows: Firstly, both the phantom and the tracked tools are scanned to obtain the CT images. Secondly, the femur and tibia images are decoupled into two data sets and labeled separately. Figure 3a shows the reconstructed femur model. During the operation, the OTS observes the tracked tools and aligns the preoperative CT images. The registration result is shown in Figure 3b, and the registration accuracy is below 0.1 mm, as is shown in

Table 1. The distance error (unit: mm).

Distance	Ground-Truth	Calculated	Error
A-C	88.00	87.99	0.01
A-B	50.00	50.07	0.07
A-D	60.00	59.96	0.04

.

The desired trajectory in the world coordinate system {W} should be transformed into the base coordinate system of the operation robot {Base}. Therefore, it is necessary to calibrate {W} and {Base}. In the experiment, the OTS is used as the position and orientation feedback. The robot forward kinematics is known. In the developed prototype system, the TM between {Base} and {W} is calibrated and given as follows:

$${}_W{}^{Base}T{}_1={}_{NDI}{}^{Base}T{}_1\cdot {}_W{}^{NDI}T{}_1=\left[\begin{array}{cccc}0.73386& 0.030063& -0.67863& 943.75\\ -0.026655& 0.99953& 0.015454& 786.47\\ 0.67878& 0.0067475& 0.73431& 245.34\\ 0& 0& 0& 1\end{array}\right]$$

(5)

The transformation relationship between the flange and the tracked tool on the surgical tool also needs to be calibrated. The same method is used for this calibration with the help of the OTS and the probe. The obtained TMs are shown in the following equations.

$${}_{RT}{}^{Flange}T{}_1=\left[\begin{array}{cccc}0.7742& 0.4966& -0.3925& 68.2000\\ -0.6326& 0.5864& -0.5059& 40.5000\\ -0.0211& 0.6400& 0.7681& 16.4000\\ 0& 0& 0& 1\end{array}\right]$$

(6)

$${}_{Tool}{}^{RT}T{}_1=\left[\begin{array}{cccc}0.7742& -0.6326& -0.0211& -89.0500\\ 0.4966& 0.5864& 0.6399& 96.0700\\ -0.3925& -0.5060& 0.7681& 275.0600\\ 0& 0& 0& 1\end{array}\right]$$

(7)

2.2.2. Real to Virtual Mapping

During the joint surgeries, the patient’s joint needs to be moved to several different poses so as to facilitate the specific operations. The TKA is selected as an example of real to virtual mapping. As shown in Figure 4a–c, the position of the patient’s knee joint needs to be adjusted so as to facilitate the oscillating saw’s cutting. The OTS captures the locomotion of the femur and tibia and sends the measurements to the host PC to drive the virtual femur and tibia. In this manner, the synchronization between the real and the virtual is realized.

The preoperative images are used to build the 3D model of the patient’s joint in the virtual scene. The position information is the representation of the tracked tool {JT_n} under the OTS {OT}, which can be expressed using ${}_{JTn}^{OT}$T. The world coordinate system {W} is located on the operating table. Its representation is also expressed under the OTS {OT}, i.e., ${}_W^{OT}$T. The inverse transformation of the TM is used to obtain the representation in the world coordinate system {W}, as shown below:

$${}_{OT}{}^{W}T={}_W{}^{OT}T{}^{-1}$$

(8)

According to Equation (8), each tracked tool can be represented in {W}. For example, {JT_n} relative to {W} can be expressed as ${}_{JTn}^W$T by Equation (9). The OTS also obtains the position of the tracked tool on the robot, which is denoted as ${}_{RT}^W$T.

$${}_{J{T}_{\mathrm{n}}}{}^{W}T={}_{OT}{}^{W}T\cdot {}_{J{T}_{\mathrm{n}}}{}^{OT}T$$

(9)

In order to evaluate the accuracy of the real to virtual mapping, a pink ball is placed at the tip of the surgical tool, which is defined as the origin of the surgical tool coordinate system. Subsequently, the tip of the surgical tool is moved towards the marker ball on the tracked tool of the femur, as shown in Figure 5b. The surgical tool is moved until the tip touches the surface of the marker ball. Based on the feedback of the OTS, the pink ball in the virtual scene also touched the surface of the marker ball. This validates the calibration and registration of the robotic surgical system.

2.2.3. Virtual to Real Mapping

In this paper, the manipulations on the virtual objects can also be used to drive the operation robot and to provide visual guidance to the surgeon during the manual adjustment of the joint. The virtual to real mapping can be mainly divided into the following two parts:

Virtual manipulation on the operation robot: Prior to operation, the surgical tool is placed in the initial position. During operation, it is inevitable that the surgical tool will be moved to several different locations to cut the bone along different planes. However, the space around the operation table is crowded. If the surgical tool is directly manipulated by the surgeon or moved by the operation robot, the surgical tool or the operation robot may cause injuries to the patient, the surgeon, or the intraoperative devices. Instead, the surgeon can manipulate the virtual surgical tool and carefully verify the safety of operation in the virtual scene. Subsequently, the position and orientation information of the verified safe operation in the virtual scene is extracted and sent to the operation robot so that it can move the surgical tool to its destination.

Visual guidance for the adjustment of the patient’s joint: The virtual to real mapping can also be used to assist the surgeon in the adjustment of the patient’s joint by providing a visual guidance. In some surgeries, such as TKA, the surgeon must move the patient’s knee joint to several different positions to facilitate the bone cutting. In preoperative planning, the desired positions of the patient’s joint and the surgical tool at different operation stages can be specified. During the surgery, both the current position of the patient’s joint and its destination are displayed on the screen. With the help of the visual guidance, the surgeon can easily move the joint to the correct position.

3. Results

3.1. The Real to Virtual Mapping

For the real to virtual mapping, a lower limb phantom is used to simulate the patient’s femur and tibia. Before the experiment, a tracked tool is rigidly connected to the surgical tool so as to provide position and orientation feedback. For TKA, two tracked tools are fixed on the phantom’s femoral and the tibia separately. The tracked tools fixed on the phantom can be used in the registration.

The 3D models of the phantom’s femur and tibia are loaded into the virtual scene. When the phantom is adjusted, the relationship between the two bones changes. The virtual models also change accordingly. In order to facilitate the observation of the movement of the model, some important reference coordinate systems are displayed in the virtual scene, as shown in Figure 4a. When the position of the femur is continuously adjusted, it can be observed that the virtual femur model can well reproduce the locomotion of the real femur. The results of the experiment are shown in Figure 4b,c.

In order to evaluate the accuracy of the real to virtual mapping, the tip of the surgical tool is placed at 10 different positions. Figure 6 shows these points in the world coordinate system. The blue dots represent the points in the real operation site, and the red circles represent the points in the virtual scene. It can be found that the error of the real to virtual mapping is on the order of 1–2 mm, as shown in

Table 2. Ten positions of the surgical tool in the virtual and real scenes and the errors. (Unit: mm).

Group ID	Coordinates of the Points			Error	Group ID	Coordinates of the Points			Error
	x	y	z			x	y	z
1	−239.613	−418.146	342.410	2.074	6	−169.222	−459.504	333.856	1.556
	−239.347	−416.181	341.804			−168.376	−458.199	333.909
2	−57.909	−392.237	54.354	0.796	7	−154.900	−531.637	474.282	1.653
	−58.599	−392.486	54.045			−155.192	−530.049	473.930
3	−148.851	−427.016	294.940	1.967	8	3.228	−498.840	196.662	0.638
	−148.436	−426.605	293.062			3.384	−498.817	196.044
4	−18.036	−220.221	79.7501	2.660	9	17.274	−585.962	138.410	2.383
	−15.481	−219.647	79.284			17.820	−584.000	139.648
5	−320.424	−486.398	335.061	1.799	10	−5.848	−389.018	190.731	1.707
	−321.172	−485.076	334.097			−7.243	−388.035	190.715

. The average error is calculated to be 1.723 mm.

The cutting paths of the surgical tool are preoperatively planned in the {CT} coordinate system. Figure 7a shows the surgical plan when the knee joint is flattened, where the blue plane is the planned cutting plane and the segments 1–14 are the planned cutting paths. During surgery, if the surgeon adjusts the pose of the knee joint, the planned cutting paths should follow the movement of the knee joint, or mismatch will occur. Since the joint is decoupled, in real to virtual mapping, the system can achieve synchronization of the planned cutting paths during the surgery. As shown in Figure 7b, when the knee joint is bent to a different pose, the preoperatively planned cutting paths can be dynamically updated. This can guarantee that the surgical robot can always move the surgical tool to the correct position and precisely complete the surgical plan after the surgeon adjusts the joint.

3.2. The Virtual to Real Mapping

The adjustment of the knee joint in TKA is adopted to evaluate the virtual to real mapping. The cutting of the femur is selected as an example. In preoperative planning, for each cutting plane, the desired pose of the joint is specified. As shown in Figure 8(a1,a2), during intraoperative operation, both the reference position (marked in pink color) and the current position (marked in grey color) of the femur are displayed on the screen. In this manner, the difference between the current position and its destination can be intuitively shown to the surgeon. Subsequently, the surgeon adjusts the knee joint with the visual guidance. Figure 8(b1,b2) shows the knee joint after adjustment, where the insets show the relationship of the actual knee joint position and the reference position. It can be observed that the knee joint almost coincides with the references. The surgeon does not have to adjust the knee joint to exactly coincide with the reference position. The operation robot can move the surgical tool to the correct position, as the real operation site and the virtual scene are synchronized. This helps to reduce the requirements on the surgeon’s experiences in the adjustment of the patient’s joint.

To evaluate the virtual manipulation of the operation robot, the relationship between the patient’s knee and each coordinate system is registered and calibrated preoperatively. As shown in Figure 9, the virtual knee model and all the coordinate systems are loaded into the virtual scene. The virtual surgical tool coordinate system is represented as a point. The preoperatively defined trajectories of the surgical tool’s tip can be loaded into the virtual scene. Alternately, the surgeon can also directly manipulate the virtual surgical tool to set a desired trajectory for the virtual tool. The position and orientation information of the virtual surgical tool in the virtual scene is sent to the host. The developed system calculates the related position in the {Base} to guide the operation robot in the real operating room. As shown in Figure 9a, a standby position is defined for the surgical tool in the virtual scene. Subsequently, the real surgical tool is moved to this standby position by the operation robot.

Similarly, in order to evaluate the accuracy of the virtual to real mapping, 10 different points are assigned in the virtual scene for the tip of the surgical tool. Subsequently, the locations of these 10 points are sent to the operation robot so as to move the real surgical tool’s tip to the defined points. Figure 10a shows the positions of the virtual points (red circles) and the actual points (blue dots). There exist small discrepancies between the virtual and real points.

Table 3. Ten positions of the surgical tool in the virtual and real scenes and the errors. (Unit: mm).

Group ID	Coordinates of the Points			Error	Group ID	Coordinates of the Points			Error
	x	y	z			x	y	z
1	−348.460	−17.688	166.969	5.951	6	−280.836	−201.004	159.612	7.203
	−352.806	−15.579	170.445			−286.582	−199.388	163.643
2	−305.445	−153.240	164.060	6.677	7	−268.969	−133.468	150.278	6.518
	−310.590	−151.312	167.853			−273.956	−131.776	154.119
3	−311.214	−157.672	217.823	6.728	8	−356.130	−55.900	223.897	6.018
	−316.372	−155.790	221.712			−360.498	−53.494	227.266
4	−275.199	−219.73	223.532	6.955	9	−294.561	−15.104	170.007	9.143
	−280.684	−217.969	227.429			−296.504	−13.204	178.737
5	−274.236	−215.848	184.939	7.119	10	−269.176	13.186	262.541	8.202
	−279.947	−214.171	188.845			−269.280	15.594	270.381

shows the numeric assessment of all the errors for each point. The average error is found to be 7.052 mm.

4. Discussion

Many existing VR and AR methods can also provide promising results. The current state of the art in real to virtual mapping can achieve high-precision, real-time monitoring intraoperatively. In virtual to real mapping, many virtual surgical systems use haptic devices to simulate the surgery for the training of surgeons. These methods can be defined as unidirectional mapping, where the synchronization between the real objects and the virtual models is not pursued. However, for joint surgeries, this unidirectional mapping is not adequate because the pose of the patient’s joint might be adjusted. In this “dynamic” scenario, it is necessary to synchronize the real operation site and its digital twin. Therefore, the proposed bidirectional mapping is an enabling technique for the synchronization between the real operation site and its digital twin. On the one hand, it is necessary to dynamically update the virtual joint to the real joint such that the preoperatively planned trajectories of the surgical robot can be dynamically updated so as to match the locomotion of the joint. On the other hand, it is also necessary that the preoperatively defined adjustments of the joint or the manipulations on the virtual models can be directly sent back to the surgeon or to the surgical robot so as to improve the efficiency.

Experimental results have shown that the synchronization between the real operation site and its digital twin has been realized. However, the registration accuracy of the overall system needs to be further improved.

In the real to virtual mapping, the registration accuracy is found to be around 1.723 mm. The RMS error of the OTS used in this paper is on the sub-millimeter magnitude. The calibration used in the real to virtual mapping is the TM of ${}_{Tool}^{RT}$T. In calibration, a probe is used to locate the tip of the surgical tool in {RT}. As a result, the majority of the error might come from the inaccuracy in the manual alignment of the probe’s axis to the surgical tool.

For the virtual to real mapping, several points in the virtual scene are selected as destinations. The information is sent to the operation robot so that the operation robot can move the tip of the surgical tool to the destinations. The average error in the virtual to real mapping is found to be 7.052 mm, which is relatively large. Similar to the real to virtual mapping, the inaccuracy of the manual alignment of the probe’s axis to the surgical tool will be included. The calibration errors of ${}_{Flange}^{RT}$T and ${}_{Base}^W$T will also be included in the virtual to real mapping. In addition, for the operation robot, sub-millimeter repeatability has been realized, whereas the rigorous calibration of its DH parameters has not been carried out. The nominal kinematics is adopted during the calibration and registration. This might increase the calibration error of the overall system.

5. Conclusions

For the safety of robotic orthopedic surgeries, it is important to dynamically monitor the locomotion of the patient and the surgical tool and synchronize the operations in the real operation site and its virtual model. This paper presents a virtual and real bidirectional driving system, where the dynamic digital twin of the patient’s joint is established from the operative CT images. The patient’s joint is decoupled into separate bones and the intraoperative OTS is used for the dynamic update of the pose of the joint and the surgical tool.

In the developed system, the adjustments and locomotion of the joint can be dynamically updated in the virtual scene such that the preoperative surgery plan can always match the dynamic variations of the joint during surgery. The updated surgery plan and the manipulations on the virtual models can then be re-projected to the operation robot and the surgeon, enabling highly intuitive and efficient human–computer interactions.

For the virtual to real mapping, the preoperatively defined poses for the patient’s joint and the trajectories for the surgical tool can be loaded into the virtual scene. The surgeon can also directly manipulate the virtual surgical tool instead of the real surgical tool. All the preoperatively defined trajectories and the intraoperative manipulation on the virtual objects can be used to drive the real operation robot in the manipulation of the surgical tool. In addition, the system can provide visual guidance for the surgeon in the adjustment of the patient’s joint, especially for joint surgeries.

TKA is adopted as the typical orthopedic surgery. Tests are conducted on lower limb phantoms. Experimental results show that the developed system can realize the synchronization between the real operating room and the virtual scene. Therefore, the developed bidirectional driving system is feasible for robotic orthopedic surgeries.

In the future work, more efforts will be directed towards the improvement of the registration and calibration accuracy of the overall system, where the influence of the robot’s kinematics, the registration method, and the OTS will be systematically investigated. In addition, more experimental verifications will be conducted to fully test the precision, capability, and feasibility of the developed system in completing orthopedic surgeries.