Haptic Interface with Twin-Driven MR Fluid Actuator for Teleoperation Endoscopic Surgery System

Abstract

Magnetorheological fluids (MRFs) are composite materials made of ferromagnetic particles, medium oils, and several types of additives. We have developed an actuation system for the fine haptic control of leader-follower robots. In this study, we developed a haptic interface with two link-type twin-driven MR fluid actuators and two MR fluid brakes for a teleoperation endoscopic surgery system and conducted evaluation tests for a remote operational task with a leader-follower robot system. For evaluations, we adopted the NASA-TLX questionnaire as a subjective assessment method. According to the experimental results, the total success rates were 0.462, 0.333, and 0.591, for the first haptic, middle no-haptic, and second haptic phases, respectively. The force information of the haptic forceps helped users to perceive grasping sensation on their fingers. Statistical analyses on the answers to the questionnaire indicate no significant differences. However, a decreasing tendency in the mental stress in the complicated manipulation tasks for fragile objects is observed.

1. Introduction

Many types of remote-controlled systems [1] have been proposed for various environments (e.g., handling of radioactive material and underwater robots) [2]. Recently robot-aided surgical systems with remote-controlled robots [3] are the most successful in their applications. The control method for the conventional surgical robot is categorized in a direct control method [1]. In this method, the remote site system follows the motion of the user interface manipulated by an operator. In this method, the local and remote sites are called the leader and follower systems, respectively.

The simplest way to accomplish the leader-follower control is via a unilateral control, in which the reference signals recorded at the leader system are unilaterally sent to the follower system, without haptic feedback to the leader system. In this method, the total control system is substantially simple and stable. However, the performance of each task significantly depends on the skill of the operator. Conversely, bilateral controllers [4,5] have been utilized to perform force feedback functions in remote-controlled systems. By adding a force/position feedback loop, the machine can support human skills with gravity compensation, friction compensation, and better haptic perception while performing a remote operation task. These functions are very useful in enhancing operational accuracy and reducing psychological stress from the human operator in very sensitive operations.

Regarding the remote-controlled surgical robot, the unilateral controller has been used in the most popular surgical robot, da Vinci Surgical System^®, with less haptic feedback for surgeons [6]. For example, evidence of increasing tendencies for concomitant medical problems in laparoscopic surgeries has been reported [7], and robot-aided surgery is a solution for safe operations [8]. However, concomitant medical problems have been reported in such robot-aided surgeries due to unexpected pressure of robotic tools for target/non target organs [9,10]. In addition, the learning curve of robotic surgery tends to be longer than that of other methods [11]. Hence, the haptic feedback function has been heavily demanded by surgeons to enhance operational functions and reduce psychological stresses.

To address the aforementioned requirements for the haptic feedback, part of recent surgical robots (e.g., Senhance^® Telerobotic System [12]) have been released with the haptic feedback function on their interfaces. Nevertheless, this haptic information was utilized to understand situations, such as conflictions of the forceps. However, the fine haptic feedback for elucidating the existence of organs has not been achieved for the surgical system. Such functions are beneficial in reducing unexpected pressure on the organs and avoiding concomitant medical problems. Therefore, establishing a high-performance force feedback system for remote-controllable robot systems is necessary to enhance operational accuracy and reduce the psychological stress on operators.

Conventional haptic devices generally use low inertia motors, e.g., coreless motors. Coreless motors have a high torque-to-inertia ratio, as a suitable characteristic for haptic devices. However, their directly controllable torque is generally small. To improve the quality of haptic information in the robotic surgery system, we developed a twin-driven magnetorheological fluid (MRF) actuator (TD-MRA) [13] as a key component of the haptic interface. In this actuator, two high-performance MRF clutches [14] are utilized for torque transmission. MRFs are suspensions of ferromagnetic particles, medium oils, and several types of additives, and their rheological properties change rapidly, stably, and repeatedly when magnetic fields are applied [15]. Their dynamic force properties are suitable for their applications on the fine haptic devices [16].

Fauteux et al. [17] developed a twin-driven MRF actuator using planetary gears, and Song et al. [18] adopted the same mechanism as a haptic master. The twin-driven mechanism has been used to reduce the basic friction of actuators, and its low friction characteristics are suitable for haptic devices. However, it is difficult to balance the frictional (and viscous) torque of the normal and reverse rotations of the planetary-gear-based mechanism with the MRF device, owing to the difference in rotational velocity between the normal and reverse rotations. Hence, our TD-MRA adopted two H-MRCs to reduce the inertia and friction, as well as a flat gear mechanism to reduce unbalanced torques. In a previous study [13], we developed two types of TD-MRAs, using belt transmission (Belt-type TD-MRA) and parallel linkage (Link-type TD-MRA). Although both types have pros and cons, their torque/inertia ratio was up to 10,000 Nm/kgm², which is at the same level as coreless motors. In addition, the maximum direct torque is from several tenfold to several hundredfolds. In this study, we developed a haptic interface with link-type TD-MRAs for teleoperation endoscopic surgery systems, including a leader-follower system. Furthermore, we conducted an evaluation test for a remote operational task with the developed system. For evaluations, we applied the NASA-TLX [19,20] as a subjective assessment method, as well as quantitative evaluation on the success rate.

2. Haptic Interface for Tele-Operational Endoscopic Surgery and Leader-Follower Systems

To investigate the effect of the haptic information in teleoperational endoscopic surgeries, we developed a haptic interface using the linkage type TD-MRA [13] and specific structure for the motion of endoscopic operations. In this section, this system is described.

2.1. Basic Structure of the Haptic Interface

Figure 1 and Figure 2 show the haptic interface with the link-type TD-MRAs for the teleoperation endoscopic surgery systems (H-MRD for ES). The right-hand side of Figure 1 presents the originally developed haptic device. The left-hand side illustrates a commercially available endoscopic simulator. The H-MRD for ES comprises two TD-MRAs, two additional H-MRCs, a gimbal structure for the roll-pitch-yaw rotation, a linear slider for insertion, and haptic forceps. The gimbal core is the center of this structure, and it generates a roll-pitch-yaw motion on a pivot center. One of the TD-MRAs is connected to the roll axis, while the other TD-MRA is connected to the pitch axis. The yaw axis is a passive axis connected with a rotational encoder. The linear slider is attached to the yaw axis and connected with a H-MRC via a belt-pulley mechanism. This MRC generates a passive resistant force on the insertion motion. The haptic forceps attached to the linear slider comprise a potentiometer and another H-MRC for sensing the gripping motion and generating gripping resistance. The specifications of the H-MRD for ES are listed in

Table 1. Specifications of the H-MRD for ES.

Item	Value
ROM for roll	−30°–60°
ROM for pitch	−45°–45°
ROM for yaw	−45°–45°
Maximum reaction force on the handgrip	3 N (min)
ROM of gripper	−15°–15°
Maximum reaction torque for griping	0.64 Nm

.

2.2. Kinematics and Statics of the Haptic Interface

The transformation from the controlled angle of the actuators to the position of the end-effector is performed with the kinematics of the mechanism. The left-hand side of Figure 3 presents the input variables and ground coordinate system (${\mathsf{\Sigma }}_G(O_G-x_G-y_G-z_G)$) of the mechanism. The figure on the right-hand side of Figure 3 shows the relationship between the ground coordinate system and that of the end-effector (${\mathsf{\Sigma }}_E(O_E-x_E-y_E-z_E)$). The origin of the ground coordination system ($O_G$) is defined as the center of the gimbal mechanism, $x_G$ in its left-hand direction, and $y_G$ as its forward. We also defined the gimbal fore coordinate system (${\mathsf{\Sigma }}_1(O_1-x_1-y_1-z_1)$) and the yaw part coordinate system (${\mathsf{\Sigma }}_2(O_2-x_2-y_2-z_2)$) as middle coordinate systems. The origins, $O_1$ and $O_2$, are defined as the same position of $O_G$. The rotational matrix from ${\mathsf{\Sigma }}_G$ to ${\mathsf{\Sigma }}_1$ is defined as:

$$\begin{gathered} {}_{}{}^{G}R{}_1(\alpha ,\beta ) \\ =\left[\begin{array}{c}\begin{array}{c}\cos (\beta )\\ \end{array}\\ 0\\ \begin{array}{c}\\ -\sin (\beta )\end{array}\end{array}\begin{array}{c}\frac{\tan (\alpha )\sin (\beta )}{k}\\ \frac{\tan (\beta )\sin (\beta )+\cos (\beta )}{k}\\ \frac{\tan (\alpha )\cos (\beta )}{k}\end{array}\begin{array}{c}\frac{\tan (\beta )}{k}\\ -\frac{\tan (\alpha )}{k}\\ \frac{1}{k}\end{array}\right] \end{gathered}$$

(1a)

$$\mathrm{k}=\sqrt{ta{n}^2(\alpha )+ta{n}^2(\beta )+1}$$

(1b)

where, $\alpha$ and $\beta$ denote the pitch and roll angles, respectively. In this mechanism, the roll and pitch motions concurrently influence the motion of the gimbal core. After the roll-pitch motion, the rotational transformation from ${\mathsf{\Sigma }}_1$ to ${\mathsf{\Sigma }}_2$ is defined as:

$${}_{}{}^{1}R{}_2(\gamma )=\left[\begin{array}{ccc}\cos (\gamma )& -\sin (\gamma )& 0\\ \sin (\gamma )& \cos (\gamma )& 0\\ 0& 0& 1\end{array}\right]$$

(2)

where, $\gamma$ denotes the yaw angle. Next, the sliding motion toward the yaw axis from ${\mathsf{\Sigma }}_2$ to ${\mathsf{\Sigma }}_E$ is defined as:

$${}_{}{}^{2}P{}_E(S)={\left[\begin{array}{ccc}0& 0& S_0+S\end{array}\right]}^T$$

(3)

where, $S_0$ and $S$ represent the initial bias and sliding motions, respectively. Finally, the position of the end-effector on the ground (${}_{}{}^{G}P{}_E(\alpha ,\beta ,\gamma ,S)$) can be acquired as:

$${}_{}{}^{G}P{}_E(\alpha ,\beta ,\gamma ,S)={}_{}{}^{G}R{}_1(\alpha ,\beta ){}_{}{}^{1}R{}_2(\gamma ){}_{}{}^{2}P{}_E(S)={\left[\begin{array}{ccc}{}_{}{}^{G}P{}_{Ex}& {}_{}{}^{G}P{}_{Ey}& {}_{}{}^{G}P{}_{Ez}\end{array}\right]}^T$$

(4)

The relationship between the desired reaction forces on the end-effector of the haptic interface, $F={\left[\begin{array}{ccc}{F}_x& F_y& F_z\end{array}\right]}^T$, and the output torques of the actuators, $\tau ={\left[\begin{array}{ccc}{\tau }_R& {\tau }_P& {\tau }_Y\end{array}\right]}^T$, is defined as:

$$\left[\begin{array}{c}{\tau }_R\\ {\tau }_P\\ {\tau }_Y\end{array}\right]=\left[\begin{array}{c}{}_{}{}^{G}P{}_{Ex}{F}_z-{}_{}{}^{G}P{}_{Ez}{F}_x\\ {}_{}{}^{G}P{}_{Ez}{F}_y-{}_{}{}^{G}P{}_{Ey}{F}_z\\ 0\end{array}\right]$$

(5)

where, ${\tau }_R$ and ${\tau }_P$ represent the output torque of TD-MRA for the roll and pitch, respectively. The H-MRD for ES has no actuator in the yaw direction because ${\tau }_Y$ is consistently zero. This is triggered by the assumption that the radius of the virtual forceps is zero.

2.3. Basic Structure of the Follower System

The structure of the robot forceps is shown in Figure 4. The robot forceps was developed, comprising a liner actuator (DR20T1B02-AZAKR, ORIENTAL MOTOR CO., LTD., Tokyo Japan), load cell (LUX-B-50N-ID, KYOWA ELECTRONIC INSTRUMENTS CO., LTD., Tokyo Japan), and part of the training forceps (KOTOBUKI Medical). The open-close force on the tip of the forceps can be measured with the loadcell and fed back to the haptic gripper.

As shown in Figure 5, the robot forceps were installed at the end-effector of a 7 DOF robot arm (xArm7, UFACTORY, Shenzhen, China) with a six-axis force-moment sensor (PFS055YA251U6, Leptrino INC., Komoro, Japan). This robot, a single-arm robot for endoscopic surgery (SAR for ES), was adopted as a follower of the remote-controlled system. The robot arm is controlled to rotate the robot forceps on a defined pivot point using the H-MRD for ES. The SAR for ES completely follows the range of motion of the H-MRD for ES (Table 1).

2.4. Controller for the Leader-Follower System

Figure 6 illustrates the control system for the H-MRD in which a real-time controller (NI, PXIe-8821) was used as the main controller with a 1-ms sampling time. The information from encoders and a potentiometer are measured with a counter board (NI, PXIe-6612) and an A/D board (NI, PXIe-6355), respectively. The reference signals to the TD-MRAs are output to the current amplifiers with a D/A board (NI, PXIe-6738). A personal computer was used as the user interface to input/output information from or to users. The real-time controller and UI are connected to the TCP/IP communication.

Figure 7 shows the control systems of the SAR. The 7-DOF robot arm was controlled with a special control box, and the reference signal on the target position/orientation of the end-effector was sent to the UI via TCP/IP communication. The signals from the force-moment sensor to the linear actuator on the robot forceps communicate with the UI computer via serial communications. The analog signals from the loadcell on the robot forceps were obtained via the A/D converter to the real-time controller and TCP / IP communication to the UI.

The H-MRD and the SAR were connected in the same LAN environment with the TCP/IP protocol and the UI computer and constructed as a bilateral control system (Figure 8). The force reverse type bilateral controller [5] was constructed, in which the positional signals of the leader system are transformed by Equation (4) and sent to the follower system. In contrast, the force signals of the follower system are transformed by Equation (5) and sent to the leader system.

3. Methods

The effect of the haptic information on the remote-controlled task was assessed by five healthy young subjects (four males and one female, 20–30 years old) in a picking task with the leader-follower system (Figure 8). The subjects operated the SAR using the H-MRD with a vision of a remote-site web camera on a monitor in front of them (Figure 9) and picked a maximum of five small fragile snacks with an average size of 15 mm, and a permissible gripping force of 2.6 N. They were asked to pick the objects and manipulate them from one tray to another tray, with a distance of 150 mm between them (Figure 10).

In the experimental protocol, they were first trained in the operation until they could pick and place five objects within three minutes using the haptic information. After the training phases, three measurements were conducted in the following order: H1 (2 min. with haptics), N (2 min. without haptics), and H2 (2 min. with haptics). To control for mental/physical stress in the subjects, a time limitation of 2 min. was applied. After every test, we asked the subjects to answer their subjective assessment with the NASA-TLX questionnaire [19]. The NASA-TLX divides the workload demands into six components: (a) mental demand, (b) physical demand, (c) temporal demand, (d) performance, (e) effort, and (f) frustration [20]. The participants put check marks on scaled bars with 20 scales from “very low (0)” on the left side to “very high (1)” on the right side. The lengths from the left side to the checked marks were normalized and evaluated.

4. Results

4.1. Success Rate

The number of objects that the subjects picked, manipulated (trial number), and successfully achieved (success number) are presented in

Table 2. Score for the picking task (Success number/Trial number).

Subject	(H1) Haptics 1	(N) No Haptics	(H2) Haptics 2
A	1/1	3/4	3/5
B	2/4	2/6	3/6
C	1/3	0/3	1/3
D	1/2	1/3	4/4
E	1/3	0/2	2/4

. In this table, the trial number does not include dropping cases. Failure consisted of breaking the objects. The total success rates for all subjects were 0.462, 0.333, and 0.591, for H1, N, and H2 conditions, respectively.

4.2. Subjective Analysis

Table 3. Scores for the NASA TLX questionnaire (Ave. ± STD.).

Item	(H1) Haptics 1	(N) No Haptics	(H2) Haptics 2
(a) Mental demand	0.491 ± 0.272	0.710 ± 0.260	0.532 ± 0.348
(b) Physical demand	0.464 ± 0.287	0.643 ± 0.172	0.548 ± 0.251
(c) Temporal demand	0.594 ± 0.324	0.709 ± 0.258	0.499 ± 0.248
(d) Performance	0.492 ± 0.319	0.708 ± 0.179	0.412 ± 0.248
(e) Effort	0.580 ± 0.220	0.749 ± 0.242	0.412 ± 0.140
(f) Frustration	0.416 ± 0.348	0.560 ± 0.277	0.470 ± 0.300

summarizes the answers to the NASA-TLA questionnaire for each item. Averages and standard deviations of the experimental results for all subjects are also listed in the table.

5. Discussion

The target object used in this experiment was a fragile object, and it was hard to pick and place this object with the remote-control system without haptics. As presented in Table 2 and the results of the total success rates, the success rate drops once in the “no haptic” phase and increases in the second haptic phase. This indicates that the haptic information successfully assisted the operation skill of the subjects. Many objects (the difference between “success” and “trial”) were broken because of overload grasping in the N-phases. The force information of the haptic forceps enabled the subjects to perceive the grasping sensation on their fingers.

For the subjective assessments, the results of the questionnaire (Table 3) were statistically analyzed between the scores of N- and H2-phases. The results of the paired t-test present p = 0.390 (mental demand), 0.391 (physical demand), 0.256 (temporal demand), 0.069 (performance), 0.109 (effort), and 0.175 (frustration). Although there are no significant differences (p < 0.05), the subjective measures exhibit better tendencies in the haptic phase for all evaluation items.

Table 4. Scores for the NASA TLX questionnaire (Average for (N-H1) and (H2-H1)).

Item	(N-H1)	(H2-H1)
(a) Mental demand	0.220	0.041
(b) Physical demand	0.179	0.084
(c) Temporal demand	0.115	−0.095
(d) Performance	0.216	−0.080
(e) Effort	0.172	−0.109
(f) Frustration	0.145	−0.047

presents the average values between N and H1 (N-H1), and between H2 and H1 (H2-H1) for each item of the questionnaire. The negative values in the H2-H1 indicate the reduction of mental stress, even after the trials in the no-haptic phase.

6. Conclusions

In this paper, we described the development of a leader-follower type remote-control system with twin-driven MR fluid actuators. We developed both leader (the H-MRD for ES) and follower (the SAR for ES) systems to elucidate the teleoperation endoscopic surgery system. The force-reverse type bilateral controller was constructed between the leader and follower systems. To assess the effect of the haptic information on the remote operation task in which users manipulate fragile objects with/without haptic information, we utilized the NASA-TLX questionnaire as a subjective assessment method. According to the experimental results, total success rates were improved with the haptic information. The force information of the haptic forceps enabled users to perceive the grasping sensation on their fingers. Statistical analyses on the answers of the questionnaire indicate that no significant differences, except a reducing tendency in the mental stress during the complicated manipulation tasks with fragile objects.