Dynamic Gesture Recognition Using a Smart Glove in Hand-Assisted Laparoscopic Surgery

Abstract

This paper presents a methodology for movement recognition in hand-assisted laparoscopic surgery using a textile-based sensing glove. The aim is to recognize the commands given by the surgeon’s hand inside the patient’s abdominal cavity in order to guide a collaborative robot. The glove, which incorporates piezoresistive sensors, continuously captures the degree of flexion of the surgeon’s fingers. These data are analyzed throughout the surgical operation using an algorithm that detects and recognizes some defined movements as commands for the collaborative robot. However, hand movement recognition is not an easy task, because of the high variability in the motion patterns of different people and situations. The data detected by the sensing glove are analyzed using the following methodology. First, the patterns of the different selected movements are defined. Then, the parameters of the movements for each person are extracted. The parameters concerning bending speed and execution time of the movements are modeled in a prephase, in which all of the necessary information is extracted for subsequent detection during the execution of the motion. The results obtained with 10 different volunteers show a high degree of precision and recall.

1. Introduction

One of the most important innovations in surgery over the past three decades has been the advent of minimally invasive surgery (MIS). This technique has revolutionized surgical practice due to its ability to avoid the trauma of traditional open surgery and diminish the possibility of incision-related complications. These benefits also have economic consequences, because they result in a reduction of hospital stay times. However, MIS is technically challenging, because it must be conducted in a very restricted space using micro instruments and endoscopes that dramatically limit the surgeon’s perception. In order to gain tactile and force feedback, new technologies and techniques have been introduced over the last few years. One of these novel techniques is hand-assisted laparoscopic surgery (HALS). In HALS, the surgeon inserts a hand in the patient’s abdomen through a small incision via a pressurized sleeve while operating a surgical tool with the other hand. Although this approach is slightly more invasive for the patient, it is still a MIS intervention, and has been proved especially advantageous in some types of operations, such as colon and colorectal cancer surgery [1].

However, HALS has a major shortcoming. As the surgeon is holding the tissue with the inserted hand and a micro instrument with the other, he/she needs the close cooperation of an assistant to manage the endoscope and additional surgical tools when performing surgical maneuvers such as stitching and knot tying. In this paper, we tackle the automation of the tasks performed by the human assistant using, instead, a collaborative robot (Figure 1). This robotic system requires, among other important issues, a simple communication scheme capable of recognizing the surgeon’s direct orders given by the hand inserted in the abdominal cavity.

Previous works on surgical robots can be found in the literature. The first robot systems for laparoscopic surgery were developed to provide more stability and precision to the movements of the surgical tools and endoscopes. They were teleoperated systems that integrated a simple robotic arm with a laparoscopic instrument attached to it [2]. Since then, a number of semi-autonomous robots have been developed and studied to assist the surgeon in the different phases of the operation [3,4,5].

Autonomous systems require recognition of the surgical gestures made by the surgeon. The use of cameras is an early developed technology to sense hand gestures [6,7,8,9,10], including gloved hand recognition [11], but image processing is always problematic when the scene is under variable illumination or with a cluttered background [12]. In HALS, the variable lighting provided by an endoscope under continuous movement, as well as the difficulty of extracting a permanently blood-stained hand from the internal scenes, prevent the use of this technology. These circumstances are aggravated as the hand would only be partially visible in the images due to the limited viewing field available inside the abdominal cavity.

In order to communicate with the collaborative robot in a natural way, the use of a sensor glove is proposed. A dynamic gesture recognition algorithm has been developed to identify the commands the surgeon gives to the robot with his/her hand inserted in the abdominal cavity. The chosen textile-based motion sensing glove is comfortable, and permits the perfect mobility of the surgeon’s hand in the reduced space inside the patient’s abdomen. Although the glove used in this study was tailored for a different application (i.e., daily-life monitoring of the grasping activity of stroke patients, as described in [13]) and has a low number of sensors, i.e., three sensors covering the thumb, index, and middle fingers, this wearable device allows the surgeon’s hand movements to be monitored. The movement of the fingers is followed by the glove sensors, without limiting the operability. However, some disturbances may appear due to cross-talk between sensors. When the operator tries to move one finger, this can generate a noise signal in another. These disturbances are filtered to avoid misclassification of the surgeon’s gesture.

To check the algorithm, 10 tests were performed by 10 different subjects to detect some movements designated in a previous selection phase. Each test consisted of three predefined movements and two additional gestures, which were included in order to demonstrate that the algorithm does not erroneously confuse a movement that is not predefined with a predefined one.

The aim of these tests is to determine whether the developed gesture recognition algorithm can be used to send commands using a sensor glove to a collaborator robot during a HALS with a high degree of precision.

This paper is organized as follows. Section 2 introduces the materials and methodologies used in the experiments that are shown in Section 3. The results are presented in Section 3 and discussed in Section 4. Finally, Section 5 presents the conclusions.

2. Materials and Methods

2.1. Sensing Glove

The sensing glove adopted in this work is made of cotton–lycra, and has three textile goniometers directly attached to the fabric. Figure 2a shows the position of the goniometers on the glove, while Figure 2b show the final prototype of the glove where the goniometers are insulated with an additional layer of black fabric.

The textile goniometers are double layer angular sensors, as previously described in [14,15]. The sensing layers are knitted piezoresistive fabrics (KPF) that are made of 75% electro-conductive yarn and 25% Lycra [16,17]. The two KPF layers are coupled through an electrically-insulating stratum (Figure 3a). The sensor output is the electrical resistance difference (ΔR) of the two sensing layers. We demonstrated earlier that the sensor output is proportional to the flexion angle (θ) [14], which is the angle delimited by the tangent planes to the sensor extremities (Figure 3b).

The glove was developed in previous studies to monitor stroke patients’ everyday activity to evaluate the outcome of their rehabilitation treatment [13,18]. In [19], the reliable performance of the glove goniometers was demonstrated, and showed errors below five degrees as compared with an optical motion capture instrument during natural hand opening/closing movements. The glove has two KPF goniometers on the dorsal side of the hand to detect the flexion-extension movement of the metacarpal-phalangeal joints of the index and middle fingers. The third goniometer covers the trapezium-metacarpal and the metacarpal-phalangeal joints of the thumb to detect thumb opposition. We conceived this minimal sensor configuration as a tradeoff between grasping recognition and the wearability of the prototype.

An ad hoc three-channel analog front-end was designed for the acquisition of ΔR from each of the three goniometers (Figure 3c). For each goniometer, the voltages V₁ = Vp₂ − Vp₃ and V₂ = Vp₅ − Vp₄ are measured when a constant and known current I is supplied through p₁ p₆. A high-input impedance stage, consisting of two instrumentation amplifiers (INS₁ and INS₂), measures the voltages across the KPF sensors. These voltages are proportional, through the known current I, to the resistances of the top and bottom layers (R₁ and R₂). A differential amplifier (DIFF) amplifies the difference between the measured voltages, obtaining the final output ΔV, which is proportional to ΔR and θ. Each channel was analogically low-pass filtered (anti-aliasing, cut-off frequency of 10 Hz). The resulting data were digitally converted (sample time of 100 Sa/s) and wirelessly transmitted to a remote PC for storage and further elaboration.

2.2. Algorithm for Movement Detection

The glove will communicate with a collaborative robot to assist during a HALS. The actions to test the collaborative robot take into account the various robotic actions covered by the literature [20], among which are the guidance of the laparoscopic camera for the safe movement of the endoscope [21] or a needle insertion [22], the prediction of the end point [23,24], the knotting and unknotting on suture procedures [25], or grasping and lifting on tissue retraction [26]. Ultimately, we selected three actions to be performed by the collaborative robot: center the image from the endoscope, indicate a place to suture, and stretch the thread to suture. These actions are performed in a cholecystectomy, which is the surgical removal of the gallbladder.

Each of the three robot actions mentioned above is associated with a hand movement to be performed by the surgeon. Therefore, the system must be prepared to unambiguously recognize the different movements defined as commands for the robot in order to prevent it from performing undesirable operations. They will be differentiated by the detection algorithm, which is tested with a protocol.

The protocol includes these three movements, which must be detected as robot commands, and are shown in

Table 1. Selected movements to be detected.

Nº	Initial Posture	Final Posture	Description	Command
1			From initial posture to final posture twice	To center the image from the endoscope.
2			From initial posture to final posture twice.	To indicate a place to suture.
3		-	Initial posture for a defined time.	To indicate to stretch the thread.
4			From initial posture to final posture twice.	-
5			From initial posture to final posture twice.	-

and numbered from 1 to 3. Actions 4 and 5 are introduced to test the developed algorithm. These were selected for their similarity to the movements selected in both the sensor value and motion patterns. As a result, differentiation in advance between the different movements is difficult.

To detect these movements, the developed algorithm analyzes the following parameters: flexion pattern, velocity, execution times, and value provided by the sensor of each finger. To evaluate these parameters, there is a previous phase in which the variables of each movement in each person are examined. This previous stage is required for each person, because the speed and timing of the finger movement is highly variable, as shown in Figure 4.

Once these variables are defined, as explained in later paragraphs, the detection algorithm can identify each of the three movements.

The motion of the index and middle fingers is sensed by the glove. The acquired data is continuously processed by the developed algorithm in order to detect some of the predefined dynamics patterns. Due to the unique textile substrate to which of all the sensors are attached, cross-talk between sensors may appear. This could be observed as a disturbing signal from a finger when the operator tries to move another finger, as shown in Figure 5. These movements are filtered in order to avoid a misclassification.

Due to the nature of the sensors used, it is possible to determine the degree of flexion being applied to the sensor on the glove. However, movements 4 and 2 could be confused due to their similarity, as shown in Figure 6.

Movement 1 can be identified by analyzing the data from the index and middle fingers. Each rise and fall in the glove sensor values corresponds to the flexion and extension movements of the fingers. This movement consists of a descent (called D1) and ascent (A1), followed by another descent (D2) and ascent (A2), as shown in Figure 7. This is the flexion pattern considered for movement 1. The D time and A time are, respectively, the times taken during a descent or ascent.

The flexion velocity involved in this dynamic gesture is higher than the cross-talk ones, as shown in Figure 7b. To establish the typical velocity for this movement, the average and the standard deviation of the velocity along D1 and D2, and A1 and A2, are calculated. This typical velocity, V_1u, is the minimum value obtained from the subtraction of the standard deviation from the average in three tests performed by the same person. The minimum time during descents, t_1Du, (D1 and D2) and ascents, t_1Au, (A1 and A2) is also calculated, and will represent the characteristic ascent and descent execution times of movement 1.

To determine the execution time, t_1u, the maximum time in which the whole movement is performed is considered; that is D1, A1, D2, and A2.

The last parameters to be defined are the maximum, x_max, and minimum, x_min, values of the sensor, which set the thresholds to consider if the obtained values are part of movement 1. They are obtained by analyzing three movement samples from the same person.

With these parameters, shown in

Table 2. Characterization of defined movements.

Mov.	Finger	Flexion Pattern	Flexion Velocity	Execution Time	D Time	A Time	Sensor Value
1	Index Middle	D1 A1 D2 A2	|Ve| > V1u	te < t1u	tD > t1Du	tA > t1Au	xmin < x < xmax
2	Index	D1 A1 D2 A2	|Ve| > V2u	te < t2u	tD > t1Du	tA > t1Au	xmin < x < xmax
3	Index Middle	-	-	te > t3u	-	-	xmin < x < xmax

, movement 1 can be defined and differentiated from others, considering the flexion velocity V_e as the instantaneous velocity scanned during the entire movement performed, and the execution time t_e as the time in which the velocity exceeds the velocity threshold.

Using the graphs obtained during the performance of movement 2, as shown in Figure 8, we can conclude that it is necessary to determine the movements of the index and middle finger in order to obtain a definition. The flexion pattern for this movement is D1, A1, D2, and A2 for the index finger, and no movement for the middle finger. The velocity, time of execution, minimum time during descents (D1 and D2) and ascents (A1 and A2), and the sensor value are defined as described in movement 1.

Movement 3, in Figure 9, differs from the other two in that the velocity must be 0, so it is a static position maintained for a certain time. To identify it, we examine the values of the index and middle finger sensors, which will be proportional to the flexion carried out by the finger with the sensor.

The algorithm for the detection of defined movements evaluates all of the abovementioned parameters, and detects when one of these movements is executed.

3. Experiments and Results

The test consists of carrying out the movements shown in Table 1 in the same order, as well as performing a flat position between them, in a scenario of experiments (Figure 10). Therefore, the correct order of execution is: flat position, movement 1, flat position, movement 2, flat position, movement 3, flat position, movement 4, flat position, movement 5, flat position.

Movements 1, 2, and 3 have been selected to be detected by the algorithm, while movements 4 and 5 were introduced to prove that they are not detected in the same manner as the three selected ones. The two newly introduced movements are similar to movements 1 and 2, but there are small differences between them.

First, the data are collected from the glove. Then, they are analyzed by the algorithm to detect the movements that will be interpreted as commands for the robot. These orders are then sent to the collaborative robot.

The test has been carried out by 10 people, 10 times. Ten right-handed volunteers (five men, five women) completed the test. All of the participating people in these research activities gave informed consent for the experiments. No one reported physical limitations that would affect their skill in performing the task.

The characteristic parameters of each movement are calculated from three tests performed by the same person. These parameters are characteristic of each person, so 10 sets of patterns have been obtained for each type of movement, one per person.

Movements 1, 2, and 3 must be detected by the algorithm, while movements 4 and 5 should not be classified as selected movements. As shown in

Table 3. Total results.

Total		Actual Movement					Precision	Recall	F1-Score
		1	2	3	4	5
Predicted Movement	1	98	0	0	1	0	0.99	0.98	0.98
	2	0	87	0	33	0	0.73	0.87	0.79
	3	0	0	97	0	0	1.00	0.97	0.98

, movement 1 was detected with a precision—the percentage of positive predictions that were correct—of 0.99, and a recall—the percent of the positive cases recognized—of 0.98. Movement 4 was identified as movement 1 only 1% of the time. On the other hand, movement 2 was detected with a precision of 0.73 and a recall of 0.87. Movement 4 was recognized as movement 2 33% of the time. Movement 5 was never mistaken with other movements, and movement 3 was detected with a precision of 1.0 and a recall of 0.97.

F₁ as scored for movements 1 and 3 is 0.98, while for movement 2 it was considerably lower, 0.79.

4. Discussion

Movement 4 is detected as movement 2 or 1 because of their similarity, as explained in the previous sections. Despite the study of different patterns, times, and speeds, movement 4 is detected as movement 2 35% of the time. Whenever movement 3 has not been detected, this was due to an insufficient time in the static position.

Reviewing the results, it can be concluded that the effectiveness of the algorithm depends largely on the person performing the test as shown in Appendix A. Results with surgeons are expected to be better, because they have greater motor skills, considering their specific training [27]. Tests have shown that the newly developed algorithm can adequately identify the three movements defined in a series of different continuous movements. Movement recognition is precise, because identification is based not only on the initial and final pose, but also on intermediate positions and speeds that are continuously analyzed to determine whether their pattern is analogous to the model. Different filters are also introduced to make the dynamic gesture recognition algorithm more reliable. The patterns obtained with the sensing glove present sufficient information as to be robustly identified, and prevent failures in those cases where the positions are similar to those of the model, but the execution speed of the movement is different.

One of the purposes of this study was to test the validity of our non-specific glove to demonstrate the possibility of using this kind of device, and define the specification for a HALS-dedicated textile glove for use in future studies. In future works, glove-based hand motion sensing could be fused with other sensing modalities, such as artificial vison, to make the system more robust.

5. Conclusions

Most current surgical robots are not suitable for HALS operations. Its teleoperated nature prevents its application in these operations where the surgeon is in direct contact with the patient. In this scenario, it is necessary to have a robot co-worker that cooperates closely with the surgeon in order to emulate the interaction with a human assistant. A natural communication interface between surgeon and robot is crucial in this context. This paper tackles the design of a dynamic gesture recognition algorithm using a sensor glove that identifies the commands given by the surgeon’s hand inside the patient’s abdominal cavity. Three different dynamic gestures have been predefined to: point the robot where to suture, order it to focus the endoscope, and stretch the thread. All of these tasks present automatic procedures in the literature to carry them out. The algorithm designed to recognize these gestures analyzes continuously the timing and the bending speed of the index and middle fingers, and it tries to match them with some of the patterns previously recorded by a particular operator.

The experiments conducted with 10 different volunteers show a good recognition rate and time performance. However, considering its application in surgical operations, there is room for improvement. Although this study has considered the option of the sensing glove, another hand motion sensor would need to be added in order to make the system completely reliable. Furthermore, other important issues such as safety or electromagnetic compatibility should be addressed in future works.