A Simulator-Based Comparison of a Novel 3D and a Conventional 3D Vision System-Surgical Performance and Subjective Ratings

Abstract

In laparoscopy, novel three-dimensional laparoscopic vision systems (3D LVS) without glasses (3D−) have been invented. While standard 3D LVS with glasses (3D+) have shown advantages over original two-dimensional systems, quantitative comparisons of surgical performance between 3D− and 3D+ systems are still lacking. The aim is to compare the systems in simulated robotic surgery tasks. In a crossover study, 18 medical students performed four basic laparoscopic tasks in the validated Simball Box simulator with authentic surgical instruments by using the 3D+ and 3D− systems. Performance was measured by the number of errors and the task’s duration. Subjective ratings of perceptions and preference were assessed after each test. There were significant, but still minor, advantages for the conventional 3D+ system regarding spatial orientation and sense of depth. Overall, ten and eight subjects preferred 3D+ and 3D− systems, respectively. No significant differences were found in performance, post-operative physicals or eye symptoms. The novel 3D− system was similar to the conventional 3D+ system regarding performance and overall preference, while there were minor advantages for the 3D+ system in the subjective ratings. Since the 3D− system is a new invention, it should have a higher potential of usability improvements.

1. Introduction

Today, minimally invasive surgery is an established technique in various surgical procedures, and the development in this field continues. In laparoscopy, the original two-dimensional laparoscopic vision system (2D LVS) is being challenged by three-dimensional systems (3D LVS) that provide a stereoscopic perception [1,2,3,4,5]. The advantages reported with 3D vision systems in surgery are shorter operative time, less blood loss, fewer perioperative complications, and shorter length of hospital stay, according to a large meta-analysis performed by Cheng et al. [4]. In laparoscopic simulation studies, the 3D LVS is proven to be better in terms of shorter task durations, reduced error rates, and a better subjective experience [3,5,6]. However, one major user complaint regarding the 3D LVSs is that it requires shutter glasses, and the system has also been associated with a bad image quality and experiences of headache, dizziness, eyestrain, nausea, and visual disturbance [2,3]. A new generation of glasses-less 3D LVSs has been invented, and the side effects mentioned above were not reported to be significantly higher than for the 2D LVSs in comparison studies [1].

Several studies have indicated a better outcome of 3D LVSs without glasses (3D−) in comparison with the conventional 3D LVSs with glasses (3D+) [1,2,5,7,8]. In a recent article by Liu et al. [9], potential advantages of the 3D− systems were technically described, such as high brightness, a large viewing area, and a strong anti-interference capability, but the authors did not conduct any actual testing; instead, they expressed a need for further comparison studies.

A single-task study (running sutures on a rubber model) using a 3D LVS system (3D+) and a novel glasses-less system (3D−) was carried out by He et al. [7]. They compared image quality, 3D effect, accessibility, and overall surgical performance, and found no significant difference (all p > 0.5). However, the experiment was short, and the performance was only measured by operative time. Another study reported significantly less visual fatigue with a 3D− than with a 3D+ [8]. However, a limitation of that study was that the tests only included watching videos with no connection to surgery.

In consideration of the limitations from the previous studies, and with the remaining need to test the new 3D− invention, the purpose of this study is to compare a novel 3D− laparoscopic vision system with a conventional 3D+ system by objective and quantitative assessment of surgical performance (in a validated simulation box with authentic surgical instruments), as well as a questionnaire-based user experience assessment.

2. Materials and Methods

2.1. Participants

A total of 18 participants that were considered laparoscopy novices voluntarily participated in this study between May and June 2017 at the Clinical Practice Centre (KTC) at Karolinska University Hospital in Solna, Sweden. The participants (6 women and 12 men) were all medical students from Karolinska Institutet. All except one participant was under 30 years of age, nine participants under 25, and eight between 26 and 30 years of age. All but one was right-handed. Ten people normally used corrective contact lenses or glasses and one of them did not wear his contact lenses on the day of the study. Two people had been exposed to a 3D laparoscopic system in simulators before. The study was approved by the ethical committee at Karolinska Institutet (No.: 2017/181-31/4; 1, March, 2017). The participants were compensated for the time that they were participating in the study. The study was explained in detail to each of the participants and all participants signed a consent form.

2.2. Apparatus

2.2.1. The 3D Laparoscopic Viewing Systems

The conventional 3D+ glasses monitor system was by Panasonic (Kadoma, Japan) and was a 4K/3D system. The full system was equipped with a 0° of view, 10 mm stereoscopic laparoscope, and a pair of 3D glasses for the viewer. The 3D− monitor system was an autostereoscopic 3D display system (Medtronic, Minnesota, USA.). It was equipped with the same light source system and laparoscope. It also provided a 4K/3D resolution, i.e., the same resolution as the 3D+ system.

The 3D− monitor was equipped with eye-tracking sensors incorporated on top of the screen that tracks the user’s eye position, and the monitor then distributed different images to the user’s left and right eyes [10]. Since the sensors could only track the eyes of one user, only one participant was allowed to be in front of the monitor when the task was performed. As described by Geng, the best 3D sensation was provided by the 3D− system when the user’s eyes were in a certain position [10]. The participants were optimally positioned in front of the center of the monitor.

2.2.2. The Laparoscopic Simulation System

The validated Simball Box (SB) (Simball, G-coder Systems, Västra Frölunda, Sweden) was used [11] (see Figure 1). SB is a video box trainer that gives performance feedback using authentic standard surgical instruments. The position and motion metrics of the surgical instruments were detected by the Simball 4D input devices integrated in a Simball Box. Using patented machine vision technology, the 3D angular position of the ball joint was detected. A laser marked pattern on the ball joint’s surface formed a dot pattern code with a configuration that was unique depending on its position on the ball surface. The image of the dot pattern was updated 100 times per second and every image was analyzed, giving the exact 3D angular position of the ball joint. Through the ball joint, a so-called instrument holder was inserted. The holder was equipped with a linear potentiometer that exactly measured the linear motion of the instrument holder (i.e., “in-and-out-motion”). A variety of trays with different tasks can be placed in the box. Video recording and image capturing were done using a See3CAM 80, which is a high-performance MP auto focus UVC USB camera module based on the OV8825 CMOS image sensor (OmniVision Inc, Santa Clara, CA, USA). LED lights were positioned around the camera to mimic the light from a laparoscope. Communication was enabled via USB protocol with a standard computer; in this case, a PC laptop (Windows 7, 8 GB RAM, AMD Radeon HD 8690 Intel^® Graphics 4600, 4 USB3).

2.3. Pre-Task Preparations

The participants had, by e-mail, been given the link to videos of the tasks 2–3 weeks before the day of their participation. Nine people were randomized to start with the 3D+ system, and nine people with the 3D− system.

2.4. Simulation Task Procedure

During the box training, authentic surgical instruments were inserted and fixed in the instrument holder. We assigned four defined tasks (in the listed order) to participants: Basal suturing, peg transfer, rope race, and precision cutting, all described on available online: www.g-coder.com/videos. For all the tasks, except the precision cutting, two 5 mm needle drivers were used. For precision cutting, the participants used scissors with the dominant hand and Kelly grasping forceps with the other. Each participant performed the four tasks twice, first with one system and then with the other.

2.4.1. Basal Suturing

The participants were asked to run a suture through six black dots that were painted by the researchers, each 3 mm in diameter. A Vicryl 4-0 suture, 275 mm in length, was used by all the participants.

2.4.2. Peg Transfer

The participants were asked to follow the fundamentals of laparoscopic surgery (FLS) procedure, to grasp a peg on its thick end from a box with the right hand to transfer it to the left hand-grasping the thin end of the peg and to place it on a specific pin on the left side of the pegboard. Then, they were required to transfer the pegs back in the same manner, left side to right using needle drivers. They were free to choose which hand to start with.

2.4.3. Rope Race

The participants were asked to grasp a white rope, 2.5 mm in diameter, with a black tip (4 mm in diameter), holding the black tip, and to run the rope through eight metal rings placed in a circle. They were all in different angles. The rings on the right side from the middle were: 20° to the right, 45° to the right, 45° to the left, and 30° to the right. The rings on the left side were mirrored.

2.4.4. Precision Cutting

The participants were asked to cut, using the laparoscopic scissors, along a pre-defined black circle, 50 mm in diameter, on a piece of rubber membrane. They were all instructed to hold up the center of the circle with the Kelly grasping forceps. The instruction was to start at four o’clock for right-handed partipants, and at eight o’clock for left-handed participants to make the task was as easy as possible.

2.5. Task Performance Evaluation

All tasks, except for the rope race, were video recorded and reviewed by two independent surgeons. In the review, errors were found in each task (see Appendix A). The surgeons counted the errors in the video of each task, for every participant individually, by following these error definitions. The final errors for each participant of each task were then calculated as the average value of those two reviewers’ results. In addition, the reviewers’ ratings were compared to each other for inter-rater reliability. Quantitative metrics, to assess the performance of each attempt, were measured and analyzed by the Simball Box system, including the task duration, the path length, and the number of predefined errors.

2.6. Questionnaires

Subjective questionnaire ratings were included with pre-experience and post-experience questions. The post-experience evaluation was performed in both a direct and a non-direct comparison (the questions were asked separately for each 3D technique). After completing the tasks for each system, every participant was asked to rate the image quality, spatial orientation, depth perception, and overall impression on a 5-point Likert-type scale (see Appendix B). They were also asked to rate the image quality, spatial orientation, and experience of physical symptoms such as nausea, headache, and eyestrain [13], based on a 4-point Likert-type scale (and for eyestrain and eye burn/sting on Borg’s CR-10 scale [14,15]). After finishing the tasks using both systems, the participants were asked which system they preferred. The scale was from −2 to 2, where a positive number meant that the 3D− system was preferred. Hence, −2 meant that the participant favored the 3D+ system much more than the 3D−, 0 meant that the two systems were preferred to an equal degree, and 2 meant that the 3D− system was preferred much more than the 3D+ system. The details of the two questionnaires are shown in Appendix B and Appendix C. The participants were also asked to comment on one of the systems or on the comparison of the two.

2.7. Statistical Analysis

Descriptive statistics were calculated, including mean, median, quartiles, and maximum/minimum for both the objective performance metrics and the subjective questionnaire. Wilcoxon’s signed-rank test was used for paired comparisons of the objective performance metrics and the subjective ratings of the two systems. The difference was considered statistically significant when p < 0.05. All tests were carried out two-sided. Inter-rater reliability was calculated by using Spearman’s rank correlation coefficient and Cohen’s linearly weighted kappa [16,17].

3. Results

In most of the outcomes and tasks, full data sets were obtained from all 18 participants. However, due to technical difficulty, we did not obtain a video recording during the first peg transfer task for three of the participants. Hence, they were excluded from the counting of errors in that task. These three participants all started with the 3D− system. We also had technical problems with another subject in the peg transfer task in the path length measurement; therefore, the subject was excluded from the calculations of path length as well. This was a participant that started with the 3D− system as well. Therefore, the peg transfer statistical comparisons were performed on 15 participants and the path length with 17.

3.1. Performance

The counted errors of the two reviewers were highly correlated; Spearman’s rank correlation coefficient was 0.883, and Cohen’s linearly weighted kappa was 0.642.

As shown in

Table 1. Task duration and path length of participants using the two systems, 3D+ with glasses and 3D− without glasses, for each task.

Tasks	Task Duration (s)					Path Length (mm)
	3D+		3D−		p	3D+		3D−		p
	$\overline{\mathrm{x}}$	$\tilde{\mathrm{x}}$	$\overline{\mathrm{x}}$	$\tilde{\mathrm{x}}$		$\overline{\mathrm{x}}$	$\tilde{\mathrm{x}}$	$\overline{\mathrm{x}}$	$\tilde{\mathrm{x}}$
Basal suturing	347	335	386	364	0.43	1077	941	1132	1011	0.94
Peg transfer	236	219	230	212	0.65	645	671	700	701	0.61
Precision cutting	410	370	408	396	0.68	958	908	989	836	0.98
Rope race	173	153	195	169	0.16	589	538	620	471	0.65
Total value	1167	1056	1219	1189	0.78	3269	3175	3440	3267	0.98

$\overline{\mathrm{x}}$: average, $\tilde{\mathrm{x}}$: median.

and

Table 2. Errors of participants using the two systems, 3D+ with glasses and 3D− without glasses, for each task.

Tasks	Errors				p
	3D+		3D−
	$\overline{\mathrm{x}}$	$\tilde{\mathrm{x}}$	$\overline{\mathrm{x}}$	$\tilde{\mathrm{x}}$
Basal suturing	0.42	0.00	0.83	0.50	0.051
Peg transfer	1.19	1.00	1.36	1.00	0.590
Precision cutting	5.75	3.50	5.06	4.50	0.324
Total errors	7.36	5.00	7.25	6.75	0.631

$\overline{\mathrm{x}}$: average, $\tilde{\mathrm{x}}$: median.

, there were no significant differences in performance between the two vision systems (3D+ and 3D−) for any of the performance tasks in terms of task duration, path length, or number of errors.

There was an overall improvement from the first to the second performance, which was expected since it was their first experience with a laparoscopic simulator. There was no significant difference in improvement between the participants who started with the 3D+ system and those who began with the 3D− system.

3.2. Questionnaires

The participants reported significantly higher spatial orientation when the 3D− monitor was used. However, direct comparisons of spatial orientation and sense of depth favored the 3D+ monitor (

Table 3. Subjective ratings concerning optical characteristics. (a) Indirect comparison refers to questions regarding to each of the system, with glasses (3D+) or without glasses (3D−). (b) Direct comparison regards to the preference of participants of one system over the other one.

(a)
Question About	3D Monitor	Indirect Comparison
		Scale: 1 = Excellent to 4 = Bad
		n	$\overline{\mathrm{x}}$	Q1		$\tilde{\mathrm{x}}$		Q3		Range	p
Image quality	3D+	18	1.61	1		1.5		2		[1, 3]	0.092
	3D−	18	2.00	1.25		2		2.75		[1, 3]
Spatial orientation	3D+	18	1.83	1.25		2		2		[1, 3]	0.008**
	3D−	18	2.33	2		2.5		3		[1, 3]
Sense of depth	3D+	18	2.00	2		2		2		[1, 3]	0.072
	3D−	18	2.44	2		2.5		3		[1, 4]
(b)
Question About	Direct Comparison
	−2 = 3D+ is much better, 0 = Equal, 2 = 3D− is much better.
	n	$\overline{\mathrm{x}}$	Q1		$\tilde{\mathrm{x}}$		Q3		Range		p
Image quality	18	−0.11	−1		0		1		[−1, 1]		0.791
Spatial orientation	18	−0.50	−1		−0.5		0		[−2, 1]		0.045*
Sense of depth	18	−0.67	−1		−0.5		0		[−2, 1]		0.014*
Overall impression	18	−0.11	−1		0		1		[−2, 1]		0.653

n: number of valid answers, $\overline{\mathrm{x}}$: average, Q1: 1st quartile, $\tilde{\mathrm{x}}$: median, Q3: 3rd quartile, and range: [minimum, maximum]. In Table 3b, the scale was from −2 to 2, where a positive number meant that the 3D− system was preferred. Hence, −2 meant that the participant favored the 3D+ system much more than the 3D−, 0 meant that the two systems were preferred to an equal degree, and 2 meant that the 3D− system was preferred much more than 3D+.* p < 0.05, ** p < 0.01.

). The question of the indirection comparison of the overall impression on each system was excluded from further analysis due to too much missing data. There were no significant differences between the systems in physical symptom ratings, i.e., headache and nausea (

Table 4. Physical symptoms that participants experienced during the operation by using each system.

		Eye Symptoms
Symptoms	3D Monitor	Scale 1 = Nothing, 4 = Very High
		N	$\overline{\mathrm{x}}$	Q1	$\tilde{\mathrm{x}}$	Q3	Range	p
Headache	3D+	18	1.17	1	1	1	[1, 2]	0.219
	3D−	18	1.39	1	1	2	[0, 3]
Nausea	3D+	18	1.06	1	1	1	[0, 2]	0.625
	3D−	18	1.17	1	1	1	[0, 2]

Scale 1: Nothing, 4: Very high, n: number of valid answers, $\overline{\mathrm{x}}$: average, Q1: 1st quartile, $\tilde{\mathrm{x}}$: median, Q3: 3rd quartile, and range: [minimum, maximum].

), nor in eye symptoms (

Table 5. Eye symptoms that participants experienced during the operation by using each system.

		Eye Symptoms
Symptoms	3D Monitor	Scale 0 = Nothing, 10 = Extremely High
		N	$\overline{\mathrm{x}}$	Q1	$\tilde{\mathrm{x}}$	Q3	Range	p
Eye strain	3D+	18	1.64	0.5	0.75	2.75	[0, 5]	0.119
	3D−	18	2.22	1.25	2.5	3	[0, 5]
Eye burn	3D+	18	1.11	0	0	2.5	[0, 5]	0.547
	3D−	18	1.25	0	0.5	2.75	[0, 4]

n: number of valid answers, $\overline{\mathrm{x}}$: average, Q1: 1st quartile, $\tilde{\mathrm{x}}$: median, Q3: 3rd quartile, and range: [minimum, maximum].

).

3.3. Participant Comments

Two of the participants pointed out that it was nice not needing to use 3D glasses in the 3D− system, although two other participants stated that they got dizzy or got diplopia when moving their head using the 3D− system. One person reported bad peripheral depth vision with the 3D− system; conversely, one person also felt that their focus was bad in the upper left corner with the 3D+ system. Three participants thought that the 3D+ system had a darker image. One person commented that the 3D+ system required a longer time to focus, but another participant made the same comment about the 3D− system.

4. Discussion

The results in this study showed no significant differences in performance between the conventional 3D+ system and the novel 3D− system when the subjects performed four tasks in a laparoscopic simulator. In post-experience questionnaires, there was a statistically significant advantage for the 3D+ system regarding spatial orientation, sense of depth, and overall impression, although the actual difference was quite small. There was no significant difference in physical symptoms or in eye symptoms and all subjects reported very little discomfort in both systems. On the direct question regarding which system they preferred overall, ten participants answered 3D+ and eight answered 3D−.

These findings indicated no objective advantages of the 3D− system compared to the conventional 3D+ system. The subjective advantages of the conventional system were small in magnitude, and one can assume that the comfort was better without glasses, especially for surgeons with visual impairment who would also need their corrective lenses. The limitations of the 3D− system were that you must keep a certain distance between the monitor and eyes, and the surgeon must stay within a small range in front of the monitor and it is only suitable for one person to view at a time. In a robot console, this is immaterial since the surgeon is alone in the console, but in traditional laparoscopy, the assistant needs to have their own monitor since they cannot stand close enough to the surgeon to see a single display.

In this study, all participants were novices with only two who had used a laparoscopic 3D− system before. This means that their improvement from the first to the second performance should be comparable. We also had two surgeons who independently watched the videos of the tasks and counted the errors. While there were some differences in the counting, the two surgeons reached a good agreement with each other with a Spearman’s rank correlation coefficient of 0.883, and Cohen’s linearly weighted kappa of 0.642.

One limitation of this study was the short training time in which the participants learned to handle the simulation box. The learning effect during the experiment made it more difficult to compare the systems. However, the subjects were randomized into a balanced order. A remaining difficulty occurred in the peg transfer task, where three subjects who started with 3D− were excluded from the error counting due to technical reasons. This may have affected the results in favor of the 3D− system, although the difference was small, and the comparison would likely not have shown a significant difference even if all subjects were included in that task.

The number of participants (18 subjects) may be seen as another limitation. However, in comparing studies where paired comparisons were used, the power of statistics was higher than in unpaired comparisons. In other system comparison studies that we have conducted, significant differences, although small, were found with fewer subjects [18,19]. With more subjects included in this study, the power would naturally have been higher, but the differences may still be smaller than what may be of clinical interest. Hence, to show important differences between the systems, we estimated that 18 subjects would be enough.

It seems that 3D systems are superior to 2D systems when it comes to laparoscopy, in terms of performance time, error rate, and subjective assessments of cognitive workload, as reviewed by Fergo et al. and Sørensen et al. [20,21]. Both Fergo et al. and Sørensen et al. showed either an advantage of 3D systems or no significant differences in those aspects among screened and selected studies [20,21]. A recent comparison study of a 2D, a 3D+, and a 3D− system from Liu et al. concluded that the tested 3D systems showed an overall superiority over the tested 2D system in performance time and precision [22]. The development of 3D diagnostic imaging may also facilitate support for surgeons during operation [23].

However, whether it is a benefit to adopt a glasses-free 3D system is debatable. A study by He et al. indicated that the perceptions of benefits of glasses-free 3D systems could be conflicting among users with different backgrounds [7]. In the current study, there were no significant differences between the systems in the objective performance data, which is in agreement (although with somewhat different parameters) with the findings of Liu et al. [22]. Meanwhile, there were significant differences in the subjective comparisons (spatial orientation and sense of depth) in favor of the 3D+; however, the indicated advantages of the conventional 3D+ system were relatively small. The current results did not show a significant difference in terms of eye discomfort, but results from Liu et al. suggested that 3D+ system might increase eye fatigue [22]. This may explain the bipolar division of opinions in regard to eyestrain in the study by He et al. [7].

5. Conclusions

There were significant, but minor, advantages for the conventional 3D+ system regarding spatial orientation and sense of depth. There were no significant differences found in the system comparison in performance, post-operative physical symptoms, or eye symptoms; instead, in these comparisons, the systems were very similar. With more development to the 3D− systems (which is being done continuously), they should have a higher potential of usability improvements.