Next Article in Journal
Load Resistance Optimization of a Magnetically Coupled Two-Degree-of-Freedom Bistable Energy Harvester Considering Third-Harmonic Distortion in Forced Oscillation
Next Article in Special Issue
Local Style Preservation in Improved GAN-Driven Synthetic Image Generation for Endoscopic Tool Segmentation
Previous Article in Journal
Self-Powered Point-of-Care Device for Galvanic Cell-Based Sample Concentration Measurement
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:

Non-Technical Skill Assessment and Mental Load Evaluation in Robot-Assisted Minimally Invasive Surgery

by 1,2,*,†,‡ and 1,3,4,‡
Antal Bejczy Center for Intelligent Robotics, University Research and Innovation Center, Óbuda University, 1034 Budapest, Hungary
Doctoral School of Applied Informatics and Applied Mathematics, Óbuda University, 1034 Budapest, Hungary
John von Neumann Faculty of Informatics, Óbuda University, 1034 Budapest, Hungary
Austrian Center for Medical Innovation and Technology, 2700 Wiener Neustadt, Austria
Author to whom correspondence should be addressed.
Antal Bejczy Center for Intelligent Robotics, University Research and Innovation Center, Óbuda University, Bécsi út 96/b, 1034 Budapest, Hungary.
These authors contributed equally to this work.
Sensors 2021, 21(8), 2666;
Received: 26 January 2021 / Revised: 31 March 2021 / Accepted: 8 April 2021 / Published: 10 April 2021
(This article belongs to the Special Issue Medical Robotics)


BACKGROUND: Sensor technologies and data collection practices are changing and improving quality metrics across various domains. Surgical skill assessment in Robot-Assisted Minimally Invasive Surgery (RAMIS) is essential for training and quality assurance. The mental workload on the surgeon (such as time criticality, task complexity, distractions) and non-technical surgical skills (including situational awareness, decision making, stress resilience, communication, leadership) may directly influence the clinical outcome of the surgery. METHODS: A literature search in PubMed, Scopus and PsycNet databases was conducted for relevant scientific publications. The standard PRISMA method was followed to filter the search results, including non-technical skill assessment and mental/cognitive load and workload estimation in RAMIS. Publications related to traditional manual Minimally Invasive Surgery were excluded, and also the usability studies on the surgical tools were not assessed. RESULTS: 50 relevant publications were identified for non-technical skill assessment and mental load and workload estimation in the domain of RAMIS. The identified assessment techniques ranged from self-rating questionnaires and expert ratings to autonomous techniques, citing their most important benefits and disadvantages. CONCLUSIONS: Despite the systematic research, only a limited number of articles was found, indicating that non-technical skill and mental load assessment in RAMIS is not a well-studied area. Workload assessment and soft skill measurement do not constitute part of the regular clinical training and practice yet. Meanwhile, the importance of the research domain is clear based on the publicly available surgical error statistics. Questionnaires and expert-rating techniques are widely employed in traditional surgical skill assessment; nevertheless, recent technological development in sensors and Internet of Things-type devices show that skill assessment approaches in RAMIS can be much more profound employing automated solutions. Measurements and especially big data type analysis may introduce more objectivity and transparency to this critical domain as well. SIGNIFICANCE: Non-technical skill assessment and mental load evaluation in Robot-Assisted Minimally Invasive Surgery is not a well-studied area yet; while the importance of this domain from the clinical outcome’s point of view is clearly indicated by the available surgical error statistics.

1. Introduction

Minimally Invasive Surgery (MIS) induced a paradigm change in medicine; however, it presented new challenges for surgeons [1,2]. In the case of MIS—against traditional, open-access surgery—inside organs are reached through small skin incisions with laparoscopic instruments, and the operating area is visualized with an endoscopic camera. During MIS, the operator (surgeon) has to work in a team as a leader, he/she gives instructions to a camera handler assistant and the other operating room members, while he/she has to constantly monitor the operating area on a 2D screen in an uncomfortable position. Thus, despite the clear benefits of MIS, including the smaller scars and faster recovery time, there are drawbacks for the physicians, such as the limited motion space, complicated instrument control, not ergonomic environment and the two dimensional endoscopic camera image.
Robot-Assisted Minimally Invasive Surgery (RAMIS) was the next step in the evolution of MIS: it provided an improved vision system, more accurate and intuitive instrument control and an ergonomic master console [3,4,5]. The most successful RAMIS system is the da Vinci Surgical System (Intuitive Surgical Inc., Sunnyvale, CA), which is a teleoperated, master–slave type surgical robot (Figure 1). The basic concept of a remote-controlled telesurgical system was created at the National Aeronautics and Space Administration (NASA) in around 1971, originally planned to be used for remote surgeries, where the slave robot is on the spaceship. In the case of the da Vinci Surgical System, the surgeon sits at an ergonomic master console, where he can operate with intuitively-controlled master arms. The surgeon can use pedals for special surgical instruments, such as a clutch, and to control the endoscopic arm, thus camera handling is only in the hands of the surgeon. At the patient side of the da Vinci, there are the remotely controlled slave arms, which accomplish the interventions minimally invasively with a motion mechanism called “Remote Center of Motion” (RCM), which can guarantee patient safety. The assistant crew works at the patient side of the da Vinci, where they can help the surgeon and support the intervention, such as they can change the surgical instruments during the operation. At the slave side, there is a 3D endoscopic camera, in which images are visualized in the screens placed in the master console; thus, the surgeon can see a 3D image of the operating area. The motion of the surgeon can be re-scaled on the patient side of the da Vinci, which can provide more accurate motion. However, the original idea of remote surgery was to operate with long distances; for safety reasons, at the moment it is not part of the clinical practice. It is important to note, against the name ‘robot’, the da Vinci Surgical System does not perform any kind of automation or decision making, neither decision support, the only very low-level automation in the da Vinci is tremor and abrupt motion filtering. A steep learning curve has been identified with the da Vinci [5,6]. Thus, despite the fact that RAMIS can decrease the mental workload of the surgeon as shown through by studies, RAMIS remains a challenging operation to perform not just physically, but mentally as well, because of the constant communication, teamwork, leadership, decision making and workload conditions (Figure 1) [7,8,9].
The improvements of RAMIS can help the surgeon, however, RAMIS is still a hard task to master; continuous training and feedback about the performance is crucial. Furthermore, the skills of the surgeon directly influence the outcome of the surgery. In surgical skill assessment, the Dreyfus model is often introduced [11]. The Dreyfus model shows the evolution of the learning process, and it can describe the typical features of the expertise levels at the different learning phases, such as a novice usually can only follow simple instructions, but an expert can well react to previously unseen situations. The Dreyfus model was fitted to surgical skills as well [12]. Surgical skill assessment improves training and provides quality assurance; therefore, it has benefits for surgeons and patients. While surgical skill assessment is available during training (such as with RAMIS simulators [13,14]), it is not the part of the everyday clinical practice yet [15,16]. Technical skill assessment is a well-studied area not just in traditional MIS, but in RAMIS as well [17]. Technical skills in RAMIS are related to the basic skills of the surgeon (knowing the instruments, using the right tools, etc.), the control of the robot and MIS tools (bimanual dexterity, endoscopic camera handling, clutch handling, instruments kept in view, etc.) and tissue handling (force sensitivity). Nevertheless, non-technical skill assessment is less objective.
The workload on the surgeon—which represents the effort to perform a task—can be high in several domains of a procedure: there are mental, physical and temporal demands. Furthermore, task complexity (including multitasking, task novelty), situational stress and distractions can influence the outcome of the surgery [18,19] (Figure 2). Naturally, the same task can cause different workload to different operators. Non-technical skills related to the workload on the surgeon, furthermore, can directly affect surgical outcome [20]. Non-technical skills include communication, teamwork, task management, leadership, decision making, situational awareness and cope with stress, fatigue and distractions based on validated metrics, such as NOTSS and ICARS [21,22] (Figure 2). Situation Awareness (SA) has been recently investigated in other safety-critical domains, such as self-driving technologies, nevertheless, the SA assessment and quantification methods are very similar in both application areas [23]. While it is straightforward that technical skills are crucial for better surgical outcomes, non-technical surgical skills can be as important as technical skills. Clinical failures in the operating room may come from low non-technical skills of the surgeon rather than the lack of technical skills [24,25,26].
In the literature, three approaches for surgical performance assessment can be identified [17,27,28]:
  • self-rating questionnaires,
  • expert-based scoring and
  • automated (sensor-based) skill assessment.
Questionnaires are filled out by the operator; thus, it is easy to implement and is subjective. Objective scoring is done by an expert panel, based on a standardized method [29]. Expert ratings are supposedly objective, but may be biased for personal reasons. Furthermore, they can be hard to implement, being human resource intensive. Automated skill assessment is based on objectively measurable parameters (such as applied forces, movement velocity, etc.), however, in most cases it is technically not easy to implement. Robotic surgical systems can provide a unique platform for objective skill assessment due to their built-in sensors providing a continuous flow of recordable kinematic and video data [3]. The original da Vinci Surgical System alone had 48 sensors. The mentioned surgical skill assessment approaches can be found in technical skill, non-technical skill and mental workload assessment as well. For mental workload assessment, questionnaires and automated solutions can be useful tools, and for non-technical skill assessment all of the methods (questionnaires, expert-rating and automated techniques) can be utilized.
The difference between traditional MIS and RAMIS mental workload was examined in some studies [30,31], demonstrating lower mental workload in the case of RAMIS. However, in these studies questionnaires created for traditional MIS were used, the main workload parameters in RAMIS are not yet defined. For RAMIS, non-technical skill assessment expert-rating methods originally created for traditional MIS can be found [32,33]. There is one metric specifically created for RAMIS non-technical expert-rating assessment (ICARS, [22]), which describes the most important non-technical skills in RAMIS (Figure 2). Non-technical skills are naturally hard to be measured automatically. The possibilities for automated RAMIS non-technical skill assessment are similar to traditional MIS, such as relying on physiological signals measured by additional sensors [34].
The goal of any kind of skill assessment is to employ automated and objective methods to measure the skills of the surgeon; thus avoiding biased assessment and the need for human resources. The built-in sensors of RAMIS can significantly ease automated skill assessment, since there are recordable kinematic and video parameters of the surgery (such as tool trajectory, orientation, velocity, etc.), which can provide input for skill assessment algorithms (statistical analysis or machine learning methods), towards traditional MIS, where these data are only available with additional sensors. Da Vinci Surgical System is a closed system; therefore, to analyze surgical data recorders is necessary, such as the da Vinci Research Kit (DVRK, developed by a consortium led by Johns Hopkins University and Worcester Polytechnic Institute), which can provide open-source hardware and software elements with complete read and write access to the first generation da Vinci arms [35].
To understand where non-technical skills can be identified in the case of RAMIS, high priority (interaction and communication) channels and interfaces have to be identified and analyzed. International Electrotechnical Commission (IEC) and International Organization for Standardization (ISO) published a new safety standard for surgical robots, the IEC 80601-2-77. In the standard, the components of RAMIS are defined, and a basic diagram of RAMIS is introduced [36,37]. Based on the proposed working diagram, we highlighted the most important components in non-technical skill assessment (Figure 3). For this, the following definitions were used from IEC 80601-2-77, following the taxonomy of the IEC 60601 medical device core standard:
  • Robotically Assisted Surgical Equipment—RASE: ‘Medical electrical equipment that incorporates programmable electrical medical system actuated mechanism intended to facilitate the placement or manipulation of a robotic surgical instrument’ (the ISO 8373 standard strictly defines the term "robot" in the ISO domain, therefore the working group decided to use the more inclusive "Robotically Assisted" expression within RAMIS, while it is less commonly used in the domain).
  • Robotic surgical instrument: ‘Invasive device with applied part, intended to be manipulated by RASE to perform tasks in surgery’.
  • High frequency (HF): ‘less than 5 MHz and generally greater than 200 kHz’.
  • HF surgical equipment: ‘medical electrical equipment which generates HF currents intended for the performance of surgical tasks, such as the cutting or coagulation of biological tissue by means of these HF currents’.
  • Interface conditions: conditions that shall be fulfilled to achieve basic safety for any functional connection between RAMIS and other medical electrical equipment or non-medical electrical equipment in the robotic surgery configuration.
  • Mechanical interface: mounting surface on RAMIS that allows for attachment of detachable accessories, components or parts that are mechanically manipulated by the RAMIS.
  • Endoscopic equipment: ‘energized endoscope together with its supply unit(s), as required for its intended use’ [36,37].
It is worth mentioning that the terminology of the ISO standard with respect to RASE slightly differs from RAMIS, mostly due to the fact that in ISO sense, the term “robot” is defined in a much narrower meaning.
In Figure 3 the components of RAMIS and the most important components in non-technical skill assessment are shown. Based on the literature findings, non-technical skill and workload can be assessed with the communication channel between the surgeon and the assistants, and with the cognitive and personal resource skills of the operating room crew, such as based on physiological signals or questionnaires, as it can be seen on the image, the surgeon’s decisions are inseparable from the control loop of RAMIS systems. It suggests that non-technical skills and workload might be shown in objectively measurable parameters, which means non-technical skill assessment is not necessarily different from technical skill assessment [38]. This may ease objective, automated non-technical surgical skill assessment in RAMIS. However, in the case of RAMIS, not many studies have examined this correlation.
In this paper, we review the recent results of non-technical skill and mental workload assessment in the case of Robot-Assisted Minimally Invasive Surgery. In the materials and methods section, we introduce the literature search strategy, following the standard PRISMA method. In Section 3.1, we show the mental workload assessment techniques in RAMIS (NASA-TLX, SURG-TLX, Multiple Resources Questionnaire, etc.). In Section 3.2, we show the expert rating techniques in RAMIS non-technical skill assessment. In Section 3.3, we overview the recent results in automated non-technical skill assessment techniques, furthermore, the possibilities and limitations of algorithm-based non-technical skill assessment. At the end of the paper, we review the relevant publications in a tabular form (Table 5), containing the following columns for easy comparability: reference, year of the publication, number of subjects involved, experimental environment, used assessment technique, measured non-technical skill, conclusion and quality of evidence. The paper ends with an appropriate discussion and conclusion.

2. Materials and Methods

To find relevant publications in the field of non-technical skill and mental workload assessment in RAMIS, the PubMed, Scopus and PsycNet databases were searched. The last search was performed in August 2020. To find relevant publications for mental workload assessment in RAMIS, we used the keywords ‘surgical robotics’ or ‘robotic surgery’ or ‘robot-assisted minimally invasive surgery’ and ‘workload assessment’ or ‘cognitive assessment’ or ‘NASA-TLX’ or ‘SURG-TLX’. In the case of expert rating and automated non-technical assessment, we use the keywords ‘surgical robotics’ or ‘robotic surgery’ or ‘robot-assisted surgery’ and ‘non-technical skill’ or ‘non-technical skill assessment’ or ‘NOTSS’ or ‘ICARS’. We included original articles about non-technical skills and mental workload assessment in RAMIS. We could not find any patents or software products matching the above criteria. We excluded publications that studied these assessment techniques in traditional MIS, not RAMIS, but included those which compared the two types of surgery with the non-technical skill assessment perspective. Due to the fact that we wanted to focus on RAMIS non-technical skills, we excluded publications about surgical process modeling, ergonomy (which considered physical workload only), technical skill assessment techniques, workflow assessments and reviews.
Fifty relevant publications were found in the field of non-technical skill and mental workload assessment in surgical robotics (Figure 4). From the relevant publications, the following research topics were identified: workload (42), brain activity (11), communication (9), stress (7), leadership (3), decision making (3), situation awareness (3) and teamwork (2) (Figure 5). The summarized results can be found in Table 5. We defined the quality of evidence based on the GRADE approach [39]. Study limitations, inconsistency of results, indirectness of evidence, imprecision and publication bias can decrease, and large magnitude of effect, plausible confounding and dose–response gradient can increase the quality of evidence in GRADE. To decide the quality of evidence, we carefully considered the impact of RAMIS workload and non-technical skill assessment research (which can increase or decrease the quality class). Based on the quality of evidence and the strength of recommendation, the following classes were defined:
  • High: high-level of confidence in the effects;
  • Moderate: confidence in the effects may change with future research findings;
  • Low: confidence in the effects is very likely to change with future research findings;
  • Very low: uncertainty about the effects.

3. Technical Approaches for Non-Technical Skill and Mental Workload Assessment in RAMIS

3.1. Mental Workload Assessment—Self-Rating Techniques

Performing a surgical procedure can be very stressful to the whole crew of the operating room. Fatigue (mental and physical) can naturally influence the outcome of the surgery; furthermore, time limits can cause serious stress and cognitive load on the surgeon, and working in a team can be disturbing in some cases. Workload is a term that represents the psychological cost to perform a task; it is human-specific, however, there are situations which can take a serious amount of mental workload from every operator. Workload can be defined with self-rating techniques, where a subject fills a questionnaire about his/her personal experience about the task workload. It is naturally a subjective technique, however, there are works in the literature which studied both subjective workload measurements and objective non-technical skill assessment metrics [32,40], or objective physiological parameters [30,34,41,42,43,44,45,46]. Workload measurements do not only help to assess the personal workload index, but also to define the main stressors and disturbing factors in surgery in general, furthermore, to provide personal training for novices as well.
NASA Task Load Index (NASA-TLX, created by NASA’s Ames Research Center in 1988) is a workload self-rate estimation metric, originally created for assessing workload in aviation [18,47]. NASA-TLX measures the workload on a subject with questions related to mental, physical and temporal demand, effort, performance and frustration level. The subject (which can be only one person or all team members) has to answer the questions on a 100-point-scale with 5-point steps (Table 1). NASA-TLX is a widely used technique for workload measurement in aviation, military and healthcare. NASA-TLX can be found in traditional MIS mental workload estimation [48,49,50,51,52], and employed in the case of surgical robotics workload assessment as well [8,32,34,40,41,43,45,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76]. There are additional mental workload assessment techniques that are not originally created for surgery, and used in workload assessment for RAMIS. Such examples are:
  • Multiple Resources Questionnaire (MRQ) [31,60,62,77,78].
  • Dundee Stress State Questionnaire (DSSQ) [31,78,79].
  • Rating Scale for Mental Effort (RSME) [42,80].
  • Psychometric Testing of Interpersonal Communication Skills Questionnaire (PTICSQ) [81].
  • Safety Attitudes Questionnaire (SAQ) [81,82]
  • Wisconsin Card Sorting Test (WCST) [57,83].
  • Coping Inventory of Task Stress (CITS) [31,84].
  • Subjective Mental Effort Questionnaire (SMEQ) [85].
  • Local Experienced Discomfort (LED) [85].
  • Short Stress State Questionnaire (SSSQ) [62,86].
MRQ estimates workload with 17 items, and it is specifically useful for multitasking workload measurements [77]. SSSQ is based on DSSQ, and both target stress measurement [86], such as CITS [84]. RSME and SMEQ estimate mental effort on a 9-point scale from extreme effort to absolutely no effort. RSME is validated in healthcare as well [80]. LED examines physical discomfort during a task [85]. For team communication quality estimation PTICSQ was created [81]. SAQ was developed for healthcare, which examines employees’ satisfaction with the job, teamwork, management, safety, stress and working conditions [82]. WCST is a neuropsychological tool, which was originally created for cognitive strategy adaptation measurements [83].
Surgery Task Load Index (SURG-TLX) (created by the cooperation of the University of Hong Kong, University of Exeter and the Department of Urology, Royal Devon and Exeter Hospital in 2011) is a modified NASA-TLX metric for surgical workload measurements [87]. SURG-TLX estimates the workload based on mental demands, physical demands, temporal demands, task complexity, situational stress and distractions (Table 2, Figure 2). SURG-TLX was tested on the Fundamentals of Laparoscopic Surgery (FLS) peg transfer task under stress, such as fatigue, multitasking, distraction and task novelty. However, the metric was validated for surgery but we could only find a few RAMIS publications on this topic [42,44,54]. Nevertheless, this topic is well-studied in traditional MIS [88,89,90,91,92], and to the best of the authors’ knowledge there is no workload self-rating measurement metric specifically created for RAMIS.
Self-rating techniques are not resource-intensive to implement, and they do not require human support, however, they typically show a bias. After all, it is still necessary to consider the usage of self-rating techniques in automated or expert-rating focused NTS and workload assessment studies, because these questionnaires can provide an easy validation tool for correlation examinations. With self-rating tools, the real stressors of the surgery can be observed, and other approaches have to fit to the clinical relevance. Self-rating studies can be found in Table 5 under the following references: [30,31,32,34,40,41,42,43,44,50,56,57,59,60,61,62,63,64,66,68,69,70,71,72,75,76,81,85,93,94,95].

3.2. Non-Technical Skill Assessment—Expert Rating

In surgical skill assessment, expert rating techniques are widely used, not just in the case of technical skill assessment, but for non-technical skill assessment as well. Therein, an expert panel (usually 8–10 expert surgeons) assesses the skills of the practicing surgeon, based on a video recording of the procedure/training session, based on a validated set of requirements. Expert rating assessment is relatively easy to complete (compared to automated techniques), more objective than self-assessment, but it definitely requires significant human resources, and it can still be biased for personal reasons. At the moment, expert rating technique is the gold standard for automated skill assessment.
In the case of non-technical skill assessment, there are several different expert-rating metrics for traditional MIS, such as NOTECHS, OTAS and NOTSS (Table 3). A few publications were identified which studied NOTSS in the case of RAMIS [32,33,96]. For surgical robotics, there is one metric which specifically measures the non-technical skills of robotic surgeons [22]; the Interpersonal and Cognitive Assessment for Robotic Surgery (ICARS), developed by Raison et al. in 2017. It was created by 16 expert surgeons with the Delphi methodology [97]. In ICARS, there were 28 non-technical skills identified (Figure 2), in 3 main non-technical skill categories, namely interpersonal skills (communication/teamwork and leadership), cognitive skills (decision making and situational awareness) and personal resource skills (cope with stress and distractions, Table 4. However, we could only find one clinical study which used ICARS for non-technical surgical skill assessment [96]. Despite the disadvantages of expert-rating techniques (need for an expert surgeon’s input, time, bias), they can still be a more objective tool for automated technique validation. They can provide a model for NTS assessment through the critical NTS categories and the given points. Expert-rating studies can be found in Table 5 under the following references: [8,22,32,40,42,45,53,54,55,57,58,60,64,65,67,73,74,96,98,99].

3.3. Automated Non-Technical Skill and Mental Workload Assessment in RAMIS

Establishing the correlation between physiological signals, kinematic data or other objectively measurable features and non-technical skills or mental workload can lead to autonomous non-technical skill assessment in RAMIS.
A common approach to assess the non-technical skills of the surgeon is through the measurement of physiological signals. However, this has limitations: the physiological signals are often linked to a particular non-technical skill, such as stress level, but they do not show other important factors (situational awareness, teamwork, etc.). In the literature, we can find physiological measurements related to the stress level, such as:
  • brain activity [103];
  • skin temperature [104,105];
  • nose temperature [106];
  • heart rate [107];
  • skin conductance [107];
  • blood pressure [107];
  • respiratory period [107];
  • tremor [108];
  • eye movement [109].
While these physiological signals are proven to be related to stress, they naturally have limitations in the usage of non-technical skills and cognitive load assessment. Such an example is skin conductance, which can be a useful technique to estimate workload [30], but it can be influenced by other physiological factors. Brain activity, heart rate and eye movement are the most studied signals in RAMIS, which can refer to more complex underlying behavior, such as technical skills [110], but the correlation between these signals and non-technical skills is harder to established.
In the literature, there are examples of the usage of an electroencephalogram (EEG) [34,43,45,69,70,72,93,111], given the fact that EEG measures the electrical activity of the brain [112]. While EEG is the most trivial physiological signal measurement technique for non-technical surgical skill assessment, the proven correlation between the measurable brain activity and non-technical skills is limited. Another approach for physiological signal-based mental workload assessment is the measurement of the heart rate (HR) [7,42,44,72,85]. However, the accuracy of HR measurements for cognitive load assessment was not enough in some cases, because there is no scale for maximum tolerated workload levels, and their related effects on the surgeon’s health [7]. The following forms of HR can be found in the non-technical skill assessment literature, however, the usage of them can be cumbersome [112,113]:
  • simple HR;
  • Heart Rate Variability (HRV);
  • mean square of successive differences between consecutive heartbeats (MSSD);
  • average heart rate (HRA).
Another objective method for non-technical skill or mental workload assessment is Functional Near-Infrared Spectroscopy (fNIRS) [44,114,115]. FNIRS is a functional neuroimaging technique to track the brain activity by monitoring the blood flow in the prefrontal lobe [116]. FNIRS shows a strong correlation with PET and fMRI data, yet it has better temporal resolution than fMRI but is limited compared to EEG; spatial resolution is more limited compared to fMRI, but better compared to EEG [117,118]. Furthermore, time of isovolumetric contraction (PEP) [119], electromyography and electrodermal [72] can also be used in mental workload assessment [85]; however, these signals can be influenced by the surgeon’s general health. Pupillary response is also studied in workload assessment [46].
As a summary, the following sensors/imaging techniques were studied in NTS and workload assessment in RAMIS (detailed in Table 5):
  • magnetic pose trackers;
  • EEG;
  • ECG;
  • fNIRS;
  • skin conductance sensor;
  • electromyograph (EMG);
  • eye-gaze tracker;
  • nose temperature and dryness sensor;
  • heart rate monitor.
Adequate sensor solutions in RAMIS do not only constitute external ones, but there are built-in internal sensors as well, which can greatly facilitate NTS skill assessment (see Section 4 for future works) and have become proven tools for technical skill assessment in RAMIS:
  • position sensors (encoders);
  • gyroscopes;
  • 2D/3D endoscopic camera.
In RAMIS research, there are typically integrated/employed sensors which are not directly related to NTS and workload assessment, but in most of the cases, their modalities show correlation with technical skills [17]. These sensor types include, but are not limited to, the following devices [120,121,122,123]:
  • force sensors (strain gauges, capacitive sensors, piezoelectric sensors, optical sensors);
  • tool position sensing (optical, electromagnetic);
  • master/surgeon arm position sensing (external);
  • wearable eyeglasses (Oculus Rift, Google Glass);
  • tool thermal sensor;
  • pressure sensors;
  • camera (RGBD, external);
  • communication (RF sensors);
  • speech (microspeaker);
  • sound (microphones).
Automated, sensory data-based NTS and workload assessment can be a key to an objective, reproducible approach to measure the surgeon’s skills without bias and the need of human resources. However, these techniques are typically costly, harder to implement and the usage of additional digital tools can be a problem in a clinical environment, even in an Internet of Things setup. Nevertheless, NTS and workload might be demonstrable in objective, technical skills, as suggested in [93], which means these sensors can provide an option for NTS assessment as well. As shown in this article, this research field is not studied widely yet. Automated technology-based studies can be found in Table 5 under the following references: [7,30,34,41,42,43,44,45,46,69,70,72,85,93,111,114,115].

4. Discussion

RAMIS related skill assessment is a relatively young research field, and the strong societal need for NTS and workload assessment have not appeared extensively in the literature yet. A few publications suggested objective, sensor-based non-technical skill and mental load evaluation in RAMIS. These approaches can provide a bias-free, reproducible solution in the clinical environment, and allow for the effortless collection of large datasets. Furthermore, during surgical education, personalized skill training would provide a more effective learning procedure, which can be achieved more easily when provided objective metrics. Nevertheless, such metrics are hard to implement, additional sensor usage can always be problematic in the surgical environment, and at the moment, there are no validated objective and automated metrics in NTS assessment. On the other hand, there are close relations in manual MIS and RAMIS, and in manual MIS, it is already suggested to approach NTS assessment with technical skill assessment metrics [38,124], which is a much more deeply studied area in RAMIS. It is assumed that technical and non-technical skills are not different in RAMIS, thus the connections of these two seemingly diverse research approaches shall be studied further. A validated manual technique could be achieved by a relatively simple statistical analysis, but in the case of automated techniques, appropriate test environment, amount of data, sensor usage, feature extraction and classification techniques should all be examined and validated.
For technical skill assessment, there are accurate results with kinematic [125] and video data [126] already. However, these studies only focused on the surgeon and not on the whole staff of the operating room. With external sensors (such as cameras) workflow and NTS (such as communication and teamwork) correlation can be further studied [127]. However, the first step of these studies is to examine the different sensor outputs, which can both correlate with technical and non-technical metrics. RAMIS built-in sensors (3D endoscopic camera and kinematic sensors) can significantly ease NTS and workload assessment, leading to established correlations between sensor outputs and/or self/expert-rating results.

5. Conclusions

In this article, we presented the findings of an extensive literature search, performed based on the standard PRISMA method, focusing on the domain of non-technical skill and mental load assessment in Robot-Assisted Minimally Invasive Surgery. Non-technical skills and mental workload directly influence the surgeon’s performance, and thereby the surgical outcome. The importance of non-technical skill assessment in robotic surgery is already recognized, however, there are not too many studies targeting this particular field. In traditional manual MIS, there are already validated metrics for non-technical skill assessment, yet it is clear that robotic surgery requires different non-technical skills from the surgeon, which might be monitored with alternative sensor systems. Alternative skills include advanced teamwork, capabilities to deal with new stress sources and different decisions to make. In the case of RAMIS for mental load assessment, authors often use self-rating techniques, such as NASA-TLX and SURG-TLX, meanwhile, there are no self-rating questionnaires specifically created for RAMIS yet. The existing primary technique for traditional surgery, SURG-TLX, defines the following workload categories: mental demands, physical demands, temporal demands, task complexity, situational stress and distractions. While SURG-TLX is similar to the general NASA-TLX, there are significant differences, such as the examination of distractions in the operating room. It concludes that workload factors for RAMIS can be different as well. For non-technical skill assessment, an often-used technique is expert rating, where a group of expert surgeons assesses the skills of the surgeon based on a validated metric, but this technique can be biased, and may require significant human resources.
The only established expert-rating tool for RAMIS is ICARS, which defines the following non-technical skills for surgeons: communication, teamwork, leadership, decision making, situation awareness and ability to cope with stress and distractions. The final frontier is a sensor-based objective, automated non-technical skill assessment method for RAMIS. Towards this, there are preliminary studies that use physiological signals, such as heart rate or the electrical activity of the brain. Most of the publications examined workload in RAMIS, a significant amount studied brain activity, but specific non-technical skills (in descending order: communication, stress, leadership, decision making, situation awareness and teamwork) can be found in the state-of-the-art as well. At the moment, there exists no widely accepted non-technical skill and mental workload assessment method in the clinical practice of RAMIS.

Author Contributions

Conceptualization, R.N.E. and T.H.; methodology, R.N.E. and T.H.; investigation, R.N.E. and T.H.; resources, R.N.E. and T.H.; writing—original draft, R.N.E.; writing—review and editing, T.H. All authors have read and agreed to the published version of the manuscript.


Authors thankfully acknowledge the financial support of this work by the Hungarian State and the European Union under the EFOP-3.6.1-16-2016-00010 project. T. Haidegger and R. Nagyné Elek are supported through the New National Excellence Program of the Ministry of Human Capacities. T. Haidegger is a Bolyai Fellow of the Hungarian Academy of Sciences.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Conflicts of Interest

The authors declare no conflict of interest.


  1. Haidegger, T.; Sándor, J.; Benyó, Z. Surgery in Space: The Future of Robotic Telesurgery. Surg. Endosc. 2011, 25, 681–690. [Google Scholar] [CrossRef] [PubMed]
  2. Ochsner, J.L. Minimally Invasive Surgical Procedures. Ochsner J. 2000, 2, 135–136. [Google Scholar]
  3. Haidegger, T. Autonomy for Surgical Robots: Concepts and Paradigms. IEEE Trans. Med. Robot. Bionics 2019, 1, 65–76. [Google Scholar] [CrossRef]
  4. Haidegger, T. Probabilistic Method to Improve the Accuracy of Computer-Integrated Surgical Systems. Acta Polytech. Hung. 2019, 16, 119–140. [Google Scholar] [CrossRef]
  5. Hernandez, J.D.; Bann, S.D.; Munz, Y.; Moorthy, K.; Datta, V.; Martin, S.; Dosis, A.; Bello, F.; Darzi, A.; Rockall, T. Qualitative and Quantitative Analysis of the Learning Curve of a Simulated Surgical Task on the Da Vinci System. Surg. Endosc. Other Interv. Tech. 2004, 18, 372–378. [Google Scholar] [CrossRef]
  6. Gomes, P. Surgical Robotics: Reviewing the Past, Analysing the Present, Imagining the Future. Robot. Comput. Integr. Manuf. 2011, 27, 261–266. [Google Scholar] [CrossRef]
  7. Heemskerk, J.; Zandbergen, H.R.; Keet, S.W.M.; Martijnse, I.; van Montfort, G.; Peters, R.J.A.; Svircevic, V.; Bouwman, R.A.; Baeten, C.G.M.I.; Bouvy, N.D. Relax, It’s Just Laparoscopy! A Prospective Randomized Trial on Heart Rate Variability of the Surgeon in Robot-Assisted versus Conventional Laparoscopic Cholecystectomy. Dig. Surg. 2014, 31, 225–232. [Google Scholar] [CrossRef]
  8. Stefanidis, D.; Wang, F.; Korndorffer, J.R.; Dunne, J.B.; Scott, D.J. Robotic Assistance Improves Intracorporeal Suturing Performance and Safety in the Operating Room While Decreasing Operator Workload. Surg. Endosc. 2010, 24, 377–382. [Google Scholar] [CrossRef]
  9. Cleary, K.; Nguyen, C. State of the Art in Surgical Robotics: Clinical Applications and Technology Challenges. Comput. Aided Surg. 2001, 6, 312–328. [Google Scholar] [CrossRef]
  10. Intuitive|Robotic-Assisted Surgery|Da Vinci Surgical System. Available online: (accessed on 22 January 2021).
  11. Peña, A. The Dreyfus Model of Clinical Problem-Solving Skills Acquisition: A Critical Perspective. Med. Educ. Online 2010, 15. [Google Scholar] [CrossRef]
  12. Azari, D.; Greenberg, C.; Pugh, C.; Wiegmann, D.; Radwin, R. In Search of Characterizing Surgical Skill. J. Surg. Educ. 2019, 76, 1348–1363. [Google Scholar] [CrossRef] [PubMed]
  13. Smith, R.; Truong, M.; Perez, M. Comparative Analysis of the Functionality of Simulators of the Da Vinci Surgical Robot. Surg. Endosc. 2015, 29, 972–983. [Google Scholar] [CrossRef] [PubMed]
  14. Meier, M.; Horton, K.; John, H. Da Vinci© Skills Simulator™: Is an Early Selection of Talented Console Surgeons Possible? J. Robot. Surg. 2016, 10, 289–296. [Google Scholar] [CrossRef]
  15. Matsuda, T.; Yoshida, K.; Habuchi, T.; Kinoshita, H.; Kanayama, H.O.; Terachi, T. Scientific Skill Assessment of Basic Surgical Dissection and Overall Laparoscopic Performance. J. Endourol. 2017, 31, S25–S29. [Google Scholar] [CrossRef]
  16. Fard, M.J.; Ameri, S.; Ellis, R.D. Toward Personalized Training and Skill Assessment in Robotic Minimally Invasive Surgery. arXiv 2016, arXiv:1610.07245. [Google Scholar]
  17. Nagyné Elek, R.; Haidegger, T. Robot-Assisted Minimally Invasive Surgical Skill Assessment—Manual and Automated Platforms. Acta Polytech. Hung. 2019, 16, 141–169. [Google Scholar]
  18. Hart, S.G. Nasa-Task Load Index (NASA-TLX); 20 Years Later. Proc. Hum. Factors Ergon. Soc. Annu. Meet. 2006, 50, 904–908. [Google Scholar] [CrossRef][Green Version]
  19. Gibaud, B.; Forestier, G.; Feldmann, C.; Ferrigno, G.; Gonçalves, P.; Haidegger, T.; Julliard, C.; Katić, D.; Kenngott, H.; Maier-Hein, L.; et al. Toward a Standard Ontology of Surgical Process Models. Int. J. Comput. Assist. Radiol. Surg. 2018, 13, 1397–1408. [Google Scholar] [CrossRef] [PubMed][Green Version]
  20. Gharagozloo, F.; Patel, V.R.; Giulianotti, P.C.; Poston, R.S.; Gruessner, R.; Meyer, M. Robotic Surgery, Defining and Validating Non-technical Skills Training in Robotics. Springer Nature Switzerland AG 2021, 75–81. [Google Scholar] [CrossRef]
  21. Sharma, B.; Mishra, A.; Aggarwal, R.; Grantcharov, T.P. Non-Technical Skills Assessment in Surgery. Surg. Oncol. 2011, 20, 169–177. [Google Scholar] [CrossRef]
  22. Raison, N.; Wood, T.; Brunckhorst, O.; Abe, T.; Ross, T.; Challacombe, B.; Khan, M.S.; Novara, G.; Buffi, N.; Van Der Poel, H.; et al. Development and Validation of a Tool for Non-Technical Skills Evaluation in Robotic Surgery-the ICARS System. Surg. Endosc. 2017, 31, 5403–5410. [Google Scholar] [CrossRef]
  23. Drexler, D.A.; Takács, A.; Nagy, T.D.; Haidegger, T. Handover Process of Autonomous Vehicles-technology and application challenges. Acta Polytech. Hung. 2019, 16.9, 235–255. [Google Scholar] [CrossRef]
  24. Yule, S.; Paterson-Brown, S. Surgeons’ Non-Technical Skills. Surg. Clin. 2012, 92, 37–50. [Google Scholar] [CrossRef] [PubMed]
  25. Hull, L.; Arora, S.; Aggarwal, R.; Darzi, A.; Vincent, C.; Sevdalis, N. The Impact of Nontechnical Skills on Technical Performance in Surgery: A Systematic Review. J. Am. Coll. Surg. 2012, 214, 214–230. [Google Scholar] [CrossRef]
  26. Collins, J.W.; Dell’Oglio, P.; Hung, A.J.; Brook, N.R. The Importance of Technical and Non-Technical Skills in Robotic Surgery Training. Eur. Urol. Focus 2018, 4, 674–676. [Google Scholar] [CrossRef] [PubMed]
  27. Urban, C.; Galambos, P.; Györök, G.; Haidegger, T. Simulated Medical Ultrasound Trainers A Review of Solutions and Applications. Acta Polytech. Hung. 2018, 15, 111–131. [Google Scholar]
  28. Collins, J.; Wisz, P. Training in Robotic Surgery, Replicating the Airline Industry. How Far Have We Come? World J. Urol. 2020, 38, 1645–1651. [Google Scholar] [CrossRef][Green Version]
  29. Sánchez, R.; Rodríguez, O.; Rosciano, J.; Vegas, L.; Bond, V.; Rojas, A.; Sanchez-Ismayel, A. Robotic Surgery Training: Construct Validity of Global Evaluative Assessment of Robotic Skills (GEARS). J. Robot. Surg. 2016, 10, 227–231. [Google Scholar] [CrossRef]
  30. Berguer, R.; Smith, W. An Ergonomic Comparison of Robotic and Laparoscopic Technique: The Influence of Surgeon Experience and Task Complexity. J. Surg. Res. 2006, 134, 87–92. [Google Scholar] [CrossRef]
  31. Klein, M.I.; Warm, J.S.; Riley, M.A.; Matthews, G.; Gaitonde, K.; Donovan, J.F.; Doarn, C.R. Performance, Stress, Workload, and Coping Profiles in 1st Year Medical Students’ Interaction with the Endoscopic/Laparoscopic and Robot-Assisted Surgical Techniques. Proc. Hum. Factors Ergon. Soc. Annu. Meet. 2008, 52, 885–889. [Google Scholar] [CrossRef]
  32. Ahmed, Y.; Lone, Z.; Hussein, A.A.; Feng, Y.; Khan, H.; Broad, S.; Kannappan, R.; Skowronski, A.; Cole, A.; Wang, D.; et al. Do Surgeon Non-Technical Skills Correlate with Teamwork-Related Outcomes during Robot-Assisted Surgery? BMJ Lead. 2019, 3, 69–74. [Google Scholar] [CrossRef]
  33. Raison, N.; Ahmed, K.; Abe, T.; Brunckhorst, O.; Novara, G.; Buffi, N.; McIlhenny, C.; van der Poel, H.; van Hemelrijck, M.; Gavazzi, A.; et al. Cognitive Training for Technical and Non-Technical Skills in Robotic Surgery: A Randomised Controlled Trial. BJU Int. 2018, 122, 1075–1081. [Google Scholar] [CrossRef] [PubMed]
  34. Shafiei, S.B.; Hussein, A.A.; Guru, K.A. Dynamic Changes of Brain Functional States during Surgical Skill Acquisition. PLoS ONE 2018, 13, e0204836. [Google Scholar] [CrossRef] [PubMed][Green Version]
  35. The Da Vinci Research Kit (dVRK)—Intuitive Foundation. Available online: (accessed on 22 January 2020).
  36. Chinzei, K. Safety of Surgical Robots and IEC 80601-2-77: The First International Standard for Surgical Robots. Acta Polytech. Hung. 2019, 16, 171–184. [Google Scholar]
  37. IEC 80601-2-77:2019. Available online: (accessed on 22 January 2021).
  38. Riem, N.; Boet, S.; Bould, M.D.; Tavares, W.; Naik, V.N. Do Technical Skills Correlate with Non-Technical Skills in Crisis Resource Management: A Simulation Study. Br. J. Anaesth. 2012, 109, 723–728. [Google Scholar] [CrossRef][Green Version]
  39. Guyatt, G.H.; Oxman, A.D.; Kunz, R.; Vist, G.E.; Falck-Ytter, Y.; Schünemann, H.J. What Is “Quality of Evidence” and Why Is It Important to Clinicians? BMJ Br. Med J. 2008, 336, 995–998. [Google Scholar] [CrossRef][Green Version]
  40. Tiferes, J.; Hussein, A.A.; Bisantz, A.; Kozlowski, J.D.; Sharif, M.A.; Winder, N.M.; Ahmad, N.; Allers, J.; Cavuoto, L.; Guru, K.A. The Loud Surgeon Behind the Console: Understanding Team Activities During Robot-Assisted Surgery. J. Surg. Educ. 2016, 73, 504–512. [Google Scholar] [CrossRef]
  41. Wu, C.; Cha, J.; Sulek, J.; Zhou, T.; Sundaram, C.P.; Wachs, J.; Yu, D. Eye-Tracking Metrics Predict Perceived Workload in Robotic Surgical Skills Training. Hum. Factors 2019, 18720819874544. [Google Scholar] [CrossRef] [PubMed][Green Version]
  42. Moore, L.J.; Wilson, M.R.; McGrath, J.S.; Waine, E.; Masters, R.S.W.; Vine, S.J. Surgeons’ Display Reduced Mental Effort and Workload While Performing Robotically Assisted Surgical Tasks, When Compared to Conventional Laparoscopy. Surg. Endosc. 2015, 29, 2553–2560. [Google Scholar] [CrossRef][Green Version]
  43. Hussein, A.A.; Shafiei, S.B.; Sharif, M.; Esfahani, E.; Ahmad, B.; Kozlowski, J.D.; Hashmi, Z.; Guru, K.A. Technical Mentorship during Robot-Assisted Surgery: A Cognitive Analysis. BJU Int. 2016, 118, 429–436. [Google Scholar] [CrossRef]
  44. Singh, H.; Modi, H.N.; Ranjan, S.; Dilley, J.W.R.; Airantzis, D.; Yang, G.Z.; Darzi, A.; Leff, D.R. Robotic Surgery Improves Technical Performance and Enhances Prefrontal Activation During High Temporal Demand. Ann. Biomed. Eng. 2018, 46, 1621–1636. [Google Scholar] [CrossRef] [PubMed][Green Version]
  45. Guru, K.A.; Shafiei, S.B.; Khan, A.; Hussein, A.A.; Sharif, M.; Esfahani, E.T. Understanding Cognitive Performance during Robot-Assisted Surgery. Urology 2015, 86, 751–757. [Google Scholar] [CrossRef] [PubMed]
  46. Nguyen, J.; Chen, J.; Marshall, S.; Ghodoussipour, S.; Chen, A.; Gill, I.; Hung, A. Using Objective Robotic Automated Performance Metrics and Task-Evoked Pupillary Response to Distinguish Surgeon Expertise. World J. Urol. 2020, 38, 1599–1605. [Google Scholar] [CrossRef]
  47. Hart, S.G.; Staveland, L.E. Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research. In Human Mental Workload; Advances in Psychology, 52; North-Holland: Oxford, UK, 1988; pp. 139–183. [Google Scholar] [CrossRef]
  48. Carswell, C.M.; Clarke, D.; Seales, W.B. Assessing Mental Workload during Laparoscopic Surgery. Surg. Innov. 2005, 12, 80–90. [Google Scholar] [CrossRef]
  49. Wadhera, R.K.; Parker, S.H.; Burkhart, H.M.; Greason, K.L.; Neal, J.R.; Levenick, K.M.; Wiegmann, D.A.; Sundt, T.M. Is the “Sterile Cockpit” Concept Applicable to Cardiovascular Surgery Critical Intervals or Critical Events? The Impact of Protocol-Driven Communication during Cardiopulmonary Bypass. J. Thorac. Cardiovasc. Surg. 2010, 139, 312–319. [Google Scholar] [CrossRef] [PubMed][Green Version]
  50. Klein, M.I.; Riley, M.A.; Warm, J.S.; Matthews, G. Perceived Mental Workload in an Endocopic Surgery Simulator. Proc. Hum. Factors Ergon. Soc. Annu. Meet. 2005, 49, 1014–1018. [Google Scholar] [CrossRef][Green Version]
  51. Ruiz-Rabelo, J.F.; Navarro-Rodriguez, E.; Di-Stasi, L.L.; Diaz-Jimenez, N.; Cabrera-Bermon, J.; Diaz-Iglesias, C.; Gomez-Alvarez, M.; Briceño-Delgado, J. Validation of the NASA-TLX Score in Ongoing Assessment of Mental Workload During a Laparoscopic Learning Curve in Bariatric Surgery. Obes. Surg. 2015, 25, 2451–2456. [Google Scholar] [CrossRef]
  52. Zheng, B.; Jiang, X.; Tien, G.; Meneghetti, A.; Panton, O.N.M.; Atkins, M.S. Workload Assessment of Surgeons: Correlation between NASA TLX and Blinks. Surg. Endosc. 2012, 26, 2746–2750. [Google Scholar] [CrossRef]
  53. Chowriappa, A.; Raza, S.J.; Fazili, A.; Field, E.; Malito, C.; Samarasekera, D.; Shi, Y.; Ahmed, K.; Wilding, G.; Kaouk, J.; et al. Augmented-Reality-Based Skills Training for Robot-Assisted Urethrovesical Anastomosis: A Multi-Institutional Randomised Controlled Trial. BJU Int. 2015, 115, 336–345. [Google Scholar] [CrossRef][Green Version]
  54. Lee, G.I.; Lee, M.R.; Clanton, T.; Clanton, T.; Sutton, E.; Park, A.E.; Marohn, M.R. Comparative Assessment of Physical and Cognitive Ergonomics Associated with Robotic and Traditional Laparoscopic Surgeries. Surg. Endosc. 2014, 28, 456–465. [Google Scholar] [CrossRef]
  55. Panait, L.; Shetty, S.; Shewokis, P.A.; Sanchez, J.A. Do Laparoscopic Skills Transfer to Robotic Surgery? J. Surg. Res. 2014, 187, 53–58. [Google Scholar] [CrossRef]
  56. Mouraviev, V.; Klein, M.; Schommer, E.; Thiel, D.D.; Samavedi, S.; Kumar, A.; Leveillee, R.J.; Thomas, R.; Pow-Sang, J.M.; Su, L.M.; et al. Urology Residents Experience Comparable Workload Profiles When Performing Live Porcine Nephrectomies and Robotic Surgery Virtual Reality Training Modules. J. Robot. Surg. 2016, 10, 49–56. [Google Scholar] [CrossRef]
  57. Lau, E.; Alkhamesi, N.A.; Schlachta, C.M. Impact of Robotic Assistance on Mental Workload and Cognitive Performance of Surgical Trainees Performing a Complex Minimally Invasive Suturing Task. Surg. Endosc. 2019. [Google Scholar] [CrossRef] [PubMed]
  58. Kawamura, K.; Kobayashi, Y.; Fujie, M.G. Development of Real-Time Simulation for Workload Quantization in Robotic Tele-Surgery. In Proceedings of the 2006 IEEE International Conference on Robotics and Biomimetics, Kunming, China, 17–20 December 2006; pp. 1420–1425. [Google Scholar] [CrossRef]
  59. Cavuoto, L.A.; Hussein, A.A.; Vasan, V.; Ahmed, Y.; Durrani, A.; Khan, S.; Cole, A.; Wang, D.; Kozlowski, J.; Ahmad, B.; et al. Improving Teamwork: Evaluating Workload of Surgical Team During Robot-Assisted Surgery. Urology 2017, 107, 120–125. [Google Scholar] [CrossRef]
  60. Klein, M.I.; Lio, C.H.; Grant, R.; Carswell, C.M.; Strup, S. A Mental Workload Study on the 2d and 3d Viewing Conditions of the Da Vinci Surgical Robot. Proc. Hum. Factors Ergon. Soc. Annu. Meet. 2009, 53, 1186–1190. [Google Scholar] [CrossRef]
  61. Sexton, K.; Johnson, A.; Gotsch, A.; Hussein, A.A.; Cavuoto, L.; Guru, K.A. Anticipation, Teamwork and Cognitive Load: Chasing Efficiency during Robot-Assisted Surgery. BMJ Qual. Saf. 2018, 27, 148–154. [Google Scholar] [CrossRef]
  62. Lee, G.I.; Lee, M.R. Can a Virtual Reality Surgical Simulation Training Provide a Self-Driven and Mentor-Free Skills Learning? Investigation of the Practical Influence of the Performance Metrics from the Virtual Reality Robotic Surgery Simulator on the Skill Learning and Associated Cognitive Workloads. Surg. Endosc. 2018, 32, 62–72. [Google Scholar] [CrossRef]
  63. Xu, S.; Perez, M.; Perrenot, C.; Hubert, N.; Hubert, J. Face, Content, Construct, and Concurrent Validity of a Novel Robotic Surgery Patient-Side Simulator: The Xperience™ Team Trainer. Surg. Endosc. 2016, 30, 3334–3344. [Google Scholar] [CrossRef]
  64. Hughes-Hallett, A.; Mayer, E.K.; Marcus, H.J.; Pratt, P.; Mason, S.; Darzi, A.W.; Vale, J.A. Inattention Blindness in Surgery. Surg. Endosc. 2015, 29, 3184–3189. [Google Scholar] [CrossRef]
  65. Sethi, A.S.; Peine, W.J.; Mohammadi, Y.; Sundaram, C.P. Validation of a Novel Virtual Reality Robotic Simulator. J. Endourol. 2009, 23, 503–508. [Google Scholar] [CrossRef] [PubMed]
  66. Teoh, A.Y.B.; Chan, S.M.; Yip, H.C.; Wong, V.W.Y.; Chiu, P.W.Y.; Ng, E.K.W. Randomized Controlled Trial of EndoWrist-Enabled Robotic versus Human Laparoendoscopic Single-Site Access Surgery (LESS) in the Porcine Model. Surg. Endosc. 2018, 32, 1273–1279. [Google Scholar] [CrossRef] [PubMed]
  67. Kawamura, K.; Kobayashi, Y.; Fujie, M.G. Operability Evaluation Using an Simulation System for Gripping Motion in Robotic Tele-Surgery. In Proceedings of the 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Minneapolis, MN, USA, 3–6 September 2009; Volume 2009, pp. 5106–5109. [Google Scholar] [CrossRef]
  68. Gerull, W.; Zihni, A.; Awad, M. Operative Performance Outcomes of a Simulator-Based Robotic Surgical Skills Curriculum. Surg. Endosc. 2019. [Google Scholar] [CrossRef]
  69. Shafiei, S.B.; Hussein, A.A.; Muldoon, S.F.; Guru, K.A. Functional Brain States Measure Mentor-Trainee Trust during Robot-Assisted Surgery. Sci. Rep. 2018, 8, 3667. [Google Scholar] [CrossRef] [PubMed][Green Version]
  70. Borghini, G.; Arico, P.; Di Flumeri, G.; Colosimo, A.; Storti, S.F.; Menegaz, G.; Fiorini, P.; Babiloni, F. Neurophysiological Measures for Users’ Training Objective Assessment during Simulated Robot-Assisted Laparoscopic Surgery. In Proceedings of the 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Orlando, FL, USA, 16–20 August 2016; Volume 2016, pp. 981–984. [Google Scholar] [CrossRef]
  71. Marçon, B.; Sime, W.; Guillemin, F.; Hubert, N.; Lagrange, F.; Huselstein, C.; Hubert, J. An Ergonomic Assessment of Four Different Donor Nephrectomy Approaches for the Surgeons and Their Assistants. Res. Rep. Urol. 2019, 11, 261–268. [Google Scholar] [CrossRef] [PubMed][Green Version]
  72. Zhou, T.; Cha, J.; Gonzalez, G.; Wachs, J.; Sundaram, C.; Yu, D. Joint Surgeon Attributes Estimation in Robot-Assisted Surgery. In Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction, Chicago, IL, USA, 5–8 March 2018; pp. 285–286. [Google Scholar] [CrossRef]
  73. Kawamura, K.; Kobayashi, Y.; Fujie, M.G. Pilot Study on Verification of Effectiveness on Operability of Assistance System for Robotic Tele-Surgery Using Simulation. In Proceedings of the 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology, Buenos Aires, Argentina, 31 August–4 September 2010; Volume 2010, pp. 2308–2312. [Google Scholar] [CrossRef]
  74. Dulan, G.; Rege, R.V.; Hogg, D.C.; Gilberg-Fisher, K.M.; Arain, N.A.; Tesfay, S.T.; Scott, D.J. Proficiency-Based Training for Robotic Surgery: Construct Validity, Workload, and Expert Levels for Nine Inanimate Exercises. Surg. Endosc. 2012, 26, 1516–1521. [Google Scholar] [CrossRef]
  75. Turiani Hourneaux de Moura, D.; Aihara, H.; Jirapinyo, P.; Farias, G.; Hathorn, K.; Bazarbashi, A.; Sachdev, A.; Thompson, C. Robot-Assisted Endoscopic Submucosal Dissection versus Conventional ESD for Colorectal Lesions: Outcomes of a Randomized Pilot Study in Endoscopists without Prior ESD Experience (with Video). Gastrointest. Endosc. 2019, 90, 290–298. [Google Scholar] [CrossRef] [PubMed]
  76. Law, K.E.; Lowndes, B.R.; Kelley, S.R.; Blocker, R.C.; Larson, D.W.; Hallbeck, M.S.; Nelson, H. NASA-Task Load Index Differentiates Surgical Approach: Opportunities for Improvement in Colon and Rectal Surgery. Ann. Surg. 2020, 271, 906–912. [Google Scholar] [CrossRef]
  77. Boles, D.B.; Adair, L.P. The Multiple Resources Questionnaire (MRQ). In Proceedings of the Human Factors and Ergonomics Society Annual Meeting; Sage: Los Angeles, CA, USA, 2001; Volume 45, pp. 1790–1794. [Google Scholar] [CrossRef]
  78. Klein, M.I.; Warm, J.S.; Riley, M.A.; Matthews, G.; Doarn, C.; Donovan, J.F.; Gaitonde, K. Mental Workload and Stress Perceived by Novice Operators in the Laparoscopic and Robotic Minimally Invasive Surgical Interfaces. J. Endourol. 2012, 26, 1089–1094. [Google Scholar] [CrossRef]
  79. Matthews, G. Multidimensional Profiling of Task Stress States for Human Factors: A Brief Review. Hum. Factors 2016, 58, 801–813. [Google Scholar] [CrossRef]
  80. Ghanbary Sartang, A.; Ashnagar, M.; Habibi, E.; Sadeghi, S. Evaluation of Rating Scale Mental Effort (RSME) Effectiveness for Mental Workload Assessment in Nurses. J. Occup. Health Epidemiol. 2016, 5, 211–217. [Google Scholar] [CrossRef]
  81. Schiff, L.; Tsafrir, Z.; Aoun, J.; Taylor, A.; Theoharis, E.; Eisenstein, D. Quality of Communication in Robotic Surgery and Surgical Outcomes. JSLS J. Soc. Laparoendosc. Surg. 2016, 20. [Google Scholar] [CrossRef][Green Version]
  82. Sexton, J.B.; Helmreich, R.L.; Neilands, T.B.; Rowan, K.; Vella, K.; Boyden, J.; Roberts, P.R.; Thomas, E.J. The Safety Attitudes Questionnaire: Psychometric Properties, Benchmarking Data, and Emerging Research. BMC Health Serv. Res. 2006, 6, 44. [Google Scholar] [CrossRef][Green Version]
  83. Kohli, A.; Kaur, M. Wisconsin Card Sorting Test: Normative Data and Experience. Indian J. Psychiatry 2006, 48, 181–184. [Google Scholar] [CrossRef] [PubMed][Green Version]
  84. Li, C.; Liu, Q.; Hu, T.; Jin, X. Adapting the Short Form of the Coping Inventory for Stressful Situations into Chinese. Neuropsychiatr. Dis. Treat. 2017, 13, 1669–1675. [Google Scholar] [CrossRef][Green Version]
  85. Van der Schatte Olivier, R.H.; Van’t Hullenaar, C.D.P.; Ruurda, J.P.; Broeders, I.A.M.J. Ergonomics, User Comfort, and Performance in Standard and Robot-Assisted Laparoscopic Surgery. Surg. Endosc. 2009, 23, 1365–1371. [Google Scholar] [CrossRef][Green Version]
  86. Helton, W.S. Validation of a Short Stress State Questionnaire. Proc. Hum. Factors Ergon. Soc. Annu. Meet. 2016. [Google Scholar] [CrossRef]
  87. Wilson, M.R.; Poolton, J.M.; Malhotra, N.; Ngo, K.; Bright, E.; Masters, R.S.W. Development and Validation of a Surgical Workload Measure: The Surgery Task Load Index (SURG-TLX). World J. Surg. 2011, 35, 1961–1969. [Google Scholar] [CrossRef][Green Version]
  88. Malhotra, N.; Poolton, J.M.; Wilson, M.R.; Fan, J.K.M.; Masters, R.S.W. Conscious Motor Processing and Movement Self-Consciousness: Two Dimensions of Personality That Influence Laparoscopic Training. J. Surg. Educ. 2014, 71, 798–804. [Google Scholar] [CrossRef]
  89. Abdelrahman, A.M.; Bingener, J.; Yu, D.; Lowndes, B.R.; Mohamed, A.; McConico, A.L.; Hallbeck, M.S. Impact of Single-Incision Laparoscopic Cholecystectomy (SILC) versus Conventional Laparoscopic Cholecystectomy (CLC) Procedures on Surgeon Stress and Workload: A Randomized Controlled Trial. Surg. Endosc. 2016, 30, 1205–1211. [Google Scholar] [CrossRef][Green Version]
  90. Weigl, M.; Stefan, P.; Abhari, K.; Wucherer, P.; Fallavollita, P.; Lazarovici, M.; Weidert, S.; Euler, E.; Catchpole, K. Intra-Operative Disruptions, Surgeon’s Mental Workload, and Technical Performance in a Full-Scale Simulated Procedure. Surg. Endosc. 2016, 30, 559–566. [Google Scholar] [CrossRef]
  91. Berg, R.J.; Inaba, K.; Sullivan, M.; Okoye, O.; Siboni, S.; Minneti, M.; Teixeira, P.G.; Demetriades, D. The Impact of Heat Stress on Operative Performance and Cognitive Function during Simulated Laparoscopic Operative Tasks. Surgery 2015, 157, 87–95. [Google Scholar] [CrossRef]
  92. Weigl, M.; Antoniadis, S.; Chiapponi, C.; Bruns, C.; Sevdalis, N. The Impact of Intra-Operative Interruptions on Surgeons’ Perceived Workload: An Observational Study in Elective General and Orthopedic Surgery. Surg. Endosc. 2015, 29, 145–153. [Google Scholar] [CrossRef]
  93. Guru, K.A.; Esfahani, E.T.; Raza, S.J.; Bhat, R.; Wang, K.; Hammond, Y.; Wilding, G.; Peabody, J.O.; Chowriappa, A.J. Cognitive Skills Assessment during Robot-Assisted Surgery: Separating the Wheat from the Chaff. BJU Int. 2015, 115, 166–174. [Google Scholar] [CrossRef]
  94. Walters, C.; Webb, P.J. Maximizing Efficiency and Reducing Robotic Surgery Costs Using the NASA Task Load Index. AORN J. 2017, 106, 283–294. [Google Scholar] [CrossRef]
  95. Yu, D.; Dural, C.; Morrow, M.M.B.; Yang, L.; Collins, J.W.; Hallbeck, S.; Kjellman, M.; Forsman, M. Intraoperative Workload in Robotic Surgery Assessed by Wearable Motion Tracking Sensors and Questionnaires. Surg. Endosc. 2017, 31, 877–886. [Google Scholar] [CrossRef]
  96. Beulens, A.; Brinkman, W.; Koldewijn, E.; Hendrikx, A.; van Basten, J.; van Merriënboer, J.; Van der Poel, H.; Bangma, C.; Wagner, C. A Prospective, Observational, Multicentre Study Concerning Nontechnical Skills in Robot-Assisted Radical Cystectomy versus Open Radical Cystectomy. Eur. Urol. Open Sci. 2020, 19, 37–44. [Google Scholar] [CrossRef]
  97. Thangaratinam, S.; Redman, C.W. The Delphi Technique. Obstet. Gynaecol. 2005, 7, 120–125. [Google Scholar] [CrossRef]
  98. Catchpole, K.; Perkins, C.; Bresee, C.; Solnik, M.J.; Sherman, B.; Fritch, J.; Gross, B.; Jagannathan, S.; Hakami-Majd, N.; Avenido, R.; et al. Safety, Efficiency and Learning Curves in Robotic Surgery: A Human Factors Analysis. Surg. Endosc. 2016, 30, 3749–3761. [Google Scholar] [CrossRef]
  99. Zattoni, F.; Guttilla, A.; Crestani, A.; De Gobbi, A.; Cattaneo, F.; Moschini, M.; Vianello, F.; Valotto, C.; Dal Moro, F.; Zattoni, F. The Value of Open Conversion Simulations During Robot-Assisted Radical Prostatectomy: Implications for Robotic Training Curricula. J. Endourol. 2015, 29, 1282–1288. [Google Scholar] [CrossRef]
  100. Sevdalis, N.; Davis, R.; Koutantji, M.; Undre, S.; Darzi, A.; Vincent, C.A. Reliability of a Revised NOTECHS Scale for Use in Surgical Teams. Am. J. Surg. 2008, 196, 184–190. [Google Scholar] [CrossRef]
  101. Yule, S.; Flin, R.; Paterson-Brown, S.; Maran, N. Non-Technical Skills for Surgeons in the Operating Room: A Review of the Literature. Surgery 2006, 139, 140–149. [Google Scholar] [CrossRef] [PubMed]
  102. Hull, L.; Arora, S.; Kassab, E.; Kneebone, R.; Sevdalis, N. Observational Teamwork Assessment for Surgery: Content Validation and Tool Refinement. J. Am. Coll. Surg. 2011, 212, 234–243.e5. [Google Scholar] [CrossRef] [PubMed]
  103. Ahn, J.W.; Ku, Y.; Kim, H.C. A Novel Wearable EEG and ECG Recording System for Stress Assessment. Sensors 2019, 19, 1991. [Google Scholar] [CrossRef][Green Version]
  104. Herborn, K.A.; Graves, J.L.; Jerem, P.; Evans, N.P.; Nager, R.; McCafferty, D.J.; McKeegan, D.E. Skin Temperature Reveals the Intensity of Acute Stress. Physiol. Behav. 2015, 152, 225–230. [Google Scholar] [CrossRef][Green Version]
  105. Pavlidis, I.; Tsiamyrtzis, P.; Shastri, D.; Wesley, A.; Zhou, Y.; Lindner, P.; Buddharaju, P.; Joseph, R.; Mandapati, A.; Dunkin, B.; et al. Fast by Nature—How Stress Patterns Define Human Experience and Performance in Dexterous Tasks. Sci. Rep. 2012, 2, 305. [Google Scholar] [CrossRef][Green Version]
  106. How the Temperature of Your Nose Shows How Much Strain You Are under—The University of Nottingham. Available online: (accessed on 20 December 2019).
  107. Laetitia Lisetti, C.L.; Nasoz, F. Using Noninvasive Wearable Computers to Recognize Human Emotions from Physiological Signals. EURASIP J. Appl. Signal Process. 2004, 2004, 1672–1687. [Google Scholar] [CrossRef][Green Version]
  108. Youngson, G.G. Nontechnical Skills in Pediatric Surgery: Factors Influencing Operative Performance. J. Pediatr. Surg. 2016, 51, 226–230. [Google Scholar] [CrossRef]
  109. Di Stasi, L.L.; McCamy, M.B.; Macknik, S.L.; Mankin, J.A.; Hooft, N.; Catena, A.; Martinez-Conde, S. Saccadic Eye Movement Metrics Reflect Surgical Residents’ Fatigue. Ann. Surg. 2014, 259, 824–829. [Google Scholar] [CrossRef]
  110. Law, B.; Atkins, M.S.; Kirkpatrick, A.E.; Lomax, A.J. Eye Gaze Patterns Differentiate Novice and Experts in a Virtual Laparoscopic Surgery Training Environment. In ETRA ’04: Proceedings of the 2004 Symposium on Eye Tracking Research & Applications; Association for Computing Machinery: San Antonio, TX, USA, 2004; pp. 41–48. [Google Scholar] [CrossRef][Green Version]
  111. Shahbazi, M.; Poursartip, B.; Siroen, K.; Schlachta, C.M.; Patel, R.V. Robotics-Assisted Surgical Skills Evaluation Based on Electrocortical Activity. In Proceedings of the 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Honolulu, HI, USA, 18–21 July 2018; Volume 2018, pp. 3673–3676. [Google Scholar] [CrossRef]
  112. Sigl, J.C.; Chamoun, N.G. An Introduction to Bispectral Analysis for the Electroencephalogram. J. Clin. Monit. 1994, 10, 392–404. [Google Scholar] [CrossRef]
  113. Shaffer, F.; Ginsberg, J.P. An Overview of Heart Rate Variability Metrics and Norms. Front. Public Health 2017, 5. [Google Scholar] [CrossRef] [PubMed][Green Version]
  114. Mylonas, G.P.; Kwok, K.W.; James, D.R.C.; Leff, D.; Orihuela-Espina, F.; Darzi, A.; Yang, G.Z. Gaze-Contingent Motor Channelling, Haptic Constraints and Associated Cognitive Demand for Robotic MIS. Med. Image Anal. 2012, 16, 612–631. [Google Scholar] [CrossRef]
  115. James, D.R.C.; Orihuela-Espina, F.; Leff, D.R.; Mylonas, G.P.; Kwok, K.W.; Darzi, A.W.; Yang, G.Z. Cognitive Burden Estimation for Visuomotor Learning with fNIRS. In International Conference on Medical Image Computing and Computer-Assisted Intervention; Springer: Berlin/Heidelberg, Germany, 2010; Volume 13, pp. 319–326. [Google Scholar] [CrossRef][Green Version]
  116. Bunce, S.; Izzetoglu, M.; Izzetoglu, K.; Onaral, B.; Pourrezaei, K. Functional Near-Infrared Spectroscopy. IEEE Eng. Med. Biol. Mag. 2006, 25, 54–62. [Google Scholar] [CrossRef] [PubMed]
  117. Venclove, S.; Daktariunas, A.; Ruksenas, O. Functional Near-Infrared Spectroscopy: A Continuous Wave Type Based System for Human Frontal Lobe Studies. EXCLI J. 2015, 14, 1145–1152. [Google Scholar] [CrossRef]
  118. Wilcox, T.; Biondi, M. fNIRS in the Developmental Sciences. Wiley Interdiscip. Rev. Cogn. Sci. 2015, 6, 263–283. [Google Scholar] [CrossRef][Green Version]
  119. Frank, M.N.; Kinlaw, W.B. Indirect Measurement of Isovolumetric Contraction Time and Tension Period in Normal Subjects. Am. J. Cardiol. 1962, 10, 800–806. [Google Scholar] [CrossRef]
  120. How Will Sensor Technology Impact the Medical Robots of the Future?|MediSens Conference. Available online: (accessed on 29 March 2021).
  121. Fu, G.; Azimi, E.; Kazanzides, P. Mobile Teleoperation: Evaluation of Wireless Wearable Sensing of the Operator’s Arm Motion. arXiv 2021, arXiv:2103.08119. Comment: 6 pages, 11 figures. Submitted to 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pending review. [Google Scholar]
  122. Abeywardena, S.; Yuan, Q.; Tzemanaki, A.; Psomopoulou, E.; Droukas, L.; Melhuish, C.; Dogramadzi, S. Estimation of Tool-Tissue Forces in Robot-Assisted Minimally Invasive Surgery Using Neural Networks. Front. Robot. AI 2019, 6. [Google Scholar] [CrossRef] [PubMed][Green Version]
  123. Thai, M.T.; Phan, P.T.; Hoang, T.T.; Wong, S.; Lovell, N.H.; Do, T.N. Advanced Intelligent Systems for Surgical Robotics. Adv. Intell. Syst. 2020, 2, 1900138. [Google Scholar] [CrossRef]
  124. Mishra, A.; Catchpole, K.; Dale, T.; McCulloch, P. The Influence of Non-Technical Performance on Technical Outcome in Laparoscopic Cholecystectomy. Surg. Endosc. 2008, 22, 68–73. [Google Scholar] [CrossRef]
  125. Zia, A.; Essa, I. Automated Surgical Skill Assessment in RMIS Training. Int. J. Comput. Assist. Radiol. Surg. 2018, 13, 731–739. [Google Scholar] [CrossRef][Green Version]
  126. Funke, I.; Mees, S.T.; Weitz, J.; Speidel, S. Video-Based Surgical Skill Assessment Using 3D Convolutional Neural Networks. Int. J. Comput. Assist. Radiol. Surg. 2019, 14, 1217–1225. [Google Scholar] [CrossRef] [PubMed][Green Version]
  127. Nagy, D.A.; Takács, K.; Rudas, I.J.; Haidegger, T. Employing Process Models for Surgical Training. In Proceedings of the 2020 IEEE 18th World Symposium on Applied Machine Intelligence and Informatics (SAMI), Herlany, Slovakia, 23–25 January 2020; pp. 101–104. [Google Scholar]
Figure 1. The da Vinci Surgical System with the identified non-technical skills and workload. The surgeon operates at the master side of the system, while the assistants can help them work at the patient side. The patient side arms are controlled by the surgeon with the master arms. Robot-Assisted Minimally Invasive Surgery requires not just technical skills, but non-technical skills as well from the operating crew, namely inter-personal skills, leadership, cognitive skills and personal resource skills, while they have to deal with the workload. Original image credit: Intuitive Surgical Inc. [10].
Figure 1. The da Vinci Surgical System with the identified non-technical skills and workload. The surgeon operates at the master side of the system, while the assistants can help them work at the patient side. The patient side arms are controlled by the surgeon with the master arms. Robot-Assisted Minimally Invasive Surgery requires not just technical skills, but non-technical skills as well from the operating crew, namely inter-personal skills, leadership, cognitive skills and personal resource skills, while they have to deal with the workload. Original image credit: Intuitive Surgical Inc. [10].
Sensors 21 02666 g001
Figure 2. Workload categories and non-technical skills in Robot-Assisted Minimally Invasive Surgery (RAMIS), based on the SURG-TLX workload questionnaire and the ICARS expert-rating assessment tool. At the moment, there is no universal solution for mental workload assessment specifically created for RAMIS.
Figure 2. Workload categories and non-technical skills in Robot-Assisted Minimally Invasive Surgery (RAMIS), based on the SURG-TLX workload questionnaire and the ICARS expert-rating assessment tool. At the moment, there is no universal solution for mental workload assessment specifically created for RAMIS.
Sensors 21 02666 g002
Figure 3. A Robot-Assisted Minimally Invasive Surgical system architecture and typical layout diagram with the most important sensor components in the case of non-technical skill assessment and mental load evaluation based on the International Electrotechnical Commission (IEC) 80601-2-77 robotic surgery safety standard [37].
Figure 3. A Robot-Assisted Minimally Invasive Surgical system architecture and typical layout diagram with the most important sensor components in the case of non-technical skill assessment and mental load evaluation based on the International Electrotechnical Commission (IEC) 80601-2-77 robotic surgery safety standard [37].
Sensors 21 02666 g003
Figure 4. PRISMA chart of the literature search results.
Figure 4. PRISMA chart of the literature search results.
Sensors 21 02666 g004
Figure 5. Bar chart of the literature search results. Each bar refers to the number of references identified and examined the particular feature/non-technical skill in RAMIS.
Figure 5. Bar chart of the literature search results. Each bar refers to the number of references identified and examined the particular feature/non-technical skill in RAMIS.
Sensors 21 02666 g005
Table 1. NASA-TLX mental workload self-rating questionnaire [18].
Table 1. NASA-TLX mental workload self-rating questionnaire [18].
Mental demandslow/highHow much mental activity was required?
Physical demandslow/highHow much physical activity was required?
Temporal demandslow/highHow much time pressure did you feel?
Effortlow/highHow hard did you have to work?
Performancegood/poorHow stressful do you think you were?
Frustration levellow/highHow frustrated did you feel?
Table 2. SURG-TLX mental workload self-rating questionnaire [87].
Table 2. SURG-TLX mental workload self-rating questionnaire [87].
Mental demandslow/highHow mentally fatiguing was the procedure?
Physical demandslow/highHow physically fatiguing was the procedure?
Temporal demandslow/highHow hurried or rushed was the pace of the procedure?
Task complexitylow/highHow complex was the procedure?
Situational stresslow/highHow anxious did you feel while performing the procedure?
Distractionslow/highHow distracting was the operating environment?
Table 3. Behavioral rating systems in traditional surgery compared to ICARS, the only established non-technical skill assessment metric particularly for RAMIS [21,22].
Table 3. Behavioral rating systems in traditional surgery compared to ICARS, the only established non-technical skill assessment metric particularly for RAMIS [21,22].
Non-technical skills
  • Communication and interaction
  • Situational awareness
  • Team skills
  • Leadership and management
  • Decision making
  • Situational awareness
  • Decision making
  • Task management
  • Leadership
  • Communication
  • Teamwork
  • Task checklist
  • Shared monitoring
  • Communication
  • Cooperation
  • Coordination
  • Shared leadership
  • Communication and teamwork
  • Leadership
  • Decision making
  • Situational awareness
  • Cope with stress and distractors
Content validity
Construct validity
Inter-rater reliability
Sensitivityn.a.not acceptable
in some
Feasibility✓ (especially for self-assessment)limited to
Table 4. Interpersonal and Cognitive Assessment for Robotic Surgery (ICARS) expert rating metrics [22].
Table 4. Interpersonal and Cognitive Assessment for Robotic Surgery (ICARS) expert rating metrics [22].
NTS CategoryNTS GroupNTS
Interpersonal skillsCommunication and teamworkEffective verbal communication
Appropriate interaction with bedside surgeon
Appropriate interaction with operating room staff
Engages/initiates in confirmatory feedback with OR staff
LeadershipAppropriate and polite instructions
Effective workload management
Coordination of the team from the console
Coordination of the team at the bedside
Delegating tasks to team members
Maintenance of professional standards
Cognitive skillsDecision makingAppropriate decision making in case of equipment failure
Appropriate decision making at the bedside
Quick diagnosis of unexpected patient events
Quick decision making in case of emergency
Generation, selection and implementation of solutions
Outcome review of decision
Situation awarenessAwareness of patient status
Ability to deal with patient at the bedside
Ability of quick adaptation to problems
Anticipation of potential problems
Role awareness of surrounding team members at the console
Personal resource skillsCope with stress and distractorsUnderstands personal limitations and asks for help
(if necessary)
Identification of stressor
Maintenance of cognitive skills
Maintenance of technical skills
Professional and appropriate choice of resolution
Table 5. Non-technical skill and mental workload assessment in surgical robotics. Used abbreviations: RAMIS: Robot-Assisted Minimally Invasive Surgery, OR: Operating Room, VR: Virtual Reality, EEG: electroencephalogram, NASA-TLX: NASA Task Load Index, SURG-TLX: Surgery Task Load Index, NOTSS: Non-Technical Skills for Surgeons, MRQ: Multiple Resources Questionnaire, DSSQ: Dundee Stress State Questionnaire, ECG: electrocardiogram, HR: heart rate, HRV: heart rate variability, RSME: Rating Scale for Mental Effort, PTICSQ: Psychometric Testing of Interpersonal Communication Skills Questionnaire, SAQ: Safety Attitudes Questionnaire, fNIRS: Functional Near-Infrared Spectroscopy, PVT: Psychomotor Vigilance Test, WCST: Wisconsin Card Sorting Test, CITS: Coping Inventory of Task Stress, MSSD: mean square of successive differences between consecutive heartbeats, PEP: time of isovolumetric contraction, HRA: average heart rate, SMEQ: Subjective Mental Effort Questionnaire, LED: Local Experienced Discomfort, SSSQ: Short Stress State Questionnaire, p.: procedures (where no subject data were available), QoE: Quality of Evidence, mod.: moderate.
Table 5. Non-technical skill and mental workload assessment in surgical robotics. Used abbreviations: RAMIS: Robot-Assisted Minimally Invasive Surgery, OR: Operating Room, VR: Virtual Reality, EEG: electroencephalogram, NASA-TLX: NASA Task Load Index, SURG-TLX: Surgery Task Load Index, NOTSS: Non-Technical Skills for Surgeons, MRQ: Multiple Resources Questionnaire, DSSQ: Dundee Stress State Questionnaire, ECG: electrocardiogram, HR: heart rate, HRV: heart rate variability, RSME: Rating Scale for Mental Effort, PTICSQ: Psychometric Testing of Interpersonal Communication Skills Questionnaire, SAQ: Safety Attitudes Questionnaire, fNIRS: Functional Near-Infrared Spectroscopy, PVT: Psychomotor Vigilance Test, WCST: Wisconsin Card Sorting Test, CITS: Coping Inventory of Task Stress, MSSD: mean square of successive differences between consecutive heartbeats, PEP: time of isovolumetric contraction, HRA: average heart rate, SMEQ: Subjective Mental Effort Questionnaire, LED: Local Experienced Discomfort, SSSQ: Short Stress State Questionnaire, p.: procedures (where no subject data were available), QoE: Quality of Evidence, mod.: moderate.
Ref.DateSubj.EnvironmentInputMeasured Feature/NTSConclusionQoE
[30]200610Dry labSkin conductance Self-rating (custom)Workload StressStress is less in the case of RAMIS compared to traditional MIS.mod.
[58]20065VR simulatorNASA-TLXWorkloadWorkload can be increased in proportion to delay time with the proposed simulators.low
[31]200815Dry labDSSQ MRQ CITSWorkload StressStress is less, workload and stress coping strategies are the same in the case of RAMIS compared to traditional MIS.low
[65]200920VR simulatorNASA-TLXWorkloadMimic dV-Trainer shows reasonable workload results.low
[60]200915Dry labNASA-TLX MRQWorkloadThe usage of the da Vinci 3D view causes less workload compared to the 2D view in some cases.low
[67]20096VR simulatorNASA-TLXWorkloadTime delay in teleoperation can significantly increase the workload.low
[85]200916Dry labMSSD PEP HRA SMEQ LEDWorkload StressRAMIS causes less cognitive workload compared to traditional MIS.low
[8]201034Live porcineNASA-TLXWorkloadRAMIS poses less mental workload compared to traditional MIS.mod.
[73]20103VR simulatorNASA-TLXWorkloadWorkload is not improved under delays of 300 ms and 400 ms in the simulated environment.low
[115]201021VR simulatorfNIRSWorkloadFNIRS can show the cognitive burden during training.high
[78]201215Dry labMRQ DSSQWorkload StressNovices have less stress when working with the da Vinci compared to traditional MIS.low
[74]201212Dry labNASA-TLXWorkloadAfter the proposed training, mental workload is similar between novices and experts.low
[114]201221VR simulatorfNIRSCortical activityThere is a significant difference between expert and non-expert subjects with Gaze-Contingent Motor Channeling.mod.
[7]20142ORHR HRVStressRAMIS poses less mental workload compared to traditional MIS. Workload measurement with HRV is cumbersome.mod.
[55]201428Dry labNASA-TLXWorkloadRAMIS poses significantly better workload perception compared to traditional MIS.low
[54]201413Dry labNASA-TLXWorkloadPhysiological and cognitive ergonomics with robotic surgery are significantly less challenging compared to traditional MIS.low
[53]201452VR simulatorNASA-TLXWorkloadUrethrovesical anastomosis VR training improves technical skill acquisition with cognitive demand.mod.
[93]201510Dry labEEGCognitive engagement Mental workload Mental stateCognitive assessment can define the expertise levels.high
[42]201532Dry labSURG-TLX RSME Heart rate monitorWorkload HRVRAMIS poses less mental workload compared to traditional MIS.mod.
[99]20156Simulated ORExpert rating (custom)Communication LeadershipRepeated simulations and increased leadership mean faster and less flawed conversions in the OR.mod.
[64]201524Image displayNASA-TLXWorkloadIncreasing the level of cognitive load is significantly increasing the inattention blindness.mod.
[45]20151OREEG NASA-TLXWorkload Distractions Mental stateExpert surgeons use different mental resources based on their needs.mod.
[98]201689ORExpert rating (custom)Communication Decision makingRAMIS increases communication requirements for the team of the OR.mod.
[63]201628VR simulatorNASA-TLXWorkloadXperience Team Trainer emphasizes the importance of teamwork.mod.
[81]201632ORPTICSQ SAQCommunicationThere is a significant correlation between team communication and surgical outcome.mod.
[43]20161OREEG NASA-TLXWorkloadA surgical expert during mentoring concerned while he was observed the surgery.low
[40]201689ORExpert rating (custom) NASA-TLXCommunication WorkloadThe proposed method is capable of capturing team activities during RAMIS.mod.
[56]201621Live porcine VR simulatorNASA-TLXWorkloadLive animal and VR simulator training provide a comparable workload.low
[70]20168VR simulatorEEG NASA-TLXProcedural memory Attention level WorkloadEEG can show the learning progress in the case of RAMIS.high
[59]201755ORNASA-TLXWorkloadThe study proposes a workload variety analysis with different members of the OR.mod.
[94]201725 p.ORNASA-TLXWorkloadNASA-TLX is a useful tool for determining the appropriate staff member mix for RAMIS procedures.mod.
[95]201710ORSURG-TLXWorkloadMental demands are higher for surgeons at the console than are assisting.mod.
[66]201824Live porcineNASA-TLXWorkloadSingle-site access surgery can significantly reduce the workload.mod.
[34]201827VR simulatorEEG NASA-TLXCognitive features Mental workload Engagement Asymmetry index Brain functional features Communication Integration Recruitment WorkloadEEG features can be used for objective non-technical skill assessment.high
[61]201827OROR efficiency (custom) NASA-TLXCommunication WorkloadAnticipation causes shorter operating time. Team familiarity causes less inconveniences. Less anticipation causes less cognitive load.mod.
[62]201832VR simulatorNASA-TLX SSSQ MRQWorkload StressTraining with a VR simulator can decrease the workload and stress.mod.
[33]201862Dry lab Simulated ORNOTSSSituational awareness Decision making Leadership Communication TeamworkMotor imaginary training technique is not effective in non-technical skill training.mod.
[44]20188Dry labfNIRS SURG-TLX HRVPrefrontal activation Workload Stress responseRAMIS improves performance during high workload conditions.high
[69]20184OREEG NASA-TLXCognitive features Functional features Mental workload Mental load Engagement Situation awareness Blink rate Asymmetry index Completion time CommunicationDuring a simple surgical task, functional brain features are sufficient to classify mentor–trainee trust.high
[111]201832VR simulatorEEGElectrocortical activity in temporoparietal and left frontal regionsThere are significant differences in electrocortical activity between novices and experts.high
[72]201812VR simulatorHRV NASA-TLX Wrist motion EMG Electrodermal EEGWorkload ExpertiseThe proposed skill and workload evaluation framework is accurate.high
[32]201920ORNOTSS NASA-TLXSituational awareness Decision making Leadership Communication Teamwork WorkloadNon-technical skills are associated with team efficiency, surgical flow disruptions and self-perceived performance.high
[75]20195ORNASA-TLXWorkloadWorkload is less in the case of robot-assisted submucosal dissection compared to the traditional case.low
[68]201931VR simulatorNASA-TLXWorkloadSpecific self-directed robotic simulation curriculum was introduced, which can significantly decrease the workload.mod.
[41]20198VR simulatorNASA-TLX Eye movementsWorkloadEye movements correlate with the workload.high
[71]2019264 p.ORNASA-TLXWorkloadMental workload is similar in the case of RAMIS, traditional MIS, hand-assisted MIS and open surgery.mod.
[57]201930Wet labNASA-TLX PVT WCSTWorkload Concentration Cognitive functionRobotic assistance does not provide less mental workload with novices. Robotic assistance may be mentally taxing for robotic novices.mod.
[76]20207ORNASA-TLXWorkloadRAMIS requires less mental demand and effort compared to open access surgery and traditional MIS.mod.
[46]202026Dry labTask-evoked pupillary responseWorkloadUnder high cognitive workload, there can be a divergence in robotic movement profiles between expertise levels.high
[96]2020n.a.OROTAS NOTSS ICARS NOTECHS IISituation awareness Decision making Communication Teamwork Leadership StressThe study proposed a structured approach to the analysis of non-technical skill using extracorporeal videos of both open radical cystectomy and RAMIS radical cystectomymod.
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Nagyné Elek, R.; Haidegger, T. Non-Technical Skill Assessment and Mental Load Evaluation in Robot-Assisted Minimally Invasive Surgery. Sensors 2021, 21, 2666.

AMA Style

Nagyné Elek R, Haidegger T. Non-Technical Skill Assessment and Mental Load Evaluation in Robot-Assisted Minimally Invasive Surgery. Sensors. 2021; 21(8):2666.

Chicago/Turabian Style

Nagyné Elek, Renáta, and Tamás Haidegger. 2021. "Non-Technical Skill Assessment and Mental Load Evaluation in Robot-Assisted Minimally Invasive Surgery" Sensors 21, no. 8: 2666.

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop