Shared Spatiotemporal Multitask Learning for Simultaneous Surgical Gesture Recognition and Skill Assessment from Minimally Invasive Surgery Videos

Abstract

The automatic recognition of surgical gestures and the assessment of surgical skills are essential for improving patient safety and optimizing surgical training processes. In this paper, a shared spatiotemporal multitask learning framework for joint surgical gesture recognition and skill assessment from minimally invasive surgery videos is presented. The proposed architecture simultaneously recognizes surgical gestures and estimates the expertise level of the surgeon from surgical video sequences. It is based on a 3D CNN with two parallel branches based on the ResNet18 backbone and incorporates LSTM module to model spatial temporal features from surgical video sequences. In addition, fully connected layers are used to perform surgical gesture classification in one branch and surgical skill level estimation. The experimental evaluation includes surgical gesture recognition and skill assessment using both segmented video clips and full-length surgical videos. The performance of the method is evaluated with the accuracy, precision, recall, and F1-score, for both tasks. These findings suggest that shared spatiotemporal feature learning provides an effective approach for jointly analyzing surgical gestures and skill levels from video.

1. Introduction

Laparoscopic and minimally invasive surgery requires a high precision to ensure patient safety, reduce postoperative care and decrease the recovery time. In this context, surgical gestures detection and skill assessment are necessary for procedure analysis, since incorrect execution may damage tissues and organs and cause bleeding or infections [1,2,3,4,5].

Currently, surgical skill assessment relies on the observation of the evaluator, which can lead to different interpretations increasing risks during surgeries [6,7,8]. On the other hand, kinematic analysis can identify surgical gestures; however, this approach requires complex studies and precise parameter definitions of surgical instrument positions [9,10,11,12].

For this reason, computer vision and artificial intelligence techniques extract spatiotemporal features to identify patterns [13,14,15,16,17,18]. Surgical gesture detection is fundamental for support systems of robot-assisted surgery.

Most of the existing studies focus on only one of these tasks. To address this problem, this work proposes a shared spatiotemporal multitask learning framework for joint surgical gesture recognition and skill assessment from minimally invasive surgery videos. The methodology consists of a neural network model which is composed with a 3D convolutional neural network (CNN) layer with two parallel branches based on the ResNet18 architecture as backbone. Therefore, a long short-term memory (LSTM) module is integrated to capture spatiotemporal information from surgical video sequences. In addition, fully connected layers are employed to independently perform surgical gesture classification in one branch and surgeon skill level estimation in the other. The combination of surgical gesture detection and surgeon skill evaluation within a single model provides an objective assessment that enhances the training process.

The main contributions of this work consist of:

• A shared spatiotemporal multitask learning architecture for joint surgical gesture recognition and surgeon skill assessment from video.
• A shared 3D-CNN representation combined with task-specific ResNet18-LSTM branches to capture common motion information while learning independent representations for each prediction task.
• An experimental evaluation of the contribution of spatiotemporal feature learning using segmented surgical clips and full-length surgical videos.
• A comparison between the proposed multitask architecture and independently trained task-specific models to evaluate the effect of shared feature learning on classification performance and computational efficiency.

Overall, the proposed framework investigates whether surgical gesture recognition and skill assessment can benefit from a shared spatiotemporal representation while preserving task-specific decision functions. This formulation provides a unified video-based approach for automated surgical performance analysis.

2. Literature Review

This section analyzes different artificial intelligence and neural network-based approaches for surgical gesture classification and surgeon skill assessment. Shafiei et al. [19] proposed a multimodal framework combining EEG signals and video analysis for surgical gesture extraction. Gazis et al. [20] introduced a two-stage framework based on self-supervised learning by combining an Inception network for spatiotemporal feature extraction with a Transformer architecture for temporal modeling and gesture classification. Van Amsterdam et al. [21], integrate visual and kinematic information using CNN and ensemble temporal models.

Additionally, some studies used deep learning architectures. Hutchinson et al. [22] applied a TCN encoder–decoder architecture for surgical gesture recognition using kinematic signals. Men et al. [23] proposed the R3D-MHA framework, which combines 3D ResNet blocks with multi-head attention to capture long-range temporal dependencies. Men et al. [24] evaluated several 3D-CNN architectures, to model spatial features from surgical videos. Rao et al. [25] proposed a contrastive prompt fine-tuning for gesture segmentation using MS-TCN++. Anh et al. [26] combined CNN-LSTM autoencoders with frequency-domain for expertise classification. Lajkó et al. [27], used optical flow to evaluate surgical skill through neural network analysis. Benmansour et al. [28] proposed a deep neural network architecture for automated surgical skill evaluation based on objective structured assessment criteria. Pan et al. [29] combined instrument tracking with ResNet-based visual feature extraction for surgeon skill classification. Sato et al. [30] proposed an objective surgical skill assessment approach based on the automatic recognition of dissection and exposure times in robot-assisted radical prostatectomy.

Most methods reported in the literature address only a single task and employ kinematic and sensors data, which increase system complexity. These limitations highlight the need for simultaneously performing gesture classification and skill assessment from surgical sequences.

3. Materials and Methods

3.1. Materials

The method was implemented using Python (version 3.12.7) and PyTorch (version 2.4.1) on a system equipped with an Intel Core i7 processor (Intel Corporation, Santa Clara, CA, USA) and an NVIDIA GeForce RTX 4060 GPU (NVIDIA Corporation, Santa Clara, CA, USA). The publicly JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS) is used [31]. This dataset was collected using the da Vinci Surgical System and includes recordings from eight surgeons with different levels of experience performing three standardized surgical tasks: suturing, knot tying, and needle passing.

JIGSAWS provides synchronized stereo endoscopic videos and contains manually annotated surgical gestures and skill assessment scores, enabling the evaluation of gesture classification, temporal segmentation, and surgical skill assessment methods.

3.2. Proposed Method for Surgical Gesture Recognition and Surgeon Skill Assessment

This approach enables an objective evaluation of each stage of the surgical procedure by assessing the expertise of the surgeons at the level of individual gestures. Figure 1 illustrates the proposed architecture of the neural network, which integrates temporal features and spatial patterns to capture motion dynamics and visual information.

Figure 1. Neural network architecture for gesture recognition and expertise assessment.

The proposed framework is organized into four main stages: frame preprocessing, spatial–temporal feature extraction, multi-task architecture, and neural network training.

3.2.1. Frame Preprocessing

The preprocessing stage consists of sampling each video into 30 frames to generate a consistent representation. Each frame is then resized to 128 × 128 pixels and normalized for neural network processing. In addition, data augmentation techniques are applied, including random rotations, brightness adjustments, blurring, and horizontal flipping.

3.2.2. Spatial–Temporal Feature Extraction

Traditional 2D CNN process frames independently and extract spatial features, these models do not capture temporal correlations across video sequences. This limitation is critical in surgical video analysis, where the recognition of surgical gestures depends not only on the appearance of instruments and surrounding structures. In addition, temporal pattern modeling is required for gesture recognition and skill assessment to understand its temporal evolution.

In the surgical context, identifying motion patterns of surgical instruments, including trajectories and temporal dynamics, is necessary for recognizing surgical gestures and assessing expertise of the surgeon. Additionally, these spatiotemporal patterns provide relevant information for surgical gesture recognition and expertise-level classification. However, the direct identification of technical execution errors and the assessment of surgical safety are beyond the scope of the present study. To identify and model spatial–temporal patterns in video sequences, a 3D CNN module is employed. By performing convolutions simultaneously over spatial dimensions and the temporal axis, the 3D CNN effectively captures motion dynamics and spatial temporal features from the input video.

A 3D CNN processes a video sequence X represented as (1):

$$\mathrm{X}\in {\mathrm{ℝ}}^{\mathit{TxHxWxC}}$$

(1)

where T is the number of frames, H and W denote the spatial resolution, and finally C is the number of channels. Consequently, 3D convolutional kernels extract temporal and spatial features from the video sequences X′, which are shared with both classification branches (2).

$${\mathrm{X}}^{\prime }=\mathit{Conv}3D({\rm X})$$

(2)

Then, a batch normalization and max pooling are applied to improve the convergence and reduce spatial dimensionality (Figure 2).

Figure 2. 3D CNN module.

The resulting spatiotemporal feature representation is subsequently used by two task-specific branches: a gesture classification branch and a surgical skill assessment branch. The network is divided into two independent branches, and share the spatiotemporal representation X′, which allows both tasks to learn common visual and motion information while learning specific decision functions.

3.2.3. Multitask Architecture

Instead of training two independent models, both tasks share a common spatial–temporal feature extraction block, while spatial feature learning is performed independently within each branch. For this reason, the proposed architecture is divided into two parallel branches for gesture recognition and skill assessment. The parameters of the 3D CNN are shared between both tasks.

After the extraction of spatiotemporal features from the video sequences, an additional spatial feature refinement is performed to reduce redundancy from the correlated motion patterns across consecutive frames. Consequently, a 2D CNN based on ResNet-18 is incorporated to identify spatial features from the surgical instruments and their relationships with the surrounding environment. The same ResNet-18 architecture is used in both branches; however, the weights are not shared, allowing each branch to learn task-specific spatial representations.

While the 3D CNN captures local spatiotemporal patterns from consecutive video frames, then each branch processes the extracted feature maps to learn spatial representations relevant to each prediction task (3) and (4).

$$f_{\mathrm{expertise}}={\mathit{CNN}}_{\mathrm{ResNet}18}(X^{\prime })$$

(3)

$$f_{\mathrm{gesture}}={\mathit{CNN}}_{\mathrm{ResNet}18}(X^{\prime })$$

(4)

where f_gesture and f_expertise denote the representations learned for gesture recognition and skill assessment, respectively. In addition, both branches share the same backbone architecture but do not share their weights. This architecture combines shared spatiotemporal feature extraction with specific spatial representation learning, enabling each branch to adapt to the distinct characteristics of its prediction objective.

Furthermore, the architecture follows a multitask learning strategy in which gesture recognition and skill assessment are addressed within a common framework. Learning related tasks provide complementary signals to improve generalization by sharing representations to capture important features to both objectives.

Although ResNet-18 extracts discriminative spatial features for identifying surgical instruments, modeling the temporal evolution of these features is essential in surgical video analysis. Therefore, an LSTM layer is incorporated after the ResNet-18 backbone to capture temporal dependencies across frames {f₁, f₂, …, f_T} for modeling the gesture progression and the surgeon skills motion patterns. The LSTM processes the extracted spatial features in a recurrent manner to learn the temporal evolution of motion patterns (5)

$$h_t=\mathit{LSTM}(f_t,h_{t-1})$$

(5)

where h_t represents the spatial feature state at time t and f_t denotes the spatial feature vector. The LSTM learns temporal representations from motion patterns through its recurrent structure. Therefore, the final state h_T consists of a temporal representation of the video sequence. The overall feature extraction and task-specific processing are summarized in (6), where the 3D CNN generates the shared representation X^′, which is subsequently processed independently by the gesture recognition and expertise assessment branches.

$$X\to \mathit{conv}3d\to X^{\prime }\to \left\{\begin{array}{c}{f}_{\mathrm{gesture}}={\mathit{CNN}}_{\mathrm{ResNet}18}(X^{\prime })\to h_g=\mathit{LSTM}(f_t,h_{t-1})\to y_{\mathit{gesture}}\\ f_{\mathit{expertise}}={\mathit{CNN}}_{\mathrm{ResNet}18}(X^{\prime })\to h_e=\mathit{LSTM}(f_t,h_{t-1})\to y_{\mathit{expertise}}\end{array}\right.$$

(6)

Thus, the final state h_T is employed as input into the fully connected layers FC used for surgeon skill assessment y_expertise and surgical gesture recognition y_gesture (7), since it captures the evolution of visual patterns over time.

$$y_{\mathrm{expertise}}=\mathit{FC}(h_T);\hspace{1em}{y}_{\mathit{gesture}}=\mathit{FC}(h_T)$$

(7)

Thus, the 3D CNN parameters are shared between the two tasks, whereas the ResNet-18, LSTM, and fully connected layers are independently parameterized for gesture recognition and expertise assessment.

3.2.4. Neural Network Training

To enhance the training stage, pretrained weights from ImageNet were employed, as transfer learning has been widely demonstrated to be effective for biomedical image and video analysis tasks. To reduce overfitting and learn features from the video sequence, some residual blocks (indices 4, 5, 6, and 7) were unfrozen during training. This strategy allows the network to learn discriminative patterns from surgical instruments and model their trajectories.

To optimize the learning process of the neural network, a loss function that combines cross-entropy loss and focal loss is employed. The cross-entropy loss L_CE (8) promotes stable convergence by penalizing incorrect predictions according to the discrepancy between the predicted probability distribution and the ground truth labels. Specifically, it measures how far the predicted probability p deviates from the true class distribution (9).

$$L_{\mathit{CE}}=-{\displaystyle \sum _{i=1}^C{y}_i\log (p_i)}$$

(8)

$$p={\displaystyle \frac{e^{z_i}}{{\displaystyle \sum _{j=1}^C{e}^{z_j}}}}$$

(9)

where C is the number of classes, y_i represents the ground truth label, z_i the ground truth logit representation and z_j is the output of the neural network before the normalization.

However, in the surgical context, the class distribution is imbalanced, since certain surgical gestures occur less frequently and some surgeon skill levels dominate for some gestures. In this scenario, cross-entropy loss was dominated by majority classes, which can limit the learning of relevant patterns. To address this problem, focal loss L_focal is incorporated (10).

$$L_{\mathit{focal}}=-{\displaystyle \sum _{i=1}^C{(1-p_i)}^{\gamma }{y}_i\log (p_i)}$$

(10)

where γ consists of a parameter that penalizes the correct classified classes by reducing their contribution to the loss, emphasizing misclassified examples. Then, the loss for each task k ∈ {gesture, expertise} is (11):

$$L_k=\alpha L_{\mathit{CE}}+\beta L_{\mathit{focal}}$$

(11)

α and β regulate the loss function, focusing on the focal loss to generate a robust learning of the imbalance classes. Therefore, the final loss function L_total is (12):

$$L_{\mathit{total}}=L_{\mathit{experstise}}+L_{\mathit{gesture}}$$

(12)

The loss function is designed to learn temporal representations from surgical instrument motion across frames, which are shared between gesture recognition and skill assessment tasks, reducing computational cost and improving training efficiency.

Table 1 illustrates the architecture of the proposed neural network. In this context, the model was trained with 50 epochs with a batch size of 16, with an initial learning rate of 1 × 10⁻⁴ and a weight decay of 5 × 10⁻⁵. Each video was sampled into 30 frames per sequence to stabilize the fine-tuning process.

Table 1. Neural network architecture.

Module	Layer	Parameters
Temporal 3D CNN Block	3D CNN	Filters = 16, kernel = 3 × 3 × 3, stride = 1, padding = 1
	Batch Normalization	Channels = 16
	ReLU	---
	Max Pooling 3D	Kernel: (1 × 2 × 2); Stride: (1 × 2 × 2); Padding: 0
Expertise Assessment Branch	ResNet 18	---
	LSTM	Input features = 512, hidden size = 512
	Fully connected	Filters = 3
Gesture Recognition Branch	ResNet 18	---
	LSTM	Input features = 512, hidden size = 512
	Fully connected	Filters = 15

4. Results

This section presents the experimental evaluation conducted to validate the proposed framework for surgical gesture recognition and surgeon skill assessment. The evaluation is performed in two stages: (1) experiments using the label video clips, and (2) experiments using full surgical videos.

All experiments were conducted using the JIGSAWS dataset, which comprises three standardized surgical training tasks: suturing, needle passing, and knot tying. The original stereoscopic recordings were segmented into individual surgical gesture clips using the temporal annotations provided by the dataset, which specify the start and end frames of each gesture. Each extracted clip was assigned to one of the 15 gesture classes represented in JIGSAWS and an expertise label corresponding to the participant who performed the gesture.

The expertise categories were defined according to the prior experience of the surgeon in robotic surgery, as established in JIGSAWS. Specifically, participants with less than 10 h of experience were classified as novices, those with 10–100 h as intermediates, and those with more than 100 h as experts. Accordingly, the expertise label assigned to each clip reflects the experience category of the corresponding surgeon rather than an independent assessment of the individual gesture.

Given the limited number of participants and the unequal distribution of gesture instances across expertise levels, the dataset was partitioned using a reproducible random split with a fixed seed. The available clips were allocated to training (80%), validation (5%), and testing (15%), ensuring that each individual clip was assigned to only one subset. This strategy was adopted to preserve the representation of gesture classes and expertise levels, particularly considering that JIGSAWS includes only two expert and two intermediate surgeons.

The partitioning was performed at the clip level without grouping samples by their original recording or surgeon identity. Consequently, different clips from the same recording or surgeon could be distributed across the training, validation, and testing subsets. Therefore, the reported results characterize the performance of the model under a clip-level partitioning protocol and do not directly establish its generalization capability to previously unseen surgeons.

Surgical gesture recognition and surgeon skill assessment are evaluated using different quantitative metrics to rigorously assess the effectiveness and robustness of the proposed framework.

The accuracy is used to measure the proportion of correctly classified samples, providing a global overview of model performance; however, given its limitations in imbalanced scenarios, it is not considered sufficient (13).

$$\mathit{Accuracy} = {\displaystyle \frac{\mathit{Tp}+\mathit{Tn}}{\mathit{Tp}+\mathit{Tn}+\mathit{Fp}+\mathit{Fn}}}$$

(13)

where Tp denotes the number of true positives, corresponding to correctly predicted samples of a given class; Tn represents the true negatives, the correct predicted samples that do not belong to the class; Fp indicates false positives, consists of the samples incorrectly assigned to the class; and Fn corresponds to false negatives, the samples belonging to the class that were incorrectly predicted as another class. The precision penalizes false positive predictions to evaluate the classification of the model (14).

$$\mathit{precision}={\displaystyle \frac{\mathit{Tp}}{\mathit{Tp}+\mathit{Fp}}}$$

(14)

The recall assesses the correctly ground truth identification, exposing the relevant classes from the surgical gestures or skill that have remained undetected (15).

$$\mathit{recall}={\displaystyle \frac{\mathit{TP}}{\mathit{TP}+\mathit{FN}}}$$

(15)

The F1-score generates a relation from the precision and recall and provides a performance indicator under class imbalance, which is inherent to surgical datasets (16).

$$F1-\mathit{score}=2{\displaystyle \frac{(\mathit{precision})(\mathit{recall})}{\mathit{precision}+\mathit{recall}}}$$

(16)

These metrics ensure demonstrate the performance of the neural network and its efficiency for simultaneous gesture recognition and skill assessment.

The reported precision, recall, and F1-score metrics were obtained using weighted averaging, in which the contribution of each class is weighted according to its number of samples. This approach was applied independently to the 15 gesture classes and the 3 expertise levels, ensuring that each class contributed proportionally to its representation in the dataset.

4.1. Clip-Based Assessment

The clip-based analysis consists of evaluating segmented video sequences corresponding to individual surgical gestures and expertise levels. This setting allows the assessment of the proposed system under controlled conditions, where each clip contains a single gesture instance performed by a surgeon with a specific skill level. The clip analysis evaluates the proposed system without considering gesture transitions. Therefore, each clip was divided into 30 frames, which improved the robustness of the system by providing a consistent temporal representation across samples.

Table 2 demonstrates the effectiveness of the proposed neural network architecture compared to the baseline model without the 3D CNN layer. The 3D CNN improves the performance of the model, achieving an accuracy of 0.93 and an F1-score of 0.93 for the skills assessment. In addition, an accuracy of 0.95 and an F1-score of 0.95 for gesture recognition. In contrast the models without this block obtain an accuracy of 0.69 and an F1-score of 0.68 for skills classification assessment and 0.88, 0.87 of accuracy and F1-Score for surgical gesture recognition.

Table 2. Performance comparison between the proposed model and the baseline architecture without the 3D CNN layer.

Model	Task	Accuracy	Precision	Recall	F1-Score
ResNet18 + LSTM	Expertise assessment	0.69	0.75	0.73	0.68
3D CNN + ResNet18 + LSTM	Expertise assessment	0.93	0.94	0.93	0.93
ResNet18 + LSTM	Gesture recognition	0.88	0.89	0.89	0.87
3D CNN + ResNet18 + LSTM	Gesture recognition	0.95	0.95	0.95	0.95

The performance of the proposed method highlights the necessity of spatiotemporal feature learning for surgical video analysis. Surgical instrument trajectories modeling captures motion patterns correlated with surgical gestures and surgeon skill levels, resulting in more consistent classification across both tasks.

Table 3 compares two independently trained 3D CNN + ResNet18 + LSTM models with the proposed dual-branch architecture. In the independent configuration, two separate models are trained for expertise assessment and gesture recognition. In contrast, the proposed architecture shares the 3D CNN spatiotemporal representation and use two independent branches for each task.

Table 3. Multitask learning vs. independent models.

Model	Task	Accuracy	Precision	Recall	F1-Score
Independent 3D CNN + ResNet18 + LSTM	Expertise assessment	0.91	0.89	0.90	0.89
Independent 3D CNN + ResNet18 + LSTM	Gesture recognition	0.85	0.84	0.84	0.85
Proposed Dual Branch 3D CNN + ResNet18 + LSTM	Expertise assessment	0.93	0.94	0.93	0.93
Proposed Dual Branch 3D CNN + ResNet18 + LSTM	Gesture recognition	0.95	0.95	0.95	0.95

Figure 3 compare the inference and training times of the proposed dual-branch architecture with those of the independently trained models. The proposed architecture requires an average inference time of 0.3611 s to jointly perform both tasks, whereas the independent gesture recognition and expertise assessment models require 0.22 s and 0.22 s, respectively. Although the dual-branch architecture has a higher inference time than either individual model, it performs both tasks avoid the need to execute two independent models. In addition, the training time differs by approximately 1 s between the evaluated configurations. These results indicate that the proposed architecture provides a more integrated computational strategy for gesture recognition and expertise assessment while sharing the spatiotemporal feature extraction stage.

Figure 3. Comparison processing time (a) Inference time using the proposed dual-branch neural network. (b) Inference time using separate neural network models. (c) Comparison of training time per epoch between separate neural network models and the proposed dual-branch neural network. Figure 4 illustrates the performance evaluation of the proposed architecture for surgical gesture recognition and surgeon skill assessment. The results demonstrate a stable and consistent classification across all classes for both tasks. The extracted spatial–temporal features contain relevant information about surgical instruments and their interactions for an accurate identification of both surgical gestures and surgeon skill levels.

Figure 3. Comparison processing time (a) Inference time using the proposed dual-branch neural network. (b) Inference time using separate neural network models. (c) Comparison of training time per epoch between separate neural network models and the proposed dual-branch neural network. Figure 4 illustrates the performance evaluation of the proposed architecture for surgical gesture recognition and surgeon skill assessment. The results demonstrate a stable and consistent classification across all classes for both tasks. The extracted spatial–temporal features contain relevant information about surgical instruments and their interactions for an accurate identification of both surgical gestures and surgeon skill levels.

Figure 4. Skill level and gesture recognition classification performance. (a) Precision per skill level, (b) Recall per skill level. (c) Precison per gesture recognition. (d) Recall per skill level.

Figure 4. Skill level and gesture recognition classification performance. (a) Precision per skill level, (b) Recall per skill level. (c) Precison per gesture recognition. (d) Recall per skill level.

Figure 5a presents the confusion matrices for surgeon skill level classification and surgical gesture recognition. For skill level classification, predictions are correct classified, but it contains misclassification observed between novice and expert levels suggests partial overlap in motion patterns. On the other hand, Figure 5b illustrates the surgical gesture recognition tasks, where most gestures are accurately classified. The most notable confusion occurs for gestures G4 and G5, which is occasionally misclassified as G8. This can be attributed to their similar instrument trajectories and spatial–temporal configurations. Nevertheless, these errors are infrequent, and the overall results highlight the robustness of the proposed.

Figure 5. Confusion matrix (a) skill level and (b) gesture recognition classification performance.

4.2. Video-Based Assessment

The video analysis assessment compares the ground truth and predicted surgical gestures on video sequences for different tasks, including suturing, knot tying, and needle passing. This visualization highlights the temporal evolution of gesture patterns and enables the identification of transitions between consecutive surgical actions. The predictions are obtained by averaging the detected gesture labels across the video sequence, allowing the model to accurately capture gesture changes.

For the full video assessment, the complete video was used as input, and the temporal annotations provided by the JIGSAWS dataset were used to identify the start and end transitions of each gesture movements. The corresponding task segments the video in clips and uses a sliding window with a length of 25 frames. Predictions were generated for successive temporal windows throughout each task segment, and each predicted gesture label was compared with its corresponding label to assess classification performance over the entire task.

Figure 6 compares the ground truth and predicted surgical gestures along the video timeline. The ground truth is used as the reference to identify the temporal transitions between surgical gestures, while the predicted sequence is compared with these transitions to determine whether the model changes its predictions at the corresponding points in the video. The analysis shows that the misclassifications may occur between gestures with similar motion trajectories or due to visual interference from background objects. In addition, Figure 6 provides a clear temporal perspective of the predictions, confirming that the proposed model identifies gesture transitions and preserves temporal consistency throughout the surgical procedure.

Figure 6. Ground truth vs. predicted gestures over video frames. (a) Video analysis for surtiring task. (b) Video analysis for knot tying task. (c) Video analysis for needle passing.

Table 4 shows the performance of the proposed architecture across three surgical tasks: Suturing, Knot-Tying, and Needle Passing. The results indicate that expertise assessment generally achieves higher performance than gesture recognition in Suturing and Needle Passing, whereas Knot-Tying shows high performance in both classification tasks. Notably, gesture recognition achieves lower performance in Suturing and Needle Passing (F1-scores of 0.75 and 0.74, respectively), compared with 0.96 in Knot-Tying. These differences may be associated with the visual characteristics of each task and the surrounding environment. In particular, the presence of additional objects, variations in background appearance, and shadows may introduce visual interference, making it more difficult for the model to distinguish surgical instruments and their movements. These factors can affect the visual features extracted by the network and lead to misclassifications, particularly when different gestures exhibit similar motion patterns. In contrast, the more consistent visual and motion patterns observed in Knot-Tying may facilitate gesture discrimination, resulting in higher classification performance. Overall, these results suggest that gesture recognition is sensitive to task-specific visual conditions, particularly during full-video evaluation, where variations in background objects, illumination, and shadows are encountered throughout the recordings. However, the results should be interpreted within the context of the dataset partitioning strategy, since generalization to unseen surgeons has not yet been established.

Table 4. Evaluation of model performance across surgical tasks.

Surgical Task	Classification Task	Accuracy	Precision	Recall	F1-Score
Suturing	Expertise assessment	0.84	0.90	0.94	0.84
	Gesture recognition	0.72	0.87	0.72	0.75
Knot-Tying	Expertise assessment	0.85	0.87	0.88	0.86
	Gesture recognition	0.96	0.96	0.96	0.96
Needle Passing	Expertise assessment	0.98	0.95	0.95	0.95
	Gesture recognition	0.72	0.80	0.72	0.74

Table 5 reports the performance of the proposed neural network using different training loss functions for gesture recognition and expertise assessment. The CE Loss provides stable and consistent performance across both tasks. Focal Loss produces lower performance for gesture recognition but improves the assessment of expertise. The combined loss functions demonstrate that the relative contribution of each loss component affects the performance of both tasks. In particular, the configuration with α = 0.15 and β = 0.85, which assigns a higher weight to the Focal Loss component. These results indicate that increasing the contribution of the Focal Loss provides a more suitable balance for the optimization of gesture recognition and expertise assessment under the experimental conditions evaluated.

Table 5. Performance comparison of the network using different loss functions.

Loss Function	Task	Accuracy	Precision	Recall	F1-Score
CE Loss	Gesture Recognition	0.92	0.93	0.93	0.93
	Expertise Assessment	0.90	0.90	0.90	0.90
Focal Loss	Gesture Recognition	0.86	0.87	0.86	0.85
	Expertise Assessment	0.91	0.92	0.92	0.92
αCE + βFocal Loss (α = 0.25, β = 0.65)	Gesture Recognition	0.90	0.92	0.90	0.90
	Expertise Assessment	0.89	0.89	0.89	0.89
αCE + βFocal Loss (α = 0.65, β = 0.25)	Gesture Recognition	0.84	0.85	0.85	0.85
	Expertise Assessment	0.82	0.85	0.85	0.87
αCE + βFocal Loss (α =0.15, β =0.85)	Gesture Recognition	0.95	0.95	0.95	0.95
	Expertise Assessment	0.93	0.94	0.93	0.93

4.3. Method Comparison

To demonstrate the effectiveness of the proposed architecture, Table 6 compares its performance against other methods for surgical gesture recognition and skill assessment.

Table 6. Method comparison. Performance comparison of different methods.

Reference	Surgical Gesture Recognition	Skill Assessment	Metric	Performance		Dataset
A. Gazis et al. [20]	✔	✖	Accuracy	75.8%		JIGSAWS and private datasets
B. van Amsterdam et al. [21]	✔	✖	Accuracy F1-Score	86.8% 94.3%		JIGASAWS and Private datasets
Y. Men et al. [24]	✔	✖	Accuracy	76.3%		JIGSAWS and private datasets
M. Rao et al. [25]	✔	✖	Accuracy F1-Score	81% 87%		JIGSAWS
N. X. Anh et al. [26]	✖	✔	Accuracy	96.30%		JIGSAWS
G. Lajkó et al. [27]	✖	✔	Accuracy	91.89%		JIGSAWS
M. Pan et al. [29]	✖	✔	Accuracy	84.8%		JIGSAWS
Proposed Method	✔	✔	Accuracy F1-Score	Gesture	Skill	JIGSAWS
				95%	93%
				95%	93%

The comparison presented in Table 6 should be interpreted considering the differences in evaluation protocols across the compared studies. Although several methods use the JIGSAWS dataset, differences in data partitioning, experimental settings, evaluation procedures, and reported metrics may affect the performance values. Therefore, the reported results provide a contextualizing performance of the method rather than a direct comparison under identical experimental conditions.

Table 6 presents a comparison with other methods evaluated on the JIGSAWS dataset. Although several of the referenced studies report additional results using private or proprietary datasets, the comparison is conducted exclusively on JIGSAWS.

A key advantage of the proposed method is the surgical gesture recognition and surgeon skill assessment simultaneously. Unlike many existing approaches that are based on kinematic or multimodal data, the proposed architecture operates with visual information which improve the gesture modeling for its recognition and assessment.

Nevertheless, a limitation of the proposed approach is that it depends on specific training that requires background patterns, such as visual features, lighting conditions, and camera viewpoints. In contrast, kinematics-based methods often generate a more efficient generalization across different environments, as they are based on physical instrument motion patterns that are less affected by visual variability.

5. Discussion

The experimental results indicate that modeling spatiotemporal features contributes to effective surgical gesture recognition and surgeon skill assessment. This is supported by the comparison between the ResNet18 + LSTM and 3D CNN + ResNet18 + LSTM configurations, where the inclusion of the 3D CNN increased gesture recognition accuracy from 88% to 95% and skill assessment accuracy from 69% to 93%. These results indicate that explicitly modeling spatial and temporal information from consecutive frames provides discriminative representations for identifying surgical gestures and differentiating skill levels.

In addition to the contribution of spatiotemporal modeling, the comparison with independently trained models provides evidence of the benefit of shared multitask learning. The independent models achieved accuracy of 85% for gesture recognition and 91% for skill assessment, whereas the proposed shared architecture achieved 95% and 93%, respectively. These results suggest that sharing the spatiotemporal representation provides complementary information between the two tasks while preserving task-specific processing.

The confusion matrix analysis further supports this interpretation. Although the model correctly classifies most samples, some confusion remains between novice and expert skill levels, indicating that their visual and motion representations can partially overlap. Similarly, the gesture recognition results show occasional confusion among gestures G4, G5, and G8. These errors may be associated with the similarity of their instrument trajectories and spatiotemporal configurations.

The video evaluation provides additional evidence regarding the temporal behavior of the proposed framework. The comparison between ground truth and predicted gestures shows that the model can preserve the temporal evolution of gestures throughout the analyzed sequences. In this context, classification errors may occur when gestures exhibit similar motion trajectories or when surrounding visual elements interfere with the extracted representations.

An important characteristic of the proposed framework is the use of visual information. However, this characteristic generates a limitation, the representations learned depend on the visual information (illumination, background, camera and image quality), which can affect the classification for both classes.

From a surgical training perspective, the proposed framework could provide complementary information for the analysis of surgical performance by simultaneously identifying surgical gestures and estimating the operator’s expertise level from video sequences. Gesture recognition may facilitate the identification and temporal organization of surgical actions, while expertise-level classification may provide additional information about movement patterns associated with different experience categories.

However, recognizing a surgical gesture does not necessarily indicate whether it was performed correctly, efficiently, or safely. In the present study, expertise labels correspond to the surgeon previous experience in robotic surgery rather than independent evaluations of the technical quality of each gesture. Therefore, the proposed framework should not be interpreted as a direct measure of gesture execution quality or surgical safety.

A potential educational application would be to support the review of recorded training sessions by automatically identifying gesture sequences and providing supplementary information for instructors. Nevertheless, the evaluation of technical errors, tissue manipulation, movement efficiency, and procedural safety would require additional performance indicators and validation against expert assessments. Such extensions represent potential directions for future development of the proposed framework.

Another limitation is the dependence on the characteristics of the JIGSAWS dataset, which includes only eight participants performing three standardized surgical training tasks. Although these tasks reproduce fundamental instrument manipulation activities involved in robot-assisted minimally invasive surgery, they do not fully represent the complexity of clinical procedures, where anatomical variability, tissue deformation, visual occlusions, and unexpected events may influence surgical performance. Furthermore, the learned visual representations may have limited generalization to other surgical environments, particularly when background patterns, illumination conditions, or camera viewpoints differ from those observed during training. Additionally, due to the limited number of participants and the unequal distribution of gesture instances across expertise levels, a random clip-level partitioning strategy was adopted. Although individual clips were assigned to mutually exclusive subsets, different clips from the same recording or surgeon could appear in training, validation, and testing. Consequently, the reported performance does not directly establish the ability to generalize to previously unseen surgeons.

Therefore, the present findings should be interpreted as an evaluation of the proposed framework under standardized surgical training conditions rather than evidence of its clinical applicability. Future studies should consider surgeon-independent evaluation protocols, such as Leave-One-User-Out (LOUO), as well as additional datasets containing more diverse surgical scenarios, to assess the generalization and potential applicability of the proposed approach.

It is also important to mention that JIGSAWS provides recordings of simulated surgical tasks performed in a controlled training environment. Consequently, the results demonstrate the ability of the proposed model to recognize surgical gestures and assess skill levels under controlled conditions. Further validation using real surgical recordings and independent clinical datasets would be necessary to investigate its potential clinical applicability.

An important limitation of this study is that the proposed model was not evaluated on surgeons completely excluded from the training process. Although the dataset was partitioned into mutually exclusive subsets at the clip level, different clips from the same recording or surgeon could be distributed among the training, validation, and testing sets. Therefore, the reported classification accuracies reflect the performance achieved under this specific evaluation protocol and should not be interpreted as evidence of generalization to previously unseen surgeons. Furthermore, the model’s performance may differ when applied to surgeons with different motion patterns, instrument handling techniques, and task execution characteristics.

Future work may focus on the generalization of the proposed method across different surgical environments. In addition, kinematic information could be integrated as a complementary module to provide robust features for modeling gesture transitions and instrument trajectories. Moreover, temporal attention mechanisms could be explored to model gesture trajectories and transitions. Finally, additional datasets could be used to assess model generalization.

6. Conclusions

This paper presented a shared spatiotemporal multitask learning framework for joint surgical gesture recognition and surgeon skill assessment from minimally invasive surgery videos. The proposed architecture uses a shared 3D-CNN to extract spatiotemporal information from surgical video sequences and two task-specific ResNet18-LSTM branches to learn independent representations for gesture classification and skill level assessment.

The experimental results show that explicit spatiotemporal feature learning improves both tasks compared with a ResNet18-LSTM baseline. Furthermore, comparison with independently trained 3D-CNN-ResNet18-LSTM models shows that the shared multitask architecture improves gesture recognition accuracy from 85% to 95% and skill assessment accuracy from 91% to 93%. These results suggest that the shared representation enables beneficial information transfer between the two related tasks while retaining task-specific prediction functions.

However, these results should be interpreted considering the limitations of the adopted dataset partitioning strategy. Since the dataset was divided at the clip level, video clips from the same surgeon could be included in both the training and testing subsets. Therefore, the reported classification accuracies reflect the performance achieved under this specific evaluation protocol and do not establish the generalization capability of the proposed model to previously unseen surgeons. Furthermore, the evaluation was conducted using standardized surgical training tasks, and the applicability of the proposed framework to real clinical environments remains to be established.

The main contribution of this work is therefore the formulation and experimental evaluation of a shared spatiotemporal multitask architecture for jointly analyzing surgical gestures and skill levels from video. By sharing the 3D-CNN feature extraction stage, the framework reduces redundant processing compared with executing two independent models while providing separate outputs for each task.

Future work should explore surgeon-independent evaluation protocols, such as Leave-One-User-Out (LOUO), and validation using independent clinical surgical datasets. In addition, multimodal integration with kinematic information, temporal attention mechanisms, and improved multitask optimization strategies should be investigated to further enhance the robustness and applicability of the proposed framework.