Assistive Control through a Hapto-Visual Digital Twin for a Master Device Used for Didactic Telesurgery

Abstract

This article explores the integration of a hapto-visual digital twin on a master device used for bilateral teleoperation. The device, known as a quasi-spherical parallel manipulator, is currently employed for the remote center of motion control in teleoperated mini-invasive surgery. After providing detailed insights into the device’s kinematics, including its geometric configuration, Jacobian, and reachable workspace, the paper illustrates the overall control system, encompassing both hardware and software components. The article describes how a digital twin, which implements a haptic assistive control and a visually enhanced representation of the device, was integrated into the system. The digital twin was then tested with the device: in the experiments, one “student” end-user must follow a predefined “teacher” trajectory. Preliminary results demonstrate how the overall system can pose a good starting point for didactic telesurgery operation. The control action, yet to be optimized and tested on different subjects, indeed seems to grant satisfying performance and accuracy.

1. Introduction

Bilateral teleoperation enables a human operator to control a machine or robot remotely through a real-time bidirectional flow of information [1]. The system consists of a master device, controlled by the user, and a slave device, which interacts with the remote environment. Conceived in the mid-20th century [2], bilateral teleoperation has evolved significantly thanks to advancements in control, telecommunication, and sensory technologies [1].

One of the primary advantages of bilateral teleoperation lies in its ability to provide haptic feedback, which conveys tactile and force sensations back to the operator [3,4]. This feedback enhances the operator’s perception of the remote environment, enabling more precise and intuitive manipulation of objects. The seamless integration of the components, along with advanced control algorithms, is essential for minimizing latency and ensuring stable, responsive interactions.

The primary objective of bilateral teleoperation is to mitigate challenges associated with direct human presence in hazardous or remote environments. Originally conceived to address the risks inherent in handling hazardous materials [2], its use has expanded into multiple scientific fields, including extreme environment exploration [5], aerospace [6], and medicine [7]. Its utility extends beyond merely enhancing safety; it serves as a pivotal tool in facilitating exploration, research, and intervention in environments where human access is limited, unmanageable, or impractical.

A recent advancement in the field of bilateral teleoperation is the integration of digital twins [8,9,10]. Digital twins are virtual replicas of physical systems that simulate their behavior in real-time. Standard digital twin architectures rely on the collection of feedback signals from sensors in their physical counterpart and continuous status updates to synchronize the two systems, allowing real-time behavior monitoring [11,12]. Thanks to a proper virtual model of the system, digital twins can also enhance system control performance by predicting the outcomes of the user’s actions [8], augmenting visualization [9], and detecting possible issues in the physical system [13].

Within the field of medicine, bilateral teleoperation finds primary application in surgical procedures [14,15], exemplified by its use in cancer care [16], laparoscopy [17,18], and spinal surgery [19,20], often implementing a remote center of motion (RCM) control, as to minimize the surgical access incision. In this field, authors have already presented different types of three-legged spherical parallel manipulators used as master devices [21,22,23,24,25,26]. Two main architectures were indeed investigated (R stands for rotational joint and U for universal joint): a spherical coordinate-based 3RRR architecture [21,22], as in Figure 1a, and a quasi-spherical 2RRR-1URU architecture [23,24,25,26], as in Figure 1b. Both devices aim to control the slave robot’s instrumented tool orientation and are actuated by three motors acting on the proximal links, allowing for haptic feedback. In more detail, for the 2RRR-1URU architecture: refs. [23,24] focused on the definition of the forward and inverse kinematic models of the device; ref. [25] investigated the definition of its reachability workspace and devised a proper avoidance algorithm so as not to trespass it during the device operation; [26] researched the definition of a position control mode through joint path planning.

This article focuses on software control integration and the definition of a hapto-visual digital twin of the quasi-spherical parallel manipulator (qSPM) architecture. The digital twin, defined via the robotic operating system (ROS), works in parallel with the main control loop of the system and ensures the operator haptic and enhanced visual feedback in real-time. The developed digital twin is then used to assist a student operator in reproducing a predefined teaching trajectory and to evaluate training through an assistive control action. The system analyzed in this paper is intended to be one component of a comprehensive architecture, which includes a simulation of the operational environment as part of a broader training program for didactic telesurgery. In fact, this type of procedure aims to eventually address the limitations of traditional training, which often relies heavily on verbal instruction and passive visual observation. This trend has recently begun to receive attention [27,28,29].

2. Materials and Methods

2.1. Kinematic Description, Jacobian, and Haptic Feedback

As stated in Section 1, the qSPM device consists of two spherical RRR legs (leg B and C), and a URU leg (leg A, as in Figure 1b). Kinematic models are delineated and formalized in [23,24,25]. Reporting a nomenclature scheme in Figure 2a for clarity, the following main premises can be outlined ($K=A,B,C,i=1,2,3$):

• We define the origin-based unit vectors ${\mathit{r}}_{\mathit{iK}}$ as parallel to the axes of the R joints $K_i$;
• We define ${\mathit{r}}_{\mathit{E}}$ as the unit vector pointing to the platform’s center E. Its orientation is expressed by the Euler angles $zxz$ triplet $\mathcal{E}\equiv (\psi ,\theta ,\varphi )$, as in Equation (1):

  $${\mathit{r}}_{\mathit{E}}\left(\mathcal{E}\right)\equiv R_E(\psi ,\theta ,\varphi )·\widehat{\mathit{z}}=R_z\left(\psi \right)·R_x\left(\theta \right)·R_z\left(\varphi \right)·\widehat{\mathit{z}},$$

  (1)

  in which $R_z(·)$ and $R_x(·)$ denote rotation matrices around axes $\widehat{\mathit{z}}$ and $\widehat{\mathit{x}}$;
• We define the workspace center coincident to the bisector of the first octant, as in Equation (2):

  $${\mathit{r}}_{\mathit{wc}}={\displaystyle \frac{1}{\sqrt{3}}}{[1,1,1]}^t\equiv {\mathit{r}}_{\mathit{E}}{(135,54.7,\varphi )}^{\circ },$$

  (2)

  in which $\varphi$, being a self-rotation angle, we can assume any possible value within the operative workspace $W_{op}$, and it is considered zero by default [25];
• According to the superposition principle, U joints on leg A can each be decomposed into two R joints sharing perpendicular axes, namely couples $(A_1,A_2)$ and $(A_4,A_5)$, as in Figure 2a;
• Considering a symmetrical structure for simplicity, we define the geometrical angles $(\alpha ,\beta ,\gamma )$ as the ones describing the angular span of proximal links $B_1{B}_2$ and $C_1{C}_2$, distal links $B_2{B}_3$ and $C_2{C}_3$, and angle $B_3\widehat{O}E\equiv C_3\widehat{O}E\equiv A_5\widehat{O}E$, respectively;
• As stated in Section 1, joints $A_1,B_1,C_1$ are each actuated by motors, and their revolution axes are perpendicular. Imposing ${\mathit{r}}_{\mathbf{1}\mathit{A}}\equiv \widehat{\mathit{z}}$, ${\mathit{r}}_{\mathbf{1}\mathit{B}}\equiv \widehat{\mathit{x}}$, ${\mathit{r}}_{\mathbf{1}\mathit{C}}\equiv \widehat{\mathit{y}}$, the actuated angle triplet is $\mathrm{\Theta }\equiv ({\theta }_{1A},{\theta }_{1B},{\theta }_{1C})$;
• In the bilaterally teleoperated system, the platform’s orientation $r_E$ is transmitted to the slave’s instrumented tool for RCM control and transformed through a proper rotation matrix $R_T$, as in Figure 2b.

With such formulation, thanks to geometric constructions, we can define the Jacobian J of the device and its dexterity $\eta \left(J\right)$, as in Equations (3) and (4) [23,24,25]:

$$J=J_p^{-1}{J}_s={\left[\begin{array}{c}{({\mathit{r}}_{\mathbf{4}\mathit{A}}\times {\mathit{r}}_{\mathbf{5}\mathit{A}})}^t\\ {({\mathit{r}}_{\mathbf{2}\mathit{B}}\times {\mathit{r}}_{\mathbf{3}\mathit{B}})}^t\\ {({\mathit{r}}_{\mathbf{2}\mathit{C}}\times {\mathit{r}}_{\mathbf{3}\mathit{C}})}^t\end{array}\right]}^{-1}\left[\begin{array}{ccc}{\mathit{r}}_{\mathbf{1}\mathit{A}}·({\mathit{r}}_{\mathbf{4}\mathit{A}}\times {\mathit{r}}_{\mathbf{5}\mathit{A}})& 0& 0\\ 0& {\mathit{r}}_{\mathbf{1}\mathit{B}}·({\mathit{r}}_{\mathbf{2}\mathit{B}}\times {\mathit{r}}_{\mathbf{3}\mathit{B}})& 0\\ 0& 0& {\mathit{r}}_{\mathbf{1}\mathit{C}}·({\mathit{r}}_{\mathbf{2}\mathit{C}}\times {\mathit{r}}_{\mathbf{3}\mathit{C}})\end{array}\right],$$

(3)

$$\eta \left(J\right)={\displaystyle \frac{1}{\left|\right|J\left|\right|·\left|\right|J^{-1}\left|\right|}},\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}0<\eta \left(J\right)<1,$$

(4)

in which $J_p$ is the parallel Jacobian, $J_s$ is the serial Jacobian, and inscription $\left|\right|·\left|\right|$ is the spectral norm.

The Jacobian can be used to define general torque control and haptic feedback for bilateral teleoperation, as in Equation (5) [25].

$$\mathit{\tau }\stackrel{\mathrm{def}}{=}\left[\begin{array}{c}{\tau }_{1A}\\ {\tau }_{1B}\\ {\tau }_{1C}\end{array}\right]=J^t·\mathit{T}+{\mathit{\tau }}_{\mathit{ctrl}}+{\mathit{\tau }}_{\mathit{comp}},$$

(5)

in which ${\tau }_{1K}{|}_{K=(A,B,C)}$ are the active torques acting on the active angles ${\theta }_{1K}{|}_{K=(A,B,C)}$, $\mathit{T}$ are the operational torques sensed on the slave robot’s tool to be reproduced through haptic feedback on the master device, ${\mathit{\tau }}_{\mathit{ctrl}}$ are extra control torques directly applied to the master device, and ${\mathit{\tau }}_{\mathit{comp}}$ are static compensation torques.

2.2. Inverse and Forward Kinematics, Working Modes, and Self-Collision

IKM and FKM are the functions linking the platform’s orientation $\mathcal{E}$ to the actuated triplet $\mathrm{\Theta }$, respectively, and vice versa. As the IKM function leads to eight different solutions, i.e., working modes $m_{i=(1,\cdots ,8)}$, depending on the posture of the legs and on the sign of the elements of the diagonal matrix $J_s$ [30], each FKM can be computed according to the constraints topologically imposed by each working mode. It has been demonstrated that the FKM can be computed with great computational gains by adding an additional encoder sensor on link $C_2$ and thus measuring $\tilde{\mathrm{\Theta }}\equiv ({\theta }_{1A},{\theta }_{1B},{\theta }_{1C},{\theta }_{2C})$ [24].

In order to compare all possible working modes and choose the best one, the research not only considered $\eta \left(J\right)$ by studying parallel and serial singularity but also the possibility of self-collision among the legs and a cylindric volume centered in ${\mathit{r}}_{\mathit{E}}$ in anticipation of adding a degree of freedom on the tool’s axial displacement [25].

2.3. Reachable Workspace and Best Working Mode

With the assumptions described in Section 2.2, we can define the reachable workspace $W_{reach}$ as the portion of the operative workspace $W_{op}$ in the Euler Space $(\psi ,\theta ,\varphi )$ connected without discontinuity to the workspace center $w_c$, i.e., without crossing any singularity area $S_J$ (in which $\eta \left(J\right)\le {\eta }_{thr}$) or any self-collision area $C_V$, as in Equation (6) [25].

$$W_{reach}=\{P=(\psi ,\theta ,\varphi )\in W_{op}\mid \exists s(w_c,P)\in C^0:s(w_c,P)\cap (S_J\cup C_V)=\varnothing \},$$

(6)

in which $s(w_c,P)$ is a random continuous path between point P and the operative workspace center $w_c$, Equation (2). As $S_J$ and $C_V$ are subjects of the kinematics of the system, the reachable workspace depends on the selected working mode. An example of $W_{reach}$ for $m_3$, in offset space $({\psi }_r,{\theta }_r,{\varphi }_r)$ with respect to central angles ${(135,54.7,0)}^{\circ }$ Equation (2), is shown in Figure 3a. Due to $\varphi$ being a self-rotation angle, the reachable workspace can be analyzed for simplicity by sectioning it on planes $(\psi ,\theta )$, as in Figure 3b and [23,24,25].

Among all possible working modes, it has been demonstrated that $m_3$, as represented in Figure 1b and Figure 2a, is the best in combined terms of $\eta \left(J\right)$, self-collision among its parts, and operative feasibility, outputting the largest reachable workspace [25]. Therefore, $m_3$ has been chosen as the operative working mode for this study.

2.4. Hapto-Visual Digital Twin for Assistive Control

As the main focus of this study, a hapto-visual digital twin is a detailed simulation model that integrates both haptic feedback and visual representations to provide an immersive and interactive user experience. Implementing a digital twin in the architecture not only allows for more flexible and precise usage of the master device but also introduces a new assistive control mode that implements predefined teaching trajectories for didactic purposes.

In the analyzed environment, a student aims to reproduce a teaching trajectory as closely as possible with the aid of visual and haptic feedback. We define the following sampled trajectories as in Equation (7).

$$\begin{array}{l}\{\begin{array}{}\mathrm{(7a)}& q_{stud}\left(k\right)\equiv {\mathcal{E}}_{stud}\left(k\right)=f_{FKM}\left(\tilde{\mathrm{\Theta }}\left(k\right)\right),\hfill & \hfill {\mathit{r}}_{\mathit{E}}=R_E\left(q_{stud}\left(k\right)\right)·\widehat{\mathit{z}},\mathrm{(7b)}& q_{teach}\left(k\right)\equiv {\mathcal{E}}_{teach}\left(k\right)=M_{teach}(k,:),\hfill & \hfill {\mathit{r}}_{\mathit{E},\mathit{teach}}=R_E\left(q_{teach}\left(k\right)\right)·\widehat{\mathit{z}},\end{array}\end{array}$$

in which we assume, for clarity, notation $k:=k{T}_s$; $k=(1,\dots ,N)\in \mathbb{N}$ is the sampling index; $T_s$ is the sampling period; $q_{stud}$ is the student trajectory, i.e., the one which is the current output of the FKM function; $q_{teach}$ is the teacher trajectory, i.e., the one which the student aims to follow, predetermined in a stored matrix $M_{teach}\in {\mathbb{R}}^{N\times 3}$; $M_{teach}(k,:)$ is the k-th line of $M_{teach}$, storing ${\mathcal{E}}_{teach}\left(k\right)$; $R_E$ is the rotational matrix related to the Euler angle, as in Equation (1). Details of relevant parameters can be found in

Table A1. Assistive control parameters used for the experiments.

Parameter	Symbol	Value	Unit
Distance between operative RF origin and platform center	$r_p$	0.208	[m]
Sampling Period	$T_s$	0.01	[s]
Number of teacher trajectory’s samples	N	376	-
Neighborhood $\mathcal{D}\left(k\right)$ angular span	${\delta }_{span}$	5	[∘]
Neighborhood $\mathcal{D}\left(k\right)$ angular admissible threshold on $\mathcal{E}\left(k\right)$	${\delta }_{thr}$	4	[∘]
Maximum spring coefficient on reduced distance $d(\psi ,\theta )$ Equation (10)	$k_d$	3	[N/m]
Maximum spring coefficient on self-rotation $\varphi$	$k_{\varphi }$	0.1	[Nm/rad]
Damping coefficient	$c_{\omega }$	1	[Nms/rad]

.

2.4.1. Haptic Assistive Control: Premises

In order to implement haptic assistive control to be used in the application described in Section 2.4, we must make the following definitions:

• For clarity of notation, given Equation (7), we define the instantaneous errors between student and teaching trajectories Equation (8):

  $$\begin{array}{l}\{\begin{array}{}\mathrm{(8a)}& \begin{array}{c}{e}_{\psi }\left(k\right)\equiv {\psi }_{stud}\left(k\right)-{\psi }_{teach}\left(k\right)\hfill \end{array}\mathrm{(8b)}& \begin{array}{c}{e}_{\theta }\left(k\right)\equiv {\theta }_{stud}\left(k\right)-{\theta }_{teach}\left(k\right)\hfill \end{array}\mathrm{(8c)}& \begin{array}{c}{e}_{\varphi }\left(k\right)\equiv {\varphi }_{stud}\left(k\right)-{\varphi }_{teach}\left(k\right)\hfill \end{array}\end{array}\end{array}$$
• We define the time-varying cubic neighborhood $\mathcal{D}\left(k\right)$ as the subdominion of $W_{reach}$ centered in $q_{teach}\left(k\right)$, as in Equation (9a). As a first approach, the self-rotation angle $\varphi$ can be neglected, defining the restricted squared neighborhood $\tilde{\mathcal{D}}\left(k\right)$ on plane $(\psi ,\theta )$, as in Equation (9b). All involved parameters’ value definitions can be found in Table A1.

  $$\begin{array}{l}\{\begin{array}{}\mathrm{(9a)}& \begin{array}{cc}\mathcal{D}\left(k\right)\hfill & =\{P:|e_{\psi }\left(k\right)|\le {\delta }_{span},|e_{\theta }\left(k\right)|\le {\delta }_{span},|e_{\varphi }\left(k\right)|\le {\delta }_{span}\}\subset W_{reach},\hfill \end{array}\mathrm{(9b)}& \begin{array}{cc}\tilde{\mathcal{D}}\left(k\right)\hfill & =\{\tilde{P}=(\psi ,\theta ):|e_{\psi }\left(k\right)|\le {\delta }_{span},|e_{\theta }\left(k\right)|\le {\delta }_{span}\}\subset \mathcal{D}\left(k\right).\hfill \end{array}\end{array}\end{array}$$
• We finally define, neglecting $\varphi$ and according to Equations (8) and (9b), the reduced euclidean distance $d(k,\psi ,\theta )$, lying on the $(\psi ,\theta )$ plane, between reduced $q_{stud}\left(k\right)$ and $q_{teach}\left(k\right)$, as in Equation (10):

  $$d(k,\psi ,\theta )=\sqrt{e_{\psi }^2\left(k\right)+e_{\theta }^2\left(k\right)}=\sqrt{{({\psi }_{stud}\left(k\right)-{\psi }_{teach}\left(k\right))}^2+{({\theta }_{stud}\left(k\right)-{\theta }_{teach}\left(k\right))}^2}.$$

  (10)

2.4.2. Haptic Assistive Control: Basic Description

With the reduced euclidean distance $d(k,\psi ,\theta )$, we can impose an assistive control action on the system, adjusting the platform orientation ${\mathit{r}}_{\mathit{E}}$ angles $(\psi ,\theta )$ by pushing the user inside an admitted neighborhood on the basis of a segmented spring-like function, as in Equation (11) and in Figure 4:

$$\begin{array}{c}{\mathit{F}}_{\mathit{E}}(k,\psi ,\theta )=\{\begin{array}{}\mathrm{(11a)}& \hfill 0& & d(k,\psi ,\theta )\le {\delta }_{thr}\hfill & \tilde{P}\in {\mathcal{A}}_F\left(k\right)\hfill \mathrm{(11b)}& \hfill -k_d(d(k,\psi ,\theta )-{\delta }_{thr})·{\mathit{t}}_{\mathit{E}}(\psi ,\theta )& \hfill {\delta }_{thr}<& d(k,\psi ,\theta )\le {\delta }_{span}\hfill & \mathrm{(11c)}& \hfill k_d·{\mathit{t}}_{\mathit{E}}(\psi ,\theta )& & d(k,\psi ,\theta )>{\delta }_{span},\hfill & \end{array}\end{array}$$

in which $k_d$ is a suitable spring constant; ${\delta }_{thr}$ is the radius of the force-related admitted area ${\mathcal{A}}_F\left(k\right)$ in which no assisted action is required; ${\mathit{t}}_{\mathit{E}}(k,\psi ,\theta )$ is the unit vector perpendicular to ${\mathit{r}}_{\mathit{E}}$ lying on the plane $({\mathit{r}}_{\mathit{E},\mathit{teach}},{\mathit{r}}_{\mathit{E}})$, according to Equation (12) and Figure 4b; ${\mathit{\omega }}_{\mathit{E}}$ is the platform’s instant angular rotation.

$${\mathit{t}}_{\mathit{E}}={\mathit{r}}_{\mathit{E}}\times ({\mathit{r}}_{\mathit{E}}\times {\mathit{r}}_{\mathit{E},\mathit{teach}}).$$

(12)

A similar but more straightforward procedure can be adopted for the self-rotation angle $\varphi$ by imposing a segmental spring-like torque around ${\mathit{r}}_{\mathit{E}}$, as in Equation (13):

$$\begin{array}{c}{\mathit{M}}_{\mathit{E}}(k,\varphi )=\left\{\begin{array}{}\mathrm{(13a)}& \hfill 0& & |e_{\varphi }(k,\varphi )|\le {\delta }_{thr}\hfill & \varphi \in {\mathcal{A}}_M\left(k\right)\hfill \mathrm{(13b)}& \hfill -k_{\varphi }·(e_{\varphi }(k,\varphi )-{\delta }_{thr})·{\mathit{r}}_{\mathit{E}}& \hfill {\delta }_{thr}<& \left|e_{\varphi }(k,\varphi )\right|\le {\delta }_{span}\hfill & \mathrm{(13c)}& \hfill -k_{\varphi }·({\delta }_{span}-{\delta }_{thr})·{\mathit{r}}_{\mathit{E}}& & |e_{\varphi }(k,\varphi )|\ge {\delta }_{span},\hfill & \end{array}\right.\end{array}$$

in which ${\mathcal{A}}_M\left(k\right)=[-{\delta }_{thr},{\delta }_{thr}]$ is the torque-related admitted range for $\varphi$ in which no assisted action is required; $\mathcal{A}\left(k\right)={\mathcal{A}}_F\left(k\right)\cup {\mathcal{A}}_M\left(k\right)$.

Also, taking into account a damping action ${\mathit{D}}_{\mathit{E}}=-c_{\omega }·{\mathit{\omega }}_{\mathit{E}}$, which activates if expressions Equations (11) and (13) are non-zero, the device is then subjected to ${\mathit{\tau }}_{\mathit{ctrl}}$ from Equation (5), as in Equation (14).

$${\mathit{\tau }}_{\mathit{ctrl}}=J^t·\left(r_p{\mathit{r}}_{\mathit{E}}\times {\mathit{F}}_{\mathit{E}}+{\mathit{M}}_{\mathit{E}}+{\mathit{D}}_{\mathit{E}}\right),$$

(14)

in which $r_p$ is the actual distance between the platform’s center and the origin.

2.4.3. Haptic Assistive Control: Modifications

Considering the definition of the haptic assistive control made in the previous sections, we have made the following possible modifications:

(I)

To have a more fluid assistive action, we can consider not only the current $q_{teach}\left(k\right)$ but also $q_{teach}(k+1)$ for determining a suitable action associated with neighborhood $\mathcal{D}\left(k\right)$. Admitted dominions $\mathcal{A}\left(k\right)$ should then depend on $\Delta q_{teach}=q_{teach}(k+1)-q_{teach}\left(k\right)$. The control algorithm can be easily modified between different admitted areas’ shapes by considering discrete numeric maps to apply on $q_{stud}$ instead of analytic functions, as in Equations (11) and (13). Examples on $(\psi ,\theta )$ are shown in Figure 5;

(II)

To have a less stringent and time-based assistive action, we can check whether the student has reached $\mathcal{A}\left(k\right)$ before updating to the next $q_{teach}\left(k\right)$ through a boolean condition $studReady$. In this way, the effects of the control loop are drastically reduced, the performance is not evaluated in a time-based point of view, and the student is assisted only when $q_{stud}$ actively leaves the current admitted dominion $\mathcal{A}$. Having adopted this strategy for the experimental part, from this point on the sampling index k will be removed for clarity;

(III)

To have an even less timely action, and with the aid of the visual feedback to be presented in Section 2.4.4, $studReady$ can toggle only on a subset of $\mathcal{A}$ depending on $q_{teach}(k+1)$, e.g., on $(\psi ,\theta )$, for a circular ${\mathcal{A}}_F$, such as the one presented in Figure 5b.

2.4.4. Visual Digital Twin

The visual digital twin aims to visualize on a screen available to the end user the actual pose of the master device. Using the RViz environment inside ROS, each element is processed by a marker object, which contains the following information:

• The 3D mesh model of the object, processed in a digital asset exchange (.dae) format, referenced by a suitable relative reference frame (RF) centered in $\mathit{O}$ for all spherical elements (i.e., legs B and C, and platform), for non-spherical elements following the standard Denavit–Hartenberg convention;
• The pose of the object, contained by a quaternion computed from a suitable rotational matrix associated with the object and the relative RF described by the previous point.

The visual environment can be enhanced by adding extra information regarding the pose of the object, for instance, reporting, for the assistive control mode, RFs associated with $q_{stud}$ and $q_{teach}$ according to Equation (7) and Section 2.4.3, and as in Figure 6. In order to enhance the visual feedback, the RF related to $q_{stud}$, reporting the actual pose of the platform, can be colored in an automatically-updating red/green hue, describing a near (green) or distant (red) $q_{stud}$ from $q_{teach}$ in a way consistent with Equations (11) and (13). In this way, the end user is constantly aware, by looking at the screen, of where the current position of $q_{teach}$ is and how to correct the platform’s pose in order to reach $\mathcal{A}$.

2.5. Overall Control Scheme Implementation

The software architecture of the prototype was developed inside an ROS environment. The main structure of the system can be summarized by the following blocks reported in the colored areas of Figure 7, each corresponding to single or multiple ROS nodes:

• Input blocks (Light green):

  –

  User input: Allowing an end user to toggle between control modes or shut off the device for emergency purposes via a keyboard. The device’s control modes, each corresponding to a suitable integer value inside topic $cmd$, include different operational modes of the device: the free roaming and reset modes developed, respectively, in [25,26]; the stop mode, blocking the master device; the bilateral teleoperation mode, in which the operator interacts with a RCM slave device, such as the one described in [31], and receives haptic feedback; the assistive control mode, outlined in Section 2.4, in which the end user aims to follow a predefined path $q_{teach}$.

  –

  Encoder input: Interfacing with the absolute encoders measuring $\tilde{\mathrm{\Theta }}$ (MAB18A by Megatron [32],

  Table A2. Hall-effect absolute encoder MAB18A using Megatron specifics. More details are reported in [32].

  Specific	Value	Unit
  Angle range	360	[∘]
  Angle resolution	4096	steps
  Supply voltage	5	[V]
  Supply current	<20	[mA]
  Signal load	>5	[$\mathrm{k}\mathsf{\Omega }$]
  Output voltage range	0–5	[V]

  ) through a microcontroller development board (Arduino Uno using Arduino [33]). The block is devoted to the encoders’ signal input and conditioning of $\tilde{\mathrm{\Theta }}$, as outlined in Section 2.2. It also contains a signal differentiator to output the joint speed $\dot{\tilde{\mathrm{\Theta }}}$.

  –

  Teacher input: Outputting a predefined teaching path through the matrix $M_{teach}$, as described in Section 2.4.

• Processing blocks (dark blue):

  –

  FKM: Inputting the measured angles and outputting the estimated values of $\mathcal{E}$, $J\left(\mathcal{E}\right)$, $\eta \left(J\right(\mathcal{E}\left)\right)$, and ${\omega }_E$, as outlined in Section 2.1 and Section 2.2.

  –

  Teacher processer: Updating the current $q_{teach}$ to feed into the system. The block checks whether $q_{stud}$ has reached the current admitted dominion $\mathcal{A}$ and updates only when said condition $studReady=True$, as described in Section 2.4.3.

• Control blocks (light blue):

  –

  Position controller: Used by control modes such as the reset or stop modes, computing a suitable reference position or joint path.

  –

  Torque controller: Used by control modes such as the free roaming, bilateral teleoperation, or assistive control modes. Its purpose is to process and compute, basing on Equation (5), a suitable torque control input.

• Output blocks (light red):

  –

  Motor manager: Devoted to processing the control inputs and interfacing with the prototype’s motors (SC040B by simplex motion [34],

  Table A3. Motor SC040B by Simplex motion specifics. More details are reported in [34].

  Specific	Value	Unit
  Maximum Speed (@24V)	6000	[rpm]
  Maximum Torque	0.8	[Nm]
  Rated Torque	0.28	[Nm]
  Power supply input voltage	24	[V]
  Maximum continuous mechanical output power	120	[W]
  Gear-ratio (by crafted 3D-printed gears)	200/23	-

  ) via the Minimalmodbus Python module [35].

  –

  Visual manager: For pre-processing and visualizing the visual digital twin inside RViz environement, as described in Section 2.4.4.

The overall control scheme also contains the following additional elements:

• Keyboard input: Allowing the user to switch between different control modes;
• Slave plant: Interacting with the architecture by inputting $\mathcal{E}$ for RCM position control, and outputting $\mathit{T}$, as in Equation (5);
• Master plant: Detailed in the previous Sections, it allows said haptic feedback to reach the end user;
• Screen output: Allowing visual haptic feedback to reach the user.

2.6. Testing Methods

We report in this section a preliminary testing method related to the device in the assistive control mode, described in Section 2.4 and Section 2.5, adopting the modifications on the haptic assistive control (II) and (III) depicted in Section 2.4.3.

For the experiment, the selected teacher trajectory $q_{teach}$, stored in a suitable $M_{teach}$, is reported in Figure 8, and consists of a simple trajectory in which $(\psi ,\theta )$ and $\varphi$ linearly span $\pm {15}^{\circ }$ and $\pm {20}^{\circ }$, respectively.

In the experiment, an end user is tasked with moving the platform through the handle shown in Figure 1b, as to follow trajectory $q_{teach}$ as closely as possible. The following experiments took place:

(Ex1)

Four times the assistive control is composed only of Equation (11), neglecting self-rotation $\varphi$ outside admitted ${\mathcal{A}}_M$ Equation (13);

(Ex2)

One time the end user, not aiming to follow $q_{teach}$, actively moves the platform outside the admitted ${\mathcal{A}}_F$, as to prompt ${\mathit{F}}_{\mathit{E}}$, neglecting self-rotation $\varphi$;

(Ex3)

Four times the assistive action is fully implemented, taking into account all Euler angles $\mathcal{E}$ and the full formula Equation (14).

An additional recording node is added to the control scheme in Figure 7, tasked with the memorization of the following variables:

• The Euler angles triple $({\psi }_{r,stud},{\theta }_{r,stud},{\varphi }_{r,stud})\equiv ({\psi }_{stud},{\theta }_{stud},{\varphi }_{stud})-{(135,54.7,0)}^{\circ }$, related to the student trajectory $q_{stud}$ Equation (7a). The angles are offset to the workspace center Equation (2) for plotting clarity;
• The Euler angles triple $({\psi }_{r,teach},{\theta }_{r,teach},{\varphi }_{r,teach}){(}^{\circ })$, related to the teacher trajectory $q_{teach}$ stored inside $M_{teach}$ (7b). The angles are offset for clarity;
• The elements of the assistive force ${\mathit{F}}_{\mathit{E}}=(F_{E,x},F_{E,y},F_{E,z})\left(N\right)$ Equation (11) within the operative RF $(\mathit{O},\widehat{\mathit{x}},\widehat{\mathit{y}},\widehat{\mathit{z}})$, defined in Section 2.1;
• The elements of the assistive torque ${\mathit{M}}_{\mathit{E}}=(M_{E,x},M_{E,y},M_{E,z})\left(Nm\right)$ Equation (13);
• The elements of the assistive damping ${\mathit{D}}_{\mathit{E}}=(D_{E,x},D_{E,y},D_{E,z})\left(Nm\right)$ Equation (14).

Other relevant variables and values reported in the results are as follows:

• Variables $(e_{\psi },e_{\theta },e_{\varphi })$ in Equation (8), described in Section 2.4.1;
• Variable $d(\psi ,\theta )$ in Equation (10), described in Section 2.4.1;
• Values ${\delta }_{thr}$ and ${\delta }_{span}$, described in Section 2.4.2 and reported in Table A1.

In the testing setup, the student user is first given the opportunity to familiarize themselves with the device in the free roaming mode and are allowed to watch a preview of the teaching path on screen. After resetting the device to the workspace center using the reset mode, the experiments are conducted. Various frames of the testing environment are shown in Figure 9.

3. Results

3.1. Experiment 1

Since the experiment is not time-based due to (II), only the results of one trial from (Ex1) are reported in Figure 10 for clarity, whereas the others are shown in different subplots in Figure A1.

Due to the nature of (Ex1), ${\varphi }_{r,teach}$ and $e_{\varphi }$ are not present. Since $d(\psi ,\theta )$ remained below ${\delta }_{thr}$ for the entire duration of the experiment, the values of ${\mathit{F}}_{\mathit{E}}$ in Equation (11) are not reported. The correlation between the two parameters is indeed studied in (Ex2).

3.2. Experiment 2

The results of (Ex2) are reported in Figure 11. Due to the nature of (Ex2), the absolute trajectories of $q_{stud}$ and $q_{teach}$ are omitted for brevity, only reporting differences $(e_{\psi },e_{\theta })$.

3.3. Experiment 3

Similarly to the results of (Ex1), only the results of one trial from (Ex3) are reported in Figure 12 for clarity, whereas the others are shown in different subplots in Figure A2.

4. Discussion on the Experimental Results

The results of (Ex1) in Section 3.1 show that, thanks to the implementation of the visual digital twin, as described in Section 2.4.4, and the modifications (II) and (III), $d(\psi ,\theta )<{\delta }_{thr}=4^{\circ }$ Equation (10) always, resulting in ${\mathit{F}}_{\mathit{E}}=0$ Equation (11). The mean and standard deviation values of all relevant parameters in all trials are listed in

Table 1. The mean and standard deviation of the Euler angles’ instantaneous errors $(e_{\psi },e_{\theta })$ Equation (8) and the reduced euclidean distance $d(\psi ,\theta )$ Equation (10) in (Ex1), alongside the time durations of each trial. All average values are weighted on the time durations of the trials. Average duration is arithmetic.

Trial	${\overline{\mathit{e}}}_{\mathit{\psi }}{\mathbf{[}}^{\mathbf{\circ }}\mathbf{]}$	${\mathit{\sigma }}_{{\mathit{e}}_{\mathit{\psi }}}{\mathbf{[}}^{\mathbf{\circ }}\mathbf{]}$	${\overline{\mathit{e}}}_{\mathit{\theta }}{\mathbf{[}}^{\mathbf{\circ }}\mathbf{]}$	${\mathit{\sigma }}_{{\mathit{e}}_{\mathit{\theta }}}{\mathbf{[}}^{\mathbf{\circ }}\mathbf{]}$	$\overline{\mathit{d}}{\mathbf{[}}^{\mathbf{\circ }}\mathbf{]}$	${\mathit{\sigma }}_{\mathit{d}}{\mathbf{[}}^{\mathbf{\circ }}\mathbf{]}$	Duration [s]
1	−0.34	1.246	0.238	1.010	1.459	0.782	26.8
2	−0.164	1.06	0.113	0.937	1.291	0.610	34.3
3	0.002	0.968	0.110	0.943	1.202	0.626	32.2
4	0.161	1.083	0.176	0.889	1.269	0.636	33
Average:	−0.074	1.082	0.155	0.941	1.298	0.657	31.6

. Without actual control of the self-rotation angle $\varphi$, ${\mathit{M}}_{\mathit{E}}=\mathbf{0}\Rightarrow {\mathit{\tau }}_{\mathit{ctrl}}=\mathbf{0}$ Equations (13) and (14), no assistive control action is present, and the end user can follow the teacher trajectory $q_{teach}$ independently. Even if the primary target of the experiment was not focused on time accuracy, as (II) was adopted and the end user had not received sufficient training on the device, (Ex1) still yielded satisfactory results in terms of overall time duration, being approximately 1.6 times slower than the pre-defined teaching path, which has a total duration of 12.3 s. This outcome preliminarily demonstrates how easy and effective it is to respect the threshold conditions without excessively strict constraints and no assistive control over the self-rotation angle $\varphi$.

Due to the results of (Ex1), the end user was tasked with actively trespassing ${\delta }_{thr}$ in (Ex2). The outcome, described in Section 3.2, demonstrates how the control action of ${\mathit{F}}_{\mathit{E}}$ indeed respects Equation (11). The assistive control action indeed pushes, with a sufficiently strong but yet to be optimized value, the end user inside the current admitted area ${\mathcal{A}}_F$ outlined by relation $d\le {\delta }_{thr}$ with a fitting assistive action. In parts ${\mathit{F}}_{\mathit{E}}\ne 0$, it is imposed ${\mathit{D}}_{\mathit{E}}=-c_{\omega }{\mathit{\omega }}_{\mathit{E}}$. This is done in order not to trigger, with a narrow ${\mathcal{A}}_F$, an unwanted and uncontrolled vibrating action between the thresholds of said dominions.

The same experiment of (Ex1) is reproduced in (Ex3) with the addition of an assistive torque action on the self-rotation angle $\varphi$, as in Equation (13). The control action is then assumed in its complete form Equation (14). Similarly to (Ex1), $d(\psi ,\theta )<{\delta }_{thr}$ always, there are time intervals in which $e_{\varphi }>{\delta }_{thr}$, issuing ${\mathit{M}}_{\mathit{E}}$, and thus ${\mathit{D}}_{\mathit{E}}$ following Equation (13). The assistive control action is thus sufficiently strong to steer the user within the admitted area ${\mathcal{A}}_M$, allowing for the continuation of the experiment. The mean and standard deviation values ofall relevant parameters involved in the trials are listed in

Table 2. The mean and standard deviation of the Euler angles’ instantaneous errors $(e_{\psi },e_{\theta },e_{\varphi })$ Equation (8) and the reduced euclidean distance $d(\psi ,\theta )$ Equation (10) in (Ex3). The average values are weighted on the time durations of the trials. Average duration is arithmetic.

Trial	${\overline{\mathit{e}}}_{\mathit{\psi }}{\mathbf{[}}^{\mathbf{\circ }}\mathbf{]}$	${\mathit{\sigma }}_{{\mathit{e}}_{\mathit{\psi }}}{\mathbf{[}}^{\mathbf{\circ }}\mathbf{]}$	${\overline{\mathit{e}}}_{\mathit{\theta }}{\mathbf{[}}^{\mathbf{\circ }}\mathbf{]}$	${\mathit{\sigma }}_{{\mathit{e}}_{\mathit{\theta }}}{\mathbf{[}}^{\mathbf{\circ }}\mathbf{]}$	${\overline{\mathit{e}}}_{\mathit{\varphi }}{\mathbf{[}}^{\mathbf{\circ }}\mathbf{]}$	${\mathit{\sigma }}_{{\mathit{e}}_{\mathit{\varphi }}}{\mathbf{[}}^{\mathbf{\circ }}\mathbf{]}$	$\overline{\mathit{d}}{\mathbf{[}}^{\mathbf{\circ }}\mathbf{]}$	${\mathit{\sigma }}_{\mathit{d}}{\mathbf{[}}^{\mathbf{\circ }}\mathbf{]}$	Admitted Points $\mathbf{[}\mathbf{\%}\mathbf{]}$	Duration [s]
1	0.002	0.869	0.155	0.716	−0.229	1.905	0.957	0.613	94.62	55.7
2	0.036	0.765	0.029	1.065	−0.524	2.516	1.036	0.805	92.46	50.3
3	−0.095	1.037	0.183	0.772	0.392	1.880	1.109	0.696	97.15	52.6
4	−0.134	1.229	0.192	0.922	−0.693	2.663	1.291	0.864	85.13	63.2
Average:	−0.052	0.988	0.144	0.867	−0.281	2.254	1.106	0.747	92.03	55.5

. The greater amplitude of $e_{\varphi }$ can be explained by the fact that modification (III) does not account for any simplification on the admitted area ${\mathcal{A}}_M$, only focusing on ${\mathcal{A}}_F$. Nevertheless, the overall behavior on the self-rotation is satisfying for these preliminary results, having, for all trials in (Ex3), $max|e_{\varphi }|-{\delta }_{thr}=3.{929}^{\circ }$ and a percentage of admitted sampling points of $92.03\%$. The average time duration, still considering the caveats of (Ex1), does not seem affected by the full assistive control implementation, being approximately 1.7 times slower than the reference one of 20.3 s, a result comparable to the one obtained in (Ex1).

5. Conclusions

This paper was devoted to the definition and implementation of a hapto-visual digital twin for the quasi-spherical parallel manipulator (qSPM) to be used for assistive control in didactic telesurgery. After outlining the kinematics and reachable workspace of the device, the research presented the formal definition of assistive control, in which a student user is required to follow a predefined teaching trajectory inside relevant threshold parameters. The paper described both the haptic and visual attributes of the digital twin, characterizing their functionalities and presenting possible modifications to make the control algorithm less stringent. In fact, this approach aimed to overcome both traditional surgical training limitations, heavily relying on verbal instructions and visual observations, enabling stricter “hand-over-hand” guidance methods by allowing the user to take the initiative within an accepted range where no corrective actions are taken by the device.

After presenting the hardware and software implementation of the overall system, preliminary tests were conducted to analyze its performance in an exemplified assistive control environment. Results show the following: assistive control without any intervention on self-rotation angle $\varphi$ indeed satisfies all threshold constraints, requiring no control action; said control action is sufficiently strong enough to push the user inside the time-variant admitted area following the teacher trajectory; the full control action on all the platform’s orientation Euler triplet $(\psi ,\theta ,\varphi )$ has been deemed satisfactory for this preliminary testing, with a high percentage of admitted sampling points of $92.03\%$. Our results further demonstrate how the overall training protocol could support the objective definition of learning curves and lay the foundation for telementorship-based training, overcoming the limitations of traditional training methods in a teleoperation environment.

Future developments will directly involve thorough testing of the assistive control algorithm, revolving around a broader sample of end user subjects and using the device with different and more complex teaching trajectories $q_{teach}$, ultimately defined through the full use of the bilaterally teleoperated system. The research will also focus on the modifications presented in Section 2.4.3: testing different dominions, as in (I) and (III), and relaxing all the constraints defined in (II) and (III). As the testing results presented are preliminary, the study will involve the optimization of the assistive action control parameters: $({\delta }_{thr},{\delta }_{span})$ for the amplitude of the neighborhood dominion and admitted area Equation (9) and $(k_E,k_{\varphi },c_{\omega })$ for the actual amplitudes of the assistive action forces Equations (11)–(14). This optimization could depend on different discrete “strictness levels” of the control strategy, which may be selected based on the individual end user or the specific current operation of the device.