3.1. Procedures and Metrics
There are three experiments designed in this section for developing and evaluating the proposed dual-mode p
HRI control system. In particular, Experiment 2 involves two tissue surface scenarios of horizontal and slope, as shown in Figure 3
. All experiments employ the same procedure for performing the ultrasound (US) imaging task, i.e., 6 sessions are required in each mode (p
HRI) while each session includes 3 trials. One trial is defined as the US probe moving over the soft tissue surface, starting at one end, moving to the other end, and then returning to the starting point. The p
HRI mode represents the physical human–robot interaction method established by an admittance controller, while the
HRI mode represents the tele-operation method established by a tracking camera system.
The performance metrics involved in the experiments are listed as follows:
Normal contact force, mean and variance (squared standard deviation) between the US probe and the soft tissue. The former indicates task performance accuracy while the latter indicates task performance stability.
User effort, in units of Newton. It is indicated by the force exerted on the robot EE by the human user in p
HRI mode, and also serves as input signals for the admittance controller. It is measured by sensor-2, as shown in Figure 1
Time percentage. A percentage for retaining the normal contact force within the desired range in one trial.
In the admittance controller (4
), the coefficient matrices
are parameterized as
for the translational part and
for the orientational part. For simplicity, the US probe is assumed to be exactly perpendicular to the tissue surface during the task (a more sophisticated 3D soft tissue reconstruction method may be required for cases beyond this assumption [8
]); then, the normal contact force can be measured by the z
-axis of the F/T sensor directly in the sensor frame. In the force controller (8
), the desired force is set as
in the F/T sensor frame and then transformed into the robot base frame. A t
-test is employed for statistical analysis and a p
is adopted as the significance level. Video S1
demonstration for the experiments is available online https://drive.google.com/file/d/1rwz_5fpUVSh2QEDMGiJansBVoiGXbDpO/view?usp=sharing
accessed on 12 December 2021.
3.2. Experiment 1: AF vs. AF + HF
In Experiment 1, audio feedback and haptic feedback (AF + HF) are presented in pHRI mode while audio-only feedback (AF) is presented in HRI mode during the US imaging task. In this experiment, we investigate how different feedback affects task performance. No force controller is implemented in this experiment, which means that both the lateral movement of the US probe and the normal contact force are controlled by the user.
This experiment requires the user to perform the US imaging task on a horizontal tissue surface in pHRI mode and HRI mode, respectively, with different feedback. As described earlier, a total of six sessions are required in each mode, while each session includes 3 trials. During the task, the user needs to manually control the lateral movement of the US probe and also needs to maintain the normal contact force between the the probe and the tissue in the desired range. As mentioned earlier, the task performance accuracy is indicated by the mean normal contact force throughout this paper, while the task performance stability is indicated by the corresponding variance.
Statistical analysis on the results (see Table A1
for details) shows that there is no significant difference (
) in the mean normal contact force between the two modes, but there is a significant difference (
) between their variances, which means that the human user has significantly more stable task performance (i.e., smaller variance) with AF in
HRI mode than with AF + HF in p
HRI mode. A sample of data is presented in Figure 4
. As can be seen in the figure, the normal contact force cannot stably remain in the desired range in either mode. This is also reflected by the time percentages for retaining the force in the desired range (see Table A2
for details), which are lower than 75% in both modes (
HRI mode and
The results from Experiment 1 show that the task performance accuracy in HRI mode with AF is comparable to that in pHRI mode with AF + HF in terms of averaged normal contact force. The task performance stability in HRI mode with AF is significantly better than that in pHRI mode with AF + HF in terms of their variances. These results indicate that the audio-only feedback (AF) is as good as audio-haptic feedback (AF + HF); thus, the audio feedback is able to serve as a replacement for the haptic feedback in our case.
3.3. Experiment 2: AF + FC vs. AF + FC + HF
In Experiment 2, the hybrid admittance-force controller is implemented. More specifically, an additional force controller (FC) is implemented into the control system in both p
HRI modes based on Experiment 1. This means that the normal contact force is regulated by the robot FC while the lateral movement is controlled by the human user in Experiment 2. As a further step based on Experiment 1, this experiment investigates how the different feedback will affect task performance when the proposed hybrid controller is implemented. The task procedures are the same as those described in Experiment 1. In particular, two tissue surface scenarios, namely the horizontal scenario and slope scenario (Figure 3
), are considered in Experiment 2 in order to test the flexibility of the proposed system.
(1) Experiment 2a: Horizontal scenario
In Experiment 2a, the US imaging task is conducted on a horizontal soft tissue surface. Statistical analysis (see Table A3
for details) shows that the task performance accuracy in
HRI mode with AF + FC is significantly better than in p
HRI mode with AF + FC + HF (
) in terms of mean normal contact force. Despite this significance, it is worth noting that the max–min magnitude difference on the normal contact force across all sessions is only
N, which is close to the F/T sensor resolution
N. There is no significant difference between their variances in the two modes (
), which indicates that the task performance stability in the two modes is comparable.
A sample of data for Experiment 2a is shown in Figure 5
. As can be seen from the figure, user effort in p
HRI mode is in the range of
N, which indicates that the user can easily control the lateral movements of the US probe when an additional force controller is implemented.
(2) Experiment 2b: Slope scenario
In Experiment 2b, the US imaging task is conducted on an inclined slope soft tissue surface in pHRI mode and HRI mode separately. This slope tissue scenario could be further generalized to slopes with other angles of inclination or even an inverted tissue surface, which may be encountered in the clinical setting.
It is worth noting that in HRI mode, a regular camera on one side of the slope for side view is mounted with the same angle of inclination as the slope such that the inclined tissue surface in the camera view appears as a horizontal tissue surface. This setting is reasonable since the user is able to use any angle of view for a good viewpoint in pHRI mode. Moreover, since the pose mapping algorithm between the robot and the stick is based on relative displacements to their own initial poses, the motion of the stick on a horizontal surface can be automatically mapped to control the motion of the US probe on the inclined slope. This operational flexibility can help the user to obtain a better view and perform comfortable movements on the remote doctor side in HRI mode if needed.
A sample of data from Experiment 2b is shown in Figure 6
. In Figure 6
a, user effort is represented by the user-exerted force along the movement direction of the US probe (i.e., along the slope in this experimental scenario). As can be seen from the figure, user effort in p
HRI mode is in the range of
N, which is relatively small. This means that the human user can easily control the lateral movements of the probe on the slope when an additional force controller is implemented.
Statistical analysis (see Table A4
for details) shows that there is no significant difference (
) between the two modes in the mean normal contact force, and also no significant difference (
) in their variances.
The results in Experiments 2a and 2b show comparable task performance accuracy and task performance stability in HRI mode (with AF + FC) and in pHRI mode (with AF + FC + HF), which indicates the potential capability of HRI mode to be taken as an alternative for pHRI mode even without HF. In addition, compared to Experiment 1, task performance stability in Experiments 2a and 2b is significantly improved (all ).
The results in Experiment 2 indicate the same conclusion as that obtained in Experiment 1, i.e., audio feedback can be a good replacement for haptic feedback. More importantly, the hybrid admittance-force controller implemented in Experiment 2 further relieves the need for haptic feedback in HRI mode.
3.4. Experiment 3: Dual-Mode Switching
Experiment 3 is designed to evaluate the overall performance of the proposed dual-mode pHRI-HRI control system when mode switching is involved. This experiment requires the human user to perform the task using a “1-2-1-2” sequence, i.e., first to perform the task using the stick (in HRI mode) for one session, then perform the task using the robot EE handle (in pHRI mode) for another session, then perform the task in HRI mode again for one session, then perform the task in pHRI mode again. This procedure is repeated another two times in order to generate six sessions for each mode.
The task procedure in this experiment can be better understood via the sample data shown in Figure 7
. In the figure, two short bar areas represent the
HRI mode, while two long bar areas represent the p
HRI mode. User effort indicates the user-exerted force on the robot EE handle (in p
HRI mode) along the lateral movement direction of the US probe. As can be seen in the figure, the switching between the p
HRI mode is seamless, smooth, and stable, and it can be performed whenever necessary, without involving stability issues. This is reasonable and expected due to the relative-displacement-based mapping method, which will be discussed in more detail in the next section.
Statistical analysis (see Table A5
for details) shows that there is no significant difference in the task performance accuracy between the two modes (
) in terms of the normal contact force. Although there is a significant difference statistically in their variances (
), it is noticed that all the standard deviation values are less than
N (i.e., less than the F/T sensor resolution). Considering this, it can be safely concluded here that there is no significant difference found between the two modes in terms of either normal contact force or their variances when switching is involved, which indicates the robustness of the proposed dual-mode system during mode switching.
3.5. Statistical Comparison across Experiments
The longitudinal comparison of the task performance accuracy and the task performance stability across Experiments 1, 2a, 2b, and 3 is conducted in pHRI mode and in HRI mode separately by using a t-test. Hereafter, for compactness, EX.1, 2a, 2b, and 3 will be used to represent Experiments 1, 2a, 2b, and 3, respectively.
HRI mode, there is no significant difference in the normal contact force between EX.1 and either of the other experiments (see Figure 8
a). However, it should be noted that the mean values cannot truly reflect the task performance stability, which mainly depends on their variances. For their variances, there is a significant difference (all
) between EX.1 and either of the other experiments in p
HRI mode (see Figure 8
b). Similar statistical results are obtained for
HRI mode (see Figure 8
c,d). These results indicate that both in p
HRI modes, the task performance is significantly improved in terms of task performance stability and reliability by implementing the designed hybrid admittance-force controller (EX.2a, 2b, 3). The statistical analysis results for the longitudinal comparison across experiments are summarized in Table A6