2.1. Experimental Setup
The experimental apparatus was composed of two essential sub-setups, positioned at a proper distance for achieving telepresence conditions (Figure 1
The first sub-setup consisted of a piezoelectric disk (7BB-12-9, MuRata, Kyoto Prefecture, Japan) encapsulated in silicone rubber, with a customized process, which was placed on the index fingertip of a textile glove to deliver the feedback [52
]. An optical sensor (Leap Motion, CA, USA) tracked the user’s hand gesture. We defined as haptic sub-system this first sub-setup located in a laboratory of The BioRobotics Institute of Sant’Anna School of Advanced Studies, Pontedera (Pisa, Italy). A graphical user interface (GUI) was developed using LabVIEW (National Instruments Corp., USA) for acquiring speed and position of the hand’s center of mass, handling communication with the other remote sub-setup, and recording hand position data.
We defined as tactile sub-system, the second remote sub-setup used for the exploration of the phantom. This platform included: a cartesian manipulator (X–Y and Z, 8MTF-102LS05 and 8MVT120-25- 4247, STANDA, Vilnius, Lithuania); a 6-axis load cell (Nano 43, ATI Industrial Automation, Apex, USA) to measure contact forces between a customized indenter with a spherical tip of 3 mm radius, mounted on the load cell; and the phantom during the active sliding. A second GUI was designed for real-time control of the motorized stages and data communication.
In the ILS session, the tactile sub-system was placed near the user to perform experiments in streamlined telepresence. In the NILS session, instead, it was located in a remote laboratory in Florence, Italy, which was about 50 km apart from the haptic sub-system, thus increasing the challenge of the proposed task. Data communication between the two sub-setups was provided through the User Datagram Protocol (UDP) that ensured a maximum latency of 15 ms. The adopted communication protocol allowed bidirectional streaming of data: hand gestures from the haptic sub-system to the mechatronic platform, and normal force from the tactile sub-system to the glove to be encoded and then delivered (Figure 2
During experiments, the user actively explored and searched for stiffer areas in the polymeric phantom. Four different rubber materials were used to cast 12 hemispherical inclusions (3 replicas of each material), of 5 mm radius, randomly inserted across the X–Y plane in a silicon cuboidal block, 100 × 100 × 15 mm3 in size. The chosen polymers for the inclusions were: Sorta Clear 40 (Smooth-on, PA, USA) as the stiffest, polydimethylsiloxane (PDMS) Sylgard 184 (Dow Corning, USA), Dragon Skin 30, and Dragon Skin 20, while Dragon Skin 10, the softest, was used for the cuboidal block.
The platform control during the experimental session was based on the user’s hand movements, tracked by the 3-D optical sensor. The infrared light-based gesture sensor used had a field of view of about 150 degrees and a range between 25 and 600 mm above the device, and it showed proper performance when it had a high-contrast view of the hand to be detected and tracked. Therefore, the device was placed just below and close to the rest position of the user’s hand, and the workspace was set according to the specifications of the gesture sensor. Details about the reference coordinates of the gesture sensor, highlighting the rest position and volume, are shown in Figure 1
B. As the user’s hand moved out of this volume, the motorized stages followed along the same direction at a speed proportional to the displacement of the user’s hand. The assigned speed was calculated using the difference ρ − ρ0
, where ρ was the distance between the hand center of mass and the center of the sensor, and ρ0
was the radius of the neutral spherical region, set to 50 mm. The user was provided with visual feedback about the position of the indenter on the phantom in the remote environment, without any information on the location of the inclusions.
The contact between the indenter tip and the polymeric test object generated a force. To avoid mechanical damage to the phantom and load cell due to the user’s upward movement, a force threshold (0.5 N) was introduced. The measured force was sent to the haptic sub-system to be encoded into spike patterns, which triggered the piezoelectric actuator. We implemented a neuromorphic feedback strategy based on a regular Izhikevich spiking model discretized using Euler’s method at 5 kHz [53
]. The chosen model efficiently encodes the temporal dynamics of a mechanoreceptor including the biological plausibility of the computationally intense Hodgkin–Huxley model [54
] and the computational efficiency of the integrate-and-fire model [55
The neuromorphic activation of the transducer was achieved by setting the input to the neuron proportional to the magnitude of the normal force measured by the remote subsystem. Further details about the neural model and a definition of its parameters can be found in our previous works [43
]. Initial calibrations were also performed to counterbalance the effect of saturation of the neural model [48
The spikes generated by the Izhikevich model were sent to the vibro-tactile glove by means of a piezoelectric driver (DRV2667, Texas Instruments). This driver facilitated the setting of the actuation parameters in analog mode to have a gain of 40.7 dB, 200 V peak-to-peak voltage amplitude, and 105 V offset voltage.
2.2. Platform and Inclusions Characterization Protocols
A characterization protocol was performed aiming at assessing the uncertainty of the apparatus in tracking and reproducing the user’s hand pose. Five subjects (4 men and 1 woman between 28 and 30 years of age), enrolled among the staff of The BioRobotics Institute of Sant’Anna School of Advanced Studies, took part in the experiment. They were asked to drive the stages of the platform in order to perform a set of target trajectories within the X–Y cartesian plane. The target trajectory and the subject’s relative position on this plane were part of the GUI providing visual feedback in the haptic sub-system. The cluster consisted of 2 square-shaped (s1
= 60 mm and s2
= 30 mm, in length) and 2 circular (r1
= 30 mm and r2
= 15 mm, in radius) trajectories, which had to be followed 3 times by each subject, starting from their centers. Deviations in the trajectories, used to evaluate the system in terms of performance, were calculated using the area comprised between the perimeter of the target and executed trajectories, by means of the boundary function (MATLAB, The MathWorks, Inc., Natick, Massachusetts, USA. The error rates between the tracked area and target were calculated as their ratio and are reported in Section 3
Moreover, before involving human subjects, the phantom was mechanically characterized to assess the vertical stiffness (ΔFz
/Δz) of the nodules and the surrounding soft material by means of the proposed platform equipped with a flat indenter. The adopted experimental protocol consisted of 5 trials in which the entire set of inclusions experienced an indentation at a fixed force threshold (Fz
= 0.5 N) and speed (v = 0.125 mms−1
). To estimate the stiffness of the investigated polymers, the vertical component of the force collected during each compression was processed through scripts in MATLAB and results of such a characterization are reported in Section 3
2.3. Inclusions Identification Experimental Methods
Psychophysical experiments were performed to validate the proposed system for delivering stiffness information about the palpated nodules in both ILS and NILS tactile telepresence conditions. The experiments involved 10 participants (7 men and 3 women between 24 and 33 years of age) in ILS, and 15 (9 men and 6 women between 25 and 37 years of age) in NILS, enrolled among the university students or staff of The BioRobotics Institute of Sant’Anna School of Advanced Studies, Pisa, Italy. The participants took a comfortable posture at the control workstation in the laboratory, where the haptic sub-system was located. They wore the vibro-tactile glove on their dominant hand and a headset to eliminate environmental noise. To familiarize participants with the driving of the tactile platform, each participant took part in a fifteen minute training session. Moreover, this preliminary task got the users accustomed to the vibro-tactile signal exerted by the piezoelectric actuator of the glove. Answers provided during the training sessions were not included in the analyzed results. Both ILS and NILS psychophysical experiments consisted of a tactile identification task: within six minute time-period of the protocol, the participants unreservedly explored the silicon block and pressed a button on a keyboard on any occasion of perceived frequency variation in the vibro-tactile feedback.
The performances of the participants were evaluated in MATLAB in terms of rate of correct identification of the inclusions, using parameters calculated for both ILS and NILS populations and for each participant. Specifically, we evaluated: (a) the number of true positives (TP), (b) the number of false positives (FP), and (c) accuracy (TP/P, with P = collected responses—FP). These parameters were computed as a function of the center-to-center distance between the position of the perceived inclusions and the nearest actual inclusion. If the perceived position was felt to be within a radius of 10 mm from the center of the nearest inclusion (i.e., the distance was equal to the diameter of the inclusion), the perceived inclusion was classified as TP; otherwise it was classified as FP. The classification of collected responses was also evaluated for a lower tolerance, 5 mm, and two greater ones, 15 mm and 20 mm. A logistic fitting, using a Cumulative Distribution Function (CDF) [56
] and the nlinfit
MATLAB function, was performed for each material to carry out the rate of correct perception and the perceptual thresholds for both the investigated experimental conditions.