EEG Investigation on the Tactile Perceptual Performance of a Pneumatic Wearable Display of Softness

Abstract

Multisensory human–machine interfaces for virtual- or augmented-reality systems are lacking wearable actuated devices that can provide users with tactile feedback on the softness of virtual objects. They are needed for a variety of uses, such as medical simulators, tele-operation systems and tele-presence environments. Such interfaces require actuators that can generate proper tactile feedback, by stimulating the fingertips via quasi-static (non-vibratory) forces, delivered through a deformable surface, so as to control both the contact area and the indentation depth. The actuators should combine a compact and lightweight structure with ease and safety of use, as well as low costs. Among the few actuation technologies that can comply with such requirements, pneumatic driving appears to be one of the most promising. Here, we present an investigation on a new type of pneumatic wearable tactile displays of softness, recently described by our group, which consist of small inflatable chambers arranged at the fingertips. In order to objectively assess the perceptual response that they can elicit, a systematic electroencephalographic study was conducted on ten healthy subjects. Somatosensory evoked potentials (SEPs) were recorded from eight sites above the somatosensory cortex (Fc2, Fc4, C2 and C4, and Fc1, Fc3, C1 and C3), in response to nine conditions of tactile stimulation delivered by the displays: stimulation of either only the thumb, the thumb and index finger simultaneously, or the thumb, index and middle finger simultaneously, each repeated at tactile pressures of 10, 20 and 30 kPa. An analysis of the latency and amplitude of the six components of SEP signals that typically characterise tactile sensing (P50, N100, P200, N300, P300 and N450) showed that this wearable pneumatic device is able to elicit predictable perceptual responses, consistent with the stimulation conditions. This proved that the device is capable of adequate actuation performance, which enables adequate tactile perceptual performance. Moreover, this shows that SEPs may effectively be used with this technology in the future, to assess variable perceptual experiences (especially with combinations of visual and tactile stimuli), in objective terms, complementing subjective information gathered from psychophysical tests.

1. Introduction

Conventional wearable (especially hand-held) human–machine interfaces used for virtual- or augmented-reality systems are typically unable to generate tactile feedback on the softness of virtual objects. This limitation is critical, as the ability to physically render a virtual softness is essential to enable a variety of multisensory interfaces, for various possible uses. Examples include simulators for medical training [1], tele-operation systems [2], computer-aided design [3], 3D model exploration [4], as well as tele-presence systems for social interactions [5]. Such devices should return haptic feedback to the fingertips, in order to mimic the haptic sensation perceived by touching soft objects. To that aim, they should ideally provide our brain with both tactile and kinaesthetic feedback.

Combining both these feedback modalities into a single wearable device is not straightforward, as the required actuation technology challenges the compactness, weight, comfort and prospected costs of the resulting system. However, a simplification can be made, by considering that Srinivasan and LaMotte have demonstrated that, in order to perceive the softness of objects with a deformable surface, tactile feedback alone is sufficient; in contrast, kinaesthetic feedback alone is insufficient [6]. Therefore, in order to develop simple wearable haptic displays of softness, the most straightforward approach is to design actuation systems that can generate purely tactile feedback via a deformable surface. The aim is to enable control of both the contact area with the skin and its indentation depth, which are both essential for the perception of softness [7]. Therefore, here we define a wearable (fingertip-mounted) tactile display of softness as a device that can provide such tactile feedback, by stimulating the fingertips via quasi-static (non-vibratory) forces, delivered through a deformable surface, so as to control both the contact area and the indentation depth.

In order to meet these requirements, the state of the art does not offer many options. One of them uses electrical motors that actuate flexible/stretchable structures, such as polymer membranes or fabrics; though effective, this approach requires complex, bulky and heavy mechanisms [8,9].

Other options include the use of soft interfaces (elastomers) deformed by a source of energy; although they exploit different actuation technologies, they avoid the need for complex transmission mechanisms. These are dielectric elastomer actuators (DEAs), electrostatic actuators and pneumatic actuators. DEAs are made of electrically deformable elastomeric membranes [10], which have been used in various configurations to create non-vibratory fingertip displays [11,12,13,14,15,16]; their main drawback is the need for high voltages. The same problem applies also to purely electrostatic actuators [17]; moreover, they share with DEAs limited active deformations, as compared with another key technology: pneumatic actuators. The latter currently offer, in general, a unique combination of high forces, large deformations and excellent electrical safety, with the possibility of creating compact and lightweight structures on the fingertip, as the pneumatic source can be arranged separately. Furthermore, as a difference from electrical motors, pneumatic driving has the advantage of not causing electromagnetic interference with functional imaging instrumentation (e.g., fMRI), which could be used to investigate the perceptual response in the brain. Among the (few) possible technological solutions described to date for pneumatic tactile displays, one of them uses focused air jets [18,19], which, however, limit the realism of the experience, due to the lack of a soft interface. Another strategy is based on tiny, pressurised chambers [20,21,22,23], which we consider very promising. Indeed, a new type of wearable pneumatic tactile display of softness was previously presented by our group [24]; the device consists of a small inflatable chamber that is arranged at the fingertip, as detailed in the next section.

So far, the perceptual performance of such a tactile interface has only been investigated through psychophysical tests, by collecting purely subjective evaluations expressed by users. In order to make the assessment objective, in this work we conducted an electroencephalographic (EEG) study, by recording somatosensory evoked potentials (SEPs) in response to different modalities of tactile stimulation delivered by the display.

Whilst several previous papers have employed SEPs to analyse perceptual responses to haptic stimulations with vibrotactile devices [25,26,27,28,29,30], those that have used SEPs to study pneumatic tactile displays have been rather limited so far, both in number and in scope [31,32,33]. Here, we present a systematic investigation on our wearable non-vibratory tactile display, by quantifying the amplitudes and latencies of six components of SEP signals, recorded from four bilateral electrode positions, upon the stimulation of three combinations of healthy subjects’ fingers with three intensities of tactile pressure.

2. Materials and Methods

2.1. Wearable Pneumatic Tactile Display of Softness

The tactile display consists of a small plastic chamber, closed by a thin elastomeric membrane, which can be inflated upon pressurisation with air, as shown in Figure 1a,b. The membrane (Elastosil, Wacker, Germany) was made of polydimethylsiloxane and had a thickness of 70 μm. The plastic chamber was ergonomically shaped to conform to the shapes and sizes of adult finger pulps, and was secured to the finger using elastic bands, as visible in Figure 1c. Up to three tactile displays could be independently controlled using a bespoke electropneumatic unit (Figure 1c), which contained a pump, six valves, six pressure sensors and a microcontroller. The unit was connected via USB to a personal computer, to be driven by any type of external software via serial communication. The total weight of the control unit was 380 g, which made the technology easily portable. Details about the constitutive components have been presented by Frediani and Carpi [24].

Each tactile display had a weight on finger of ~3 g, and a length and thickness of 22.5 mm and 8 mm, respectively; the exposed part of the membrane had a diameter of 12 mm. The dynamic response of the device is presented in Figure 1d. The pressure rise time, defined as the time needed by the pressure signal to reach 90% of its steady-state value, was ~320 ms at 10 kPa, ~390 ms at 20 kPa and ~440 ms at 30 kPa.

2.2. Experimental Setup and Tactile Stimulation Protocol

The tests involved ten subjects (four males and six females), aged between 25 and 40 years (mean age 29.1 years). All subjects were right-handed. None of them had been diagnosed with neurological diseases. The study was conducted according to the guidelines of the Declaration of Helsinki and was approved by the Ethics Commission of the University of Florence (n. 160, 28 May 2021).

For each subject, three tactile displays were placed on the dominant hand, so as to stimulate the thumb, index and middle fingers. After the application of the tactile displays to the fingertips, the subject was asked to comfortably sit on an armchair, keeping their arms along the armrests, and to completely relax, for 4 min. The subject was then asked to maintain the relaxed state also during the administration of the stimuli, in order to minimise skeletal muscle movements, as well as to keep the eyes closed, so as to minimise eye movements and eye blinks. The experimental setup is shown in Figure 2.

The tactile stimulation protocol consisted of nine stimulation conditions, divided into three groups: (i) stimulation of only the thumb; (ii) stimulation of the thumb and index finger simultaneously; and (iii) stimulation of the thumb, index and middle finger simultaneously. For each group, the stimulation was repeated for three intensities of tactile pressure: 10, 20 and 30 kPa. They were applied in random sequence. Each stimulation consisted of a sequence of 30 pressure steps at the given intensity (10, 20 or 30 kPa), having a duration of 500 ms each, with an inter-stimulus interval of 2 s. Each subsequent sequence started 2 min after the end of the preceding sequence. The duration of the entire protocol was approximately 34 min, including the initial 4 min of relaxation.

2.3. EEG Signals Recording

EEG signals representing the cortical response to the non-painful tactile stimuli described above were recorded with Ag/AgCl electrodes, which were positioned according to the 10-10 standard reference system and were combined with a conductive gel for contact impedance adaptation (electrode impedance < 5 kΩ). In particular, the study analysed signals detected by the following electrodes located above the fronto-central (Fc) and central (C) areas of the cerebral cortex (somatosensory cortex): Fc2, Fc4, C2 andC4, and Fc1, Fc3, C1 and C3. The electrodes’ positions are indicated in Figure 2. As all subjects were right-handed and the stimulations occurred on their dominant hand, the even-numbered electrodes (i.e., those located on the right hemisphere) were ipsilateral with the stimulus, whilst the odd-numbered electrodes (i.e., those located on the left hemisphere) were contralateral. The EEG biopotentials were measured relative to the central Cz electrode.

The signals were recorded at a sampling frequency of 1024 Hz, by using a 64-channel electroencephalograph (BE PLUS LTM/PRO dense array EEG, EBNeuro, Florence, Italy). In order to synchronise the electrophysiological recordings with the electropneumatic control signals of the tactile displays, the electroencephalograph’s MK channel was used to receive a trigger signal from the electropneumatic unit. This allowed for recording the instant at which the stimulation began, i.e., the onset of the pressure stimulus (Figure 1d).

2.4. EEG Signals Processing and SEP Peaks Identification

Signal processing was performed using the MATLAB toolbox EEGLAB v2023 [34]. The signals were bandpass filtered (0.5–30 Hz) to extract the typical SEP frequency range. All channels were referenced relative to the average signal from all electrodes (average referencing).

An independent component analysis was used to identify and remove artefacts, due to ocular and facial involuntary movements and complex brain activities. From each EEG signal, epochs having a duration of 1200 ms were extracted, starting from 200 ms before the stimulus onset, up to 1000 ms after it. Pre-processing also applied a baseline correction relative to the 200 ms interval that preceded the stimulus onset. Furthermore, the signals were subjected to visual inspection, so as to identify and remove possible epochs containing residual motion artefacts.

The analysis of the SEP waveforms was focused on the following signal components: P50, N100, P200, N300, P300 and N450. They typically are mostly related to tactile stimuli in their early sensing phase, rather than in their subsequent cognitive processing [35]. In order to identify each characteristic peak of those components, a MATLAB script was used to automatically detect them within the following time intervals (selected according to Oniz et al. [33]): P50 within 30–150 ms, N100 within 50–150 ms, P200 within 150–250 ms, N300 within 250–350 ms, P300 within 250–350 ms and N450 within 350–550 ms.

2.5. Statistical Analysis

A statistical analysis was conducted using MATLAB. The normal distribution of the data was assessed through the Shapiro–Wilk test. Data with a normal distribution were analysed with an ANOVA test with Bonferroni correction (MATLAB functions ‘anova1’ and ‘multcompare’). Data with a non-normal distribution were analysed in different ways, depending on the experiment: in order to compare the responses of the ipsilateral and contralateral brain areas, we used the Wilcoxon test (MATLAB function ‘ranksum’), whilst in order to compare the responses to the three groups of stimulation and the three intensities of pressure, we used the Kruskal–Wallis test with Bonferroni correction (MATLAB functions ‘kruskalwallis’ and ‘multcompare’). The significance level (p-value) for each analysis was set to 0.05.

3. Results

3.1. SEP Waveforms

Figure 3, Figure 4 and Figure 5 present the grand-average SEP waveforms (as obtained by averaging the data across the ten subjects), detected by the contralateral and ipsilateral electrodes in the different conditions of tactile stimulation.

The waveforms related to all subjects were processed in order to extract the latency and amplitude values of the various signal components, as described in Materials and Methods. The values were then plotted as functions of either the tactile pressure intensity or the number of simultaneously stimulated fingers, as separately presented in the following sections.

3.2. SEP Latencies and Amplitudes as Functions of Tactile Pressure Intensity

Figure 6, Figure 7, Figure 8, Figure 9, Figure 10 and Figure 11 present, for the different number of simultaneously stimulated fingers and the different recording electrodes, plots of the latency and amplitude values of the SEP signal components as functions of the tactile pressure intensity.

3.3. SEP Latencies and Amplitudes as Functions of the Number of Simultaneously Stimulated Fingers

Figure 12, Figure 13, Figure 14, Figure 15, Figure 16 and Figure 17 present, for the different tactile pressure intensities and the different recording electrodes, plots of the latency and amplitude values of the SEP signal components as functions of the number of simultaneously stimulated fingers.

4. Discussion

4.1. Effect of Signal Recording from the Contralateral Cerebral Hemisphere

Due to the contralateral organisation of the forebrain, in general, contralateral SEPs typically tend to have shorter latencies and larger amplitudes, as compared to ipsilateral SEPs. In order to verify whether the recordings showed such an effect in our tests, a statistical analysis was performed for each stimulation condition. In particular, the analysis compared, both for the latency and amplitude of each signal component, the values achieved from the electrodes in symmetrical positions on the two hemispheres, i.e., Fc1 vs. Fc2, Fc3 vs. Fc4, C1 vs. C2, C3 vs. C4. The outcomes of this analysis are graphically indicated with asterisks in Figure 6, Figure 7, Figure 8, Figure 9, Figure 10 and Figure 11. Both for the latency and amplitude, statistically significant differences between the contralateral and ipsilateral electrodes were mostly found on the fronto-central area (Fc electrodes). Especially, for all the three groups of stimulation, Fc3 was able to provide, relative to Fc4, significantly shorter latencies and significantly larger amplitudes, confirming the expectations.

This outcome provides an indication for future investigations on this new tactile display of softness, confirming that for the sake of studying the tactile perceptions that it can elicit, contralateral cortical responses are preferable.

4.2. Effect of the Tactile Pressure Intensity

By increasing the stimulus intensity, in general, SEPs typically tend to have shorter latencies and larger amplitudes. In order to verify the possible occurrence of such an effect in our tests, a statistical analysis was performed on the latency and amplitude of each signal component, for the variable intensity of the tactile pressure. The outcomes of this analysis are graphically indicated with asterisks in Figure 6, Figure 7, Figure 8, Figure 9, Figure 10 and Figure 11. Statistically significant differences were mostly found for contralateral recordings, as expected. Furthermore, they were especially evident for the stimulation that involved all three fingers (thumb, index and middle finger), as shown in Figure 10 and Figure 11. This finding could be explained by the fact that this stimulation condition engaged the highest number of mechanoreceptors, leading to increased signal summations, which likely resulted in enhanced signal-to-noise ratios.

The data also show additional evidence. With the simultaneous stimulation of three fingers and with contralateral recordings, i.e., for the configuration with the highest signal-to-noise ratios (Figure 10 and Figure 11), most of the trends (although with some exceptions at the lowest pressure, as discussed later) agree with the expectation that increasing pressure intensities should cause shorter latencies and larger amplitudes.

Nevertheless, it is worth noting that statistically significant differences mainly occurred among the latencies, rather than the amplitudes. Moreover, they mostly occurred for the high-latency components (see for instance N300, P300 and N450 detected by Fc3—Figure 10 and Figure 11). This might be interpreted as a consequence of the tactile display’s mechanical dynamics. Indeed, according to the time that the device took to increase the output pressure to any set value (pressure rise time, Figure 1), the applied pressure intensities were most significantly different over intervals that were mainly relevant for the long-latency components.

Another evidence from the configuration with the highest signal-to-noise ratios (Figure 10 and Figure 11) is the inconsistency of some data at the lowest pressure of 10 kPa, which occasionally altered the expected trends (decrease in latency and increase in amplitude) for increasing pressures, as shown by the plots. Our interpretation is that at 10 kPa the stimulation was affected by the following issues. First, that pressure was likely insufficiently intense to be perceivable in a distinct way by all subjects. Indeed, some of them reported that the stimulus was somehow ambiguous. Moreover, considering that the variability in the shape of different finger pulps was responsible for different initial conditions of contact between the finger and the membrane at a null pressure, it is likely that the application of a small pressure was insufficient to level out such differences; this may have caused a considerable variability in the skin contact area (and therefore also on the tactile experience), even upon actuation. Furthermore, the perceptual process at the lowest pressure was certainly complicated by the relatively slow mechanical dynamics of the tactile displays at that pressure (Figure 1). This fact could explain why inconsistencies have mostly been recorded for the short-latency components P50 e N100, especially in the latency plots (Figure 10 and Figure 11). Therefore, all of these considerations suggest that the data collected at 10 kPa might have a lower reliability relative to the data at higher intensities.

Overall, this set of comparisons at variable tactile pressure intensities provides useful indications that this new tactile display of softness was capable of inducing perceptual responses that were consistent, both in latency and in amplitude, with the stimulation conditions.

4.3. Effect of the Simultaneous Stimulation of Multiple Fingers

By increasing the number of co-stimulated fingers, in general, SEPs typically tend to have larger amplitudes. In order to investigate whether our tests actually showed such an effect, another statistical analysis was performed, for the variable number of simultaneously stimulated fingers, on the signal components’ amplitude (as well as on the latency, for consistency with previous analyses, although this was not of explicit interest). The outcomes of this analysis are graphically indicated with asterisks in Figure 12, Figure 13, Figure 14, Figure 15, Figure 16 and Figure 17. Even in this case, statistically significant differences were mostly found for contralateral recordings, as expected. By ignoring the data at 10 kPa (Figure 12 and Figure 13), which are considered as less reliable (as discussed above), we observed from the contralateral recordings at 20 and 30 kPa (Figure 14, Figure 15, Figure 16 and Figure 17) the following evidence. The amplitude showed only two statistically significant variations (those detected by Fc1 and C3 at 30 kPa—Figure 17) and both confirmed the expectation of an increasing amplitude for an increasing number of co-stimulated fingers.

Therefore, even this set of comparisons indicates that the tactile display was able to generate perceptual responses that were consistent with the stimulation conditions.

4.4. Possible Influence of the Experimental Conditions and Methodological Choices

As for any research activity, the results presented in this study might have been affected, to some extent, by experimental conditions during the acquisition of the data and methodological choices adopted for their processing.

As an example, although during the tests the subjects kept their eyes closed and any artificial light in the room was switched off, the room was not completely dark, as some natural light could enter through a window.

Another issue deals with signal filtering, as required to exclude possible artifacts (noise), due to, for instance, involuntary muscle activations or electromagnetic interference. Owing to a lack of gold standards, different strategies are described in the literature [36]. In this work, during pre-processing we applied a bandpass filter (0.5–30 Hz), which was chosen according to previous empirical experience; however, that choice was not guided by any systematic assessment in comparison with alternatives. Therefore, it is unknown whether cutting the signals’ bandwidth at 30 Hz may have negligibly attenuated some EEG components or not.

5. Conclusions

We presented an EEG investigation on a new kind of tactile display of softness, based on pneumatic actuation. Its tactile perceptual performance was assessed by measuring cortical SEPs, in response to different conditions of tactile stimulation of one or multiple fingers at different pressures. The latency and amplitude of several components of the signals were systematically analysed, as functions of the position of the recording electrodes, the number of stimulated fingers and the tactile pressure intensity.

The collected data show that this new type of wearable pneumatic device is able to elicit perceptual responses consistent with the stimulation conditions. Especially, the data showed that, for the sake of maximising the SEP amplitude (and so also improving the investigation accuracy), it is convenient, as expected, to use contralateral electrodes, the highest tactile pressure intensity and the highest number of co-stimulated fingers. Therefore, by confirming such expectations, we verified that the device can generate perceptual responses that are predictable, according to established electro-neurological knowledge. This has the following main implications.

Firstly, it proves that the wearable tactile device is capable of adequate actuation performance, which enables adequate tactile perceptual performance.

Secondly, the consistency of the data show that conventional cortical SEPs may effectively be used with this technology in the future in order to assess variable perceptual experiences (especially with combinations of visual and tactile stimuli), in objective terms; this will complement subjective information that can be gathered from psychophysical tests.

Therefore, future developments could take advantage of these outcomes, by integrating SEP measurements into psychophysical experiments, so as to combine subjective and objective assessments of perceptual tests.

Moreover, the results from this study provide useful indications for EEG-based objective investigations on future improved wearable interfaces, which may integrate tactile feedback with kinaesthetic feedback, for comprehensive haptic stimulation; this could be exploited in various applications, such as haptic feedback for rehabilitation [37] or minimally invasive surgery [38].