Development of Multimodal Stimulator for Studying Human Tactile Perception and Cognitive Functions: Preliminary Results

Featured Application

Research on realizing immersive tactile sensation.

Abstract

Humans mostly perceive tactile sensations in daily life as a combination of warmth, vibration, and pressure. To understand the complex tactile perception and cognitive processes, in this study, we aimed to develop a multimodal stimulator and investigate changes in neuronal activity. An actuator that can display warmth (W), vibration (V), and pressure (P) on the distal region of the index finger has been developed. Preliminary experiments were conducted with nine subjects. Electroencephalograms were measured for six tactile stimuli—three single stimuli (W, V, and P) and three combination stimuli (W + V, V + P, and W + V + P)—and event-related desynchronization/synchronization (ERD/S) analysis were performed. The actuator can present all kinds of stimuli in the same location and control stimulation parameters quantitatively. For all experiments, there was an ERD in the α and β bands about 0.5 s after stimulation followed by ERS was observed in the C3 area. The change in the peak-to-peak value was the largest for warmth and the smallest for pressure. In contrast, in the duration of the ERD, W was the shortest and P was the longest. As stimulus presented simultaneously, the ERD became longer in both the alpha and beta bands. In the beta band, the peak of ERD became larger. The developed system was confirmed to be capable of providing valid tactile stimulation, inducing appropriate neuronal activation, and enabling multimodal tactile research.

1. Introduction

Various technologies utilizing visual and auditory-based virtual or augmented reality technology have been developed, and research for practical applications is being actively conducted [1,2,3]. Also, in attempting to mix the real and the digital worlds, research for technology related to mixed reality is being carried out to implement an interface between humans and computers with vision and space [1,4]. Recently, research has also been conducted to add tactile sensation to these technologies to increase realism and provide immersive experiences [5].

Humans primarily use their visual and auditory senses to perceive changes in their environment [6]. For this reason, research has been focused on visual and auditory presentation technology to make spaces and objects feel real such as in virtual reality, augmented reality, and mixed reality [2]. However, as the field of application expands and higher immersion is required, haptics technologies are being introduced to add tactile sensations in addition to visual and auditory sensations [3,5]. Currently, most of them use vibration to add tactile sensation. DC (direct current) motors are mostly used to generate vibration due to their simple configuration, small size, and reproducibility [4,5,6,7,8]. There have also been attempts to provide pressure sensation using pin arrays or pneumatics, or to display warmth using Peltier elements, but they are rarely used in practical applications. Recently, research has been conducted to provide sustained pressure by changing the plane to convex by electrochemical reaction [9,10]. A method to generate heat using conductive yarns to present warmth has also been devised [8,11]. The existing methods have advantages and disadvantages in terms of stimulation parameters such as stimulation frequency, stimulation intensity, stimulation area, location, and response time depending on the source of actuation, but only one of the three tactile sensations, vibration, pressure, and warmth, can be presented by each actuator. A study that presents two types of sensations simultaneously has been carried out using electromagnetic coils and pneumatics to present vibration and pressure [12,13] or thermal fabric with continuous convexity using active material to present warmth and pressure [11,14]. A study to present three types of tactile sensations was attempted relatively recently. Pneumatic technology was used to present pressure and vibration sensations, and thermal fabric was used to present thermal sensations. However, it was difficult to achieve subtle intensities and high-frequency vibrations. Additionally, since the fabric, along with the pneumatic chamber, completely surrounded the fingertips, pressure and thermal sensations were presented not only on the lower part but also on the upper part of the fingers. These factors resulted in differences in the perceived location of tactile sensations compared to those experienced in daily life [15]. However, since most tactile sensations perceived by humans in daily life are a combination of vibration, pressure, and warmth, stimulators that can stimulate all three sensations individually or simultaneously are needed. Also, to induce a high level of realism, quantitative evaluation studies on the perceptual and cognitive aspects of the brain within and between humans for obtaining effective tactile stimulation parameters are necessary.

In this paper, we developed a multimodal tactile stimulator that can present three types of stimuli, warmth, vibration, and pressure, with one actuator at the same stimulus location. Preliminary experiments were conducted on nine subjects to validate the developed stimulator. The changes in brain signals in response to individual and combined stimuli were measured by an electroencephalogram (EEG). An event-related desynchronization/synchronization (ERD/S) analysis was performed for each type of stimulus to identify changes in neuronal activity by the frequency of brain signals over time.

2. Materials and Methods

The overall configuration of the developed multimodal stimulator is shown in Figure 1. The developed system consists of three parts: control part, driver part, and an actuator.

2.1. Control Part

An ATMEGA128A (Microchip Technology, Chandler, AZ, USA) was used to control the entire system. The microcontroller used has a general-purpose 8-bit AVR series core. It operates at up to 16 MHz over an input operating voltage range of 2.7 to 5.5 volts and has 53 programmable general-purpose input and output ports (GPIOs). Figure 2 shows the configuration blocks and signal flow of the entire system.

For the temperature control, 14 GPIOs were used to control the 12-bit digital-to-analog converter (DAC) AD7545A (Analog Device, Wilmington, MA, USA). It outputs a direct current signal in the range of 0~5 volts and is passed to the thermo-tactile driver part, which is configured to adjust the temperature in 5 steps in the range of 26 °C to 36 °C in proportion to the magnitude of the output value. At a 0 volt output, the thermo-tactile driver does not operate. A digital temperature sensor, TCN75 (Microchip Technology, Chandler, AZ, USA), and the two-wire serial interface (TWI) function of the microcontroller were used to monitor the temperature. To control the vibration, the 16-bit timer/counter function of the microcontroller and the AD7545A were used to output a sinusoidal wave. The digital values for generating 12-bit sinusoidal waves were generated in the form of hexadecimal values using MATLAB R2023b (MathWorks Inc., Natick, MA, USA) software, a numerical analysis tool, and stored in an array in a look-up table. The values output from the DAC were filtered with a 2nd-order low-pass filter before being passed to the vibro-tactile driver part. The magnitude of the vibration was adjusted from 0.2 g to 2.1 g, and the frequency of the vibration was adjusted from 25 Hz to 250 Hz in 5 steps. Four GPIOs, a 16-bit timer/counter, and an analog-to-digital converter (ADC) were used to control the pressure sense. Two of the GPIOs turn on/off two solenoid valves to control the start and end of the presentation of the pressure. The other two GPIOs control a ball valve to maintain the intensity of the pressure. A step motor driver, SLA7062M (Allegro Microsystems, Manchester, NH, USA), was used to generate a signal from the timer/counter function and a digital signal to control the forward and reverse directions, respectively, to control the intensity of the pressure sensation [16]. The magnitude is adjusted in five steps, ranging from 0.2 psi to 3.8 psi.

2.2. Driver Part

The driver part is responsible for operating the actuator by receiving signals from the control part as input. The warm sensation utilizes heat when passing a current through the electric wire. For this purpose, the thermo-tactile driver configures a boost converter used in switched-mode power supply (SMPS). Since it utilizes the Joule heating effect, the temperature is regulated by monitoring the current flowing in the electric wire. The current flowing through the sensing resistor is converted into a voltage by Ohm’s law, and this signal is input to the microcontroller to control the amount of current flowing. This allows the temperature to be adjusted to provide a warm sensation. Vibro-tactile sensation utilizes the electromagnetic force generated when passing a current through a coil located in a magnetic field. The vibro-tactile driver amplifies the sinusoidal signal input from the control part with a 40-watt audio power amplifier, LM2876 (National Semiconductor, Santa Clara, CA, USA), to supply enough current to the coil to generate vibration. The magnitude and frequency of the sinusoidal waves are regulated using a microcontroller to control the stimulus intensity and frequency of vibration. Pressure sensation is achieved using compressed air. The pressure-tactile driver uses an air pump, JS20 (Join Medical, Seoul, Republic of Korea), and an air valve matrix [16]. The air pump generates up to approximately 9 psi of compressed air, which can be adjusted in five steps from 0.2 psi to 3.8 psi using a ball valve, and two solenoid valves are used to turn on/off the pressure sensation.

2.3. Actuator

The actuator was designed using the 3-dimensional design software SolidWorks 2021 (Dassault Systems, Paris, France) with dimensions of 18 × 18 × 45 mm³ (width × length × height) to allow for displaying three different types of stimuli individually or simultaneously on fingertip and printed using a 3D printer (Figure 3a).

The thermo-tactile driver for warmth uses a 0.5 mm diameter copper wire that is placed on the upper part of the actuator where the finger is placed and wired in an S-shape in the area in contact with the fingertip. The thermo-tactile driver is configured to adjust the temperature in 5 steps in the range of 26 °C to 36 °C. The vibro-tactile driver uses a 0.22 mm diameter copper wire wound into a coil in the middle of the actuator with a permanent magnet made of neodymium placed underneath so that it can be in a magnetic field. The magnitude of the vibration was adjusted from 0.2 g to 2.1 g and the frequency of the vibration was adjusted from 25 Hz to 250 Hz in 5 steps. The pressure-tactile driver is presented in the form of pressing the compressed air generated by the driver by blowing it in the normal direction of the fingertip. The compressed air generated from the air pump and air valve matrix is delivered to the actuator through a rubber tube. A total of 16 circular holes with a diameter of 2 mm and a spacing of 3.5 mm between the center holes are arranged in a 4 × 4 arrangement, so that uniform compressed pressure is delivered to the whole fingertip area. Figure 3b shows the configuration for calibration and verification of the actuator’s performance. A 3-axis accelerometer, MMA7260Q (NXP Semiconductors, Eindhoven, The Netherlands), was used to measure vibration. For temperature measurement, TCN75, a small outline integrated circuit (SOIC) package, was attached under the 3-axis accelerometer module so that the heat-generating electric wire was in contact with the surface. The pressure was measured by digitizing the analog signal of the pressure sensor 33A-005G-2210 (Smate, Lukang, Taiwan) connected to the air valve matrix with a microcontroller so that the pressure could be checked in real time.

2.4. Preliminary Human Experiment

To verify the feasibility of the developed multimodal stimulator, a preliminary electroencephalogram (EEG) experiment was conducted on 9 male subjects in their 20 s (mean age 24 ± 1.7 years). The distal tip of the index finger of the subject’s right hand was placed comfortably on the top of the actuator, and a single trial was designed with a stimulation phase of 2 s and a resting phase of 58 s, as shown in Figure 4a. All examinations were performed under and approved by the regulations of Institutional Review Committee of Konkuk University Hospital (KUH 1160062), and all methods were carried out in accordance with the approved guidelines.

In the stimulation phase, six stimuli were presented 10 times each in a randomized order: warmth (W), vibration (V), pressure (P), W + V, V + P, and W + V + P. The stimuli were presented in the following order. The resting phase was 58 s long to minimize the cumulative effect of the stimulus type. Stimulus intensity was fixed at 36 °C for warmth, 200 Hz for vibration, with a stimulus frequency of 1.02 g, and a force of 2.7 psi for pressure. Brain signals were measured using QuickAMP (Brain Product, Inc., Gilching, Germany) and signals from 8 locations (C3, C4, Cz, AF3, AF4, P3, P4, and Pz) were recorded at a sampling rate of 500 Hz using a 10–20 international electrode system. A black curtain was used to minimize external visual and auditory factors and a headset with white noise was worn (Figure 4b).

The brain signals were analyzed on the C3 area, which is the contralateral somatosensory area of the right hand that presented the stimulus. After bandpass filtering from 0.5 Hz to 40 Hz using MATLAB software, the analysis period was 0.5 s before stimulus and 2 s after stimulus presentation; changes were confirmed through spectrogram analysis.

3. Results

3.1. Multimodal Stimulation Device and Actuator

Figure 5 shows the developed system and the actual shape of the actuator. The control part, thermo-tactile driver, and vibro-tactile driver are configured on three circuit boards. The pressure-tactile driver is configured on the same circuit board as the control part. Two DC (direct current) power supplies are used to supply +24 volts, ±12 volts, and +5 volts.

3.2. Response Characteristics of the Actuator

The temperature, vibration, and pressure measurement results are shown in Figure 6. Figure 6a shows the output signal from the temperature sensor. It took about 2 s to go up from the baseline temperature of 26 °C to the maximum warmth temperature of 36 °C. Figure 6b shows the temperature rise profile for each stimulation step. The rise to target temperature can adjust to be the same for all stimulation trials, 2 s. The raw data from an accelerometer for a vibration stimulus of 200 Hz is shown in Figure 6c. The signals in the x and z axes are similar in magnitude because they are in the same plane, while the signal in the y axis, which is in the normal direction, is the largest. Figure 6d shows the peak value by stimulus intensity levels and frequencies. Even though the frequency varied, the change in stimulus intensity was small. Figure 6e shows the output signal from the pressure sensor. The rising time to the maximum pressure of 3.8 psi was about 0.4 s and the falling time was about 0.1 s. Figure 6f shows the magnitude of the pressure as a function of the stimulus intensity level, and it was found to change linearly.

3.3. ERD/S Spectrogram Analysis

Figure 7 shows the results of the ERD/S spectrogram in the C3 area of the brain. For all stimulus types (warmth (W), vibration (V), pressure (P), W + V, V+ P, and W+ V + P), all subjects perceived all tactile parameters presented in all trials; there was a decrease in brain signal power (dB) in the α band (8~12 Hz) and β band (12~31 Hz) about 0.5 s before and after stimulation, i.e., ERD, followed by an increase in signal strength, i.e., ERS. This is a common phenomenon in the human perception of tactile sensations [17]. ERD was followed by ERS within 0.5 s after stimulus in the β band (12–31 Hz) for the single stimuli.

ERS occurred after 0.5 s in all cases except P. In the case of P, the duration of ERD was 0.2 s longer. Combined stimuli also showed ERS after 0.5 s, but the duration of the ERD was 0.4 s longer for W + V + P. For the α-band (8–12 Hz), the ERD occurred after 0.5 s in all conditions. The ERD was more pronounced in experiments with V stimuli. The peak value of the ERD increased when different types of stimuli were presented together and was largest when all three stimuli were displayed simultaneously. For maximum and minimum peak analysis, the ERD and ERS of warmth (W) had peak values of −3.27 dB and 2.45 dB, respectively. The ERD and ERS of vibration (V) had peak values of −2.31 dB and 1.83 dB, respectively. Pressure (P) had the smallest peak values of ERD and ERS at −2.09 dB and 1.64 dB, respectively.

4. Discussion

In this study, we developed a multimodal stimulator for the study of human tactile perception and cognitive function. The tactile sensations of warmth, vibration, and pressure can be presented individually or simultaneously, and the stimulus parameters of tac-tile stimulus type, intensity, and time can be quantitatively controlled using a microcontroller. In addition, the three stimuli can be presented in the same location, which is the area below the fingers where most sensations are felt in everyday life. From the pilot experiments with human subjects, changes in neuronal activity were confirmed in all types of stimuli.

From the evaluation of the actuator, the system showed that the intensity of the warmth sensation ranged from 26 °C to 36 °C, the vibration sensation ranged from 0.2 g to 2.1 g, the frequency ranged from 25 Hz to 250 Hz, and the pressure sensation ranged from 0.2 psi to 3.8 psi in five levels. Although fluctuations of up to ±0.5 °C and ±0.4 psi occurred for warmth and pressure, respectively, the subjects did not perceive the change. In the case of warmth, the rising time to the maximum temperature was about 2 s and the falling time was about 3.8 s. The thermo-tactile driver can control the rise time of the temperature because it can digitally control the magnitude of the current flowing through the electric wire. This will also allow for additional experiments on the perceptual and cognitive aspects of the difference in temperature-change time. However, the return to the initial temperature cannot be controlled, as it uses natural convection with the finger resting on the actuator. In the case of vibration, it has the advantage of being able to present the same intensity from the low frequency of 25 Hz to the range of 225 Hz to 250 Hz [18], which is the most sensitive range for humans, including the 130 Hz to 180 Hz range [19], which is the vibration frequency used by most cellular phones. This corresponds to the frequency range [18,20] detected by the four representative mechanoreceptors, which will enable the study of various aspects of tactile sensation. In the case of pressure sensation, the rising time to the maximum pressure was about 0.4 s and the falling time was about 0.1 s, so the repetition rate can be presented up to 0.5 s, or 2 Hz, unless it is a sustained pressure-sensation experiment.

In general, research on tactile stimuli requires repeated experiments to generate meaningful results. For this reason, the framework of the actuator must be made of a rigid, low-deformation material. In this study, the actuator 3D-printed using ABS (acrylonitrile butadiene styrene) filament, which is a high-strength and heat-resistant material, so that it can be consistently used without deforming even when the force applied by the subject, compressed air, and temperature change repeatedly.

Three single stimuli (W, V, and P) and combination stimuli (W + V, V + P, and W + V + P) were presented to the bottom of the right index finger, and the ERD/S spectrum was analyzed in the somatosensory area (C3) of the brain. Event-related desynchronization (ERD) refers to the decrease brain signal power in the α band and β band in response to external stimulation. Event-related synchronization (ERS) shows an increase in signal power in the same band in response to the stimulus [17,21]. Although ERD/S analysis is mainly used in motor research, it has also been used in tactile research in the sensorimotor rhythm (SMR) region of 10 to 30 Hz because vibration and pressure also trigger fine muscle movements [22]. In this study, a similar trend was observed, i.e., ERD occurred after stimulus presentation followed by ERS [23,24]. It is believed that the ERD occurred due to attention processing by tactile perception after the stimulus presentation, and the subsequent ERS occurred due to neuronal activity due to cognitive processing of the stimulus [24].

For individual stimuli, from a, b, and c in Figure 7, all three types of stimuli (W, V, and P) showed ERD and ERS in the β band. For warmth (W), the ERD did not appear in the α band, but for vibration (V) and pressure (P), it appeared after 0.5 s. Also, for both stimuli (V and P), the ERD lasted longer than for the temperature stimulus. This is likely due to the small muscle movements. In the peak-to-peak analysis of ERD and ERS, warmth was the largest and pressure was the smallest. This is because thermoreceptors, which detect warmth, are more sensitive than mechanoreceptors [11,21,25]. Vibration and pressure sensations differed from thermal sensations in that the frequencies activated in the β band of the ERD were similar in duration and time. In sensory receptors that detect sensations, vibration is predominantly responded to by Pacinian corpuscles, while pressure is predominantly responded to by Merkel discs [8]. In physiology, these are classified as mechanoreceptors because they respond together depending on the intensity and duration of the sensation. Thermal sensation is primarily mediated by free nerve ending receptors classified as thermoreceptors that respond without physical deformation, leading to a different trend compared to vibration and pressure sensations. The peak of the vibration sensation was larger than the pressure sensation, which is likely due to the fact that the intensity of the pressure presented in this study was relatively not high and also, we used a frequency of 200 Hz, which is the most sensitive in vibration. But this may change depending on the tactile stimulation parameters (frequency, intensity, and time) of other tactile studies [17,19,26].

For the combined stimuli, from d, e, and f in Figure 7, in all cases, the ERD was larger in all α bands of the individual stimulus and commonly appeared after 0.5 s. This is likely due to the inclusion of vibration stimuli. Although the same stimuli were presented simultaneously to individual tactile receptors, the superimposed results of the responses to each individual stimulus did not appear. This is thought to be because information processing in the brain is performed integrally rather than individually. Further detailed research on the functional and effective connectivity of the brain is needed. In the case of two simultaneous stimuli, the duration of the β band ERS was longer than that of a single stimulus. This means that more complex cognitive processes are required than for a single stimulus. For the W + V + P simultaneous stimuli, the ERD peaked at −3.63 dB in the β-band and had the maintain largest duration. This suggests that neuronal activity is more focused on attention-related processing. Also, the interaction between the types of stimuli would have resulted in different results than in the case of a single stimulus. This suggests that the response of sensory receptors in the skin may be different depending on the type and combination of stimuli, and that there may be differences in information processing in the brain.

5. Conclusions

A multimodal stimulator and actuator that can quantitatively present three types of tactile sensations with various parameters were developed. Changes in brain signals were confirmed according to tactile stimulation conditions. Using the developed system, it was possible to induce tactile sensations and utilize it for multimodal tactile research. Although the results of this study are preliminary, we confirmed that the multimodal actuator can be utilized for studies related to the perceptual and cognitive aspects of tactile sensation. In the future, it is expected that in-depth studies can be conducted by changing various tactile stimulation parameters. Human senses involve wide variability and differences between and within individuals, which leads to attempts to enhance realism by adding tactile sensations. The developed system can be utilized for the development of tactile presentation technologies, such as research on extracting meaningful tactile trigger parameters. This will enable the implementation of immersive technologies in the field of human–computer interfaces in addition to vision and hearing. It can also be used for research in the field of neuroscience, such as information processing, activation, and connectivity in the brain related to tactile sensations. These results will contribute to clinical areas such as brain neurology and neuronal disorders. However, this study also has limitations. First, the hardware for each type of tactile sensation was configured as individual circuits. To facilitate large-scale experiments, it is necessary to configure them as a single integrated board like a printed circuit board (PCB). Second, only individual and combined stimuli for a single tactile parameter were presented for each type of tactile sensation. Future studies are needed to conduct detailed experiments using various tactile parameters on a large number of participants and to analyze connectivity. Third, thermal sensation is generated by adjusting the current flowing through a copper wire to utilize the Joule effect. While it is easy to control the temperature rise time and regulation, lowering the temperature requires natural convection, which prolongs the experimental time.