Development of a Modular Adjustable Wearable Haptic Device for XR Applications

Abstract

Current XR applications move beyond audiovisual information, with haptic feedback rapidly gaining ground. However, current haptic devices are still evolving and often struggle to combine key desired features in a balanced way. In this paper, we propose the development of a high-resolution haptic (HRH) system for perception enhancement, a wearable technology designed to augment extended reality (XR) experiences through precise and localized tactile feedback. The HRH system features a modular design with 58 individually addressable actuators, enabling intricate haptic interactions within a compact wearable form. Dual ESP32-S3 microcontrollers and a custom-designed system ensure robust processing and low-latency performance, crucial for real-time applications. Integration with the Unity game engine provides developers with a user-friendly and dynamic environment for accurate, simple control and customization. The modular design, utilizing a flexible PCB, supports a wide range of actuators, enhancing its versatility for various applications. A comparison of our proposed system with existing solutions indicates that the HRH system outperforms other devices by encapsulating several key features, including adjustability, affordability, modularity, and high-resolution feedback. The HRH system not only aims to advance the field of haptic feedback but also introduces an intuitive tool for exploring new methods of human–computer and XR interactions. Future work will focus on refining and exploring the haptic feedback communication methods used to convey information and expand the system’s applications.

1. Introduction

Since the concept of the “Ultimate Display” was presented by Ivan E. Sutherland at the IFIP Congress in 1965 [1] and laid the groundwork for advancements in computer graphics, virtual reality (VR), and interactive computing, numerous definitions have been introduced to describe the concept of VR over the years. For instance, Fuchs and Bishop [2] described it as “real-time interactive graphics with 3D models, combined with a display technology that gives the user the immersion in the model world and direct manipulation”. Gigante [3] defined it as an immersive, multisensory experience and “The illusion of participation in a synthetic environment rather than external observation of such an environment, relying on a 3D, stereoscopic head-tracker displays, hand/body tracking and binaural sound”. A more contemporary view by Slater [4] characterizes VR as an immersive, interactive system that instills a sense of presence, where users feel part of a computer-generated environment and that the events taking place are really happening, despite recognizing its artificial nature. From these definitions, as evidenced in various articles, albeit with some differences, three specific and unanimous components emerge for defining VR: immersion, presence, and interaction within a virtual environment [4,5,6,7,8,9,10,11,12]. Immersion, specifically, refers to the degree to which multiple senses are engaged, the level of interaction, and the extent to which the virtual environment replicates real-world stimuli. This concept relies on the characteristics of the technology employed to isolate the user from reality [4]. The degree of immersion is dependent on both the quantity and quality of the sensory engagement, aiming to replicate human sensations holistically, including touch and other sensory inputs [13]. However, creating a comprehensive experience that simulates most or all human senses remains a significant challenge [14,15,16,17]. Although there have been notable advancements in audio–visual technology driven by commercialization and the growth of the multimedia industry, it is argued that further technological innovation is imperative to enhance immersion and realism beyond audio–visual stimuli [16,18,19,20]. The pursuit of more immersive experiences through heightened sensation began with the emergence of VR technology, pioneered by figures like Morton Heiling and Ivan Sutherland, who developed devices aimed at achieving full immersion [1,21]. However, in the past decade, the field of XR has undergone a significant transformation, expanding beyond visual and auditory stimuli to incorporate multisensory experiences, with a special focus on haptic feedback [22]. The growing demand for portable electronics and the development of wearable and flexible technologies [23] have supported this transition, highlighting the crucial role of tactile interactions in enhancing the realism of VR environments, thereby meeting the increasing demand for more interactive technologies. This progression demonstrates an expansion and intensification of research across various domains [24], particularly in clinical applications, where XR is proving to be remarkably effective [25].

Human haptic perception encompasses both kinesthetic and cutaneous (tactile) elements [26]. Kinesthetic feedback pertains to the perception of the position and movement of one’s body, mediated by receptors found in the skin, joints, skeletal muscles, and tendons [27]. In contrast, cutaneous feedback is linked to stimuli detected by low-threshold mechanoreceptors beneath the skin in areas of contact [28]. Cutaneous feedback encompasses a range of sensations involving mechanoreceptors and thermoreceptors that are distributed throughout the skin and are crucial for perceiving external stimuli. The deterioration of these sensory mechanisms or their associated afferent peripheral nerves can impose significant limitations on daily life. Engineered interfaces that integrate with the skin to engage healthy receptors in a programmable fashion can provide sensory substitution, which is crucial for patients with impaired sensory nerves, amputations, or rehabilitation needs [24]. Therefore, available haptic devices aim to cater to both kinesthetic and cutaneous feedback to create haptic illusions, interacting with a user’s perceived position and tactile data while integrating information from a virtual environment [29].

Various reviews [27,30,31], as well as studies on wearable interfaces [32] and their applications [33,34,35], have focused on categorizing haptic devices into distinct types, each offering unique contributions to VR. Culbertson et al. [30] argued that grounded devices offer superior interaction quality despite facing constraints in terms of cost, portability, and workspace requirements. In contrast, other studies [32,36,37] have explored how wearable devices like gloves and exoskeletons prioritize authenticity and mobility, with a primary emphasis on providing tactile and force feedback to simulate real-world interactions. Laycock and Day [31] investigated hand-held devices to enhance immersion in gaming and VR through vibrations or motion feedback. This variety of haptic device types, supported by numerous research initiatives, underscores the critical role of haptic technology in enhancing immersion within virtual environments and human–computer interaction [25,29].

Haptic devices within VR are typically classified into three categories: wearable devices, grounded and hand-held devices, and innovative non-contact systems like mid-air haptics, as discussed by Adilkhanov et al. [29]. Wearable devices, including gloves and exoskeletons, offer tactile and force feedback closely integrated with the user’s body [38]. In contrast, grounded and hand-held devices traditionally aim to simulate real-world interactions by offering physical resistance and tactile sensations. Meanwhile, emerging technologies explore non-contact feedback mechanisms, offering a glimpse into a future of immersive interaction without physical constraints [29,39].

The applications of haptic devices are diverse, ranging from entertainment and gaming to more specialized fields, such as medical applications [24,40], rehabilitation [41,42], remote communication [43,44], education [45,46], and psychology [47]. These devices aim to enhance realism and introduce new methods of communication and interaction through sensory feedback within virtual environments.

Wearable haptic feedback devices play a pivotal role in simulating tactile sensations, such as touch, pressure, and vibration, to enhance the realism of virtual environments. Recent research has expanded the scope of these interfaces by utilizing arrays of mechanical and electrical actuators, not only on the fingertips but across larger areas of the body [48,49,50,51,52]. For instance, the fingertip-wearable device can deliver touch, sliding, and vibration feedback through a dense array of actuators aligned with the fingertip’s high sensitivity, thus enhancing immersion in VR environments and teleoperation systems [53,54]. Similarly, a glove equipped with a self-sensing actuator (SenAct) offers accurate force and vibration feedback [55]. Other solutions, such as the vDeltaGlove [56], provide multimodal haptic feedback for teleoperated robotic systems through VR, offering a multimodal user experience and performance in teleoperation tasks compared to traditional haptic devices. Actuated thimbles based on electro-cutaneous feedback prioritize wearability and natural tactile sensation transmission, easily managed via a smartphone app [57]. In terms of vibrotactile feedback devices, those employing linear resonant actuators (LRAs) have been explored for human–robot interaction. They allow for independent control over the location and intensity of virtual stimuli, enhancing the spatial resolution of vibrotactile interfaces while requiring fewer actuators [58]. Furthermore, devices combining fingertip vibrators with an arm mechanism to indicate force direction and strength offer a new approach to haptic presentation in VR, allowing for a smaller device size while effectively conveying directional force [59].

To provide haptic feedback over larger areas of the body with comprehensive and precise haptic experiences, Moriyama et al. [60] explored two-dimensional communication (2DC) technology. This technique synchronizes multiple actuators to deliver coordinated haptic feedback. While many of these technologies focus on replicating real-world interactions to enhance immersion, there is a growing interest in using haptic feedback as a new medium for communication. Moriyama’s project on a High-Density Tactile Vest, which can emulate fingers tapping on the back, exemplifies the innovative use of haptic technology to enhance tactile communication [61,62]. These advancements highlight the potential of high-resolution haptic devices not only in increasing the quality of realistic haptic sensations but also in being used in novel ways for human–computer interaction and sensory experiences [63]. Overall, the field is moving toward more intuitive, immersive, and versatile wearable haptic feedback devices. These advancements include targeting higher resolutions, more cutaneous senses, and applicability to a wider range of body parts in order to enhance VR experiences in applications ranging from entertainment to rehabilitation [64,65].

Nevertheless, several challenges continue to impede their usability. Achieving high realism and fidelity in feedback is crucial, as devices strive to accurately simulate touch and texture [24]. Key engineering challenges include achieving spatial resolution, power-efficient operation, a wide dynamic range, and fast haptic responses. Developing devices that can cover extensive areas of the body without compromising comfort remains a significant obstacle [24]. Wearability and comfort are paramount, necessitating devices that allow prolonged usage without causing fatigue [29]. The high cost and limited accessibility of advanced haptic systems restrict their widespread adoption, leaving high-quality haptic experiences out of reach for many [38]. Furthermore, achieving seamless user interaction and compatibility with existing platforms [37] and software is essential for creating a more user-friendly system [63]. Additionally, the diversity of devices highlights the need for scalability and a customizable development kit in haptic technology [64]. The development kit flexibility can be beneficial during the exploration and development stages to address the varied needs and preferences of users and researchers [23,37,64].

This paper presents the development of a high-resolution, real-time wearable haptic device with a primary focus on non-interactive body regions—areas not typically involved in fine motor tasks and interactions in XR, such as the arms, torso, thighs, and legs. The device addresses the lower tactile sensitivity of these regions compared to more sensitive areas like the fingers and palms by employing suitable high-density, individually addressable actuators to deliver effective haptic feedback. An emphasis on real-time performance guided our approach to minimizing latency, ensuring that the device provides immediate and immersive feedback to users in XR environments.

While the primary focus is on these core functionalities, we also sought to integrate practical factors, such as comfort, modularity, and affordability. The modular design was intended to provide users with the flexibility to reconfigure the system for different body parts, as well as for various actuators, sensors, and layouts, making it adaptable to a wide range of applications.

Affordability and accessibility were also a consideration. By using cost-effective, widely available components, we designed the device to be accessible to a broad audience, including developers and researchers with limited technical expertise in haptic technology. To encourage collaboration and further innovation, this project is open-source and available on GitHub (https://github.com/alinajm7/Haptisense, accessed on 1 July 2024).

2. Materials and Methods

2.1. System Design and Implementation

This section details the architecture of the new proposed haptic device, encompassing both software and hardware development.

2.1.1. Overview of the System

The HRH system functions as a comprehensive solution, comprising both a wearable haptic device and a front-end application, serving as the interface between virtual and real environments. The system processes a variety of tactile sensations, including vibration intensity and frequency, commonly known as “texture” in haptics, to replicate haptic feedback within the virtual environment. The application formulates and generates data, transmitting it to the wearable HRH device. The haptic device then processes these data, and its interface transforms them into tangible tactile sensations that the user can perceive. Figure 1 provides an overview of the system, illustrating how data move from the virtual environment to tactile sensation output, demonstrating the core functionality of the HRH system.

The system required both software and hardware development, each divided into smaller sections. The hardware section includes three distinct components: the controller, driver, and interface, each responsible for specific tasks. The software section comprises HRH application and firmware designs, focusing on the user interface and processing data and actuator behaviors. Figure 2 depicts the structure of the HRH system and its components, categorized into software and hardware development sections. It also highlights the modularity of the hardware parts. This modular structure allows for independent updates and adjustments, which are critical for adapting to evolving technological standards and enhancing the system’s processing power and functionality over time or according to specific requirements, all without necessitating a complete system overhaul. Each hardware module or the software can be modified or replaced separately.

In the following subsections, the implementation of the HRH system, specifically the wearable haptic device and system, is detailed, with particular attention given to the hardware (Section 2.1.2) and software (Section 2.1.3) aspects.

2.1.2. Hardware Design and Configuration

In the hardware design of the HRH system, we employ a modular architecture to ensure adaptability and customization across a variety of applications. This architecture is composed of the following three principal modules:

• Controller Module: Acts as the central processing unit of the device, using a custom-designed microcontroller module based on the ESP32-S3.
• Driver Module: Regulates the power and frequency distribution for 58 ERM actuators, ensuring precise control.
• Interface Module: Holds the actuators in place with a custom-designed flexible PCB (Printed Circuit Board) that integrates fifty-eight vibrotactile actuators.

Each module serves a distinct function while integrating sequentially to form a cohesive system. The interface module connects to the driver module, which in turn connects to the controller module. The controller module communicates with the application on the computer. The electronic design of these modules involved PCB design and choosing electronic components to ensure optimal performance tailored to the project’s requirements. Figure 3 illustrates the hardware components of the HRH system and their corresponding implemented parts, demonstrating their integration within the overall architecture.

The Controller Module

The controller module acts as the central processing unit, managing commands and synchronizing the operation of the driver and, subsequently, the interface module using two ESP32 chips with dual-core processors (Figure 4A). This configuration improved the system’s capability to efficiently manage haptic feedback data and promptly respond to user interactions with minimal latency.

The HRH system employs the ESP32-S3 chip (datasheet available at https://www.espressif.com/sites/default/files/documentation/esp32-s3_datasheet_en.pdf, accessed on 1 July 2024), a powerful component suitable for complex applications requiring wireless functionalities. The selection of this chip was made after thoroughly reviewing available commercial microcontrollers and comparing their characteristics of interest. The ESP32-S3 was chosen for its dual-core processor, which supports efficient multitasking, and for its robust wireless connectivity options, including dual-mode Bluetooth and Wi-Fi, essential for seamless XR experiences. Additionally, its cost efficiency and low-power management capabilities ensure energy efficiency, making it ideal for wearable standalone wireless devices that require prolonged battery life.

The controller module includes manual reset and boot buttons implemented with simple push-button switches. These features facilitate both development and troubleshooting by providing an easy way to reset the system when necessary. Moreover, the design of the module has been optimized for signal integrity, incorporating electrical noise reduction techniques such as proper PCB layout, ground planes, and clear pathways for data transmission. To further ensure clean signal transmission and minimize signal noise, additional hardware techniques were implemented. Proper shielding was applied to data lines to protect against electromagnetic interference (EMI), and grounded cables were used to reduce interference during communication. Capacitors were integrated into the design to filter out electrical noise from the power supply, and components were added to mitigate the effects of back EMF, which can generate voltage spikes when the motors stop. These measures ensure consistent and accurate signal delivery to the actuators, preventing noise from affecting the performance of the haptic feedback system.

The Driver Module

The driver module (Figure 4B) regulates the power and frequency distribution to the actuators. This module is crucial for tailoring the intensity and characteristics of the haptic feedback, allowing the system to deliver a broad range of control for tactile sensations that can mimic real-world interactions or create unique sensory experiences. It handles diverse power supply ranges and adjusts output voltages to meet the specific needs of different actuators and the dual ESP32 microcontrollers. This flexibility enhances the system’s adaptability across various operational environments. It is designed to handle currents of 5 A safely to ensure robust performance for simultaneously driving multiple actuators. The module’s design includes connectors for easy integration with the microcontroller and interface modules, simplifying assembly and ensuring reliable power distribution. Should there be a need to accommodate different types of actuators or integrate sensors into the system, the design of this module can be separately updated or adjusted to meet requirements.

The Interface Module

The interface module in the HRH system is designed to interact with users by incorporating a flexible PCB that positions the actuators and necessary components while providing physical flexibility (Figure 4C). This module can be custom-designed in various shapes to accommodate different actuator layouts, body parts, or specific applications (Figure 5). The current prototype design is rectangular, as shown in the first and second images in Figure 5 from left to right, with an offset grid pattern of actuators covering an area of approximately 15 × 17 cm. The distance between actuators is 25 mm and 20 mm, respectively. These distances were chosen based on the two-point discrimination threshold defined in the literature [26,66] for mechanoreceptors in target body parts that are less sensitive, such as the back, arm, shoulder, breast, thigh, and calf. This ensures that the resolution is appropriate for these areas. As demonstrated in Figure 6, a single shape can be attached to different parts of the body. This configuration simulates the sensation of stroking on the skin, mimicking the tactile drawing of shapes. The 58 vibrators used in the current design are 10 × 3 mm Mini Vibration Motors (DC 3V 12,000 rpm Flat Coin Button-Type). The number of vibrators used in the current setup, which is substantially higher than those in common existing ERM vibrator-based haptic wearable devices, is typically up to 12 and, in some cases, around 33 actuators or fewer (

Table 1. Comparison of various haptic devices.

Device Name	Approx. Responsiveness	Price Range	Number and Type of Actuators	Body Part Focus
* Ultrahaptics (https://www.ultraleap.com/haptics/ (accessed on 26 September 2024))	Hand-tracking 20 to 200 FPS Haptic feedback 40 kHz (https://support.ultraleap.com/hc/en-us/articles/360004368558-What-is-the-resolution-of-Ultrahaptics (accessed on 26 September 2024))	<USD 3000 (https://www.robotshop.com/products/ultrahaptics-stratos-explore-development-kit (accessed on 26 September 2024))	320 ultrasonic emitters	Hand
* Neosensory Buzz (https://neosensory.com/blog/introducing-buzz/ (accessed on 26 September 2024))	End-to-end latency 23 ms	<USD 1000 (https://neosensory.com/wp-content/uploads/2020/10/Neosensory-Buzz-Tech-Sheet.pdf (accessed on 26 September 2024))	4 vibratory motors	Wrist
* SenseGlove Nova (https://www.senseglove.com/ (accessed on 26 September 2024))	Sensor Data 10–29 ms Command 20–22.5 ms (https://senseglove.gitlab.io/SenseGloveDocs/nova-glove.html?highlight=latency (accessed on 26 September 2024))	<USD 6000	Force feedback and few vibrotactile actuators	Hand
* HaptX Gloves (https://haptx.com/ (accessed on 26 September 2024))	Update Rate 120 Hz (16 ms) (https://docs.haptx.com/docs/files/HaptX_Glove_G1_Spec_Sheet_Rev_1.4_Spec.pdf (accessed on 26 September 2024))	<USD 6000 (https://www.techpowerup.com/300272/haptx-introduces-industrys-most-advanced-haptic-gloves-priced-for-scalable-deployment (accessed on 26 September 2024))	Hundreds of microfluidic actuators	Hand
* VibroTac Belt (https://www.sensodrive.de/products/vibrotactile-feedback.php (accessed on 26 September 2024))	N/A	N/A	6 vibrotactile	Wrist
* TactGlove DK2 (https://www.bhaptics.com/tactsuit/tactglove// (accessed on 26 September 2024))	N/A	<USD 500	12 vibrators	Hand
* TactSuit X16–X40 (https://www.bhaptics.com/tactsuit/ (accessed on 26 September 2024))	N/A	<USD 1000	X16: 16 and X40: ERM Vibromotors	Torso
* TESLASUIT (https://teslasuit.io/products/teslasuit-4/ (accessed on 26 September 2024))	Motion capture 200 Hz Command N/A	>USD 10,000	80 electro muscle stimulation (EMS)	Upper and lower body
Wireless Haptic Interface [52]	Total system latency 20 ms	N/A	36 ERM Vibromotors	Large areas of the skin
3-RSR Haptic Wearable Device [41]	N/A	N/A	2 contact points, 6 actuators	Fingertips
Untethered Hand-Wearable Haptic Device [54]	System latency avg. 46.5 ms	N/A	5 contact points, 80 piezo-based pin actuators	Fingertips
HARVEST: High-Density Tactile Vest [62]	System latency N/A Actuator Response 4 ms	N/A	144 voice coil vibrators	Torso (back)
HRH Device	System latency 9–17 ms Actuator Response: 5–50 ms	<USD 500	58 ERM Vibromotors	Large areas of the skin

* Commercially available.

). This gives our proposed device a significant advantage, allowing for a high resolution of haptic inputs. The ERM type of vibrator was chosen due to its powerful sensation, effective mounting technique, and relatively compact and lightweight design [50].

System Integration

The HRH system combines the microcontroller, driver, and interface modules into a compact assembly measuring 25 × 20 cm (Figure 4D), where the essential hardware components are mounted on a flexible PCB. The PCB was designed using the EasyEDA (https://easyeda.com, accessed on 26 September 2024) online PCB design software, an Electronic Design Automation (EDA) tool. Once the design was completed, the manufacturing and assembling of the PCB were outsourced to JLCPCB (https://jlcpcb.com, accessed on 26 September 2024), which specializes in high-quality PCB production. Components for the PCB were sourced from LCSC (https://www.lcsc.com, accessed on 26 September 2024). The system is encased in a cover tailored from light-brown suede, lightweight and soft fabric, and blackout fleece-lined fabric selected for its comfort and skin-friendliness. This choice of material ensures that the device fits comfortably on the body, resembling everyday wearable technology. The lightweight design, with an approximate weight of 350 g (60 g electronics + 290 g case and fabrics), adds minimal burden to the user, making the system practical for XR sessions.

In addition to providing comfort, these materials also help reduce mechanical noise generated by the device during operation. The fabric backing absorbs vibrations, particularly in areas prone to shaking or excessive movement. The vibrators are secured to the fabric using hot glue, which cushions the back of the actuators while keeping the front in contact with the skin. This setup prevents rigid components from colliding and further reduces noise levels. Additionally, software controls can be employed to adjust the frequency and intensity of the actuators, helping to manage mechanical noise. In virtual reality environments, background sounds can also be integrated to mask any remaining noise, ensuring an immersive user experience without distraction.

Furthermore, the HRH system is developed with consideration of relevant EU directives, including the Radio Equipment Directive (RED) [67], the Electromagnetic Compatibility (EMC) [68] Directive, and the General Product Safety Directive (GPSD) [69].

2.1.3. Software Development

The proposed HRH system includes the implementation of corresponding software, enabling (i) the interaction with the device and (ii) the functionality of the device. The complete software system comprises two main components, following good practices of modularized programming:

• HRH Application: The front-end and logic components, developed using Unity (https://unity.com, accessed on 26 September 2024). This user-friendly application allows the user to interact with the virtual environment, triggers haptic sensations, and includes scripts to control actuator behavior and manage data communication with the microcontroller.
• Microcontroller Firmware: The software that runs on the microcontroller, implementing its functionality, managing hardware operations, and processing data received from the HRH application.

Figure 7 illustrates the software architecture of the HRH system, showing the interaction of the HRH application with the microcontroller firmware to control the hardware components.

HRH Application

The HRH application consists of front-end and logic components designed to control the HRH system. Software development focused on creating an intuitive and efficient interface to manage both kinesthetic and cutaneous haptic feedback using the Unity game engine. Unity’s support for VR and AR applications, combined with its programming flexibility, makes it an ideal choice for developing complex haptic feedback mechanisms. While Unity was chosen for its features, the HRH system is compatible with other platforms as well. The firmware is designed to receive a numeric array representing the states of all actuators in a specific format within each communication via serial port, Bluetooth, or Wi-Fi protocols. This design allows the HRH system to be controlled by any platform capable of generating these formatted data, including Unreal Engine, WebXR, and other game engines or software frameworks. Currently, Unity classes manage the logic and feedback for the actuators in XR environments. In the future, similar or new interfaces can be developed for other platforms to ensure the system is broadly applicable to different use cases. This flexibility, combined with the open-source nature of the project, allows developers to integrate the HRH system into various applications across different platforms, making it suitable for a wide range of XR systems. Within the application, 3D objects are defined as either “Actuators” or “Tangible” objects. When a virtual tangible object (VTO) approaches a virtual actuator object (VAO), a distance-based equation calculates the interaction type, and the results are communicated to the microcontroller. Specifically, the actuator value is calculated using the following formula:

$$\mathrm{Actuator}\mathrm{intensity}={\left(\max \left(0,1-{\displaystyle \frac{\mathrm{Closest} \mathrm{distance} \mathrm{between} \mathrm{VTO}\&\mathrm{VAO}}{\mathrm{Scanning} \mathrm{distance}}}\right)\right)}^{\mathrm{Power} \mathrm{factor}}$$

This equation allows the system to determine the intensity of the feedback based on the distance.

The development process involves several key components, including custom libraries to facilitate the layout of actuators, control them, and visualize their behavior for tangible objects in the application, and a device manager to handle communication with the microcontroller. An essential part of a realistic haptic system’s software is accurately positioning each VAO to mirror its real-world position and achieving accurate kinesthetic feedback. To achieve this, the “Actuators Layout Creator” class was developed to automate the positioning of actuators within the application, using parametric control to generate an offset grid pattern that accurately reflects the real-world layout (Figure 8A). Additionally, for more realistic and immersive interactions, the “Actuator Mapper on Surface” class was designed to shape the group of VAOs for curvy surfaces, such as different parts of the body (Figure 8B). This is accomplished using three different Raycast-based techniques. The first, Simple Surface Mapping, uses Raycast to project each actuator toward the surface, mapping it at the straight hit point in the selected direction and aligning it with the surface normal at that point. The second, Reference-Based Surface Mapping, employs a Raycast from a reference actuator to position all actuators, maintaining their initial relative positions based on this reference point. The third technique, Distance-Maintained Surface Mapping, maps actuators while preserving the original distances between them by scanning along the surface (Figure 8C). This ensures that the simulated actuator behavior accurately matches their real positions, thereby increasing kinesthetic feedback accuracy. The “Actuator Manager” class is designed to define and manage the behavior of the actuators as cutaneous feedback. In this application, the proximity of a VTO to a VAO is the primary factor influencing actuator intensity (Figure 8D). The distance can be selected to be either from the center or from the nearest surface of the VTO to the center of the VAOs. Consequently, the shape of the 3D VTO can affect the patterns and intensities of the actuators. As a VTO approaches a VAO, the intensity of the actuator’s feedback increases. For instance, when the distance is zero, the actuator operates at 100% intensity, while at the maximum scanning distance, the intensity is zero.

This class offers configurable parameters, such as scanning distance, which defines the range within which the actuator responds to an object; Power Factor to Distance, which determines the relationship between the distance and the actuator’s intensity; and minimum and maximum threshold values, which set the bounds for actuator intensity. Additionally, the “Actuator Manager” class includes visual representations of the calculated values. It draws lines from the tangible objects to the activated actuators and changes the colors of the actuators and lines based on their calculated intensity. This visualization aids in understanding and debugging the interaction between virtual objects and actuators.

VTOs are defined by having a “Tangible Object” component, a custom class that specifies the characteristics of the object, such as its frequency and whether it uses dynamic frequency. The “Actuator Manager” class utilizes these characteristics to generate different frequencies and sensations for various tangible objects. Additionally, the class includes a mechanism for handling multiple objects, which is useful for more complex scenarios. This mechanism allows for defining several affecting VTOs and their respective weights (Figure 9). For example, it can be configured to react only to the nearest VTO or to multiple VTOs within the scanning distance, assigning different weights based on their proximity. For instance, the nearest object might have a 100% effect, the second-nearest 50%, and the third 20%.

Currently, two mechanisms are designed for actuator behavior based on the proximity of VTOs to VAOs. The first mechanism adjusts the intensity (power) of the actuators in response to the proximity of VTOs, without altering the frequency. The second mechanism modifies the frequency of the actuators according to the distance between the VTOs and VAOs. In conclusion, VAO data are calculated using the mentioned methods and configured settings and transmitted to the microcontroller via the “Communication Manager”, which can utilize serial port, Wi-Fi, or Bluetooth configurations for data transfer to the device.

Microcontroller Firmware

The microcontroller module, which features a dual ESP32-S3-MINI-1U (datasheet available at https://www.espressif.com/sites/default/files/documentation/esp32-s3-mini-1_mini-1u_datasheet_en.pdf, accessed on 26 September 2024) microcontroller (Figure 10), serves as the core processing unit of the HRH system, overseeing command management and coordinating the activities of other modules. This section details the programming aspects and capabilities of the microcontroller, emphasizing its role in ensuring efficient and responsive haptic feedback.

The ESP32-S3-MINI-1U was chosen for its robust dual-core processor, which supports efficient multitasking essential for real-time applications. It offers dual-mode Bluetooth and Wi-Fi connectivity, ensuring flexible and stable wireless communication options important for XR experiences. Additionally, its low-power management capabilities are optimized for wearable devices, enhancing energy efficiency and prolonging battery life. The microcontroller is programmed using the Arduino IDE (https://www.arduino.cc/en/software, accessed on 26 September 2024), which supports the ESP32 platform. This development environment provides a comprehensive set of libraries and tools for configuring the microcontroller, managing communication protocols, and handling actuator control.

A feature of the HRH system is the inclusion of a dual-microcontroller setup, with two ESP32 microcontrollers designated as master and slave. This configuration not only handles the complexity of haptic feedback but also allows control over a total of 58 PWM (Pulse-Width Modulation) pins (29 per microcontroller) with the lowest possible latency. Efficient communication between these microcontrollers is crucial for synchronized operations. So, a custom 4-bit communication protocol is implemented for efficient data exchange between the master and slave microcontrollers. The microcontrollers are programmed to generate PWM signals using the CPU to control the actuators, with adjustable durations ranging from 1 to 99,999 microseconds, allowing precise control over the frequency and intensity of the actuators. Therefore, the microcontroller can reach a nominal frequency of 500 kHz (minimum cycle is 2 microseconds) for one pin. A custom PWM simulator was developed due to the limitation of the microcontroller’s PWM independent channels: there are usually 16 channels, and this specific ESP32-S3 is limited to 8 channels, meaning that only 8 pins could generate independent PWM signals, while the HRH requires 29 pins of each microcontroller to function as PWM (Figure 10).

The firmware uses a special operating system to manage tasks efficiently, making sure that important tasks, like controlling the PWM signals and processing data, run smoothly on separate processing cores of the microcontroller.

3. Results

This section evaluates the HRH system by comparing its key factors with various existing haptic devices and the diversity of vibrotactile actuators.

3.1. Comparison with Existing Haptic Devices

Various existing haptic devices, including both commercially available products and research prototypes, are compared in Table 1. This comparison focuses on key parameters, such as responsiveness (latency/frequency), price, the number and type of actuators, focused body part, and commercial availability. It is important to note that this is not an exhaustive list, as the diversity of haptic devices is vast. We have selected devices that share certain characteristics with the HRH system, either in terms of the number of actuators, type of feedback provided, or overall functionality, to provide a relevant comparison.

Latency is a critical factor in the performance of haptic devices, particularly in real-time XR applications. The overall latency in the HRH device is determined by several components. The microcontroller update cycle, which involves updating data for all 58 PWM pins, forms the baseline latency. In the current setup, data are transmitted via serial ports at a baud rate of 250 kbps to two ESP32-S3 processors. The most significant delay occurs during data transition between these processors. By utilizing a custom 4-bit communication protocol across six pins, the cycle time for the microcontroller is estimated to be around 4 milliseconds. The driver module, which employs MOSFET components, contributes minimally to latency, with a worst-case switching time of just 5 microseconds. The interface module, which uses Mini Vibration Motors, has an initial reaction time of under 5 milliseconds, though latency can vary between 5 and 50 milliseconds, depending on the target speed and the magnitude of the change. These elements combined result in a nominal average latency of less than 20 milliseconds [70,71] for the HRH device, aligning with human perception thresholds essential for immersive VR experiences.

The HRH system’s standout features include its high-density actuators, low latency, and modular design, which collectively offer enhanced tactile resolution, superior comfort, and adaptability to a wide range of applications. These characteristics position the HRH system as an innovative and versatile solution for immersive XR experiences.

3.2. Vibrotactile Actuator Diversity

One of the features of the proposed HRH system that advances it beyond existing solutions is its ability to support a diverse range of vibrotactile actuators. This flexibility allows the system to be customized to various applications and user needs. By accommodating different types of actuators, including ERM, LRA, and voice coil, the HRH system can deliver different types of haptic feedback. This section highlights the compatibility and benefits of using different actuators with the HRH system, demonstrating its adaptability and potential for various use cases.

Among these, ERM vibrotactile actuators are favored in haptic technology for their compact size, ease of control, cost-effectiveness, and immediate feedback [24]. Despite these advantages, challenges in achieving precise localization for immersive XR experiences persist. Effective haptic feedback actuators must meet specific requirements to simulate realistic tactile sensations (

Table 2. Requirements of actuators for haptic feedback devices.

Parameter	Vibrotactile
Frequency (Hz)	5–1000
Max Sensitivity (Hz)	250
Force (mN)	>10–100 (Normal direction)
Displacement (µm)	>10–100 (static, normal direction)
	>0.85 (dynamic, normal direction)

Normal direction: Force or displacement applied perpendicular to the surface of the skin. Static, out-of-plane (normal direction): Measurements taken when the skin is not moving. Dynamic, out-of-plane (normal direction): Measurements taken when the skin is moving.

). The human tactile sensing range is most sensitive at frequencies between 50 and 400 Hz [72], peaking at around 250 Hz [73]. Typical perception thresholds are 10–100 mN for force and 10–100 µm for displacement in static conditions, with even smaller displacements ( 0.85 µm) perceivable under dynamic conditions [74,75]. Actuators operating below 1 kHz are ideal for meeting these sensitivity ranges, ensuring precise and nuanced feedback [73].

Vibrotactile actuators play a crucial role in haptic technology, offering various mechanisms to simulate tactile sensations. Different types of actuators, such as electromagnetic (ERM, LRA, and voice coil), electrostatic, and piezoelectric/electrostrictive (Ceramic and Polymer) actuators, have distinct characteristics that make them suitable for specific applications. To better understand the capabilities and specifications of these actuators, we have compiled a comparison table,

Table 3. Comparison of vibrotactile actuator specifications.

		Electromagnetic *		Electrostatic *	Piezoelectric/ Electrostrictive	Polymer
	ERM *	LRA *	Voice Coil *		Ceramic
Driving field	Magnetic	Magnetic	Magnetic	Electric	Electric	Electric
Driven voltage (V)	3–5	1–3	5–12	10	10–120	10–100
Driven current (A)	0.05–0.5	0.05–0.2	0.5–3	1 µA	1–100 µA	1–100 µA
Energy consumption	High	Medium	High	Low	Low	Low
Vibration frequency (Hz)	90–200	150–300	50–300	100–2000	5–25,000	0.1–500
Haptic types	Vibrotactile	Vibrotactile	Vibrotactile	Vibrotactile	Vibrotactile/ Ultrasonic	Vibrotactile
Responding time (ms)	20–100	20–30	5	5	1–4	10
Displacement (µm)	>1000	>1000	>1000	10–40	0.5–100	75–200
Physical Flexibility (Potential)	No	No	No	Yes	Limited	Yes
Suitable application	Vibration notification	Vibration notification	Haptic wearable MEMS	Haptic wearable MEMS	High-definition haptic sensation for touching screen	Medical wearable device
Commercialized	Yes	Yes	Yes	No	No	No

* Compatible actuators with the current HRH system.

, which outlines their key parameters, including driven voltage, current, energy consumption, vibration frequency, response time, displacement, and suitable applications. The table is an expanded version based on the table provided in the review by Chen et al. [73].

Recent advancements in haptic technology have focused on improving the performance and integration of these actuators into various devices. Techniques like inverse filter correction [65] enhance localization by precisely focusing sensations. Innovations have led to the development of air-permeable actuators embedded in fabrics [76] and lightweight dielectric elastomer actuators [77], which do not compromise comfort or functionality. Additionally, ERM actuators have seen improvements through novel closed-loop systems [65,78], further enhancing responsiveness and feedback control. Research into spatially dynamic vibrotactile stimuli and soft electromagnetic actuators is ongoing to provide innovative solutions for enhanced user engagement in VR [79,80].

Based on this comparison, Eccentric Rotating Mass (ERM) actuators offer a balanced choice due to their straightforward design, effective tactile feedback, high displacement, cost-effectiveness, and moderate power consumption. While they have some limitations, such as responsiveness and the challenge of decoupling amplitude from frequency, these are mitigated by recent advancements in closed-loop systems, which have significantly enhanced their performance. These characteristics make ERM actuators well suited for our project’s goal of providing immediate and reliable haptic feedback for immersive XR experiences.

4. Discussion

In this paper we present the development of a new high-resolution haptic (HRH) system with the aim to potentially enhance XR experiences. The system’s modular design offers notable benefits, including flexibility and adaptability to a wide range of applications. This design ensures that components such as the microcontroller, driver, and interface modules are independently updated or modified to meet specific requirements without necessitating a complete system overhaul. This modularity not only simplifies maintenance but also extends the device’s applicability across various use cases and user needs.

With respect to system performance and modularity, the integration of dual ESP32-S3 microcontrollers provides extensive processing power and redundancy, which is important for handling complex haptic feedback and ensuring reliable performance. This setup allows for efficient multitasking, thereby enhancing the system’s responsiveness and reducing latency. Direct control over PWM signals without additional driver ICs simplifies the hardware design, offers more control, and reduces potential failure points, ensuring more consistent performance. The interface module, implemented on a flexible PCB, is another aspect of the HRH design. This flexibility is particularly beneficial for wearable applications, enabling the device to conform comfortably to various body parts. For instance, the interface can be designed for gloves, helmets, shoes, or any other wearable configuration.

The ability to reposition and secure different components and actuators easily makes the system highly adaptable, allowing for the modification of the actuator placements with minimal changes to the interface module. The current HRH design utilizes 58 mini-DC vibration motors of the Eccentric Rotating Mass (ERM) category. In addition to ERM motors, the system supports a variety of actuators, including linear resonant actuators (LRAs), voice coil actuators, piezoelectric actuators, miniature solenoids, and various small DC motors. The system can accommodate actuators operating within a voltage range of 1.25 V to 10.5 V, with a total current capacity of up to 5 A. This broad compatibility with different actuator types allows the HRH system to be adaptable to a wide range of applications, enhancing its versatility and flexibility. While the current design primarily uses vibrotactile feedback from ERM motors, the modular and flexible nature of the HRH system allows for the integration of additional haptic feedback modalities, such as pressure-based feedback, temperature variation, and more. This adaptability enables the creation of richer, more immersive sensory experiences in XR environments, depending on the specific scenario. Future work can explore the potential of these additional modalities to further enhance the system’s tactile feedback capabilities, offering a more comprehensive range of haptic interactions and evaluating their effects across different use cases and applications.

In the HRH system, multiple microcontrollers are utilized in the hardware design to derive the number of actuators needed for analog control. This setup allows for precise management of numerous devices requiring individual adjustments and quick response times, such as those found in haptic feedback systems. While other solutions, such as Field-Programmable Gate Arrays (FPGAs) or Complex Programmable Logic Devices (CPLDs), can achieve very low latency, the design and programming complexity, along with the overall cost of the system, would add significant challenges compared to the current solution [81,82,83]. Although Digital-to-Analog Converters (DACs) could be used alternatively, as they offer precise control over output voltage, which is essential for fine-tuning actuator signals, DACs typically introduce more latency than microcontrollers. This is due to the time required for converting digital signals into analog outputs. Despite being relatively fast, this conversion process can delay signal transmission compared to direct digital control methods like Pulse-Width Modulation (PWM). Furthermore, the complexity and cost of a system using DACs would be higher compared to our solution [84].

Another option could be the use of multiple PWM driver ICs, such as the PCA9685 (https://cdn-shop.adafruit.com/datasheets/PCA9685.pdf, accessed on 26 September 2024). This IC is typically used for its capability to control up to 16 separate PWM outputs, making it popular in applications requiring independent control of multiple components, such as in advanced lighting systems or robotic servos. It helps to offload the PWM management from the primary microcontroller, simplifying the system design by centralizing the control of multiple outputs. Unlike systems utilizing a single microcontroller with several PCA9685 extenders, the HRH system employs a dual ESP32-S3 microcontroller setup. Each ESP32-S3 features a dual-core processor, significantly enhancing processing capabilities. This configuration not only allows the simultaneous handling of more data-intensive tasks but also provides redundancy, thereby improving the system’s reliability. Although the ESP32-S3 has built-in PWM support, it can only control eight GPIO pins simultaneously. To meet the requirement of controlling 29 pins with a minimal refresh rate (the interval required to update all pins), the HRH system leverages the CPU power of the ESP32-S3 to generate PWM signals. One core of the ESP32-S3 is dedicated to PWM signal generation, while the other core is used for other calculations and processing tasks. This strategy eliminates the need for additional PWM driver ICs, reducing potential failure points and lowering latency, as signals are processed and output directly from the ESP32-S3. This results in finer control over actuators and quicker responses to sensor inputs. With a reduced number of components, both development and hardware troubleshooting become more straightforward. The direct engagement of the ESP32-S3 core in generating PWM signals and processing data, coupled with onboard Wi-Fi and Bluetooth modules, simplifies the hardware layout and software development, thereby facilitating modifications and maintenance. Finally, the ESP32-S3’s advanced power management functionality surpasses configurations that combine basic microcontrollers with multiple PCA9685s. This aspect proves particularly advantageous in wearable XR applications, where enhanced power efficiency directly translates to extended battery life and heightened user comfort.

Minimizing user fatigue is essential for the long-term usability of haptic devices, particularly in extended reality (XR) applications, where users are immersed for prolonged periods. Our development of lightweight and flexible haptic devices aims to mitigate fatigue by ensuring high operational efficiency and stability, thus enhancing comfort during extended use [77]. Additionally, the cognitive load imposed by haptic devices, especially in educational settings, can significantly affect user fatigue. Research indicates that haptic feedback, while beneficial for reducing cognitive load and enhancing learning, requires careful consideration of the instructional design and haptic interface sensitivity to optimize user comfort [45]. Furthermore, inadequate or unrealistic haptic feedback can disrupt the immersive experience, potentially increasing cognitive strain as users attempt to integrate visual and tactile stimuli [30]. In our experimental protocol, we have incorporated rest periods after several trials to address fatigue effectively.

Considering the software design of the HRH system, it boasts the benefit of facilitating the precise control of haptic feedback in virtual settings. Core tools like the Actuators Layout Creator, Actuator Mapper on Surface, and Actuator Manager offer versatility and customization. These tools enhance realism by allowing each component to be positioned according to their real-world positions and offering detailed control over vibration behaviors. This makes the system suitable for a wide range of XR applications, including sectors such as gaming, medical training, rehabilitation, remote communication, and education. For instance, in medical training, the precise position and varied haptic feedback can replicate various textures and resistances, providing a more realistic training environment [40]. In surgical training, the system can simulate the sensation of touch, enhancing the realism and effectiveness of the training process [34,85]. In rehabilitation, this system’s customizable haptic feedback patterns can aid therapy and recovery processes by targeting specific areas of the body. This feature is particularly beneficial in physiotherapy, where vibration patterns can assist in muscle stimulation and relaxation, improving outcomes [40,86].

The HRH device does not currently have built-in physical sensors to detect real-world touch directly on the device. However, its modular design allows for easy adaptation and the integration of additional sensors if needed. By modifying the driver board and interface module, we can enable the device to sense touch inputs from the real world, enhancing its interactivity and usability in various XR applications. The flexibility of the design also makes it possible to develop hybrid devices that offer both sensing and feedback capabilities, which is an important feature for interactive scenarios requiring real-time sensory input. This adaptability allows the HRH system to be used in a wide range of applications, from simple haptic feedback systems to more complex, interactive environments where real-world touch inputs are needed. Future work could focus on integrating such sensors to further expand the functionality and interactivity of the device.

Additionally, the system offers the opportunity to explore further haptic communication, providing a novel non-verbal communication channel through tactile feedback [30,47,87]. This can prove beneficial in remote counseling or telemedicine, enhancing communication experiences [32,88]. In education, the HRH system has the potential to enhance learning by incorporating tactile feedback into virtual labs or interactive simulations. This can allow students to experience the textures of different materials or machinery vibrations in a safe, controlled virtual environment, thereby enriching their understanding and engagement [27,45,64,85]. Moreover, in gaming and entertainment, precise haptic feedback synchronized with in-game actions can increase player engagement and realism, offering a more immersive gaming experience [59,86]. Furthermore, the system can be employed in assistive technology to help individuals with sensory impairments by offering tactile feedback to assist with navigation and interaction in both virtual and real-world environments, thereby promoting accessibility and inclusivity [35,52].

Future work on the system’s technical configuration will involve running detailed experiments to validate the device’s effectiveness and further refine the concept. This includes creating distinguishable haptic patterns and characteristics to form a comprehensive haptic language. The goal is to establish a nuanced and multifaceted haptic communication system for various applications, enhancing perception in the real world and improving the immersive experience in XR. The designed experiments will focus on the device’s functionality, including actuator responsiveness, control signal accuracy, and overall system stability. These tests are crucial for verifying the device’s technical performance and ensuring reliability. Additionally, the experiment will gather initial user feedback on the haptic feedback patterns and distinguishable characteristics such as intensity, speed, and frequency, following a methodology similar to that of Abad et al. [54]. Moreover, future work will also evaluate the HRH system’s performance in highly complex and demanding XR environments. These evaluations will focus on the system’s ability to handle intensive processing loads and deliver real-time haptic feedback in scenarios that require rapid and responsive interactions. Key performance metrics such as scalability, system latency, and stability in challenging virtual environments will be essential for ensuring the robustness of the HRH system across various XR applications. The objective is to assess the discernibility and meaningfulness of the haptic characteristics for advanced haptic communication and to evaluate the device’s potential to enhance XR perception through tactile feedback.

5. Conclusions

The HRH system represents an advancement in the realm of haptic feedback devices, particularly tailored for enhancing extended reality (XR) experiences. The introduced system, characterized by its modular design and high-density actuators, provides a versatile and responsive solution for delivering nuanced tactile feedback. The dual ESP32-S3 microcontroller setup ensures robust processing power and low-latency performance, crucial for real-time haptic interactions. The design emphasizes wearability and adaptability, featuring a flexible PCB that can conform to various body parts, thereby enhancing user comfort and immersion. The integration with the Unity game engine facilitates seamless software management of haptic feedback, offering precise control and a realistic simulation of tactile sensations. The system’s ability to support a range of actuators further broadens its scope, making it suitable for a variety of applications, including VR gaming, medical training, rehabilitation, remote communication, and educational tools. Future work will focus on refining the haptic patterns and establishing a comprehensive haptic communication language, aimed at enhancing both virtual and real-world interactions. In conclusion, the HRH system not only addresses existing challenges in haptic technology but also opens new avenues for immersive and interactive XR experiences, marking a step forward in the field of human–computer interaction. The open-source nature of this project encourages further innovation and development opportunities for researchers.