Wearable Haptic Music Player with Multi-Feature Extraction Using Spectral Flux and Yin Algorithms

Abstract

Vibrotactile feedback synchronized with audio through haptic music players (HMPs) creates a synergistic effect that has been shown to improve the music listening experience. However, current HMPs are still unable to efficiently retrieve multiple music features, decelerating app scalability and jeopardizing long-term user engagement. This study introduces a wearable HMP that utilizes piezoelectric actuators and a novel audio-tactile rendering algorithm that uses YIN to extract pitch and spectral flux for rhythm. Building upon prior work, the system additionally features a modified discretization step and software optimization to improve multi-feature extraction and tactile display of music. The pitch, melody/timbre, and rhythm displays, respectively, were validated using Mean Average Error (MAE), Dynamic Time Warping (DTW) distance, and accuracy, yielding normalized averages of MAE = 0.1020 and DTW = 0.1518, and a rhythmic pattern accuracy of 97.56%. The Yin algorithm was shown to greatly improve the tactile display of vocals, with slight improvements for bass and accompaniments, while spectral flux and software optimizations significantly improved rhythm display. The wearable HMP effectively communicates multiple music features without the pitfalls of prior approaches. Future research can improve the system’s audio-tactile signal fidelity and explore the qualitative merits of multi-feature extraction in HMPs.

1. Introduction

1.1. Background

The synergy of sound with its mechanical vibrations in the music playing and listening experience, such as from instruments and speakers, suggests that the sense of touch could be provided with an even greater appreciation for music [1,2]. For this purpose, numerous haptic music players (HMPs) that integrate vibrotactile feedback with audio stimuli have been developed [3]. The form factor of HMPs in prior research ranges from chairs [4,5,6,7,8,9,10], belts [10,11,12], bracelets [6,13,14], jackets [15,16], and most recently due to the high concentration of tactile receptors in the hands, gloves [17,18,19,20,21]. Regardless of form, multiple user studies have shown that vibration signals with similar dynamics as the original audio provide more enjoyment to the music listening experience [4,18,19,21,22].

However, the issue that heavily limits the development of high-quality HMPs is their inability to communicate multiple music features, such as pitch, rhythm, melody, and timbre, in an efficient manner. Despite an overall positive feedback, users may still feel “bored” when there is too much focus on only one or a few music features [23], jeopardizing long-term use beyond the scope of prior user studies. One of the root causes is the kind of actuators used, as an overwhelming majority of HMPs use either voice coil actuators (VCA) [4,17,18,19] or coin motors [20,21], and oftentimes transduce the audio signals without further signal processing to preserve its vibrations. Even if VCAs have high frequency ranges, the human skin can perceive only up to 1 kHz, which is minuscule compared to the 20 kHz auditory perception limit [5,17,20]. Because of the perception limit of the human skin, designers of HMPs that use VCAs need to set the upper limit of their range of frequencies to 1 kHz [24], diminishing the communicated pitch. On the other hand, the vibration frequencies of coin motors cannot be controlled for the purposes of HMPs, so they are often used to solely simulate tempo instead [20,21]. While attempts have been made to communicate a higher frequency range using these actuators, such as assigning different actuators to frequency subsets, the discrete rather than the upwards or downwards movements created by these setups displayed rhythm instead of pitch or melody, as the exact frequency emitted by each VCA cannot be determined by users [4,5,25]. These biological constraints indicate that methods beyond transducing the signal and actuators other than VCAs or coin motors need to be explored if multiple music features are to be presented in future HMPs.

Another cause is the excessive complexity involved in extracting basic music features, particularly for pitch and rhythm. Aside from transducing audio signals, a dominant method in extracting pitch is the creation and/or use of MIDI data [1,5,6,24,25,26], which contains note information that correspond to various levels of pitch. Unfortunately, integrating MIDI data to the HMP development process requires the manual use of external software such as a Digital Audio Workstation [25,26], decelerating implementation and disallowing scalable, mobile HMP applications. For instance, an HMP study even created a MIDI player from scratch to display pitch [1], but this method exacerbates the difficulty of modifying such a system to also extract other music features. Additionally, working with MIDI data requires either access to the music’s MIDI file [24] or the capacity to accurately convert audio to MIDI, which is tedious and extremely difficult with our technology at present. The field has yet to provide an algorithm that, without merely transducing audio signals or using MIDI files, could take only an audio file as input and convert it to tactile signals containing multiple music features for HMP applications.

Moreover, the beat extraction in prior studies is mostly unspecified and unverified, with an example taking the beats per minute (bpm) of musical excerpts and creating 102 ms beats that match with them [20]. Another practice would be to use a low-frequency bass band and define beats as energy peaks in the resulting signal [23,27,28], with the assumption that the sound already contains enough haptic information on its own as is the principle behind transducing the signal [27]. However, since the bass band reaches up to only 200 Hz [23], this method cannot extract beats from higher frequency ranges where drums are usually situated. As volume will inevitably be a factor in an energy-based approach [23], current methods are also susceptible to erroneously considering fluctuations in volume and noise as beats, while being unable to detect beats that simultaneously occur at a softer volume. Thus, it becomes necessary to create an HMP application that utilizes a frequency-based beat extraction approach that is robust to noise and polyphony, which are common in modern music. Furthermore, only a handful of studies extracted and displayed timbre from music [5,26], underscoring the overall limitations of current HMPs in multi-feature extraction (

Table 1. Overview of research on related HMPs.

HMP	Actuators	Form	Methods	Features
Model Human Cochlea [5]	Speaker	Belt	MIDI 3, TM 4, FM 5	Pitch, Timbre
Audio-Tactile Glove [17]	Voice coil	Glove	Transduction	Pitch
Dual-Band HMP [23]	DMA 1	Handheld	Energy-based	Rhythm
Mood Glove [18]	Voice coil	Glove	Transduction	Rhythm
HMP (Spidar-G2) [1]	DC Motors	Installation	MIDI, ADSR 6 filter	Pitch
Musical Haptic Sleeves [25]	Speaker	Sleeve	MIDI, transduction	Pitch
SENSE [19]	Voice coil	Glove	Transduction	Rhythm
Vibrotactile Captioning [20]	ERM 2	Glove	BPM Detection	Tempo
Hap-phones [24]	Voice coil	Actuators	MIDI, Tactile Illusions	Rhythm, Tempo, Melody
Haptic gloves to elicit emotions [21]	ERM	Glove	BPM Detection	Tempo
Tactile Melodies [26]	ERM	Installation	MIDI, Tactile Illusions	Pitch, Melody, Rhythm, Timbre
Music Tactalizer (ours) [29]	Piezoelectric	Glove	Spectral Flux, YIN	Pitch, Melody, Rhythm, Timbre

¹ Dual-Mode Actuator; ² Eccentric Rotating Mass; ³ Musical Instrument Digital Interface; ⁴ Track Model; ⁵ Frequency Model; ⁶ Attack, Decay, Sustain, and Release.

).

1.2. Research Question

In this study, we introduce an HMP wearable device (HMP-WD) that uses piezoelectric actuators, equipped with an algorithm capable of extracting pitch, melody, rhythm, and timbre from an audio file and embedding them in tactile signals for mobile applications. We build upon our previous work [29] by enhancing its multi-feature extraction in four stages. First, we now utilize the YIN algorithm [30], a time-domain autocorrelation-based algorithm used for monophonic pitch tracking, making it ideal for detecting pitch in isolated vocal and instrument tracks. YIN is efficient, highly accurate, and robust to noise, making it a suitable alternative to MIDI in HMP applications. Our prior method used spectral peaks for pitch estimation; however, the results demonstrated that the tactile signals had a slight difficulty replicating the coarse pitch contours created from the original audio. Second, we now use spectral flux for beat estimation as opposed to our previous energy-based approach. Unlike energy-based methods, spectral flux detects subtle spectral changes independent of volume while emphasizing transients and sharp attacks. While the coarseness of its resulting signal makes it difficult for tactile pitch display, this quality may make it subjectively better for determining beats in polyphonic audio. Third, we discretize the pitch to eight levels rather than four, hypothesizing that this will enable the tactile signals to replicate pitch contours more accurately. Fourth, we implement software optimization on the mobile application to reduce latencies in synchronizing the visual, audio, and haptic feedback. In addition to evaluating the HMP-WD’s multi-feature display, this study will also compare both approaches for future HMP applications.

The study proceeds as follows: in Section 2, we further discuss the system components and the algorithms employed for audio-tactile rendering. In Section 3, we present the signal fidelity testing, which evaluates the HMP-WD’s ability to communicate music features from a wide frequency range while preserving the music’s original melodic contours and rhythmic patterns. In Section 4, we analyze the results and their implications on the efficacy of the methods for HMP-WDs. This work may provide the solution for current limitations on multi-feature extraction and display for future HMP applications.

2. Materials and Methods

The HMP-WD is composed of (1) the haptic glove, (2) the mobile app, and (3) the rendering system. Given an audio file, the audio-tactile rendering system extracts the music features pitch, melody, rhythm, and timbre using source separation, spectral flux, YIN, discretization, and feature-pin mapping, embedding the extracted features into tactile patterns. The tactile patterns are stored in the mobile app, and when a song is played, the app synchronizes the visual and audio feedback with the tactile feedback as it sends the patterns to the haptic glove. The haptic glove processes the patterns and activates the corresponding pins. Figure 1 illustrates the system architecture of the HMP-WD.

The haptic glove consists of P20 braille cells (metec Ingenieur GmbH, Stuttgart, Germany) connected to a custom-designed printed circuit board (PCB), with both housed in 3D-printed enclosures and attached to off-the-shelf gloves. The PCB features an STM32F10 microcontroller and a JDY33 Bluetooth module, which enables it to receive tactile patterns from the mobile app through Bluetooth Low Energy (BLE) communication. The system is powered by a 5 V lithium battery with a 2 A current rating and supports automatic recharging. A DC-DC boost converter supplies 185 V for the actuators and at least 3.3 V for control, managed by an LM3488 controller. The gloves’ components observe a balance of portability, efficiency, and reliability required for HMP-WD applications.

Each finger is equipped with two piezoelectric actuators, with each containing eight pins that can deliver a minimum tactile force of 17 cN and a fast 50 ms dot rising time, which are well within the human tactile sensitivity range (≥10 cN, 1–100 ms), providing high-resolution vibrotactile feedback that can communicate rapid changes in music features. The HMP-WD takes advantage of the piezoelectric actuator’s unique ability for fine localization, which is also made possible by their position at the user’s fingertips, where 44–60% of the Pacinian corpuscles of the hand are situated [31]. While seemingly no HMP has utilized piezoelectric actuators for audio-tactile feedback, the notion has been expressed before as these actuators were known to have extremely fast response times and wide frequency ranges that would fit the requirements for the tactile display of music [32]. Recently, the same configuration of actuators has been used to provide localized cutaneous force feedback in robotics assisted surgery systems [33] and virtual reality immersion [34,35], highlighting its reliability for HMP applications. Figure 2 presents the components and control pipeline of the haptic gloves used by the HMP-WD.

The mobile app, developed using Flutter for cross-platform compatibility, delivers synchronized visual, audio, and vibrotactile stimuli through real-time, BLE communication with the haptic gloves while the music plays. The feature-pin mapping assigns instruments to different fingers to communicate timbre, centered around the percussive elements being assigned to the index and middle fingers as these are commonly used when moving to the rhythm in finger-tapping studies [36]. Among the remaining melodic elements, the bass is assigned to the thumb as it has the lowest frequency band [23], and the vocals and accompaniment are sent to the ring and pinky fingers, respectively. The percussive elements are ordered from the lowest frequency band to the highest in a down-up, left-right manner, as kick and snare are assigned to the index finger while toms and cymbals belong to the middle finger. A similar behavior can be observed in the melodic elements’ pitch display, as a pitch movement from 0 to 4 reflect an upwards movement from the pins, while movement from 0 to 8 would show an upwards, rightwards, and upwards movement to communicate melody. The rightwards movement to signify an increase in pitch mirrors the behavior in HMPs that utilize tactile illusions [24,26], as lower-pitched notes are assigned to the left side while higher pitched notes are given to the right. An advantage of using piezoelectric actuators for this application, however, is the ability to further localize pitch information in different levels on solely the fingertips and without using tactile illusions, providing the space necessary to assign instruments to their own fingers. Figure 3 shows the mobile application with the assignment of music elements to their corresponding pins.

The rendering system consists of Python software (version 3.11.5) that takes an audio file path, extracts music features, and embeds them in tactile signals. The algorithm sequentially performs (1) source separation, (2) pitch and onset detection, and (3) discretization and feature-pin mapping.

Source separation isolates different instruments from the music into their own audio files. To achieve this, we employed a pre-trained hybrid separation model [37] that separates the audio into four stems: drums, bass, vocals, and other accompaniments, with the latter three comprising the melodic tracks. The model uses a Transformer architecture and leverages both spectrogram-based and waveform-based techniques to perform high-accuracy source separation. Since drums are unpitched percussive sounds, the drum stem is further split into four frequency subsets, which will later be used to detect beat onsets. The subdivisions were heuristically selected based on where a distinct drum sound is best heard, which are 30–150 Hz for kicks, 150–800 Hz for snares, 800–5000 Hz for toms, and 5000–12,000 Hz for cymbals. These were then separated from the drum stem using a Butterworth bandpass filter, resulting in four percussive tracks.

Pitch and onset detection performs feature extraction on each of the separated audio tracks, preventing features of one track from being overlapped by another. The process begins by removing noise from the audio at the signal level using two filters. A Butterworth high-pass filter removes low-frequency noise, including microphone rumbles and external movements. Afterward, a median filter replaces each sample with the median of its window to remove impulse noise such as clicks and pops. Denoising is an essential preprocessing step to make voiced signal parts and real beats more salient for detection, so it is first applied to both melodic and percussive tracks.

To extract pitch information, autocorrelation measures the similarity between the time-lagged version and the original signal, which will periodically repeat at certain lag values. The best match after zero lag occurs at the fundamental period, revealing the signal’s pitch. The autocorrelation function can be defined as

$$R\left(\tau \right)={\displaystyle {\sum }_n}x\left[n\right]·x[n+\tau ],$$

(1)

where x[n] is the signal sample at time n, $\tau$ is the time lag, and R($\tau$) is the similarity between signal and itself shifted by the time lag. While autocorrelation has shown robustness to amplitude changes without the need for frequency analysis, it tends to perform poorly in the presence of harmonics, which occur simultaneously with the pitch. The YIN algorithm improves autocorrelation by using the difference and cumulative mean normalized difference functions to suppress harmonic peaks and amplify the true pitch [30]. While our previous method utilized Short-Time Fourier Transform and Quadratically Interpolated Fast Fourier Transform for spectral peak estimation [38], these are just as if not more susceptible to noise and harmonics that disrupt accurate pitch detection, suggesting that YIN may be a more fitting approach for HMP applications.

To detect beat onsets, spectral flux computes how much the frequency-domain representation of the audio changes at each frame. Sharp increases in the spectrum indicate potential points where a percussive instrument was used, signifying a beat onset. Changes in the spectrum are quantified by the onset strength envelope which is given by

$$o(t)={\displaystyle {\sum }_f}max(0,\left|S(f,t)\right|-\left|S(f,t-1)\right|),$$

(2)

where o(t) is the onset strength at time frame t, f is the frequency bin index, S is the spectrum, and a max function implements half-wave rectification. From the envelope, local thresholds are determined to pick peaks that represent beat onsets [39]. While spectral flux has been used for onset detection in pitched instruments [40,41], it has also shown reliability in beat onset detection for rhythm-based applications [42,43]. Energy peaks as used in our previous method is still a simple approach that is effective when beats always occur with loudness spikes, but spectral flux looks beyond loudness and into the frequencies while being efficient enough for use in HMPs.

To remove unreliable pitch and beat detections as well as noise, the Root Mean Square (RMS) energy is calculated. The RMS energy of an audio signal is defined as

$$RMS=\sqrt{{\displaystyle \frac{1}{N}}{\displaystyle {\sum }_{n=1}^N}{\displaystyle x_1}{\left[n\right]}^2},$$

(3)

where x[n] is the signal frame and N is the number of samples in the frame. The RMS energy is then expressed in decibels and paired with a predefined threshold, which is 35 dB in this study, to remove frames where the estimated pitch or beat onsets are too quiet and are likely false detections and noise. As the pitch values generally follow non-normal distributions that exhibit skewness and platykurtosis, Tukey’s method is used to detect and remove outliers. The technique uses a standard interquartile range with a threshold of 1.5 to clean the pitch contours.

Discretization involves transforming the continuous pitch contours into eight discrete levels based on their percentile relative to the maximum pitch value in the audio track. Since beat onsets are already in binary form, they are not discretized further. To ensure uniform spacing for BLE communication, the resulting discretized signals are aligned to 100 ms bins and empty values are set to zero. The signals finally undergo feature-pin mapping to convert them into tactile signals.

The third step involves converting the continuous pitch contours to eight discrete values based on the quartile they belong to relative to the maximum pitch value in the audio file. Time points with no pitch values are set to zero. Both the discretized pitch contours and the beat onsets are assigned to 100 ms time bins so that the tactile patterns are uniformly spaced. The onsets and discretized pitch values are formatted in a dictionary which are then converted to strings. The tactile patterns generated in this final step correspond to the feature-pin mappings, which are later stored in the mobile app and sent to the haptic gloves. Figure 4 illustrates the proposed audio-tactile rendering algorithm.

3. Results

3.1. Signal Fidelity Testing

As was achieved in our previous work [29], the HMP-WD should be validated in terms of its ability to display the extracted music features while maintaining the original audio’s melodic contour and rhythmic patterns. Tactile displays of pitch, melody/timbre, and rhythm, respectively, are evaluated using mean absolute error (MAE), dynamic time warping (DTW) distance, and accuracy with respect to the cleaned audio signals. In addition to testing the current methods and device, the results of both studies will be compared. In particular, our previous work used spectral peaks for pitch estimation, energy peaks for beat detection, and four levels in pitch discretization. Additionally, our previous work had no software optimization. After the creation of our previous work, we have since split the in-app timer that synchronized the visual, audio, and vibrotactile stimuli into two, which resulted in parallel processing and reduced latencies.

In this experiment, the mobile app recorded the tactile signals transmitted to the haptic glove with their corresponding timestamps, generating one log file per song across five selected tracks. The songs were chosen randomly based on the criteria of having the presence of vocals, multiple instruments, and variety of genres (

Table 2. Songs used for the technical evaluation.

Song Number	Artist/s—Title	Genre/s
1	Maroon 5—She Will Be Loved	Pop Rock, Soft Rock
2	Tame Impala—The Less I Know The Better	Psychedelic Pop, Disco, Funk, Pop
3	Mark Ronson & Bruno Mars—Uptown Funk	Funk, Contemporary Soul, Electronica
4	Adele—Rolling In The Deep	Pop, Blues, R&B
5	Rihanna & Jay-Z—Umbrella	R&B/Soul, Pop

), following the approach of related studies [6]. The log files contain the tactile patterns that were produced by the piezoelectric actuators as output, representing the data after it had been processed by both the audio-tactile rendering algorithm and the wearable HMP. After the experiments, the log files were later used in Python to reconstruct the tactile signals for comparison with their parent audio signals. The previous and current methods are then compared in terms of signal fidelity (

Table 3. Summary of normalized signal fidelity testing results for previous and current HMP-WD.

	MAE			DTW Distance			Accuracy
	Bass	Vocal	Other	Bass	Vocal	Other	Kick	Snare	Toms	Cym
Previous	0.1151	0.1296	0.1272	0.1015	0.1931	0.1230	0.82	0.79	0.76	0.76
Current	0.1147	0.0880	0.1033	0.1470	0.1413	0.1670	0.98	0.98	0.97	0.97

).

3.1.1. Pitch Display

Figure 5a plots the pitch of the cleaned original audio for the vocals stem, while Figure 5b plots the pitch levels of the tactile signals designated for the vocals pin group. The bass, vocals, and others stems for both audio and tactile signals were normalized before MAE was computed using

$$MAE={\displaystyle \frac{1}{N}}{\displaystyle {\sum }_{i=1}^N}\left|{\displaystyle y_i}-{\displaystyle {\widehat{y}}_i}\right|,$$

(4)

where N is the number of time bins, y is the normalized pitch from the audio, and ŷ is the normalized pitch from the tactile signals. The MAE measures the average difference between the two pitch contours, with lower values indicating that the tactile signals convey the original audio’s pitch more accurately. Across all melodic elements, the average MAE is 0.1020 with a standard deviation of 0.0451 (

Table 4. MAE of tactile pitch display from pitch of audio signals, by each melodic element.

Melodic Element	Song Number					Average MAE
	1	2	3	4	5
Bass	0.1829	0.1095	0.0754	0.1230	0.0829	0.1147
Vocal	0.1047	0.1122	0.0326	0.1123	0.0784	0.0880
Other	0.1450	0.2017	0.0276	0.1112	0.0311	0.1033
Average MAE	0.1442	0.1411	0.0452	0.1155	0.0641	0.1020

). Figure 5c illustrates a comparison between the normalized tactile and audio signals. While discretization causes the tactile signals to slightly deviate from the original pitch, the heights and depths of the pitch contour are still generally preserved.

3.1.2. Melody/Timbre Display

DTW is applied to assess the similarity between audio and tactile pitch sequences while considering potential temporal misalignment. In both sequences, Euclidean distances between corresponding points are used to construct a distance matrix. We recursively compute the accumulated cost matrix before extracting the optimal warping path, which is used to calculate the Normalized DTW distance using

$$Normalized DTW={\displaystyle \frac{1}{K}}{\displaystyle {\sum }_{i=1}^K}d\left({\displaystyle a_{{\displaystyle i_k}}},{\displaystyle b_{{\displaystyle j_k}}}\right),$$

(5)

where K is the length of the warping path, a and b are the DTW-aligned pair from the audio and tactile sequences at index k, their respective indices i and j, and the Euclidean distance d between the two values. The size of K corresponds to the duration of the song in seconds multiplied by 10, as there are 10 discrete time bins used per second. To prevent increases of DTW distance as K increases, the raw DTW distance is normalized by K. Furthermore, the boundary, monotonicity, and continuity constraints are applied to the warping path to account for the full duration of both signals, to maintain temporal ordering, and to avoid point skips, respectively. Across all melodic elements, the average normalized DTW distance is 0.1518 with a standard deviation of 0.0729 (

Table 5. Normalized DTW distances of tactile melodic display from melody of audio signals, by each melodic element.

Melodic Element	Song Number					Avg. Norm. DTW Dist.
	1	2	3	4	5
Bass	0.2196	0.1484	0.1160	0.1552	0.0960	0.1470
Vocal	0.1143	0.1934	0.0445	0.2137	0.1406	0.1413
Other	0.3000	0.3350	0.0262	0.1419	0.0317	0.1670
Avg. Norm. DTW Dist.	0.2113	0.2256	0.0622	0.1703	0.0894	0.1518

). Figure 6 visualizes the DTW alignment paths for the bass, vocals, and others stems using the previous and current methods. Paths that are nearly diagonal lines indicate strong temporal alignment with minimal warping, highlighting the tactile signals’ capacity to preserve movements of pitch. On the other hand, paths that show a curve deviating from the diagonal reflect a temporal misalignment caused by latency and a lower signal resolution, disrupting pitch movements.

3.1.3. Rhythm Display

Accuracy quantifies how closely the rhythm of the tactile signal matches that of the original audio. Actuations are considered accurate only if a beat onset was detected for the particular 100 ms time bin, while any violations of this are considered an inaccuracy in rhythm display. The accuracy is computed as

$$ACC={\displaystyle \frac{1}{N}}{\displaystyle {\sum }_{i=1}^N}1 \left({\displaystyle y_i}={\displaystyle {\widehat{y}}_i}\right)$$

(6)

where N is the total number of time bins, y is the normalized pitch from the audio, ŷ is the normalized pitch from the tactile signals, and 1 (y = ŷ) is an indicator function that returns 1 when the tactile and audio signals match and 0 otherwise. Figure 7 illustrates an example of tactile pin actuations compared to the beat onsets from the original audio. The tactile actuations are almost always aligned with the beat onsets; however, latency sometimes causes delayed pin resets, leading to missed beats. The rhythmic pattern accuracy of the HMP-WD is 97.56% with a standard deviation of 0.0285 (

Table 6. Accuracy of tactile rhythm display from rhythm of audio, by each percussive element.

Percussive Element	Song Number					Avg. Accuracy
	1	2	3	4	5
Kick	0.9386	0.9926	0.9915	1.0000	0.9982	0.9842
Snare	0.9229	0.9821	0.9908	1.0000	0.9978	0.9787
Toms	0.9200	0.9775	0.9875	0.9758	0.9978	0.9717
Cymbals	0.9222	0.9834	0.9878	0.9469	0.9982	0.9677
Avg. Accuracy	0.9259	0.9839	0.9894	0.9807	0.9980	0.9756

).

3.2. Algorithm Processing Time

To prove that the methods used are relatively efficient, the HMP-WD should be validated in terms of processing speed, particularly on how long both previous and current methods take to fully convert an audio file to multi-feature tactile signals. To accomplish this, a Jupyter Notebook (version 6.5.4) was made for each method, where the algorithms would be executed in sequential order for each of the five songs. The first code cell starts the timer, which ends when the last code cell runs, displaying the processing time. The results show that using spectral flux and YIN for multi-feature extraction increased processing time for only a few seconds on average compared to our previous method (

Table 7. Audio-tactile rendering algorithm processing time comparison: previous vs. current.

	Song Number					Avg. Time (s)
	1	2	3	4	5
Previous	185.04	168.20	182.00	148.35	185.31	173.78
Current	167.40	168.36	204.17	168.13	194.17	180.45

).

4. Discussion

The HMP-WD attained relatively high scores in its respective metrics on the tactile display of pitch, melody, timbre, and rhythm, and improvements were also observed when using spectral flux and YIN in the feature extraction process. Furthermore, these improvements came with only a small increase in the algorithm’s processing time compared to our previous method.

For pitch display, the current methods were formulated with the hypothesis that modifying the discretization step from four levels to eight would reduce the MAE of tactile signals, as the distances between each discretized time point and the original pitch would become smaller. Compared to the previous method whose MAE was 0.1240, the current implementation showed an overall improvement with an error of 0.1020 and deviating from the original audio by only 10.2%. While improvements on the bass stem are negligible, the accompaniment and vocal stems showed significant improvements, with the latter reducing MAE by 0.0416.

The disparity of results among the three stems can be explained by the non-zero pitch ranges in the original audio, which play an integral factor during discretization. It is important to note that when the eight levels are determined during the discretization step, only the non-zero pitch ranges are considered. The rationale behind this is that starting from zero reduces touch-perceptible level changes as the original pitch increases. For instance, a stem whose non-zero pitch range is 400–600 Hz would feel more muffled compared to one with a range of 200–400 Hz, even if both have the same melody, because the former would only actually utilize fewer upper levels than the latter. Conversely, if both utilize only their non-zero pitch range, the movements in pitch would be equally perceptible regardless of raw value, leading to better melody displays. Returning to the point on disparity, the bass stem will almost never have a higher raw pitch value compared to the other stems, and it will generally have a smaller non-zero pitch range. In one song, the estimated range for bass was 50–160 Hz, while the ranges were 50–725 Hz for vocals and 50–650 Hz for accompaniments. This means that, after discretization, pitch levels for bass are shifted relatively higher or lower than those of other stems. Despite the disparity of results across stems, increasing the number of levels during discretization was still shown to improve the HMP-WD’s pitch display.

For melody/timbre display, the hypotheses are that using YIN would lead to pitch estimates that are more stable, allowing the tactile signals to replicate them better compared to the coarse pitch contours retrieved using spectral peaks, and that increasing the discretized pitch levels would allow tactile signals to capture finer pitch movements. Compared to the previous DTW distance of 0.1392, the current implementation seems to display overall melody worse with a distance of 0.1518. Taking a closer inspection, it is revealed that the driving forces for the overall distance increase are the bass and accompaniment, whose distances increased by about 0.04 each, even when the vocal stem’s distance decreased by 0.0518.

Although it is reliable in general pitch estimation, YIN is best applied on monophonic vocal signals as observed in other studies [44,45]. Thus, the significant decrease in DTW distance for the vocal stem is not surprising and rather proves the working hypotheses. On the other hand, the accompaniment stem is usually polyphonic, with some songs containing multiple instruments such as piano and guitar. YIN still accurately estimates the pitch in polyphonic signals but because of harmonization, the resulting pitch contours are relatively rougher, resulting in an increased difficulty for the tactile signals to accurately replicate these sharp pitch movements. Similarly, while the bass stem is usually monophonic, its percussive aspects would inevitably lead to rough pitch contours even when using YIN, and when combined with its pitch display issues as discussed earlier, the resulting melody display is also impacted. The working hypotheses were proven for the vocal stem, whereas future research can explore methods in pitch estimation or discretization that can improve the melody display for the bass and accompaniment stems.

For the rhythm display, we sought to improve the results of our previous approach using spectral flux and software optimization. Since it detects beat onsets at the frequency level rather than finding energy peaks, it no longer includes volume spikes that may merely come from noise, leading to reduced payloads and latencies. At the same time, we optimized the mobile application’s synchronization with the haptic gloves and music playback. Thus, the accuracy for all drum frequency subsets was drastically improved. Unlike related applications that immediately attempted to detect beats from the audio, the methods in this study account for beats with softer volume and at four different frequency subsets to further isolate each type of drum, which were obtained from a source separation process that isolates it from vocals, bass, and accompaniments.

With these results, the HMP-WD has been shown to effectively display pitch, melody, timbre, and rhythm using only an audio file, extracting features and communicating them from a near-auditory frequency range. It retains the characteristics of the original audio while improving upon prior approaches in four key ways. First, this study uses piezoelectric actuators, which offer fine localization capabilities and can communicate frequencies beyond 1000 Hz through vibrotactile feedback, unlike other popular actuators. In HMP-WD studies with a glove form factor, this device aligns with prior studies that applied stimuli to the fingertips [19,20,21] instead of only the phalanges, backhand, or palm [17,18]. The use of a mobile app to transmit tactile signals via Bluetooth has been explored before [19], but this study advances the approach by using BLE to improve portability. Second, the study introduces an algorithm that does not require a music’s MIDI file to extract pitch information. It proposes YIN and discretization as an efficient and effective method. Unlike previous studies limited by external software [25,26] and a fixed number of notes [1,5,6,24,25,26], the proposed methods generate continuous pitch contours that can be discretized into any number of levels, adjusting for the number of actuators. This also addresses the need for tactile illusions to increase pitch display capacity [24,26]. Third, this study provides a framework for beat extraction for HMP applications. While prior studies mentioned aligning vibrations with a beat [20], their accuracy was not validated. The proposed methods allow for beat extraction without using a bass band that is limited to 200 Hz [23,27,28] and suggest a frequency-based approach to detect soft-volume beats rather than the energy-based approach [23]. Fourth, source separation is used to separate sounds from different instruments to display timbre, similar to prior studies [5,26].

The limitation in the proposed algorithm is that it requires preprocessing the audio file before it can be used in the HMP-WD. However, despite using spectral flux and YIN for multi-feature extraction, the methods take an average of only 180.45 s to process an audio file from start to finish for permanent, offline use. This is a minimal increase of less than 10 s compared to our previous methods, highlighting the overall efficiency of the audio-tactile rendering algorithm.

5. Conclusions

With a normalized respective MAE and DTW distance of 0.1020 and 0.1518, and a rhythmic pattern accuracy of 97.56%, the wearable HMP effectively communicates multiple musical features. The proposed audio-tactile rendering algorithm achieves this without the pitfalls of prior approaches. Using piezoelectric actuators, this HMP also overcomes the 1000 Hz frequency range barrier of previous devices. Additionally, the HMP-WD is more efficient, with an average preprocessing time of only 180.45 s and without required MIDI files or software. The study also proposes a volume-independent method to extract beats from music for HMP applications and a way to display timbre information using source separation and feature-pin mapping.

The findings suggest that the HMP-WD can communicate the pitch, melody, timbre, and rhythm of music at an auditory frequency range while retaining most of the characteristics of the original audio. The YIN algorithm was shown to greatly improve the tactile display of vocals and slightly improve the display of bass and accompaniments. Spectral flux and software optimizations significantly improved rhythm display.

Future research should focus on reducing the MAE and DTW distances of melodic displays and validating the results with more song data. Other actuators could be explored to improve the results, and adjustments to the methods could be made, such as using probabilistic YIN [45] or related source separation, beat extraction, and discretization methods. Subjective experiences with the device could also be validated through user testing. Furthermore, these methods could be used to enhance the music instrument playing experience [22] or integrated with other modalities like VR [2]. Future work could also study the human emotional and cognitive perception of music using the HMP-WD.