Detection of Hand Poses with a Single-Channel Optical Fiber Force Myography Sensor: A Proof-of-Concept Study

Abstract

Force myography (FMG) detects hand gestures based on muscular contractions, featuring as an alternative to surface electromyography. However, typical FMG systems rely on spatially-distributed arrays of force-sensing resistors to resolve ambiguities. The aim of this proof-of-concept study is to develop a method for identifying hand poses from the static and dynamic components of FMG waveforms based on a compact, single-channel optical fiber sensor. As the user performs a gesture, a micro-bending transducer positioned on the belly of the forearm muscles registers the dynamic optical signals resulting from the exerted forces. A Raspberry Pi 3 minicomputer performs data acquisition and processing. Then, convolutional neural networks correlate the FMG waveforms with the target postures, yielding a classification accuracy of (93.98 ± 1.54)% for eight postures, based on the interrogation of a single fiber transducer.

1. Introduction

The development of sensors for identifying hand gestures is crucial to establish reliable and intuitive communication between humans and computers, as requested by several areas related to control and automation engineering. Potential applications for these devices encompass the operation of dexterous bionic prostheses [1], medical robots [2], rehabilitation devices [3], and virtual reality environments [4].

Nowadays, different approaches are available for assessing hand poses. Examples include glove-based sensors, exoskeletons, optical tracking, brain–computer interfaces (BCI), and myographic methods such as surface electromyography (sEMG) and force myography (FMG).

Although glove devices and exoskeletons monitor the angular displacements of hand joints, these techniques constrain user movements by imposing mechanic load [5]. As to non-invasive techniques, optical tracking relies on static cameras and usually degenerates due to occlusion and illumination changes [6]. Besides, BCI requires expensive, bulky instrumentation and demanding signal processing [7]. Consequently, myographic sensors are eligible to detect hand motion or intentions in teleoperation and prostheses control, providing accurate results with accessible costs [8,9].

The sEMG detects electrical stimuli produced by the forearm muscles using electrodes attached to the skin surface. After extracting temporal and spectral features from sEMG waveforms, a classifier addresses the collected data to the target postures through machine learning algorithms [10]. Notwithstanding their successful applications, sEMG systems are susceptible to electromagnetic noise and variations of skin impedance. Alternatively, the FMG assesses radial pressures exerted by forearm muscles employing piezoresistive transducers to retrieve hand motions from the average force levels [11].

Currently, several authors have investigated FMG to identify hand poses. In recent works, Asfour et al. employed an array of 16 force sensing resistors (FSR) to discriminate 16 grasps exerted with different force levels [12]. An improved principal component analysis pipeline yielded an average accuracy of 86.4%. Nowak et al. used ten FSR to classify transitions between six poses through the ridge regression with random Fourier features algorithm, achieving >80% accuracy for the best cases [13]. Ultimately, Jiang et al. developed an integrated device comprising eight FMG and sEMG electrodes to recognize ten gestures. Linear discriminant analysis combining mechanical and electrical signals resulted in a 91.6% accuracy [14].

Electronic devices such as piezoresistive, piezoelectric, and capacitive transducers are vulnerable to electric noise and impedance changes caused by sweat and fat [15]. Therefore, optical fiber sensors are reliable alternatives to the widespread FSR, exhibiting high sensitivity, lightness, flexibility, and immunity to electromagnetic interference [16]. For instance, an integrated FMG system comprising two micro-bending transducers provided >90% accuracy regarding the classification of nine static postures [17]. Moreover, this sensor operated simultaneously with a functional electrical stimulation module for controlling a robot glove, demonstrating its robustness to concurrent electrical impulses [18].

Apart from the available technologies, developments towards reducing the number of measurement channels are attractive to simplify both force sensing apparatus and signal interrogation system. In this context, exploring the dynamic behavior of FMG waveforms is a promising approach once the temporal characteristics of muscular contractions vary accordingly to the target posture and the exerted force level. Some groups investigated this aspect in recent works. Hellara et al. combined strain, pressure, inertial, and sEMG sensors, producing an ensemble of 23 dynamic signals to feed a random forest classifier; the overall accuracy is >99% for ten postures [19]. Li et al. employed eight piezoelectric strain gauges to analyze six gestures. Based on 13 time-domain features, the authors processed the acquired signals with four classification algorithms and obtained >95% accuracy [20]. Furthermore, Fujiwara et al. used optical fiber sensors to recognize the 26 alphabet letters written in the air, achieving ∼90% accuracy with parallel correlators unified by a competitive layer [21]. Nevertheless, the current approaches rely on multiple transducers for recognizing intuitive gestures, increasing the system complexity and imposing additional load on the upper limb.

Therefore, this work presents the proof-of-concept of an FMG system for identifying hand poses based on a single optical fiber transducer. The time-varying intensity waveforms proceed to a convolutional neural network that maps the hidden features into the target postures. Experiments comprising eight classes yielded reliable results disregarding the limited number of force transducers.

Departing from the current approaches based on arrays of FSR, the proposed method uses a single fiber bending transducer to assess the FMG signals, reducing the mechanical load on the forearm and simplifying the interrogation hardware and signal processing. Furthermore, optical fiber sensors are intrinsically robust to electromagnetic noise and changes in skin conditions. To the best of our knowledge, this is the first demonstration of a single-channel FMG system based on optical interrogation.

Force Myography

Force myography is the mechanical counterpart of the sEMG. This technique assesses radial pressures produced by the forearm muscles—in virtue of hand movements or intentions—and correlates the force levels to the performed gesture. Moreover, the FMG does not rely on a strict placement of transducers and requires simpler interrogation hardware than the electrical approach [11].

Upon activation by the central nervous system, muscle tissues contract to develop force and increase their strength. Force-velocity-length relations dictate the mechanics involved in achieving the force levels required to grasp or sustain a pose [22]. Therefore, FMG transducers attached to the forearm through straps or orthoses detect the overall response of embraced flexor and extensor muscles. Choosing the placement of sensors allows for assessing specific groups of muscles, enhancing the identification of certain postures. Nevertheless, dynamic FMG signals comprise transient and steady-state features: the former accounts for activation and contraction peaks to attain the desired pose, whereas the latter stabilizes the hand configuration [23].

Current FMG systems compute the average value of stationary forces and combine the information from multiple transducers to discriminate postures. Alternatively, one may explore the FMG transient characteristics to optimize the number of transducers necessary for recognizing a gesture. Albeit FMG patterns depend on the initial and final hand pose—training the classifier may be laborious for an inclusive set of postures,—assuming an event-driven finite state (EDFS) control alleviates the calibration task by mapping a few input gestures to a compendium of actions, which is applicable for commanding practical prostheses [24,25].

2. Materials and Methods

2.1. Hardware Design

Figure 1 shows the system setup [17]. Light emitted by an Agilent HFBR-1414T LED source (80 $\mathsf{\mu }\mathrm{W}$, 820 $\mathrm{n}\mathrm{m}$) excites a silica multimode optical fiber (MMF, step-index, 62.5/125 $\mathsf{\mu }\mathrm{m}$ core/cladding diameters, straight tip terminations). An optomechanical transducer secured to the forearm by Velcro straps modulates the guided light. Then, an Agilent HFBR-2416T photodetector (7 /$\mathsf{\mu }\mathrm{W}$, 820 $\mathrm{n}\mathrm{m}$) assesses the output intensity and delivers the electrical signal to a double amplifier circuit (INA 122, total gain factor of 19,200), followed by an analog low-pass filter (cutoff frequency of 100 $\mathrm{Hz}$) for noise suppression. Lastly, the electrical signal passes through a 16 bits analog-to-digital converter (ADS1115) and feeds the digital port of the Raspberry Pi 3 microprocessor Model B (Broadcom Quad Core BCM2837 chipset with 64 bits, 1.2 GHz clock, and 1 GB RAM).

2.2. Force Transducer

The force transducer (Figure 2) comprises a periodical structure that deforms the MMF and induces optical losses by microbending effect. The consecutive curvatures cause the core-guiding modes to couple with the radiation modes, decreasing the output light intensity [26]. Consequently, the optical signal varies proportionally to the dynamic muscle contractions.

The transducer manufacturing begins with parts designed in the Inventor (Autodesk) Software. Next, an Ultimaker 3 Extend 3D printer fabricates the components using acrylonitrile butadiene styrene (ABS) material. Lastly, a careful finishing procedure ensures a smooth surface without sharp edges, providing comfort and safety to the user.

The device contains two connectable pieces to fixate the fiber and create a deforming structure with a period of 10 $\mathrm{m}\mathrm{m}$. The upper part has passers for inserting straps for embracing the forearm. Once assembled, the force transducer has final dimensions of 60 $\mathrm{m}\mathrm{m}$ × 15 $\mathrm{m}\mathrm{m}$ × 15 $\mathrm{m}\mathrm{m}$.

2.3. Experimental Procedure

2.3.1. Evaluated Postures

The experiments investigate eight intuitive hand poses (labeled with capital letters in Figure 3) selected in agreement with previous works [17,27]. These gestures comprise the flexion/extension and adduction/abduction of thumb and finger joins, as well as the flexion/extension of the wrist. Four of the poses (A, B, C, and D) are similar to those recognized by a popular, low-cost sEMG system (Myo Armband, Thalmic Labs) [28].

N is the relaxed hand position and works as a neutral reference to the classifier. Despite not being included as a class, N indicates the transitions between gestures.

Posture A characterizes the closed fist. In opposite to A, B is the hand open with the abduction of all fingers. C and D are the wrist flexion (wave in) and extension (wave out) movements, respectively. Posture E adopts fingers flexed and the thumb abducted in the carpometacarpal (CMC) joint. F has the flexion of all digits except the index. G comprises an extension of the index and middle fingers. Lastly, H has all fingers joined and extended.

2.3.2. Measurement Protocol

This proof-of-concept study recruited two healthy participants with an average age of 24.40 ± 3.36 years old. The volunteers provided consent to participate, and the experiments followed the recommendations of the Ethics Committee. During the measurements, the individuals sit comfortably by keeping their elbows supported and slightly flexed, with the forearm suspended in the air in a neutral position. This procedure maintains the wrist with neutral pronation to avoid unintentional movements.

The optical fiber transducer accommodates on the posterior side of the forearm over the extensor digitorum communis (EDC) muscle. According to previous studies, various fingers and wrist movements are detectable from the FMG signals produced by this muscle [27]. A Velcro strap tightly adjusts the transducer to create a subtle preload without injuring the user. As the sensor placement depends on muscle palpation, one may expect variations in the FMG signals acquired for different individuals and measurement sessions.

After issuing a command visual, the volunteers executed sequences of predefined postures; in the meanwhile, the microprocessor unit acquires and records the corresponding optical signals. Each volunteer received previous instructions about the experimental procedures, having to perform each pose during intervals of 5 $\mathrm{s}$. Starting from posture N, the volunteer performs the target gesture (A to H), then returns the hand to N. Therefore, each FMG pattern comprises a rising transient followed by a stationary phase and a fall transition. This step repeats 36 times per hand gesture, summarizing 288 FMG waveforms per individual. Furthermore, the individuals must doff/don the strap and rest for ∼1 $\min$ between experimental sessions to avoid fatigue and possible degeneration of force levels.

2.4. Classification of Hand Postures

2.4.1. Signal Processing

An analog-to-digital converter acquires the FMG signals at 200 $\mathrm{Hz}$ sampling rate and transfers the data to the microcomputer through an I²C serial communication protocol. Digital data processing proceeds with routines programmed in Python to design modular, scalable applications based on the Raspberry Pi single-board computer. After removing the noise from the FMG patterns (using a low-pass fifth-order Butterworth digital filter with a cutoff frequency of 100 $\mathrm{Hz}$,) the routine normalizes the data and segments it using a 5 $\mathrm{s}$ rectangular window before the classification. Figure 4a portrays the signal processing pipeline.

2.4.2. Classification System

Classification proceeds with convolutional neural networks (CNN) implemented in Python using Keras with TensorFlow backend [29]. The CNN comprises a one-dimensional convolutional model that detects the particular characteristics of each FMG pattern in the time domain. The convolutional layers contain filters to process the input signals and generate feature maps. During the training step, the algorithm adjusts these filters automatically to make them responsive to the relevant features [30].

The classifier (Figure 4b) has a convolutional layer with 64 filters and a convolutional window of size 50, wherein the weights initiates with random values. After determining the feature maps, a rectified linear unit (ReLU) activation function eliminates the negative values [31]. Next, a max pooling layer designed with a reduction factor of 10 times (to make the network more robust) eliminates redundant values and reduces the size of the input layer of the dense network, improving the computing burden. To extract more features and diminish the input, complementary convolutional layer and max pooling perform with the same parameters as the previous one. Finally, after vectorizing the feature maps, the data achieves the dense neural network comprising a competitive layer of 8 neurons with a linear activation function.

The CNN training step adjusts the hyperparameters through supervised learning using the gradient descendant algorithm. The Adam optimizer runs with a 0.001 learning rate and clip value of 0.05. The loss function employs the mean squared error limited to 100 epochs to avoid overfitting.

The evaluation of classifier performance assumes the accuracy $ACC$ and hit rate $HR$ as quantitative metrics,

$$ACC=\frac{TP+TN}{P+N}$$

(1)

$$HR=\frac{TP}{P}$$

(2)

where $TP$ and $TN$ are the true positive and true negative detections, respectively, P and N are the total numbers of positives and negatives, respectively, [32]. Furthermore, since the performance may vary depending on the training and validation sets, the tests followed a 10-fold cross-validation [33].

3. Results

3.1. Analysis of FMG Signals

Figure 5 summarizes the FMG signals measured for all volunteers. The voltage waveforms are normalized and scaled for subsequent processing. The red dotted lines indicate the reference force level, i.e., the neutral pose N. Albeit the amplitudes may vary due to anatomical characteristics inherent to the volunteers, each posture produces discernible temporal patterns.

Two types of muscle contractions occur during hand movements, namely, (i) concentric and (ii) isometric contractions [22]. The former governs the dynamics involved in shortening muscle fibers, responsible for moving articulations during the transients of FMG patterns. The latter maintains the hand posture and establishes stationary force levels.

Concerning the clenched fist (pose A), fingers joints bend by contraction of the flexor digitorum superficilis (FDS) and flexor digitorum profundus (FDP) muscles, increasing their diameters as the antagonistic extensors stretch. The normalized voltage is higher for this movement because the strap enclosing the belly of the aforementioned muscles presses the microbending transducer against the forearm, modulating the optical signal. A similar pattern emerges for pose F; in this case, the magnitude of radial pressures diminishes due to the extension of the index finger, relaxing part of the muscles ensemble. Alternatively, posture E differs by thumb abduction, and the force level increase originates from the activation of the abductor pollicis longus (APL) muscle.

As to the wrist flexion (pose C), despite the contraction of the flexor carpi radialis (FCR), the force magnitude is lower than posture A because of fingers extension, abbreviating the contribution of FDS and FDP muscles. The transient part exhibits a peak followed by an exponential decay; this behavior depends on how fast the transitions between postures occur. Furthermore, this observation corroborates the higher activation of muscle fibers during concentric contractions than the isometric ones. Similar events occur in poses A, E, and F, as expected.

Conversely, posture B produces voltage values lower than the reference line. This gesture comprises the extension and abduction of all digits. Upon activation of the extensor digitorum communis (EDC) and relaxation of antagonistic flexors, overall radial pressures decrease once the belly of the FDS and FDP muscles suffers a radial contraction, reducing the mechanical load over the microbending transducer. This pattern repeats for postures D and H, wherein differences in stationary levels result from wrist extension and fingers adduction, respectively. For instance, gesture D produces peaks during both initial and final transients, probably due to the contribution of the extensor carpi radialis (ECR) muscle.

Ultimately, although the differences between gestures G and F are limited to the extension of the middle finger and abduction of the index one, these postures present discrepant FMG patterns. This result suggests that isometric contractions for sustaining the index finger abducted predominate over the activation of flexor muscles.

3.2. Identification of Hand Poses

Following the characterization of FMG signals, the collected waveforms and their corresponding labels proceeded to the CNN training and validation accordingly to the procedures exposed in previous sections. The confusion matrix (Figure 6) shows the nominal gestures versus the classifier predictions regarding the average of volunteers, yielding an overall accuracy of (93.98 ± 1.54)%.

Postures A, C, E, and F provided the best performance, achieving a hit rate ≥0.97%. Conversely, gesture G obtained the poorest recognition with $HR$ = 85%. Most misclassifications belong to postures B, D, G, and H: these signals exhibit comparable waveforms with negative variations in the voltage level. Furthermore, unintentional movements or muscular contractions also affect the behavior of FMG stimuli, which explains the eventual confusion between antagonistic classes—poses A and B, for example.

The results suggest a successful classification of different gestures regardless of the unique measurement channel. The temporal characteristics of the FMG signals complete the lacking information from spatially-distributed forearm muscles, as obtained by arrays of FSR. Nevertheless, monitoring a single position with the optical fiber transducer generates similar waveforms evidenced in Figure 5; this aspect may compromise the identification of gestures depending on their complexity and likenesses.

3.3. Discussion

The proposed methodology identified eight gestures using a single optical fiber transducer. Typical FMG systems operating with static force levels require arrays of pressure sensors to collect the spatial response from forearm muscles and discern between the possible hand poses. For example, 32 FSR assessed with a support vector machine classifier provide an accuracy of >99% for 17 classes [34]. Moreover, the interrogation of 16 FSR through linear discriminant analysis yields an impressive accuracy of 96.7% for 48 hand gestures [35]. Multiple measurement channels enrich the classifiers with unique features, minimizing the ambiguities expected for overlaid groups of muscles. Nevertheless, eliminating excessive sensors is crucial for improving portability and reducing hardware costs.

As to the dynamic methods,

Table 1. Comparison of hand posture classification systems based on dynamic FMG signals.

Ref	Transducers	Classifier	Accuracy (Gestures)
[19]	15 (strain/pressure sensors, 9-axis IMU, electrodes)	Random forest	99.5% (10)
[20]	8 (piezoelectric sensors)	k-nearest neighbor	95.5% (6)
[36]	1 (6-axis IMU)	Correlator with competitive layer	96.6% (4)
This work	1 (optical fiber)	CNN	94.0% (8)

summarizes the characteristics of current FMG systems. It is worth noting that each work assumes a particular experimental protocol, i.e., immediate comparisons would lead to biased conclusions. Using a combination of pressure/strain transducers, inertial measurement units (IMU), and sEMG electrodes yielded the best performance concerning number of classes (ten) and accuracy (99.5%) [19]. However, such an approach requires processing 23 measurement channels to retrieve the gesture—for instance, ordinary static FMG signals assessed by arrays of FSR exceed the performance of the dynamic approach [34]. Nevertheless, the proposed optical fiber system achieves an acceptable accuracy based on a single acquisition channel, exceeding the available technologies in terms of simplicity and comfort to the user (by alleviating the mechanical load imposed on the forearm).

Concerning practical measurements, the experimentalist must consider possible variations in the FMG signatures caused by the forearm orientation and the magnitude of exerted forces during grasp events [37]. Variations in the FMG waveforms will probably confound the CNN and lead to equivocated predictions. Furthermore, using a single measurement channel makes the system less robust to extraneous effects due to the lack of comparative data. In this case, adding a complimentary optical or electronic transducer to implement a data fusion technique may improve the sensor response [38].

Another aspect is the time required for buffering the FMG signals. A 5 $\mathrm{s}$ window may frustrate applications demanding real-time response, such as virtual reality and teleoperation. One may abbreviate the buffer size and increase the acquisition rate to amend this limitation; though, refining the classifier accordingly to the duration of the signals is mandatory.

Lastly, the number of classes retrieved by the proposed method is not impressive compared with the available FMG (and EMG–assisted) systems [12,13,14]. Given the layered configuration of the forearm muscles and the single fiber transducers, part of the additional gestures may produce ambiguous FMG signatures. Identifying the individual motion of the fingers based on the forearm contractions is hard to accomplish with concentrated sensors [34]. However, applications such as prostheses control are feasible through a limited set of gestures by implementing an event-driven finite state machine approach [24]. Instead of performing laborious training for a collection of classes, one may map a limited set of gestures into a comprehensive dictionary of actions to drive the manipulator, making the bionic prosthesis more intuitive and less exhaustive to the operator [25]. Yet, integrating the FMG sensor with other technologies (sEMG, mechanomyography, optical tracking, etc.) through collaborative or competitive data fusion is another possibility to enhance the system reliability regarding a modular approach [25,39].

Further developments will focus on enhancing the robustness and miniaturizing the device for applications outside the lab. Moreover, comprehensive experiments with a larger population and following a strict protocol are expected to allow assessment of the practical accuracy and of the repeatability/reproducibility of this method.

4. Conclusions

This proof-of-concept study pursued developing, implementing, and testing a force myography system using optical fiber sensors with a single transducer. The project aimed for a low-cost device without compromising its sensitivity, comfort to the user, mobility, and robustness. Based on the transient characteristics of the FMG waveforms, the system achieved an average accuracy of (93.98 ± 1.54)%. Although the transducer essentially monitors the extensor muscles, it was possible to determine flexion and extension movements besides combinations of these, making the sensor suitable for controlling orthoses and prostheses by gestural commands.