# Combining Electromyography and Tactile Myography to Improve Hand and Wrist Activity Detection in Prostheses

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Materials

#### 2.1. Surface Electromyography Bracelet

#### 2.2. Tactile Bracelet

## 3. Methods

#### 3.1. Gaussian Process Regression

#### 3.2. Gaussian Process Regression for Matrix-Valued Data

#### 3.3. Gaussian Process Regression for Multimodal Data

#### 3.4. Kernels

- the Kullback–Leibler (KL) divergence ${d}_{\mathrm{KL}}=\mathrm{KL}\left(\right)open="("\; close=")">p\left({\mathit{X}}_{i}\right|{\mathit{\mu}}_{i},{\Sigma}_{j})\parallel q\left({\mathit{X}}_{j}\right|{\mathit{\mu}}_{j},{\Sigma}_{j})$ by treating all matrix data as a Gaussian generative model [14];
- two different distances for covariance or symmetric positive definite (SPD) matrices, ${d}_{\mathrm{logSPD}}={\parallel \mathrm{ln}\left({\Sigma}_{i}\right)-\mathrm{ln}\left({\Sigma}_{j}\right)\parallel}_{\mathrm{F}}$ and ${d}_{\mathrm{SPD}}={\parallel {\Sigma}_{i}-{\Sigma}_{j}\parallel}_{\mathrm{F}}$, where $\Sigma =\mathit{X}$ if $\mathit{X}$ is a SPD matrix and $\Sigma =\mathrm{cov}\left(\mathit{X}\right)$ otherwise [17];
- the Euclidean distance ${d}_{\mathrm{Eucl}}=\parallel \mathrm{vec}\left({\mathit{X}}_{i}\right)-\mathrm{vec}\left({\mathit{X}}_{j}\right)\parallel $ for matrices and ${d}_{\mathrm{Eucl}}=\parallel {\mathit{x}}_{i}-{\mathit{x}}_{j}\parallel $ for vectors.

## 4. Experiments

#### 4.1. Participants

#### 4.2. Experimental Setup

#### 4.3. Experiment 1: Real-Time Goal-Reaching Task with Tactile Myography

#### 4.4. Experiment 2: Combination of Electromyography and Tactile Myography

- sEMG signal as input: ${d}_{\mathrm{Eucl}}$, ${d}_{\mathrm{SPD}}$ and ${d}_{\mathrm{logSPD}}$;
- TMG signal as input: ${d}_{\mathrm{Eucl}}$, ${d}_{\mathrm{SPD}}$, ${d}_{\mathrm{logSPD}}$ and ${d}_{\mathrm{KL}}$;
- sEMG and TMG signals as input: all combinations of ${d}_{\mathrm{Eucl}}$, ${d}_{\mathrm{SPD}}$ and ${d}_{\mathrm{logSPD}}$ for sEMG coupled with ${d}_{\mathrm{Eucl}}$, ${d}_{\mathrm{SPD}}$ and ${d}_{\mathrm{KL}}$ for TMG.

## 5. Results

#### 5.1. Experiment 1: Real-Time Goal-Reaching Task with Tactile Myography

#### 5.2. Experiment 2: Combination of Electromyography and Tactile Myography

## 6. Discussion

## 7. Conclusions

## Supplementary Materials

## Acknowledgments

## Author Contributions

## Conflicts of Interest

## Abbreviations

sEMG | Surface electromyography |

TMG | Tactile myography |

RKI | Residual kinematic imaging |

ADC | Analog-to-digital converter |

DOF | Degree of freedom |

RR | Ridge regression |

GPR | Gaussian process regression |

RBF | Radial basis function |

Eucl | Euclidean |

SPD | Symmetric positive definite matrix |

KL | Kullback–Leibler divergence |

SR | Success rate |

TCT | Time to complete task |

TIT | Time in the target |

RMSE | Root-mean-square error |

## Appendix A

**Figure A1.**Comparison of the success rate (SR) between Gaussian process regression (GPR) and ridge regression (RR) for each participant.

**Figure A2.**Comparison of the time to complete task (TCT) and the time in the target (TIT) between Gaussian process regression (GPR) and ridge regression (RR), in the case of successful and failed tasks, for each participant.

**Figure A3.**Comparison of the root-mean-square error (RMSE) between Gaussian process regression (GPR) using different kernel distances, with surface electromyography (sEMG) as input, for each participant.

**Figure A4.**Comparison of the root-mean-square error (RMSE) between Gaussian process regression (GPR) using different kernel distances, with tactile myography (TMG) as input, for each participant.

**Figure A5.**Comparison of the root-mean-square error (RMSE) between Gaussian process regression (GPR) using different kernel distances to combine surface electromyography (sEMG) and tactile myography (TMG) as input, for each participant.

## References

- Atzori, M.; Gijsberts, A.; Kuzborskij, I.; Elsig, S.; Mittaz Hager, A.G.; Deriaz, O.; Castellini, C.; Müller, H.; Caputo, B. Characterization of a Benchmark Database for Myoelectric Movement Classification. IEEE Trans. Neural Syst. Rehabilit. Eng.
**2015**, 23, 73–83. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Jiang, N.; Rehbaum, H.; Vujaklija, I.; Graimann, B.; Farina, D. Intuitive, Online, Simultaneous and Proportional Myoelectric Control Over Two Degrees-of-Freedom in Upper Limb Amputees. IEEE Trans. Neural Syst. Rehabilit. Eng.
**2014**, 22, 501–510. [Google Scholar] [CrossRef] [PubMed] - Amsuess, S.; Vujaklija, I.; Goebel, P.; Roche, A.; Graimann, B.; Aszmann, O.; Farina, D. Context-Dependent Upper Limb Prosthesis Control for Natural and Robust Use. IEEE Trans. Neural Syst. Rehabilit. Eng.
**2016**, 24, 744–753. [Google Scholar] [CrossRef] [PubMed] - Ameri, A.; Scheme, E.J.; Kamavuako, E.N.; Englehart, K.B.; Parker, P.A. Real-Time, Simultaneous Myoelectric Control Using Force and Position-Based Training Paradigms. IEEE Trans. Biomed. Eng.
**2014**, 61, 279–287. [Google Scholar] [CrossRef] [PubMed] - Krasoulis, A.; Nazarpour, K.; Vijayakumar, S. Towards Low-Dimensional Proportional Myolelectric Control. In Proceedings of the IEEE International Conference on Engineering in Medicine and Biology Society (EMBS), Milano, Italy, 25–29 August 2015. [Google Scholar]
- Santello, M.; Bianchi, M.; Gabiccini, M.; Ricciardi, E.; Salvietti, G.; Prattichizzo, D.; Ernst, M.; Moscatelli, A.; Jörntell, H.; Kappers, A.; et al. Hand synergies: Integration of robotics and neuroscience for understanding the control of biological and artificial hands. Phys. Life Rev.
**2016**, 17, 1–23. [Google Scholar] [CrossRef] [PubMed] - Peerdeman, B.; Boere, D.; Witteveen, H.; Hermens, H.; Stramigioli, S.; Rietman, H.; Veltink, P.; Misra, S. Myoelectric forearm prostheses: State of the art from a user-centered perspective. J. Rehabilit. Res. Dev. (JRRD)
**2011**, 48, 719–737. [Google Scholar] [CrossRef] - Phillips, S.L.; Craelius, W. Residual kinetic imaging: A versatile interface for prosthetic control. Robotica
**2005**, 23, 277–282. [Google Scholar] [CrossRef] - Wininger, M.; Kim, N.H.; Craelius, W. Pressure signature of forearm as predictor of grip force. J. Rehabilit. Res. Dev.
**2008**, 45, 883. [Google Scholar] [CrossRef] - Kõiva, R.; Riedenklau, E.; Viegas, C.; Castellini, C. Shape Conformable High Spatial Resolution Tactile Bracelet for Detecting Hand and Wrist Activity. In Proceedings of the IEEE International Conference on Rehabilitation Robotics (ICORR), Singapore, 11–14 August 2015. [Google Scholar]
- Nissler, C.; Connan, M.; Nowak, M.; Castellini, C. Online Tactile Myography For Simultaneous and Proportional Hand and Wrist Myocontrol. In Proceedings of the Myoelectric Control Symposium (MEC), Fredericton, NB, Canada, 15–18 August 2017. [Google Scholar]
- Rasmussen, C.E.; Williams, C.K.I. Gaussian Processes for Machine Learning; MIT Press: Cambridge, MA, USA, 2006. [Google Scholar]
- Zhao, Q.; Zhou, G.; Adali, T.; Zhang, L.; Cichocki, A. Kernelization of Tensor-Based Models for Multiway Data Analysis: Processing of Multidimensional Structured Data. IEEE Signal Process. Mag.
**2013**, 30, 137–148. [Google Scholar] [CrossRef] - Zhao, Q.; Zhou, G.; Zhang, L.; Cichocki, A. Tensor-Variate Gaussian Processes Regression and Its Application to Video Surveillance. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Florence, Italy, 4–9 May 2014. [Google Scholar]
- Connan, M.; Ruiz Ramírez, E.; Vodermayer, B.; Castellini, C. Assessment of a Wearable Force- and Electromyography Device and Comparison of the Related Signals for Myocontrol. Front. Neurorobot.
**2016**, 10, 17. [Google Scholar] [CrossRef] [PubMed] - Castellini, C.; Kõiva, R. Using a high spatial resolution tactile sensor for intention detection. In Proceedings of the ICORR—International Conference on Rehabilitation Robotics, Seattle, WA, USA, 24–26 June 2013; pp. 1–7. [Google Scholar]
- Jayasumana, S.; Hartley, R.; Salzmann, M.; Li, H.; Harandi, M. Kernel Methods on Riemannian Manifolds with Gaussian RBF Kernels. IEEE Trans. Pattern Anal. Mach. Intell.
**2015**, 37, 2464–2477. [Google Scholar] [CrossRef] [PubMed] - Jaquier, N.; Castellini, C.; Calinon, S. Improving Hand and Wrist Activity Detection Using Tactile Sensors and Tensor Regression Methods on Riemannian Manifolds. In Proceedings of the Myoelectric Controls Symposium (MEC), Fredericton, NB, Canada, 15–18 August 2017. [Google Scholar]
- Nielsen, J.L.G.; Holmgaard, S.; Jiang, N.; Englehart, K.B.; Farina, D.; Parker, P.A. Simultaneous and Proportional Force Estimation for Multifunction Myoelectric Prostheses Using Mirrored Bilateral Training. IEEE Trans. Biomed. Eng.
**2011**, 58, 681–688. [Google Scholar] [CrossRef] [PubMed] - Sierra González, D.; Castellini, C. A realistic implementation of ultrasound imaging as a human-machine interface for upper-limb amputees. Front. Neurorobot.
**2013**, 7, 17. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Jiang, N.; Vujaklija, I.; Rehbaum, H.; Graimann, B.; Farina, D. Is Accurate Mapping of EMG Signals on Kinematics Needed for Precise Online Myoelectric Control? IEEE Trans. Neural Syst. Rehabilit. Eng.
**2014**, 22, 549–558. [Google Scholar] [CrossRef] [PubMed] - Ortiz-Catalan, M.; Rouhani, F.; Brånemark, R.; Håkansson, B. Offline accuracy: A potentially misleading metric in myoelectric pattern recognition for prosthetic control. In Proceedings of the 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Milano, Italy, 25–29 August 2015; pp. 1140–1143. [Google Scholar]

**Figure 1.**(

**a**) A participant equipped with the two sensor devices. The proximal device is a shape-conformable tactile bracelet with 10 pressure sensor modules. The distal device is composed of 10 surface electromyography (sEMG) sensors linked to a wireless acquisition device. (

**b**) The two sensor devices lain down, as shown from the side in contact with the skin. In the upper part, the tactile bracelet is linked via a mini USB cable to the computer. In the lower part, the sEMG Ottobock electrodes are linked to a data acquisition board transmitting the data via Bluetooth to the same computer. The spacing of the sensors/modules of each bracelet could be easily changed to adapt to the circumference of the forearm of each participant.

**Figure 2.**Functional block representation of the analog-to-digital converter (ADC) board for the surface electromyography (sEMG) sensors (reproduced with permission from [15]).

**Figure 5.**Testing part of the experiment: the participant imitates the grey animated hand model and controls the skin-colored model. The green smiling face indicates a successful task.

**Figure 6.**Comparison of the success rate (SR) of Gaussian process regression (GPR) and ridge regression (RR).

**Figure 7.**Comparison of the time to complete task (TCT) and the time in the target (TIT) between Gaussian process regression (GPR) and ridge regression (RR), in the case of successful and failed tasks, respectively.

**Figure 8.**Comparison of the root-mean-square error (RMSE) between Gaussian process regression (GPR) and ridge regression (RR) for the different movements executed by the participants.

**Figure 9.**Comparison of the root-mean-square error (RMSE) between Gaussian process regression (GPR) and ridge regression (RR) for different activation levels.

**Figure 10.**Performance of ridge regression (RR) and Gaussian process regression (GPR) with different distances using surface electromyography (sEMG) (

**a**), tactile myography (TMG) (

**b**), or both (

**c**) as input for the regression models.

**Figure 11.**Comparison of the root-mean-square error (RMSE) between Gaussian process regression (GPR) using different kernel distances to combine surface electromyography (sEMG) and tactile myography (TMG) as input for the different movements executed by the participants.

**Table 1.**Performance comparison in term of success rate (SR), time to complete the task (TCT) and time in the target (TIT) between Gaussian process regression (GPR) and ridge regression (RR) in experiment 1.

Regression Method | SR (%) | TCT (s) | TIT (s) | RMSE |
---|---|---|---|---|

GPR (${d}_{\mathrm{Eucl}}$) | $55.35\%\pm 17.28\%$ | $4.58\pm 3.02$ | $0.13\pm 0.46$ | $0.58\pm 0.08$ |

RR | $35.60\%\pm 12.24\%$ | $5.56\pm 3.46$ | $0.23\pm 0.56$ | $0.78\pm 0.12$ |

**Table 2.**Performance in term of root mean square error (RMSE) of ridge regression (RR) and Gaussian process regression (GPR) with different distances using surface electromyography (sEMG) or tactile myography (TMG) as input for the regression model.

sEMG | RMSE | TMG | RMSE | |
---|---|---|---|---|

GPR (${d}_{\mathrm{Eucl}}$) | $0.69\pm 0.12$ | GPR (${d}_{\mathrm{Eucl}}$) | $0.31\pm 0.05$ | |

GPR (${d}_{\mathrm{logSPD}}$) | $0.46\pm 0.05$ | GPR (${d}_{\mathrm{logSPD}}$) | $0.31\pm 0.05$ | |

GPR (${d}_{\mathrm{SPD}}$) | $0.58\pm 0.05$ | GPR (${d}_{\mathrm{SPD}}$) | $0.32\pm 0.06$ | |

GPR (${d}_{\mathrm{KL}}$) | $0.38\pm 0.12$ | |||

RR | $0.47\pm 0.06$ | RR | $0.41\pm 0.23$ |

**Table 3.**Performance in term of root mean square error (RMSE) of ridge regression (RR) and Gaussian process regression (GPR) with different distances combining surface electromyography (sEMG) and tactile myography (TMG) as input for the regression models.

sEMG | ${\mathit{d}}_{\mathbf{Eucl}}$ | ${\mathit{d}}_{\mathbf{logSPD}}$ | ${\mathit{d}}_{\mathbf{SPD}}$ | ||||
---|---|---|---|---|---|---|---|

TMG | |||||||

${\mathit{d}}_{\mathbf{Eucl}}$ | $0.29\pm 0.05$ | $0.31\pm 0.05$ | $0.31\pm 0.06$ | RR | $0.39\pm 0.25$ | ||

${\mathit{d}}_{\mathbf{SPD}}$ | $0.31\pm 0.05$ | $0.32\pm 0.06$ | $0.32\pm 0.06$ | ||||

${\mathit{d}}_{\mathbf{KL}}$ | $0.31\pm 0.06$ | $1.15\pm 2.08$ | $2.02\pm 5.01$ |

**Table 4.**Computation (comp.) time for the different kernel distances using surface electromyography (sEMG) or tactile myography (TMG) as input for the regression model. The time to compute training distances is the time needed to compute the distances between all training data points. The time to compute testing distances is the time needed to compute the distance of a new test data point with all training data points.

sEMG | Training Distances Comp. Time (s) | Testing Distances Comp. Time (s) |

${d}_{\mathrm{Eucl}}$ | $0.37\pm 0.01$ | $0.001\pm 0.001$ |

${d}_{\mathrm{logSPD}}$ | $34.18\pm 0.96$ | $0.046\pm 0.001$ |

${d}_{\mathrm{SPD}}$ | $2.75\pm 0.1$ | $0.006\pm 0.001$ |

TMG | Training Distances Comp. Time (s) | Testing Distances Comp. Time (s) |

${d}_{\mathrm{Eucl}}$ | $1.35\pm 0.06$ | $0.003\pm 0.000$ |

${d}_{\mathrm{SPD}}$ | $1.94\pm 0.06$ | $0.077\pm 0.007$ |

${d}_{\mathrm{KL}}$ | $49.31\pm 2.1$ | $0.184\pm 0.010$ |

© 2017 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Jaquier, N.; Connan, M.; Castellini, C.; Calinon, S.
Combining Electromyography and Tactile Myography to Improve Hand and Wrist Activity Detection in Prostheses. *Technologies* **2017**, *5*, 64.
https://doi.org/10.3390/technologies5040064

**AMA Style**

Jaquier N, Connan M, Castellini C, Calinon S.
Combining Electromyography and Tactile Myography to Improve Hand and Wrist Activity Detection in Prostheses. *Technologies*. 2017; 5(4):64.
https://doi.org/10.3390/technologies5040064

**Chicago/Turabian Style**

Jaquier, Noémie, Mathilde Connan, Claudio Castellini, and Sylvain Calinon.
2017. "Combining Electromyography and Tactile Myography to Improve Hand and Wrist Activity Detection in Prostheses" *Technologies* 5, no. 4: 64.
https://doi.org/10.3390/technologies5040064