# A Fully-Automatic Gap Filling Approach for Motion Capture Trajectories

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Related Work

## 3. Proposed Method

#### 3.1. Initialization

#### 3.1.1. Auxiliary Markers

#### 3.1.2. Unscented Kalman Filter

#### 3.2. Gap Filling Algorithm

#### 3.2.1. Frame-Wise Selection of Markers for Reconstruction Assistance

- If ${A}_{i}$ includes more than four non-missing markers in frame f, ${R}_{i}\left(f\right)$ shall consist of the subset of non-missing markers of ${A}_{i}$ in f;
- Otherwise, ${R}_{i}\left(f\right)={A}_{i}$, i.e., ${R}_{i}\left(f\right)$ may include missing markers.

#### 3.2.2. Least Squares Minimization of Systems of Equations

**x**) consists of an array that can be expressed as

**x**$=[{x}_{1},{y}_{1},{z}_{1}]$, where ${x}_{1}$, ${y}_{1}$ and ${z}_{1}$ represent the Cartesian coordinates of marker one. Markers that fulfill condition two are considered simultaneously if they can contribute towards each other’s reconstruction. Using the example of Table 1, this means that markers four and five will be considered in the same nonlinear problem, since they are included in each other’s set of reconstruction markers. Thus, in this case, the independent variable can be expressed as

**x**$=[{x}_{4},{y}_{4},{z}_{4},{x}_{5},{y}_{5},{z}_{5}]$.

**x**with reference to the n variables. Let us consider, for each frame $f\in \{1,2,\dots ,F\}$ (where F corresponds to the total number of frames in the trajectory), the following representations:

- j: A reconstruction marker of marker i in frame f, where marker $j\in {R}_{i}\left(f\right)$;
- ${d}_{ij}\left(f\right)$: Euclidean distance between markers i and j in frame f;
- ${\mu}_{{d}_{ij}}\left(f\right)$: Average distance between markers i and j over the previous f-1 frames;
- ${\sigma}_{{d}_{ij}}\left(f\right)$: Standard deviation of the distance between markers i and j over the previous f-1 frames.

**x**. The concept behind this equation is also illustrated by Figure 2. We aim to find an optimal ${X}_{1}$ that minimizes the probability represented by the highlighted area, forcing a semi-rigid behaviour on the segment delimited by markers i and j, which favors the case where ${d}_{ij}\left(f\right)$ approximates ${\mu}_{{d}_{ij}}\left(f\right)$.

- ${m}_{i}\left(f\right)$: coordinates of marker i in frame f;
- ${v}_{i}\left(f\right)$: instant velocity of marker i in f;
- ${v}_{h}\left(f\right)$: instant velocity of marker h in f;
- ${\mu}_{{v}_{i}/{v}_{h}}\left(f\right)$: average of the ratio between the instant velocity of markers i and h over the previous f-1 frames;
- ${\sigma}_{{v}_{i}/{v}_{h}}\left(f\right)$: standard deviation of the ratio between the instant velocity of markers i and h over the previous f-1 frames.

**x**. We aim to find an optimal ${X}_{2}$ that minimizes the probability function represented by (8).

**x**. The normalization by the sphere’s radius confers less penalization of greater distances between $p{s}_{ij}\left(f\right)$ and the missing marker i for larger spheres, since greater distances are likely associated with greater estimation errors.

**x**will be provided by the UKF prediction of position for the missing marker(s) under consideration. Our implementation considered a smooth approximation relative to an absolute value as a loss function of the least squares minimization algorithm.

#### 3.3. Multiple Initialization

## 4. Experiments

#### 4.1. Datasets

#### 4.2. Gap Generator

- Gap length: varying from 0.5, 1 and 2 to 5 s, which corresponds to 60, 120, 240 and 600 frames;
- Temporal location: The temporal location of two non-overlapping gaps were randomly defined for each sample. These non-overlapping gaps could be located at any frame but they were explicitly not located on the initial and final frames of the samples to allow comparison with other methods that, by design, do not allow reconstructions on the initial and final frames of the sample (see Section 4.4). These generated samples are hereinafter referred to as samples with gaps in the middle. As an additional experiment, we also generated samples with gaps in the initial or in the final frames of the trajectories.
- Number of missing markers: The possible values of 1, 2, 4, 6, 8, 10, 12 and 14 missing markers were considered, where the markers that will be dropped were randomly selected. For each possible number of missing markers, we have randomly generated four combinations of markers to drop.

#### 4.3. Algorithm Initialization

#### 4.4. Trajectories Reconstruction

#### 4.5. Evaluation

## 5. Results

## 6. Discussion

#### 6.1. Comparison with Vicon Nexus’ Methods

#### 6.2. Filling Gaps in the Initial and Final Frames

#### 6.3. Framing within State-of-the-Art

#### 6.4. Limitations

## 7. Conclusions

## Author Contributions

## Funding

## Institutional Review Board Statement

## Informed Consent Statement

## Data Availability Statement

## Acknowledgments

## Conflicts of Interest

## References

- Menolotto, M.; Komaris, D.S.; Tedesco, S.; O’Flynn, B.; Walsh, M. Motion Capture Technology in Industrial Applications: A Systematic Review. Sensors
**2020**, 20, 5687. [Google Scholar] [CrossRef] [PubMed] - Van der Kruk, E.; Reijne, M.M. Accuracy of human motion capture systems for sport applications; state-of-the-art review. Eur. J. Sport Sci.
**2018**, 18, 806–819. [Google Scholar] [CrossRef] [PubMed] - Valevicius, A.M.; Jun, P.Y.; Hebert, J.S.; Vette, A.H. Use of optical motion capture for the analysis of normative upper body kinematics during functional upper limb tasks: A systematic review. J. Electromyogr. Kinesiol.
**2018**, 40, 1–15. [Google Scholar] [CrossRef] [PubMed] - Geng, W.; Yu, G. Reuse of Motion Capture Data in Animation: A Review. In Computational Science and Its Applications—ICCSA 2003; Goos, G., Hartmanis, J., van Leeuwen, J., Kumar, V., Gavrilova, M.L., Tan, C.J.K., L’Ecuyer, P., Eds.; Springer: Berlin/Heidelberg, Germany, 2003; Volume 2669, pp. 620–629. [Google Scholar] [CrossRef]
- Camargo, J.; Ramanathan, A.; Csomay-Shanklin, N.; Young, A. Automated gap-filling for marker-based biomechanical motion capture data. Comput. Methods Biomech. Biomed. Eng.
**2020**, 23, 1180–1189. [Google Scholar] [CrossRef] [PubMed] - Cui, Q.; Sun, H.; Li, Y.; Kong, Y. Efficient human motion recovery using bidirectional attention network. Neural Comput. Appl.
**2020**, 32, 10127–10142. [Google Scholar] [CrossRef] - Tits, M.; Tilmanne, J.; Dutoit, T. Robust and automatic motion-capture data recovery using soft skeleton constraints and model averaging. PLoS ONE
**2018**, 13, e0199744. [Google Scholar] [CrossRef] [PubMed] - Howarth, S.J.; Callaghan, J.P. Quantitative assessment of the accuracy for three interpolation techniques in kinematic analysis of human movement. Comput. Methods Biomech. Biomed. Eng.
**2010**, 13, 847–855. [Google Scholar] [CrossRef] - Smolka, J.; Lukasik, E. The rigid body gap filling algorithm. In Proceedings of the 2016 9th International Conference on Human System Interactions (HSI), Portsmouth, UK, 6–8 July 2016; pp. 337–343. [Google Scholar] [CrossRef]
- Liu, G.; McMillan, L. Estimation of missing markers in human motion capture. Vis. Comput.
**2006**, 22, 721–728. [Google Scholar] [CrossRef] - Kucherenko, T.; Beskow, J.; Kjellström, H. A Neural Network Approach to Missing Marker Reconstruction in Human Motion Capture. arXiv
**2018**, arXiv:1803.02665. [Google Scholar] - Aristidou, A.; Cameron, J.; Lasenby, J. Predicting Missing Markers to Drive Real-Time Centre of Rotation Estimation. In Articulated Motion and Deformable Objects; Perales, F.J., Fisher, R.B., Eds.; Springer: Berlin/Heidelberg, Germany, 2008; pp. 238–247. [Google Scholar]
- Li, L.; McCann, J.; Pollard, N.; Faloutsos, C. BoLeRO: A Principled Technique for Including Bone Length Constraints in Motion Capture Occlusion Filling. In Eurographics/ACM SIGGRAPH Symposium on Computer Animation; The Eurographics Association: Goslar, Germany, 2010; p. 10. ISBN 9783905674279. [Google Scholar]
- Federolf, P.A. A Novel Approach to Solve the “Missing Marker Problem” in Marker-Based Motion Analysis That Exploits the Segment Coordination Patterns in Multi-Limb Motion Data. PLoS ONE
**2013**, 8, e78689. [Google Scholar] [CrossRef] [Green Version] - Xiao, J.; Feng, Y.; Hu, W. Predicting missing markers in human motion capture using l1-sparse representation. Comput. Animat. Virtual Worlds
**2011**, 22, 221–228. [Google Scholar] [CrossRef] - Vicon Motion Systems Ltd. Vicon Nexus User Guide. Available online: https://docs.vicon.com/display/Nexus211 (accessed on 8 April 2021).
- Vicon Motion Systems Ltd. Technical Information—FAQs. Available online: https://www.vicon.com/software/nexus/ (accessed on 8 April 2021).
- Burke, M.; Lasenby, J. Estimating missing marker positions using low dimensional Kalman smoothing. J. Biomech.
**2016**, 49, 1854–1858. [Google Scholar] [CrossRef] [Green Version] - Mall, U.; Lal, G.R.; Chaudhuri, S.; Chaudhuri, P. A Deep Recurrent Framework for Cleaning Motion Capture Data. arXiv
**2017**, arXiv:1712.03380. [Google Scholar] - Gløersen, Ø.; Federolf, P. Predicting Missing Marker Trajectories in Human Motion Data Using Marker Intercorrelations. PLoS ONE
**2016**, 11, e0152616. [Google Scholar] [CrossRef] [PubMed] - Tan, C.H.; Hou, J.; Chau, L.P. Motion capture data recovery using skeleton constrained singular value thresholding. Vis. Comput.
**2015**, 31, 1521–1532. [Google Scholar] [CrossRef] - Peng, S.J.; He, G.F.; Liu, X.; Wang, H.Z. Hierarchical block-based incomplete human mocap data recovery using adaptive nonnegative matrix factorization. Comput. Graph.
**2015**, 49, 10–23. [Google Scholar] [CrossRef] - Julier, S.J. The scaled unscented transformation. In Proceedings of the 2002 American Control Conference (IEEE Cat. No. CH37301), Anchorage, AK, USA, 8–10 May 2002; Volume 6, pp. 4555–4559. [Google Scholar]
- Wan, E.A.; Van Der Merwe, R. The unscented Kalman filter for nonlinear estimation. In Proceedings of the IEEE 2000 Adaptive Systems for Signal Processing, Communications, and Control Symposium (Cat. No. 00EX373), Lake Louise, AB, Canada, 4 October 2000; pp. 153–158. [Google Scholar]
- Merriaux, P.; Dupuis, Y.; Boutteau, R.; Vasseur, P.; Savatier, X. A Study of Vicon System Positioning Performance. Sensors
**2017**, 17, 1591. [Google Scholar] [CrossRef] [PubMed] - Van Der Merwe, R.; Wan, E. Sigma-Point Kalman Filters for Probabilistic Inference in Dynamic State-Space Models. Ph.D. Thesis, OGI School of Science & Engineering at OHSU, Portland, OR, USA, 2004. [Google Scholar]
- Carnegie Mellon University. CMU Graphics Lab Motion Capture Database. Available online: http://mocap.cs.cmu.edu/ (accessed on 23 October 2020).
- Müller, M.; Röder, T.; Clausen, M.; Eberhardt, B.; Krüger, B.; Weber, A. Documentation Mocap Database HDM05; Technical Report CG-2007-2; Universität Bonn: Bonn, Germany, 2007; ISSN 1610-8892. [Google Scholar]

**Figure 2.**Normal distribution curve. The probability to minimize using (5) is highlighted.

**Figure 3.**Example of a pair of markers for which their relative movement is mostly constant over the course of the trajectory.

**Figure 4.**Example of an optimal point in the surroundings of the projections over the spheres centered in marker $j\in {R}_{i}\left(f\right)$ and with radius ${d}_{ij}\left(f\right)$.

**Figure 5.**Comparison of gap filling methods performance for each (

**A**) gap length (l) and (

**B**) number of missing markers (n). Vertical error bars represent standard deviation.

**Table 1.**Markers selected to assist the reconstruction of missing markers in a certain frame (example scenario). Missing markers are presented in bold to facilitate the analysis.

Missing Marker | Auxiliary Markers | Reconstruction Markers |
---|---|---|

1 | ${A}_{1}=\{\mathbf{2},6,7,8,9,10\}$ | ${R}_{1}\left(f\right)=\{6,7,8,9,10\}$ |

2 | ${A}_{2}=\{\mathbf{1},6,7,8,9,10\}$ | ${R}_{2}\left(f\right)=\{6,7,8,9,10\}$ |

3 | ${A}_{3}=\{8,9,10,11,12,13\}$ | ${R}_{3}\left(f\right)=\{8,9,10,11,12,13\}$ |

4 | ${A}_{4}=\{\mathbf{5},14,15,16\}$ | ${R}_{4}\left(f\right)=\{\mathbf{5},14,15,16\}$ |

5 | ${A}_{5}=\{\mathbf{4},17,14,15,16\}$ | ${R}_{5}\left(f\right)=\{\mathbf{4},17,14,15,16\}$ |

Database | Filename | Activity | # Frames | Duration (s) |
---|---|---|---|---|

CMU | 14_01 | Boxing | 5593 | 46.6 |

14_09 | Reaching | 3286 | 27.4 | |

HDM05 | bd_01-01_01_120 | Walking | 9841 | 82.0 |

bk_03-01_01_120 | Dancing | 9700 | 80.8 |

**Table 3.**Summary of the performance of our method per activity using the sequences with gaps and the original sequences (without gaps) as calibration trajectories during initialization. Mean (standard deviation) of RMSE are presented in centimeters.

Calibration Trajectory | Boxing | Reaching | Walking | Dancing |
---|---|---|---|---|

Sequences with gaps | 3.2 (1.7) | 3.9 (1.7) | 2.8 (1.2) | 4.6 (2.8) |

Sequence without gaps | 3.1 (1.2) | 3.9 (1.7) | 2.9 (1.2) | 4.5 (2.7) |

Statistic | Ours | Pattern | Rigid Body | Spline |
---|---|---|---|---|

Mean | 3.6 | 9.0 | 9.5 | 158.4 |

Median | 3.1 | 3.9 | 9.5 | 23.4 |

SD | 1.9 | 10.5 | 6.1 | 361.2 |

IQR | 1.6 | 10.7 | 8.3 | 108.7 |

Min | 0.8 | 0.1 | 0.4 | 0.2 |

Max | 13.5 | 52.9 | 31.2 | 4485.5 |

**Table 5.**Summary of the performance per activity. Mean (standard deviation) of RMSE are presented, in cm.

Method | Boxing | Reaching | Walking | Dancing |
---|---|---|---|---|

Ours | 3.1 (1.2) | 3.9 (1.7) | 2.9 (1.2) | 4.5 (2.7) |

Pattern | 3.2 (1.3) | 4.9 (4.1) | 12.1 (12.2) | 15.8 (12.9) |

Rigid body | 12.5 (4.5) | 12.0 (6.9) | 4.3 (4.1) | 9.3 (4.6) |

Spline | 213.8 (313.3) | 315.5 (599.8) | 35.5 (56.0) | 69.0 (100.0) |

**Table 6.**Performance of the four gap filling methods (ours, pattern fill, rigid body fill and spline fill) in each tested condition. Mean (standard deviation) of RMSE is presented in cm.

l | Method | $\mathit{n}=1$ | $\mathit{n}=2$ | $\mathit{n}=4$ | $\mathit{n}=6$ | $\mathit{n}=8$ | $\mathit{n}=10$ | $\mathit{n}=12$ | $\mathit{n}=14$ |
---|---|---|---|---|---|---|---|---|---|

60 | Ours | 3.1 (0.7) | 3.0 (0.6) | 3.4 (0.5) | 3.4 (0.6) | 3.3 (0.7) | 3.5 (1.0) | 3.3 (0.7) | 4.3 (2.3) |

Pattern | 1.5 (1.0) | 1.4 (0.8) | 5.2 (9.8) | 4.5 (8.2) | 12.9 (11.2) | 9.4 (10.7) | 13.6 (13.1) | 13.8 (15.0) | |

Rigid body | 4.5 (5.0) | 3.9 (2.6) | 4.9 (4.0) | 4.8 (3.1) | 5.0 (3.6) | 4.9 (3.2) | 5.4 (3.5) | 5.3 (3.3) | |

Spline | 4.5 (4.7) | 4.8 (3.8) | 4.3 (3.8) | 4.3 (3.5) | 5.1 (3.7) | 4.9 (3.9) | 5.1 (3.5) | 4.8 (3.2) | |

120 | Ours | 2.5 (1.2) | 2.5 (0.9) | 2.6 (0.6) | 2.5 (0.5) | 2.7 (0.6) | 2.7 (0.6) | 3.0 (0.8) | 2.9 (0.8) |

Pattern | 2.7 (1.5) | 2.4 (0.8) | 2.5 (0.8) | 5.9 (8.9) | 12.2 (13.2) | 6.6 (7.4) | 13.1 (9.9) | 14.0 (11.6) | |

Rigid body | 7.7 (5.3) | 6.6 (3.8) | 7.6 (4.2) | 7.4 (3.8) | 8.0 (4.1) | 8.0 (3.7) | 7.9 (3.7) | 7.8 (3.6) | |

Spline | 13.2 (11.6) | 13.7 (9.3) | 15.4 (11.4) | 16.4 (11.7) | 17.8 (11.4) | 18.2 (12.2) | 17.0 (10.3) | 16.6 (10.9) | |

240 | Ours | 2.3 (1.4) | 3.0 (1.5) | 2.7 (1.0) | 3.4 (1.1) | 2.8 (1.1) | 3.4 (1.2) | 4.2 (1.5) | 4.6 (1.7) |

Pattern | 3.1 (1.5) | 7.0 (11.9) | 7.9 (11.0) | 8.7 (9.6) | 11.2 (11.1) | 15.7 (11.8) | 16.0 (12.1) | 18.1 (13.4) | |

Rigid body | 7.8 (5.4) | 13.6 (6.8) | 10.7 (3.9) | 10.8 (4.5) | 10.2 (4.0) | 10.6 (3.8) | 10.6 (3.9) | 10.6 (3.8) | |

Spline | 53.9 (51.8) | 72.3 (48.0) | 70.3 (46.6) | 90.9 (76.2) | 78.2 (61.4) | 59.7 (35.4) | 65.8 (39.7) | 65.3 (37.8) | |

600 | Ours | 3.0 (2.8) | 4.8 (3.1) | 3.5 (1.3) | 5.2 (2.5) | 5.5 (2.3) | 5.0 (2.5) | 5.9 (3.0) | 6.4 (2.8) |

Pattern | 5.4 (3.4) | 6.9 (3.2) | 5.4 (2.2) | 8.3 (5.7) | 13.2 (11.1) | 11.8 (9.8) | 13.9 (10.9) | 14.2 (10.0) | |

Rigid body | 11.5 (6.9) | 14.5 (7.6) | 14.9 (5.9) | 16.3 (5.8) | 15.9 (6.0) | 15.9 (5.6) | 15.0 (5.0) | 15.8 (5.0) | |

Spline | 545.1 (1033.7) | 477.8 (427.0) | 532.4 (578.0) | 578.7 (484.0) | 512.3 (376.1) | 581.0 (430.7) | 559.7 (477.7) | 560.2 (410.7) |

**Table 7.**Performance of our algorithm when reconstructing samples with gaps located in the initial or final frames of the trajectories. Means (standard deviation) of RMSE are presented, in cm, per tested condition.

l | $\mathit{n}=1$ | $\mathit{n}=2$ | $\mathit{n}=4$ | $\mathit{n}=6$ | $\mathit{n}=8$ | $\mathit{n}=10$ | $\mathit{n}=12$ | $\mathit{n}=14$ | Average |
---|---|---|---|---|---|---|---|---|---|

60 | 2.3 (1.9) | 3.0 (3.4) | 3.2 (1.8) | 3.6 (1.9) | 3.0 (1.1) | 3.5 (1.5) | 3.7 (1.4) | 4.8 (1.8) | 3.4 (2.1) |

120 | 2.8 (2.1) | 2.5 (1.8) | 2.6 (1.0) | 2.8 (1.3) | 3.7 (2.8) | 3.3 (1.7) | 3.4 (1.4) | 4.6 (3.1) | 3.2 (2.1) |

240 | 2.5 (1.5) | 4.0 (3.7) | 4.5 (1.6) | 4.8 (2.3) | 4.7 (2.5) | 6.0 (3.4) | 4.9 (2.3) | 5.6 (2.0) | 4.6 (2.7) |

600 | 4.9 (3.9) | 3.9 (2.0) | 5.3 (3.9) | 4.6 (2.6) | 5.2 (3.7) | 6.0 (2.4) | 7.1 (4.7) | 5.9 (3.0) | 5.3 (3.5) |

Average | 3.1 (2.7) | 3.3 (2.9) | 3.9 (2.6) | 3.9 (2.2) | 4.2 (2.8) | 4.7 (2.7) | 4.8 (3.2) | 5.2 (2.6) | 4.1 (2.8) |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Gomes, D.; Guimarães, V.; Silva, J.
A Fully-Automatic Gap Filling Approach for Motion Capture Trajectories. *Appl. Sci.* **2021**, *11*, 9847.
https://doi.org/10.3390/app11219847

**AMA Style**

Gomes D, Guimarães V, Silva J.
A Fully-Automatic Gap Filling Approach for Motion Capture Trajectories. *Applied Sciences*. 2021; 11(21):9847.
https://doi.org/10.3390/app11219847

**Chicago/Turabian Style**

Gomes, Diana, Vânia Guimarães, and Joana Silva.
2021. "A Fully-Automatic Gap Filling Approach for Motion Capture Trajectories" *Applied Sciences* 11, no. 21: 9847.
https://doi.org/10.3390/app11219847