# BROAD—A Benchmark for Robust Inertial Orientation Estimation

^{1}

^{2}

^{BIO}Med Lab—Biomedical Engineering Lab, Politecnico di Torino, 10129 Torino, Italy

^{3}

^{4}

^{*}

## Abstract

**:**

## 1. Introduction

- includes a broad range of different motions at various speeds
- contains separate trials with various deliberate magnetic disturbances
- contains separate trials with disturbances that affect the measured accelerations
- is already time-synchronized and contains ground truth data that requires no further preprocessing.

## 2. Brief Review of Existing Datasets for IOE Validation

#### 2.1. RepoIMU Dataset (T-Stick Trials)

#### 2.2. RepoIMU Dataset (Pendulum Trials)

#### 2.3. Sassari Dataset

#### 2.4. OxIOD Dataset

#### 2.5. EuRoC MAV Dataset

#### 2.6. TUM VI Dataset

#### 2.7. Summary

## 3. Dataset Description

#### 3.1. Trials

- the type of motion: rotation, translation, and combined (rotational and translational motions)
- the speed at which the motion was performed: slow and fast
- whether the trial consists of one uninterrupted continuous motion or of several segments with short breaks in between: no breaks, with breaks
- whether there are deliberate disturbances that affect the accelerometer measurements: undisturbed, tapping, and vibrating smartphone
- the magnetic environment in which the motion takes place: undisturbed (homogeneous indoor magnetic field), stationary magnet, attached magnet, office environment.

^{2}removed) ranges from 0.5 to 23 m/s

^{2}(slow trials: 0.5 to 1.6 m/s

^{2}, fast trials: 1.6 to 23 m/s

^{2}) with peak values (99th percentile) of up to 67 m/s

^{2}. The RMS values of all trials are shown in Figure 4 and cover a wider range than publicly available datasets.

#### 3.2. File Format

**imu_gyr**IMU gyroscope measurements in rad/s**imu_acc**IMU accelerometer measurements in m/s^{2}**imu_mag**IMU magnetometer measurements in $\mathsf{\mu}\mathrm{T}$**opt_quat**OMC ground truth orientation as a unit quaternion (w component first, ENU reference frame)**opt_pos**OMC ground truth position in m**movement**Boolean array (0/1) indicating movement phases**sampling_rate**sampling rate of the measurements in $\mathrm{Hz}$ ($2000/7\mathrm{Hz}\approx 286\mathrm{Hz}$, HDF5: stored as an attribute)**info**short description of the file contents (HDF5: stored as an attribute)

#### 3.3. Example Code

## 4. Materials and Methods

#### 4.1. Hardware Setup

#### 4.2. Data Preprocessing

#### 4.3. Metrics for Orientation Accuracy

#### 4.4. Benchmark Metrics

- run the IOE algorithm on all 39 trials with a given parameter setting
- for each trial, calculate the orientation RMSE (i.e., the RMS of $e\left(t\right)$) while only considering the labeled motion phases
- average all 39 RMSE values.

## 5. Case Study on the Proposed Benchmark Dataset

## 6. Conclusions

- the determination of robust algorithm parameters for a given IOE algorithm that perform well for a broad range of motions and environmental conditions,
- an in-depth analysis of strength and weaknesses of a given IOE algorithm in different scenarios, while considering heading and inclination separately,
- a detailed comparison of the performance of different algorithms with respect to a wide range of possible application and motion scenarios,
- an objective comparison of different literature algorithms as well as targeted development of new algorithms with improved performance by using the well-defined benchmark metrics described in Section 4.4.

## Author Contributions

## Funding

## Institutional Review Board Statement

## Informed Consent Statement

## Data Availability Statement

## Acknowledgments

## Conflicts of Interest

## Abbreviations

BROAD | Berlin robust orientation estimation assessment dataset |

IMU | inertial measurement unit |

IOE | inertial orientation estimation |

MAV | micro aerial vehicle |

OMC | optical motion capture |

RMSE | root mean square error |

## Appendix A. Quaternion Notation

## Appendix B. Time Synchronization

## Appendix C. Coordinate System Alignment

## References

- Seel, T.; Kok, M.; McGinnis, R.S. Inertial Sensors—Applications and Challenges in a Nutshell. Sensors
**2020**, 20, 6221. [Google Scholar] [CrossRef] [PubMed] - Passon, A.; Schauer, T.; Seel, T. Inertial-Robotic Motion Tracking in End-Effector-Based Rehabilitation Robots. Front. Robot. AI
**2020**, 7. [Google Scholar] [CrossRef] [PubMed] - Kotiadis, D.; Hermens, H.J.; Veltink, P.H. Inertial Gait Phase Detection for Control of a Drop Foot Stimulator: Inertial Sensing for Gait Phase Detection. Med. Eng. Phys.
**2010**, 32, 287–297. [Google Scholar] [CrossRef] - Seel, T.; Laidig, D.; Valtin, M.; Werner, C.; Raisch, J.; Schauer, T. Feedback Control of Foot Eversion in the Adaptive Peroneal Stimulator. In Proceedings of the 22nd Mediterranean Conference on Control and Automation, Palermo, Italy, 16–19 June 2014; pp. 1482–1487. [Google Scholar] [CrossRef] [Green Version]
- Nguyen, H.; Lebel, K.; Bogard, S.; Goubault, E.; Boissy, P.; Duval, C. Using Inertial Sensors to Automatically Detect and Segment Activities of Daily Living in People with Parkinson’s Disease. IEEE Trans. Neural Syst. Rehabil. Eng.
**2018**, 26, 197–204. [Google Scholar] [CrossRef] - Werner, C.; Schneider, S.; Gassert, R.; Curt, A.; Demkó, L. Complementing Clinical Gait Assessments of Spinal Cord Injured Individuals Using Wearable Movement Sensors. In Proceedings of the 2020 42nd Annual International Conference of the IEEE Engineering in Medicine Biology Society (EMBC), Montreal, QC, Canada, 20–24 July 2020; pp. 3142–3145. [Google Scholar] [CrossRef]
- Woodman, O.J. An Introduction to Inertial Navigation; Technical Report UCAM-CL-TR-696; Computer Laboratory, University of Cambridge: Cambridge, UK, 2007. [Google Scholar]
- Kok, M.; Hol, J.D.; Schön, T.B. Using Inertial Sensors for Position and Orientation Estimation. Found. Trends® Signal Process.
**2017**, 11, 1–153. [Google Scholar] [CrossRef] [Green Version] - Caruso, M.; Sabatini, A.M.; Laidig, D.; Seel, T.; Knaflitz, M.; Della Croce, U.; Cereatti, A. Analysis of the Accuracy of Ten Algorithms for Orientation Estimation Using Inertial and Magnetic Sensing under Optimal Conditions: One Size Does Not Fit All. Sensors
**2021**, 21, 2543. [Google Scholar] [CrossRef] - Nazarahari, M.; Rouhani, H. 40 Years of Sensor Fusion for Orientation Tracking via Magnetic and Inertial Measurement Units: Methods, Lessons Learned, and Future Challenges. Inf. Fusion
**2020**. [Google Scholar] [CrossRef] - Weber, D.; Guehmann, C.; Seel, T. Neural Networks Versus Conventional Filters for Inertial-Sensor-Based Attitude Estimation. In Proceedings of the 2020 IEEE 23rd International Conference on Information Fusion (FUSION), Rustenburg, South Africa, 6–9 July 2020. [Google Scholar]
- Caruso, M.; Sabatini, A.M.; Knaflitz, M.; Gazzoni, M.; Croce, U.D.; Cereatti, A. Accuracy of the Orientation Estimate Obtained Using Four Sensor Fusion Filters Applied to Recordings of Magneto-Inertial Sensors Moving at Three Rotation Rates. In Proceedings of the 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Berlin, Germany, 23–27 July 2019; pp. 2053–2058. [Google Scholar] [CrossRef]
- de Vries, W.H.K.; Veeger, H.E.J.; Baten, C.T.M.; van der Helm, F.C.T. Magnetic Distortion in Motion Labs, Implications for Validating Inertial Magnetic Sensors. Gait Posture
**2009**, 29, 535–541. [Google Scholar] [CrossRef] - Ligorio, G.; Sabatini, A.M. Dealing with Magnetic Disturbances in Human Motion Capture: A Survey of Techniques. Micromachines
**2016**, 7, 43. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Fan, B.; Li, Q.; Wang, C.; Liu, T. An Adaptive Orientation Estimation Method for Magnetic and Inertial Sensors in the Presence of Magnetic Disturbances. Sensors
**2017**, 17, 1161. [Google Scholar] [CrossRef] - Caruso, M.; Sabatini, A.M.; Knaflitz, M.; Gazzoni, M.; Croce, U.D.; Cereatti, A. Orientation Estimation through Magneto-Inertial Sensor Fusion: A Heuristic Approach for Suboptimal Parameters Tuning. IEEE Sens. J.
**2021**, 21, 3408–3419. [Google Scholar] [CrossRef] - Fan, B.; Li, Q.; Liu, T. How Magnetic Disturbance Influences the Attitude and Heading in Magnetic and Inertial Sensor-Based Orientation Estimation. Sensors
**2018**, 18, 76. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Szczęsna, A.; Skurowski, P.; Pruszowski, P.; Pęszor, D.; Paszkuta, M.; Wojciechowski, K. Reference Data Set for Accuracy Evaluation of Orientation Estimation Algorithms for Inertial Motion Capture Systems. In Computer Vision and Graphics; Chmielewski, L.J., Datta, A., Kozera, R., Wojciechowski, K., Eds.; Lecture Notes in Computer Science; Springer International Publishing: Cham, Switzerland, 2016; pp. 509–520. [Google Scholar] [CrossRef]
- Jędrasiak, K.; Daniec, K.; Nawrat, A. The Low Cost Micro Inertial Measurement Unit. In Proceedings of the 2013 IEEE 8th Conference on Industrial Electronics and Applications (ICIEA), Melbourne, Australia, 19–21 June 2013; pp. 403–408. [Google Scholar] [CrossRef]
- Chen, C.; Zhao, P.; Lu, C.X.; Wang, W.; Markham, A.; Trigoni, N. OxIOD: The Dataset for Deep Inertial Odometry. arXiv
**2018**, arXiv:1809.07491. [Google Scholar] - Burri, M.; Nikolic, J.; Gohl, P.; Schneider, T.; Rehder, J.; Omari, S.; Achtelik, M.W.; Siegwart, R. The EuRoC Micro Aerial Vehicle Datasets. Int. J. Robot. Res.
**2016**, 35, 1157–1163. [Google Scholar] [CrossRef] - Schubert, D.; Goll, T.; Demmel, N.; Usenko, V.; Stückler, J.; Cremers, D. The TUM VI Benchmark for Evaluating Visual-Inertial Odometry. In Proceedings of the 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain, 1–5 October 2018; pp. 1680–1687. [Google Scholar] [CrossRef] [Green Version]
- Lee, G.H.; Achtelik, M.; Fraundorfer, F.; Pollefeys, M.; Siegwart, R. A Benchmarking Tool for MAV Visual Pose Estimation. In Proceedings of the 2010 11th International Conference on Control Automation Robotics & Vision, Singapore, 7–10 December 2010; pp. 1541–1546. [Google Scholar] [CrossRef] [Green Version]
- Madgwick, S.O.H.; Harrison, A.J.L.; Vaidyanathan, R. Estimation of IMU and MARG Orientation Using a Gradient Descent Algorithm. In Proceedings of the 2011 IEEE International Conference on Rehabilitation Robotics, Zurich, Switzerland, 29 June–1 July 2011; pp. 1–7. [Google Scholar] [CrossRef]
- Sabatini, A.M. Kalman-Filter-Based Orientation Determination Using Inertial/Magnetic Sensors: Observability Analysis and Performance Evaluation. Sensors
**2011**, 11, 9182–9206. [Google Scholar] [CrossRef] [PubMed] - Hol, J.D. Sensor Fusion and Calibration of Inertial Sensors, Vision, Ultra-Wideband and GPS. Ph.D. Thesis, Linköping University, Linköping, Sweden, 2011. [Google Scholar]
- Bergamini, E.; Ligorio, G.; Summa, A.; Vannozzi, G.; Cappozzo, A.; Sabatini, A.M. Estimating Orientation Using Magnetic and Inertial Sensors and Different Sensor Fusion Approaches: Accuracy Assessment in Manual and Locomotion Tasks. Sensors
**2014**, 14, 18625–18649. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Mahony, R.; Hamel, T.; Pflimlin, J.M. Nonlinear Complementary Filters on the Special Orthogonal Group. IEEE Trans. Autom. Control
**2008**, 53, 1203–1218. [Google Scholar] [CrossRef] [Green Version] - Ligorio, G.; Sabatini, A.M. A Novel Kalman Filter for Human Motion Tracking With an Inertial-Based Dynamic Inclinometer. IEEE Trans. Biomed. Eng.
**2015**, 62, 2033–2043. [Google Scholar] [CrossRef] [PubMed] - Guo, S.; Wu, J.; Wang, Z.; Qian, J. Novel MARG-Sensor Orientation Estimation Algorithm Using Fast Kalman Filter. J. Sens.
**2017**, 2017, e8542153. [Google Scholar] [CrossRef] [Green Version] - Kuipers, J.B. Quaternions and Rotation Sequences. In Proceedings of the International Conference on Geometry, Integrability and Quantization, Varna, Bulgaria, 1–10 September 1999; Coral Press Scientific Publishing: Sofia, Bulgaria, 2000; pp. 127–143. [Google Scholar] [CrossRef]

**Figure 1.**The accuracy of inertial orientation estimation (IOE) depends on the employed algorithm, the chosen algorithm parametrization, and the specific application scenario. There is a lack of datasets and methods for systematic evaluation of IOE algorithm performance across a broad range of motion characteristics and environmental conditions.

**Figure 2.**Examples of artifacts found in existing datasets. (

**a**) Gyroscope clipping leading to large errors in angular rate strapdown integration. (

**b**) Repeated samples in IMU data, leading to a very low effective sampling rate. (

**c**,

**d**) Two examples of artifacts found in the OMC ground truth orientations, potentially caused by interpolation of gaps and swapped markers.

**Figure 3.**Office environment used to provide a realistic indoor scenario with magnetic disturbances.

**Figure 4.**RMS values of the angular velocity and accelerometer norm for the undisturbed slow, undisturbed fast, and disturbed trials of the proposed dataset in comparison to the publicly available datasets (only considering trials suitable for IOE accuracy evaluation, cf. Section 2). The BROAD dataset covers a wider range of motions than publicly available datasets.

**Figure 5.**Custom 3D-printed rigid body used in the experiments. The IMU is attached to the center of the board using tape. Four reflective optical markers are attached to the ends of the Xshaped structure to increase the distance between markers. A fifth marker is used to ensure that the orientation can uniquely be determined from the marker positions.

**Figure 6.**Illustration of the different local coordinate systems and reference frames. IOE algorithms estimate the orientation of the sensor frame $\mathcal{S}$ with respect to a frame of reference $\mathcal{E}$, defined by gravity and the local magnetic field. The OMC reference system tracks the orientation of a rigid body $\mathcal{B}$, defined by reflective markers, relative to a reference frame $\mathcal{M}$ that is defined during calibration and, in general, does not coincide with $\mathcal{E}$.

**Figure 7.**Decomposition of an exemplary orientation difference into heading and inclination. Heading is a rotation around the vertical axis and inclination is a rotation around a horizontal axis. Note that in contrast to other decompositions that are used in literature, the angles commute.

**Figure 8.**Orientation estimation RMSE (averaged over all trials) obtained with Algorithms A [24] and B [28] for different values of the tuning parameters. For various motions at different speeds, a parameter choice of $\beta =0.12$ yields the lowest overall errors for Algorithm A. For Algorithm B, the parameter combination ${K}_{\mathrm{p}}=0.74$, ${K}_{\mathrm{i}}=0.0012$ yields the lowest overall errors.

**Figure 9.**Averaged RMSE errors for Algorithms A [24] and B [28] for various groups of trials. The bars show errors with the trial-agnostic parameters, and the black dots indicate the minimum error achievable with individual parameters for each trial. The lines originating from the center show the difference of the errors obtained with Algorithm A and B. It can be seen that for most groups of trials, Algorithm A yields smaller errors.

Dataset | Sensor | Sampling Rate | # of Trials | Isolated Translations | Isolated Rotations | Different Velocities | Short and Long Motion Phases | Accelerometer Disturbances | Disturbed Magnetic Fields | Initial Rest Phase | GT Includes Position | GT Is Synchronized | GT Coordinate Frames Aligned | Reproducible Error Metrics Defined |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|

RepoIMU T-stick [18] | Xsens MTi | 100 $\mathrm{Hz}$ | 29 | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ |

RepoIMU Pendulum [18] | custom [19] | 90–166 Hz ^{a} | 22 | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ |

Sassari [16] | 3 models | 100 $\mathrm{Hz}$ | 3 | ✗ | ✓ | ✓ | ✗ | ✗ | ✗ | ✓ | ✓ | ✓ | ✗^{b} | ✓ |

OxIOD [20] | iPhone | 100 $\mathrm{Hz}$ | 132 | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✓ | ✓ | ✓ | ✗ | ✗ |

EuRoC MAV [21] | ADIS16448 | 200 $\mathrm{Hz}$ | 6 ${}^{\mathrm{c}}$ | ✗ | ✗ | ✓ | ✗ | ✓ | ✗ | ✗ | ✓ | ✓ | ✓ | ✗ |

TUM VI [22] | BMI160 | 200 $\mathrm{Hz}$ | 6 ${}^{\mathrm{d}}$ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✓ | ✓ | ✗ |

BROAD | myon aktos-t | 286 $\mathrm{Hz}$ | 39 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |

undisturbed | slow | fast | ||

rotation | 1, 2, 3, 4 *, 5 * | 6, 7, 8 *, 9 * | ||

translation | 10, 11, 12, 13 *, 14 * | 15, 16, 17 *, 18 * | ||

combined | 19, 20 | 21, 22, 23 | ||

disturbed (medium speed) | ||||

tapping | 24, 25 | |||

vibrating smartphone | 26, 27 | |||

stationary magnet | 28, 29, 30 *, 31 * | |||

attached magnet (1– 5 $\mathrm{c}$$\mathrm{m}$) | 32, 33, 34, 35, 36 | |||

office environment | 37, 38 | |||

mixed (disturbed and undisturbed) | 39 * |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Laidig, D.; Caruso, M.; Cereatti, A.; Seel, T.
BROAD—A Benchmark for Robust Inertial Orientation Estimation. *Data* **2021**, *6*, 72.
https://doi.org/10.3390/data6070072

**AMA Style**

Laidig D, Caruso M, Cereatti A, Seel T.
BROAD—A Benchmark for Robust Inertial Orientation Estimation. *Data*. 2021; 6(7):72.
https://doi.org/10.3390/data6070072

**Chicago/Turabian Style**

Laidig, Daniel, Marco Caruso, Andrea Cereatti, and Thomas Seel.
2021. "BROAD—A Benchmark for Robust Inertial Orientation Estimation" *Data* 6, no. 7: 72.
https://doi.org/10.3390/data6070072