BROAD—A Benchmark for Robust Inertial Orientation Estimation

Round 1

Reviewer 1 Report

The authors make the point that openly available datasets for objective performance assessment of IOE algorithms are lacking in terms of data quality and key features and propose the Berlin Robust Orientation Estimation Assessment Dataset (BROAD) as a benchmark dataset for orientation estimation. BROAD, consists of a diverse collection of 39 trials, covering different movement types, speeds, and both undisturbed motions as well as motions with deliberate accelerometer disturbances as well as motions performed in the presence of magnetic disturbances. The proposed paper is interesting in the sense that it indeed provides a more thorough and cleaner dataset collected with IMU and synchronized to a gold standard  (optical system) under different conditions. It also offers a code to implements the evaluation and benchmark metrics described. Overall, this is a good contribution and the dataset will provide better data to test and run optimization on inertial orientation estimation (IOE) algorithms. However, some things are overlooked by the authors.

Some minor revisions:

The main point made by the authors is that BROAD is there to address limits found in existing publicly available datasets for IOE. Compared to the cited datasets, the proposed dataset is indeed a better contribution toward achieving benchmarking of OIE algorithms. However, its uniqueness and scope are not as broad as the authors make it in the manuscript. Specifically, 1) with open data and open access policies a lot of other datasets used in biomechanical research published on IMU could be accessible on demand by the readers. A lot of these studies used a thorough optical motion capture system synchronized with IMU data collected at high sampling frequency and derived the same benchmark metrics described using the same approach provided in the manuscript (Coordinate System Alignment, Quaternion annotation, etc…). These studies were done in human subjects performing different tasks under controlled conditions and some also used Gimbal systems. Their argumentation in the introduction and literature review of publicly available datasets and their limits needs to take into account this. They should also in the discussion push that BROAD is an initiative to further broaden the development of IOE but that previous published biomechanical studies on IMU should make efforts to make their dataset available and maybe adopt their data structure and metrics so that this becomes a stepping-stone to maybe a global data repository of datasets for that purpose. 2) While the proposed dataset offers a strong ground truth and variation in conditions, the data collected were for one IMU, in motions that what one would not consider human motions with nor real control over the velocity. Depending on the application targeted by the IOE, this could be limiting, especially for human biomechanics. Please add some of this consideration in the discussion and maybe open / propose how your existing dataset could be enhanced and where you see this going. 

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

General comments

This paper proposes to describe how a benchmark for robust inertial orientation estimation was created. This benchmark is by the way made public.

The idea is very good since there is still now consensus on the best fusion algorithms. If broadly used, this benchmark should help to compare the proposals.  

Moreover, the authors did think to consider a large number of details such as movement type and velocity, magnetic disturbance. We can regret that the trials were not very long, 358s at most. Now, we know that the orientation drift is an important issue in some applications.

We can also regret that only one IMU brand was used since the tuning of the sensors and maybe the best fusion algorithm should depend on the sensors’ characteristics. However, it is obvious that it is not easy to apply a protocol like that to sensors from different brands.   

Regarding the protocol, one can wonder why the ground truth was obtained only at 120Hz. The authors could have provided the characteristics of the sensors in terms of error and noise.

The authors could have detailed a little bit more the consequences of the magnetic disturbance in terms of duration and values.

Minor remark

There is a small error in the authors’ affiliations (4 for 3).

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 3 Report

please, see attached file

Comments for author File: Comments.pdf

Author Response

Please see the attachment.

Author Response File: Author Response.pdf