Next Article in Journal
Subjective Response Measurement to Prosthesis or Device Use: Validation of the Prosthetic–Bionic Paradigm Questionnaire (PBP-Q)
Next Article in Special Issue
Effects of Patient Education on Pain and Function and Its Impact on Conservative Treatment in Elderly Patients with Pain Related to Hip and Knee Osteoarthritis: A Systematic Review
Previous Article in Journal
Risk Perception towards COVID-19: A Systematic Review and Qualitative Synthesis
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
IJERPH
Volume 19
Issue 8
10.3390/ijerph19084655
ijerph-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Validity and Absolute Reliability of the Cobb Angle in Idiopathic Scoliosis with TraumaMeter Software

by
José Hurtado-Avilés
1,
Fernando Santonja-Medina
1,2,3,
Vicente J. León-Muñoz
4,*,
Pilar Sainz de Baranda
1,5,
Mónica Collazo-Diéguez
1,6,
Mercedes Cabañero-Castillo
7,
Ana B. Ponce-Garrido
7,
Victoria Eugenia Fuentes-Santos
1,6,
Fernando Santonja-Renedo
8,
Miriam González-Ballester
9,
Francisco Javier Sánchez-Martínez
9,
Pietro Gino Fiorita
9,
Jose Manuel Sanz-Mengibar
1,10,
Joaquín Alcaraz-Belzunces
9,
Vicente Ferrer-López
11 and
Pilar Andújar-Ortuño
1,6
1
Sports & Musculoskeletal System Research Group (RAQUIS), University of Murcia, 30100 Murcia, Spain
2
Department of Orthopaedic Surgery and Traumatology, Hospital Clínico Universitario Virgen de la Arrixaca, 30120 Murcia, Spain
3
Department of Surgery, Pediatrics and Obstetrics & Gynecology, Faculty of Medicine, University of Murcia, 30100 Murcia, Spain
4
Department of Orthopaedic Surgery and Traumatology, Hospital General Universitario Reina Sofía, 30003 Murcia, Spain
5
Department of Physical Activity and Sport, Faculty of Sport Sciences, Regional Campus of International Excellence “Campus Mare Nostrum”, University of Murcia, 30100 Murcia, Spain
6
Department of Rehabilitation Sciences and Physiotherapy, Albacete University Hospital Complex, 02006 Albacete, Spain
7
Department of Rehabilitation Sciences and Physiotherapy, General Hospital, Almansa, 02640 Albacete, Spain
8
“La Vega Lorenzo Guirao” Hospital, 30530 Cieza, Spain
9
Medicine Faculty, University of Murcia, 30100 Murcia, Spain
10
Centre for Neuromuscular Diseases, National Hospital for Neurology and Neurosurgery, University College London Hospitals NHS Foundation Trust, London WC1N 3BG, UK
11
Department of Physiotherapy, Faculty of Medicine, University of Murcia, 30100 Murcia, Spain
 
add Show full affiliation list
 
remove Hide full affiliation list
*
Author to whom correspondence should be addressed.
Int. J. Environ. Res. Public Health 2022, 19(8), 4655; https://doi.org/10.3390/ijerph19084655
Submission received: 17 March 2022 / Revised: 8 April 2022 / Accepted: 10 April 2022 / Published: 12 April 2022
(This article belongs to the Special Issue Bone and Joint Health and Rehabilitation)
Browse Figures
Review Reports Versions Notes

Abstract

:
The Cobb angle value is a critical parameter for evaluating adolescent idiopathic scoliosis (AIS) patients. This study aimed to evaluate a software’s validity and absolute reliability to determine the Cobb angle in AIS digital X-rays, with two different degrees of experienced observers. Four experts and four novice evaluators measured 35 scoliotic curves with the software on three separate occasions, one month apart. The observers re-measured the same radiographic studies on three separate occasions three months later but on conventional X-ray films. The differences between the mean bias errors (MBE) within the experience groups were statistically significant between the experts (software) and novices (manual) (p < 0.001) and between the novices (software) and novices (manual) (p = 0.005). When measured with the software, the intra-group error in the expert group was MBE = 1.71 ± 0.61° and the intraclass correlation coefficient (ICC (2,1)) = 0.986, and in the novice group, MBE = 1.9 ± 0.67° and ICC (2,1) = 0.97. There was almost a perfect concordance among the two measurement methods, ICC (2,1) = 0.998 and minimum detectable change (MCD95) < 0.4°. Control of the intrinsic error sources enabled obtaining inter- and intra-observer MDC95 < 0.5° in the two experience groups and with the two measurement methods. The computer-aided software TraumaMeter increases the validity and reliability of Cobb angle measurements concerning manual measurement.

1. Introduction

Adolescent idiopathic scoliosis (AIS) is a three-dimensional deformity involving the axial, sagittal, and frontal planes [1]. AIS can progress over the years, especially during growth, and can cause musculoskeletal, lung, and psychological problems and significant pain in adulthood [2]. The Cobb angle (described by John Robert Cobb in 1948) measurement on the standing posteroanterior full-length spine X-ray is the gold standard for diagnosing and monitoring AIS changes [3]. Cobb angle measurement is necessary to assess the severity of scoliosis and to quantify the risk of progression [4,5,6,7], for the selection of treatment [3,5,8], and the analysis of orthopaedic and surgical procedures [3,4,6,9,10,11] and the effectiveness of treatment [12,13,14]. The Cobb angle is the most important measurement next to vertebral rotation on AIS radiographs [3,4], as it is necessary to establish the diagnosis and decide on the treatment. A sufficiently large error in the Cobb angle measurement can mean, for example, that the indicated treatment varies from observation to orthopaedic therapy or from bracing to instrumented arthrodesis.
Using a computerised tomography (CT) scan, a three-dimensional reconstruction of the spine can be obtained to quantify AIS with a high level of accuracy [1]. However, the CT scan is not suitable for monitoring scoliotic progression because of the excessive and repeated radiation (e.g., an estimated radiation dose of 5.2 mSv for each study [15]). Radiographic medical imaging, especially the standing posteroanterior full-length spine X-ray [16,17,18], continues to be the method of choice for diagnosing and monitoring scoliosis [19].
Traditionally, an increase in the Cobb angle of 5° between successive measurements indicates scoliosis progression [12,20,21,22,23]. Although conventional Cobb angle scoliosis measurement is a simple technique, there are numerous studies of manual measurements of Cobb angle with an average inter-observer variability greater than 5° [12,13,24,25,26,27,28,29,30]. Potential sources of intrinsic error for Cobb angle measurement are poor-quality digital X-rays images, the incorrect definition of the cranial and caudal vertebrae, variable width markers/pencils, different protractors, inaccurate drawing of the lines along the vertebral endplates, imprecise drawing of perpendicular lines, and inaccurate angle measurement itself [12,17,30,31,32,33,34,35,36]. Since the establishment of digital medical imaging, several authors have developed computer-assisted measurement systems to measure the Cobb angle on digital AIS images. These computer programmes avoid sources of intrinsic error compared to conventional measurement on X-ray films [17,28,31,34,36,37,38,39,40]. Using these systems, different authors reported intra-observer MBE between 1° and 2° [31,34,37,38] and between 2° and 4° [17,28,36].
The present study aims to: (1) evaluate the intra- and inter-observer absolute reliability and validity of a computer-aided Cobb measurement method designed to reproduce the manual Cobb method in AIS digital images, focused on reducing intrinsic error sources in two groups of observers, novices and experts; (2) investigate if the developed software is sensitive to observer skill levels or experiences; and (3) compare the software method with the manual Cobb method. We have hypothesised that the intra-observer error in Cobb angle measurements is less than 2.5° in novices and even better (less than 2°) in experts and that the use of the software TraumaMeter improves the validity and reliability of the measurements obtained compared to the manual method.

2. Materials and Methods

2.1. Software

We developed a computer-aided measurement system (TraumaMeter v.873, José Hurtado Avilés and Fernando Santonja Medina, registration number 08/2021/374, Murcia, Spain) that digitally reproduces the manual Cobb angle measurement method on digital X-ray images [41,42]. The software was developed in C++ language under the Microsoft Visual Studio 2019 (version 16.3.5, Microsoft Corporation, Redmond, WA, USA) development environment using the OpenCV 3.4.10 (Intel Corporation, Santa Clara, CA, USA) artificial vision libraries and the DCMTK libraries, from OFFIS, Institute for Information Technology, to operate with DICOM (digital imaging and communication on medicine) files. The software incorporates additional tools, such as the ability to zoom in on regions of interest and to vary the contrast (fractional difference in optical density of the brightness between two regions of an image) of the digitalised X-ray image.
The system allows the evaluator to choose several cranial and caudal vertebrae from the curve, with the software selecting the most tilted ones, returning a Cobb angle result expressed in degrees, as shown in Figure 1. To measure the Cobb angle, the observer opens the X-ray image, enlarges the vertebra, and selects with a mouse click the two points defining the lines tangent to the cranial endplate of the curve and the two caudal endplate points.

2.2. Study Design and Measurement Protocol

The validity and reliability of the traditional manual measurement method were studied to validate the software, focusing on decreasing the sources of intrinsic measurement error. We conducted a prospective and observational study of 35 scoliotic curves in 21 selected standing frontal full-length spine X-rays of patients with AIS. The X-ray sample was homogeneous, had equivalent image quality, and had no defects.
The radiographic images were collected from an image repository in a retrospective manner during the routine medical care of patients with AIS. Our study followed the World Medical Association Declaration of Helsinki’s ethical standards, as revised in 2013. The study was granted exemption from requiring ethics approval since the complete and irreversible anonymisation of the images did not involve data processing. The X-ray images were obtained natively in digital format (in DICOM, with a resolution of 283.46 pixels/mm) and printed in 350 × 430 mm format.
The selected X-rays showed, according to the angular classification proposed by the International Society on Scoliosis Orthopaedic and Rehabilitation Treatment [43]: low scoliosis in 9 cases (curves between 11° to 20°), moderate scoliosis in 11 cases (curves between 21° and 35°), moderate to severe scoliosis in 6 cases (curves between 36° and 40°), severe scoliosis in 4 cases (curves between 41° and 50°), severe to very severe scoliosis in 3 cases (curves between 51° and 55°), and very severe scoliosis in 2 cases (curves with 56° or more).
We assessed absolute reliability according to the Hopkins criteria (minimum n of 30 cases, at least six blinded observers as assessors, and at least three tests per observer, separated by at least two weeks) [44,45]. We also assessed validity.
The research was carried out with eight independent evaluators with different experience levels in measuring Cobb angles. Four observers, considered “Experts”, were an orthopaedic specialist and three physical therapy and rehabilitation specialists who are accustomed to measuring spinal misalignments in their daily practice. The division of the observers into the expert and novice groups was made based on the frequency with which they use the Cobb method rather than based on their medical speciality. We considered experts to be observers that were very often involved in the follow-up and monitoring of patients with AIS. Four “novice” observers were professionals from different health sciences branches (not orthopaedists) and who, although they knew the theory of how to make measurements on X-rays of the spine, had never measured with Cobb’s method.
In each of the 21 X-rays, each observer identified the primary curve and the secondary or compensatory curve and measured them with the software on three occasions separated by one month (Table S1a). To validate the software, the observers re-measured the same radiographic studies three months later but on X-ray films (analogical radiographs) in a conventional manual way (Table S1b). The conventional measurement was also repeated on three occasions, one month apart. To avoid bias, the sequence in which the radiographs were presented was randomly assigned in each measurement round by the study coordinator, who kept the randomisation key confidential. In total, 1680 Cobb angles were measured for this study (210 by each observer).
A 5 h briefing was held before the software TraumaMeter v.873 measurements, with comprehensive information on the study and training in software use. Similarly, one month after completing the measurements with the software and before the manual measurements, a briefing session was held with Cobb’s method’s relevant indications for the correct measurement. The observers received the 21 X-ray films, the same kind of ruler, square, bevel, permanent black fine-point ink marker, and the same protractor and transparent acetate sheets for the manual measurements to mark the reference points and measure without leaving any marks or signals on the X-ray images that could alter the results of the investigation.

2.3. Statistics

Statistical analysis was performed using the Statistical Package for the Social Sciences (SPSS), version 25 for Windows (SPSS, Inc., Chicago, IL, USA). The results were rounded to one decimal place in the measurements obtained with the software and obtained with one decimal place in the manual conventional measurement due to the scale of each measuring instrument. The average of the errors at each retest of the four observers in each group was employed to estimate the agreement between each experience group and the different experience groups.
The distributions of measurements for each curve and the error distributions were improved by identifying values lower than Q1 − 1.5 × IQR (interquartile range)) and higher than Q3 + (1.5 × IQR). These values were considered outliers and were eliminated from each distribution. We removed outliers because of their effect on the normality loss in the data distributions. To be able to apply statistical inference methods, these distributions must be sufficiently normal. Table S2a–c show the outliers removed from each distribution.
The Shapiro–Wilk test was used to check that the p-values of the data were above the significance level of 0.05, with the null hypothesis that the data fit a normal distribution being accepted. All distributions met the normality criterion of this test.
We used the 24-measurement mean obtained from the three measurements made by each of the eight assessors for each curve and each method to assess both methods’ concordance.
To analyse intra- and inter-group agreement of the software and manual measurements, we calculated the validity (MBE, mean bias error), the reliability (SD, standard deviation), the standard error of the sample (SE), the minimum detectable change (MDC95), and the intra-class correlation coefficient of absolute concordance using a two-factor random-effects model (ICC (2,1)) [46]. We assessed the intra- and inter-observer reliability according to the criteria by Landis and Koch (<0 indicate no agreement, 0.00 to 0.20 indicate slight agreement, 0.21 to 0.40 indicate fair agreement, 0.41 to 0.60 indicate moderate agreement, 0.61 to 0.80 indicate substantial agreement, and 0.81 to 1.0 indicate almost perfect or perfect agreement) [47]. Although the Landis and Koch criteria are for qualitative estimates, we consider that this criterion can serve as a reference for quantitative determinations by measuring the same thing, i.e., the degree of concordance. We also obtained the Bland–Altmann plot for the agreement between the analysis of the manual and software measurement methods.
We analysed whether the differences in MBE values between each set of measurements were statistically significant using ANOVA and Tukey’s method for multiple comparisons. Student’s t-test for independent samples was used to analyse the two intergroup distributions (obtained with the software and manually).

3. Results

In the ANOVA for the intra-group distributions, we obtain a p < 0.001, and, according to Tukey’s method, the differences between the MBE values are statistically significant at a confidence level of at least 95% between the expert (software) and novice (manual) groups (p < 0.001) and between the novice (software) and novice (manual) groups (p = 0.005) (Figure 2).
As Figure 3 shows, the MBE value of the two inter-group distributions obtained with the software and manually is different when using TraumaMeter or the manual method (p < 0.001).
The Table 1 shows the validity and reliability of the intra- and inter-group measures obtained with both measurement methods.
When measuring with the software, the intra-group error in the expert group was MBE = 1.71°, SD = 0.61°, ICC (2,1) = 0.986 (95% CI: 0.977–0.992) and in the novice group was MBE = 1.9°, SD = 0.67°, ICC (2,1) = 0.97 (95% CI: 0.95–0.985). When measured manually, the intra-group error in the expert group was MBE = 2.13°, SD = 0.75°, ICC (2,1) = 0.981 (95% CI: 0.97–0.99) and in the novice group was MBE = 2.50°, SD = 0.88°, ICC (2,1) = 0.974 (95% CI: 0.954–0.988).
The mean intra-observer error with the software was MBE = 1.8°, SD = 0.65° and when measuring manually was MBE = 2.31°, SD = 0.83°.
In the inter-group study (experts versus novices), when measuring manually the error was MBE = 2.47°, SD = 0.76°, ICC (2,1) = 0.973 (95% CI: 0.951–0.988), and when measuring with the software the error was MBE = 1.82°, SD = 0.59°, ICC (2,1) = 0.973 (95% CI: 0.954–0.987).
The evaluation of the agreement between both measurement methods showed that MBE = 0.08°, SD = 0.844°, SEM = 0.143°, MCD95 = 0.395° and an ICC (2,1) = 0.998 (95% CI: 0.996–0.999). The Bland–Altman graphical representation shows the absence of bias in both method agreements (Figure 4).

4. Discussion

Our research demonstrates that TraumaMeter software allows an inexperienced observer to measure the Cobb angle with high validity and reliability. We have developed and evaluated a computer-aided measurement system that allows a reduction in the factors responsible for the intrinsic error of the traditional manual measurement, such as the selection of the reference points on the vertebral bodies, inaccurate drawing of the lines along the vertebral endplates, the determination of the perpendicular, and the measurement of the angle. Another robustness of the present study is the use groups of observers with different grades of experience. According to our results, no significant improvement is attributable to practice. Neither with the software (ICC equal to 0.979 in the first evaluation, 0.977 in the second, and 0.982 in the third) nor with the manual method (ICC equal to 0.975 in the first evaluation, 0.977 in the second, and 0.978 in the third).
There is a consensus in the literature that the difference between Cobb angle measurements should be at least 5° to ensure a real change [12,20,21,22,23]. However, our research shows that exiguous measurement changes (at most 0.5°) are representative. This aspect could be related to the five-hour training sessions before measuring and the experience gained from performing the three measurement rounds with the software prior to the conventional manual measuring.
A comparison of the validity and reliability results of the Cobb angle on AIS X-ray between different studies is difficult due to the diversity of criteria in their design (different number of X-rays, observers, number of measurement sessions, number of weeks between measurement sessions, pre-selection of the limit vertebrae, or measurement tools used) and due to the format of the results (intra- or inter-observer values, ICC, 95%CI for the mean, or only SD). The reliability analyses between the computer-aided and manual measurement produced ICC > 0.99 with 95% CI: 0.996–0.999. The MCD95 was < 0.4°. In our research, we have employed the criteria of absolute reliability [45], which requires a minimum of 30 cases measured by at least six blinded observers with at least three tests per observer, separated from each other by at least two weeks. Different rulers, variable width markers/pencils, and poor-quality X-rays have also been reported as causes of intrinsic error [12,17,32,35], so we controlled for these variables in our study.
In our study, the observers measured the same scoliotic curves with the software and manually to obtain a more meaningful comparison of both measurement methods’ validity and reliability results. In the study, we set a test-retest time of one month so that the variability of the measurements could not be attributed to the observers remembering the measured radiographs nor the results obtained on them. There were no statistically significant improvements between the successive tests because the training sessions avoided common measurement errors from the beginning of the study. The second possible explanation is that when the precision error is so small (less than one degree), it is difficult to improve the precision with training.
The computer-assisted measurement of the Cobb angle eliminates the sources of intrinsic error [12,30,31,32,34] except the selection of the terminal vertebrae reference points of the scoliotic curve, which is supposed to improve the accuracy of manual measurements since the software allows zooming in on the points of interest and varying the brightness of the medical image for better visualisation and, therefore, better selection of the points [42].
Other authors have reported similar error values to ours in determining the Cobb angle using computer systems. These studies were also designed to avoid intrinsic causes of measurement error by considering a large sample of subjects, several observers, and several measurements repetitions. For instance, Srinivasalu et al. [31] and Zhang et al. [38] considered 318 and 60 X-rays, respectively, measured by three observers, three and two times, respectively, obtaining similar MBE and SD values to those of our study. In contrast to our study, these authors [31,38] did not compare their results with manual measurements obtained under the same conditions.
The value of our study lies in the fact that the developed measurement software reproduces the manual measurement method with minimal computer intervention but eliminates some sources of intrinsic error; following the same methodology and using the same subjects and observers, we have studied the error of the manual Cobb angle measurement method. This methodology allows a better comparison between the methods’ (software and manual) validity. In addition, we considered the Hopkins criteria [45] for calculating reliability as well as two experience groups.
We consider that the methodology followed in the manual measurements (reproduction of the performance of the software in the selection of the most tilted vertebrae) limits the error in the selection of the terminal vertebrae.
From the standpoint of statistical inference, it was necessary to treat the values obtained by the observers and the error distributions to reduce the error in any statistical estimation. If we consider outliers, the DCM results were as follows: intra-observer error for the expert group for the software measures was MCD95 = 0.54° and MCD95 = 0.36° for the manual measures; intra-observer error for the novice group for the software measures was MCD95 = 0.45° and MCD95 = 0.64° for the manual measurements. When measuring with the software, the inter-observer error was MCD95 = 0.42° and MCD95 = 0.49° when measuring manually. These values lead us to believe that eliminating outliers does not produce a significant bias.
To avoid bias in the measurements, we established the procedure to follow by employing training sessions for the observers, distinguishing their level of experience, using a sample of subjects sufficiently representative of the population, and considering the temporal stability of the measurements by repeating them at different times.
There are some limitations to our study. First, to establish the “gold standard” and compare manual and software measurements, we used the mean value of each measurement distribution, which means that each measurement may contain a small error. Second, we did not consider each evaluator’s computer equipment (e.g., viewable image size, display resolution, luminance, and contrast ratio or the characteristics of the mouse or touchpad), which may have influenced the accuracy of the measurements. However, the obtained results (error fewer than 2 degrees) seem to be of little significance and would not preclude extrapolating the results to another population of observers with different computer equipment. Third, we did not consider the effect on measurements obtained manually if they had not been measured beforehand with the TraumaMeter software (we think that the results of manual measurements were improved by the previous learning of measurement with the software). Fourth, although we designed our study to meet the Hopkins criteria, which require a minimum of 30 subjects, we suppose that a larger sample of AIS radiographs would have decreased the likelihood of type II error in our results. Fifth, there was a restriction in selecting the radiographs in terms of the severity of the scoliotic curves, as those with a magnitude of less than 10° were discarded. It was also difficult to obtain radiographs with very severe curves, so the number studied in this severity group was small. Finally, the outliers eliminated in each distribution used in the study may be due to imperfect measurement and errors in recording the value of the measurements in the database provided by each observer. However, they accounted for only 2.44% of the total measurements made (41 of 1680 measurements).
These limitations notwithstanding, the authors believe that the study’s outcomes are valuable. Due to its high validity and reliability, the TraumaMeter v.873 software can be recommended for quantifying AIS curves in clinical practice and research.

5. Conclusions

Intra-observer measurement errors are lower when using the software TraumaMeter (MBE = 1.8°, SD = 0.65°) than when using the conventional manual Cobb angle measurement method (MBE = 2.31°, SD = 0.83°). The MBE value of the inter-group (expert and novice) distributions is statistically different when using TraumaMeter or the manual method. The error in the measurements depends on the observer skill levels or experiences. The use of the software reduces the difference in error between the novice and expert observers in a statistically significant way. The minimum detectable change (MDC95) is equal to or less than 0.5°, irrespective of the observer’s experience and measurement method (TraumaMeter or manual). There is almost a perfect agreement between the TraumaMeter measurement and the manual method.

Supplementary Materials

The following supporting information can be downloaded at: https://www.mdpi.com/article/10.3390/ijerph19084655/s1, Table S1: a: Cobb angle values (in degrees) of each curve obtained by each observer and in each series of measurements with the software; b: Cobb angle values (in degrees) of each curve obtained by each observer and in each series of measurements with the manual method, Table S2: a: Outliers of measurements obtained with the software; b: Outliers of manually obtained measurements; c: Outliers of the error distributions.

Author Contributions

Conceptualization, J.H.-A. and F.S.-M.; Formal analysis, J.H.-A.; Investigation, F.S.-M., M.C.-D., M.C.-C., A.B.P.-G., V.E.F.-S., F.S.-R., M.G.-B., F.J.S.-M., P.G.F., J.M.S.-M., J.A.-B., V.F.-L. and P.A.-O.; Methodology, J.H.-A., F.S.-M. and P.S.d.B.; Project administration, J.H.-A.; Software, J.H.-A. and F.S.-M.; Supervision, F.S.-M.; Validation, J.H.-A., F.S.-M. and V.J.L.-M.; Visualization, J.H.-A. and V.J.L.-M.; Writing—original draft, J.H.-A., F.S.-M. and V.J.L.-M.; Writing—review and editing, J.H.-A., F.S.-M. and V.J.L.-M. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

Our study followed the World Medical Association Declaration of Helsinki’s ethical standards, as revised in 2013. The study was granted exemption from requiring ethics approval since the complete and irreversible anonymisation of the images did not involve data processing.

Informed Consent Statement

Not applicable.

Data Availability Statement

Not applicable.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Ma, Q.; Lin, H.; Wang, L.; Zhao, L.; Chen, M.; Wang, S.; Rao, Z.; Luo, Y. Correlation between Spinal Coronal Balance and Static Baropodometry in Children with Adolescent Idiopathic Scoliosis. Gait Posture 2020, 75, 93–97. [Google Scholar] [CrossRef]
  2. Hefti, F. Pathogenesis and Biomechanics of Adolescent Idiopathic Scoliosis (AIS). J. Child. Orthop. 2013, 7, 17–24. [Google Scholar] [CrossRef] [Green Version]
  3. Kuklo, T.R.; Potter, B.K.; Lenke, L.G. Vertebral Rotation and Thoracic Torsion in Adolescent Idiopathic Scoliosis. J. Spinal Disord. Tech. 2005, 18, 139–147. [Google Scholar] [CrossRef] [PubMed]
  4. Mohanty, S.P.; Pai Kanhangad, M.; Gullia, A. Curve Severity and Apical Vertebral Rotation and Their Association with Curve Flexibility in Adolescent Idiopathic Scoliosis. Musculoskelet. Surg. 2021, 105, 303–308. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  5. Eijgenraam, S.M.; Boselie, T.F.M.; Sieben, J.M.; Bastiaenen, C.H.G.; Willems, P.C.; Arts, J.J.; Lataster, A. Development and Assessment of a Digital X-Ray Software Tool to Determine Vertebral Rotation in Adolescent Idiopathic Scoliosis. Spine J. 2017, 17, 260–265. [Google Scholar] [CrossRef] [PubMed]
  6. Carlson, B.B.; Burton, D.C.; Asher, M.A. Comparison of Trunk and Spine Deformity in Adolescent Idiopathic Scoliosis. Scoliosis 2013, 8, 2. [Google Scholar] [CrossRef] [Green Version]
  7. Vrtovec, T.; Vengust, R.; Likar, B.; Pernuš, F. Analysis of Four Manual and a Computerized Method for Measuring Axial Vertebral Rotation in Computed Tomography Images. Spine 2010, 35, E535–E541. [Google Scholar] [CrossRef]
  8. Tamura, Y.; Sugano, N.; Sasama, T.; Sato, Y.; Tamura, S.; Yonenobu, K.; Yoshikawa, H.; Ochi, T. Surface-Based Registration Accuracy of CT-Based Image-Guided Spine Surgery. Eur. Spine J. 2005, 14, 291–297. [Google Scholar] [CrossRef] [Green Version]
  9. Qiao, J.; Zhu, F.; Xu, L.; Zhu, Z.; Qian, B.; Liu, Z.; Qiu, Y. Comparison of the Aorta Impingement Risks between Thoracolumbar/Lumbar Curves with Different Convexities in Adolescent Idiopathic Scoliosis: A Computed Tomography Study. Eur. Spine J. 2012, 21, 2043–2049. [Google Scholar] [CrossRef] [Green Version]
  10. Vrtovec, T.; Pernuš, F.; Likar, B. A Review of Methods for Quantitative Evaluation of Axial Vertebral Rotation. Eur. Spine J. 2009, 18, 1079–1090. [Google Scholar] [CrossRef] [Green Version]
  11. Petit, Y.; Aubin, C.-É.; Labelle, H. Spinal Shape Changes Resulting from Scoliotic Spine Surgical Instrumentation Expressed as Intervertebral Rotations and Centers of Rotation. J. Biomech. 2004, 37, 173–180. [Google Scholar] [CrossRef]
  12. Morrissy, R.T.; Goldsmith, G.S.; Hall, E.C.; Kehl, D.; Cowie, G.H. Measurement of the Cobb Angle on Radiographs of Patients Who Have Scoliosis. Evaluation of Intrinsic Error. J. Bone Jt. Surg. Am. 1990, 72, 320–327. [Google Scholar] [CrossRef] [Green Version]
  13. Carman, D.L.; Browne, R.H.; Birch, J.G. Measurement of Scoliosis and Kyphosis Radiographs. Intraobserver and Interobserver Variation. J. Bone Jt. Surg. Am. 1990, 72, 328–333. [Google Scholar] [CrossRef]
  14. Cowell, H.R. Radiographic Measurements and Clinical Decisions. J. Bone Jt. Surg. Am. 1990, 72, 319. [Google Scholar] [CrossRef]
  15. Hattori, T.; Sakaura, H.; Iwasaki, M.; Nagamoto, Y.; Yoshikawa, H.; Sugamoto, K. In Vivo Three-Dimensional Segmental Analysis of Adolescent Idiopathic Scoliosis. Eur. Spine J. 2011, 20, 1745–1750. [Google Scholar] [CrossRef] [Green Version]
  16. Fletcher, N.D.; Bruce, R.W. Early Onset Scoliosis: Current Concepts and Controversies. Curr. Rev. Musculoskelet. Med. 2012, 5, 102–110. [Google Scholar] [CrossRef] [Green Version]
  17. Kuklo, T.R.; Potter, B.K.; Schroeder, T.M.; O’Brien, M.F. Comparison of Manual and Digital Measurements in Adolescent Idiopathic Scoliosis. Spine 2006, 31, 1240–1246. [Google Scholar] [CrossRef]
  18. Yazici, M.; Acaroglu, E.R.; Alanay, A.; Deviren, V.; Cila, A.; Surat, A. Measurement of Vertebral Rotation in Standing versus Supine Position in Adolescent Idiopathic Scoliosis. J. Pediatr. Orthop. 2001, 21, 252–256. [Google Scholar] [CrossRef]
  19. Essex, R.; Bruce, G.; Dibley, M.; Newton, P.; Thompson, T.; Swaine, I.; Dibley, L. A systematic scoping review and textual narrative synthesis of the qualitative evidence related to adolescent idiopathic scoliosis. Int. J. Orthop. Trauma Nurs. 2022, 45, 100921. [Google Scholar] [CrossRef]
  20. D’Andrea, L.P.; Betz, R.R.; Lenke, L.G.; Clements, D.H.; Lowe, T.G.; Merola, A.; Haher, T.; Harms, J.; Huss, G.K.; Blanke, K.; et al. Do Radiographic Parameters Correlate with Clinical Outcomes in Adolescent Idiopathic Scoliosis? Spine 2000, 25, 1795–1802. [Google Scholar] [CrossRef]
  21. Mok, J.M.; Berven, S.H.; Diab, M.; Hackbarth, M.; Hu, S.S.; Deviren, V. Comparison of Observer Variation in Conventional and Three Digital Radiographic Methods Used in the Evaluation of Patients with Adolescent Idiopathic Scoliosis. Spine 2008, 33, 681–686. [Google Scholar] [CrossRef] [PubMed]
  22. Lonstein, J.E.; Carlson, J.M. The Prediction of Curve Progression in Untreated Idiopathic Scoliosis during Growth. J. Bone Jt. Surg. Am. 1984, 66, 1061–1071. [Google Scholar] [CrossRef] [Green Version]
  23. Weinstein, S.L.; Ponseti, I.V. Curve Progression in Idiopathic Scoliosis. J. Bone Jt. Surg. Am. 1983, 65, 447–455. [Google Scholar] [CrossRef]
  24. Loder, R.T.; Spiegel, D.; Gutknecht, S.; Kleist, K.; Ly, T.; Mehbod, A. The Assessment of Intraobserver and Interobserver Error in the Measurement of Noncongenital Scoliosis in Children ≤ 10 Years of Age. Spine 2004, 29, 2548–2553. [Google Scholar] [CrossRef]
  25. Ylikoski, M.; Tallroth, K. Measurement Variations in Scoliotic Angle, Vertebral Rotation, Vertebral Body Height, and Intervertebral Disc Space Height. J. Spinal Disord. 1990, 3, 387–391. [Google Scholar]
  26. Zmurko, M.G.; Mooney, J.F., 3rd; Podeszwa, D.A.; Minster, G.J.; Mendelow, M.J.; Guirgues, A. Inter- and Intraobserver Variance of Cobb Angle Measurements with Digital Radiographs. J. Surg. Orthop. Adv. 2003, 12, 208–213. [Google Scholar]
  27. Langensiepen, S.; Semler, O.; Sobottke, R.; Fricke, O.; Franklin, J.; Schönau, E.; Eysel, P. Measuring Procedures to Determine the Cobb Angle in Idiopathic Scoliosis: A Systematic Review. Eur. Spine J. 2013, 22, 2360–2371. [Google Scholar] [CrossRef] [Green Version]
  28. Ricart, P.A.; Andres, T.M.; Apazidis, A.; Errico, T.J.; Trobisch, P.D. Validity of Cobb Angle Measurements Using Digitally Photographed Radiographs. Spine J. 2011, 11, 942–946. [Google Scholar] [CrossRef]
  29. Segev, E.; Hemo, Y.; Wientroub, S.; Ovadia, D.; Fishkin, M.; Steinberg, D.M.; Hayek, S. Intra- and Interobserver Reliability Analysis of Digital Radiographic Measurements for Pediatric Orthopedic Parameters Using a Novel PACS Integrated Computer Software Program. J. Child. Orthop. 2010, 4, 331–341. [Google Scholar] [CrossRef] [Green Version]
  30. Gstoettner, M.; Sekyra, K.; Walochnik, N.; Winter, P.; Wachter, R.; Bach, C.M. Inter- and Intraobserver Reliability Assessment of the Cobb Angle: Manual versus Digital Measurement Tools. Eur. Spine J. 2007, 16, 1587–1592. [Google Scholar] [CrossRef] [Green Version]
  31. Srinivasalu, S.; Modi, H.N.; SMehta, S.; Suh, S.-W.; Chen, T.; Murun, T. Cobb Angle Measurement of Scoliosis Using Computer Measurement of Digitally Acquired Radiographs-Intraobserver and Interobserver Variability. Asian Spine J. 2008, 2, 90. [Google Scholar] [CrossRef] [Green Version]
  32. Cheung, J.; Wever, D.J.; Veldhuizen, A.G.; Klein, J.P.; Verdonck, B.; Nijlunsing, R.; Cool, J.C.; Van Horn, J.R. The Reliability of Quantitative Analysis on Digital Images of the Scoliotic Spine. Eur. Spine J. 2002, 11, 535–542. [Google Scholar] [CrossRef] [PubMed]
  33. Zhang, J.; Lou, E.; Shi, X.; Wang, Y.; Hill, D.L.; Raso, J.V.; Le, L.H.; Lv, L. A Computer-Aided Cobb Angle Measurement Method and Its Reliability. J. Spinal Disord. Tech. 2010, 23, 383–387. [Google Scholar] [CrossRef] [PubMed]
  34. Wills, B.P.D.; Auerbach, J.D.; Zhu, X.; Caird, M.S.; Horn, B.D.; Flynn, J.M.; Drummond, D.S.; Dormans, J.P.; Ecker, M.L. Comparison of Cobb Angle Measurement of Scoliosis Radiographs with Preselected End Vertebrae: Traditional versus Digital Acquisition. Spine 2007, 32, 98–105. [Google Scholar] [CrossRef]
  35. Dang, N.R.; Moreau, M.J.; Hill, D.L.; Mahood, J.K.; Raso, J. Intra-Observer Reproducibility and Interobserver Reliability of the Radiographic Parameters in the Spinal Deformity Study Group’s AIS Radiographic Measurement Manual. Spine 2005, 30, 1064–1069. [Google Scholar] [CrossRef] [PubMed]
  36. Shea, K.G.; Stevens, P.M.; Nelson, M.; Smith, J.T.; Masters, K.S.; Yandow, S. A Comparison of Manual versus Computer-Assisted Radiographic Measurement. Intraobserver Measurement Variability for Cobb Angles. Spine 1998, 23, 551–555. [Google Scholar] [CrossRef]
  37. Chan, A.C.Y.; Morrison, D.G.; Nguyen, D.V.; Hill, D.L.; Parent, E.; Lou, E.H.M. Intra- and Interobserver Reliability of the Cobb Angle-Vertebral Rotation Angle-Spinous Process Angle for Adolescent Idiopathic Scoliosis. Spine Deform. 2014, 2, 168–175. [Google Scholar] [CrossRef]
  38. Zhang, J.; Lou, E.; Hill, D.L.; Raso, J.V.; Wang, Y.; Le, L.H.; Shi, X. Computer-Aided Assessment of Scoliosis on Posteroanterior Radiographs. Med. Biol. Eng. Comput. 2010, 48, 185–195. [Google Scholar] [CrossRef]
  39. Stokes, I.A.F.; Aronsson, D.D. Computer-Assisted Algorithms Improve Reliability of King Classification and Cobb Angle Measurement of Scoliosis. Spine 2006, 31, 665–670. [Google Scholar] [CrossRef] [Green Version]
  40. Aubin, C.-E.; Bellefleur, C.; Joncas, J.; de Lanauze, D.; Kadoury, S.; Blanke, K.; Parent, S.; Labelle, H. Reliability and Accuracy Analysis of a New Semiautomatic Radiographic Measurement Software in Adult Scoliosis. Spine 2011, 36, E780–E790. [Google Scholar] [CrossRef]
  41. Hurtado-Avilés, J.; León-Muñoz, V.J.; Andújar-Ortuño, P.; Santonja-Renedo, F.; Collazo-Diéguez, M.; Cabañero-Castillo, M.; Ponce-Garrido, A.B.; González-Ballester, M.; Sánchez-Martínez, F.J.; Fiorita, P.G.; et al. Validity and Absolute Reliability of Axial Vertebral Rotation Measurements in Thoracic and Lumbar Vertebrae. Appl. Sci. 2021, 11, 1084. [Google Scholar] [CrossRef]
  42. Hurtado-Avilés, J.; León-Muñoz, V.J.; Sanz-Mengibar, J.M.; Santonja-Renedo, F.; Andújar-Ortuño, P.; Collazo-Diéguez, M.; Ferrer-López, V.; Roca-González, J.; Kurochka, K.S.; Cabañero-Castillo, M.; et al. Validity and Reliability of a Computer-Assisted System Method to Measure Axial Vertebral Rotation. Quant. Imaging Med. Surg. 2021, 12, 1706. [Google Scholar] [CrossRef] [PubMed]
  43. Negrini, S.; Donzelli, S.; Aulisa, A.G.; Czaprowski, D.; Schreiber, S.; de Mauroy, J.C.; Diers, H.; Grivas, T.B.; Knott, P.; Kotwicki, T.; et al. 2016 SOSORT Guidelines: Orthopaedic and Rehabilitation Treatment of Idiopathic Scoliosis during Growth. Scoliosis Spinal Disord. 2018, 13, 3. [Google Scholar] [CrossRef] [Green Version]
  44. Atkinson, G.; Nevill, A.M. Selected Issues in the Design and Analysis of Sport Performance Research. J. Sports Sci. 2001, 19, 811–827. [Google Scholar] [CrossRef] [Green Version]
  45. Hopkins, W.G. Measures of Reliability in Sports Medicine and Science. Sport. Med. 2000, 30, 1–15. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  46. Shrout, P.E.; Fleiss, J.L. Intraclass Correlations: Uses in Assessing Rater Reliability. Psychol. Bull. 1979, 86, 420–428. [Google Scholar] [CrossRef]
  47. Landis, J.R.; Koch, G.G. The Measurement of Observer Agreement for Categorical Data. Biometrics 1977, 33, 159–174. [Google Scholar] [CrossRef] [Green Version]
Ijerph 19 04655 g001 550
Figure 1. Several vertebrae points can be selected when there is doubt about which vertebrae are more tilted. The software will automatically choose the vertebrae that are most inclined to the horizontal (in this example, T6 (27.4°) and T12 (38.4°)). α: Cobb angle.
Figure 1. Several vertebrae points can be selected when there is doubt about which vertebrae are more tilted. The software will automatically choose the vertebrae that are most inclined to the horizontal (in this example, T6 (27.4°) and T12 (38.4°)). α: Cobb angle.
Ijerph 19 04655 g001
Ijerph 19 04655 g002 550
Figure 2. The 95% confidence intervals of the intra-group MBEs. The letter E identifies the measurements obtained by the group of expert observers. E1, E2, and E3 represent the measurements obtained by the group of expert observers in the first, second, and third rounds of measurements, respectively. The letter N identifies the measurements obtained by the group of novice observers. N1, N2, and N3 represent the measurements obtained by the group of novice observers in the first, second, and third rounds of measurements, respectively. The intervals for MBE in the error distribution of E1E2 (between the first and second round of expert measurements), E2E3, E1E3, and E (interval for the intra-group MBE when considering the three batches of expert measurements) are shown. In the same way, the intervals for the different measurement runs of the novice group are shown. Both distributions are shown for the data obtained both with the software and manually, where E and N are the intra-group error distributions in the three measurement runs of the expert (E) and novice (N) groups. In green, the errors of the intra-group measurements of the Expert group between measurement rounds 1 and 2, 2 and 3 and 1 and 3. In blue, the errors of the intra-group measurements of the Novice group between measurement rounds 1 and 2, 2 and 3 and 1 and 3. In black, the errors in the measurements of the Expert and Novice groups in all three tests.
Figure 2. The 95% confidence intervals of the intra-group MBEs. The letter E identifies the measurements obtained by the group of expert observers. E1, E2, and E3 represent the measurements obtained by the group of expert observers in the first, second, and third rounds of measurements, respectively. The letter N identifies the measurements obtained by the group of novice observers. N1, N2, and N3 represent the measurements obtained by the group of novice observers in the first, second, and third rounds of measurements, respectively. The intervals for MBE in the error distribution of E1E2 (between the first and second round of expert measurements), E2E3, E1E3, and E (interval for the intra-group MBE when considering the three batches of expert measurements) are shown. In the same way, the intervals for the different measurement runs of the novice group are shown. Both distributions are shown for the data obtained both with the software and manually, where E and N are the intra-group error distributions in the three measurement runs of the expert (E) and novice (N) groups. In green, the errors of the intra-group measurements of the Expert group between measurement rounds 1 and 2, 2 and 3 and 1 and 3. In blue, the errors of the intra-group measurements of the Novice group between measurement rounds 1 and 2, 2 and 3 and 1 and 3. In black, the errors in the measurements of the Expert and Novice groups in all three tests.
Ijerph 19 04655 g002
Ijerph 19 04655 g003 550
Figure 3. The 95% confidence intervals of the inter-group MBEs. The letter E identifies the measurements obtained by the group of expert observers. E1, E2, and E3 represent the measurements obtained by the group of expert observers in the first, second, and third rounds of measurements, respectively. The letter N identifies the measurements obtained by the group of novice observers. N1, N2, and N3 represent the measurements obtained by the group of novice observers in the first, second, and third rounds of measurements, respectively. Intervals are shown for MBE in the error distribution E1N1 (between the first batch of experts and the first round of novices), E2 N2, E3N3, and EN (interval for the inter-group MBE when considering the three rounds of expert and novice measurements). Confidence intervals are shown for the error distributions of the measurements obtained both with the software and manually, where EN is the distribution of inter-group errors in the three measurement rounds of the expert (E) and novice (N) groups. In green, the inter-group measurement errors when measuring with the software between measurement rounds 1, 2 and 3. In blue, inter-group errors when measuring manually between measurement rounds 1, 2 and 3. In black, the inter-group errors when considering the set of the three tests.
Figure 3. The 95% confidence intervals of the inter-group MBEs. The letter E identifies the measurements obtained by the group of expert observers. E1, E2, and E3 represent the measurements obtained by the group of expert observers in the first, second, and third rounds of measurements, respectively. The letter N identifies the measurements obtained by the group of novice observers. N1, N2, and N3 represent the measurements obtained by the group of novice observers in the first, second, and third rounds of measurements, respectively. Intervals are shown for MBE in the error distribution E1N1 (between the first batch of experts and the first round of novices), E2 N2, E3N3, and EN (interval for the inter-group MBE when considering the three rounds of expert and novice measurements). Confidence intervals are shown for the error distributions of the measurements obtained both with the software and manually, where EN is the distribution of inter-group errors in the three measurement rounds of the expert (E) and novice (N) groups. In green, the inter-group measurement errors when measuring with the software between measurement rounds 1, 2 and 3. In blue, inter-group errors when measuring manually between measurement rounds 1, 2 and 3. In black, the inter-group errors when considering the set of the three tests.
Ijerph 19 04655 g003
Ijerph 19 04655 g004 550
Figure 4. Bland–Altman graphic for the curves’ measurements acquired with the software and manually.
Figure 4. Bland–Altman graphic for the curves’ measurements acquired with the software and manually.
Ijerph 19 04655 g004
Table
Table 1. The intra- and inter-group validity and reliability analysis with the software and manual measures.
Table 1. The intra- and inter-group validity and reliability analysis with the software and manual measures.
Intragroup Analysis with SoftwareIntergroup Analysis with Software
MBESDglSEMDC95ICC (2,1)CI 95% MBESDglSEMDC95ICC (2,1)CI 95%
E1E21.670.67340.110.320.9870.978–0.993E1N11.750.57330.100.270.9830.972–0.991
E2E31.830.74350.130.350.9840.974–0.991E2N21.770.65330.110.320.9750.959–0.987
E1E31.610.56330.100.270.9860.976–0.992E3N31.990.84340.140.400.9810.969–0.99
E1.710.61340.110.290.9860.977–0.992EN1.820.59330.100.290.9730.954–0.987
N1N21.710.55320.100.270.9710.952–0.985
N2N31.850.87340.150.410.9700.950–0.984
N1N32.020.71340.120.340.9770.962–0.988
N1.900.67340.120.320.9700.950–0.985
Intragroup Analysis with the Manual MethodIntergroup Analysis with the Manual Method
MBESDglSEMDC95ICC (2,1)CI 95% MBESDglSEMDC95ICC (2,1)CI 95%
E1E22.080.74350.130.350.9820.971–0.990E1N12.200.77340.130.370.9750.959–0.987
E2E32.080.73340.120.350.9780.964–0.987E2N22.610.81350.140.380.9740.955–0.987
E1E31.960.75340.130.360.9820.972–0.990E3N32.631.05330.180.500.9760.961–0.987
E2.130.75350.130.350.9810.970–0.990EN2.470.76340.130.360.9730.951–0.988
N1N22.490.84330.150.410.9670.944–0.984
N2N32.611.07350.180.500.9760.958–0.988
N1N32.150.69310.120.340.9740.955–0.987
N2.500.88340.150.420.9740.954–0.988
AXBY is the distribution of errors between the measurements of experience groups A and B in tests X and Y. E stands for experts and N for novices. MBE is the mean bias error, SD is the standard deviation, gl is the number of sample measurements (gl = 35 − outliers), SE is the standard error of the sample, MDC95 is the minimum detectable change (in degrees), ICC (2,1) is the intra-class correlation coefficient of absolute concordance, and CI 95% is the 95% confidence interval.
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Hurtado-Avilés, J.; Santonja-Medina, F.; León-Muñoz, V.J.; Sainz de Baranda, P.; Collazo-Diéguez, M.; Cabañero-Castillo, M.; Ponce-Garrido, A.B.; Fuentes-Santos, V.E.; Santonja-Renedo, F.; González-Ballester, M.; et al. Validity and Absolute Reliability of the Cobb Angle in Idiopathic Scoliosis with TraumaMeter Software. Int. J. Environ. Res. Public Health 2022, 19, 4655. https://doi.org/10.3390/ijerph19084655

AMA Style

Hurtado-Avilés J, Santonja-Medina F, León-Muñoz VJ, Sainz de Baranda P, Collazo-Diéguez M, Cabañero-Castillo M, Ponce-Garrido AB, Fuentes-Santos VE, Santonja-Renedo F, González-Ballester M, et al. Validity and Absolute Reliability of the Cobb Angle in Idiopathic Scoliosis with TraumaMeter Software. International Journal of Environmental Research and Public Health. 2022; 19(8):4655. https://doi.org/10.3390/ijerph19084655

Chicago/Turabian Style

Hurtado-Avilés, José, Fernando Santonja-Medina, Vicente J. León-Muñoz, Pilar Sainz de Baranda, Mónica Collazo-Diéguez, Mercedes Cabañero-Castillo, Ana B. Ponce-Garrido, Victoria Eugenia Fuentes-Santos, Fernando Santonja-Renedo, Miriam González-Ballester, and et al. 2022. "Validity and Absolute Reliability of the Cobb Angle in Idiopathic Scoliosis with TraumaMeter Software" International Journal of Environmental Research and Public Health 19, no. 8: 4655. https://doi.org/10.3390/ijerph19084655

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop