Next Article in Journal
An Impact Analysis of Artificial Light at Night (ALAN) on Bats. A Case Study of the Historic Monument and Natura 2000 Wisłoujście Fortress in Gdansk, Poland
Next Article in Special Issue
Effect of Antioxidant Supplementation on Markers of Oxidative Stress and Muscle Damage after Strength Exercise: A Systematic Review
Previous Article in Journal
“Only My Husband and My Doctor Know. And You, Girls”: Online Discussions of Stigma Coping Strategies for Russian Surrogate Mothers
Previous Article in Special Issue
Multicomponent Exercise Program for Improvement of Functional Capacity and Lipidic Profile of Older Women with High Cholesterol and High Triglycerides
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
IJERPH
Volume 18
Issue 21
10.3390/ijerph182111326
ijerph-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Systematic Review

Reliability of Isokinetic Hip Flexor and Extensor Strength Measurements in Healthy Subjects and Athletes: A Systematic Review and Meta-Analysis

by
Guido Contreras-Díaz
1,2,
Luis Javier Chirosa-Ríos
1,
Ignacio Chirosa-Ríos
1,
Leonardo Intelangelo
3,
Daniel Jerez-Mayorga
4,* and
Darío Martinez-Garcia
4
1
Department Physical Education and Sports, Faculty of Sport Science, University of Granada, 18011 Granada, Spain
2
Department of Health, Los Lagos University, Puerto Montt 5500000, Chile
3
Musculoskeletal Research Group, University Center for Assistance, Teaching and Research, University of Gran Rosario, Rosario S2000, Argentina
4
Exercise and Rehabilitation Sciences Laboratory, School of Physical Therapy, Faculty of Rehabilitation Sciences, Universidad Andres Bello, Santiago 7591538, Chile
*
Author to whom correspondence should be addressed.
Int. J. Environ. Res. Public Health 2021, 18(21), 11326; https://doi.org/10.3390/ijerph182111326
Submission received: 22 September 2021 / Revised: 22 October 2021 / Accepted: 25 October 2021 / Published: 28 October 2021
(This article belongs to the Special Issue Fitness and Health)
Browse Figures
Versions Notes

Abstract

:
Background: The objective of this systematic review and meta-analysis was to examine the reliability of isokinetic measurements of hip strength in flexion and extension in healthy subjects and athletes. Methods: The databases used were Web of Science, SCOPUS, Medline and PubMed. R was used for all statistical analyses. Results: Hip flexion shows moderate reliability in the supine position (ICC = 0.72; 95% CI: 0.46–0.99) and good reliability in the standing position (ICC = 0.79; 95% CI: 0.54–1.04). Hip extension shows excellent reliability in the supine position (ICC = 0.90; 95% CI: 0.85–0.96) and moderate reliability in the standing position (ICC = 0.72; 95% CI: 0.48–0.96). Flexion of 120°/s and 180°/s showed excellent reliability (ICC = 0.93; 95% CI: 0.85–1.00), (ICC = 0.96; 95% CI: 0.92–1.01). The 60°/s and 120°/s extension showed good reliability (ICC = 0.90; 95% CI: 0.82–0.98), (ICC = 0.87; 95% CI: 0.75–0.99). The 180°/s extension presented excellent reliability (ICC = 0.93; 95% CI: 0.82–1.03). Conclusions: The standing position shows good reliability for hip flexion and the supine position shows excellent reliability for hip extension, both movements have excellent reliability at velocities between 120°/s to 180°/s.

1. Introduction

Optimal muscle strength levels are directly related to sports performance and rehabilitation [1]. In terms of sports performance, it has been shown that runners with greater hip extensor strength have greater anterior trunk inclination when running, which improves running mechanics, reduces the work done by the knee extensors and consequently decreases the probability of knee injury [2]. Likewise, these muscles have a great participation during propulsion in jumping, contributing 31.2% of the work in vertical jumping and 44.2% of the work in horizontal jumping [3]. On the other hand, having greater cross-sectional area (CSA) in the hip flexor muscles is related to greater performance during sprinting in pre-adolescent [4] and adult male [5] sprinters, so having strong hip flexor muscles increases running speed [6]. In the hip, it has been shown that strength deficit is associated with lower extremity injuries [7]. When the deficit is in hip flexors (HF), it is manifested through pathologies such as femoroacetabular impingement [8] and anterior cruciate ligament injuries [9]. When the deficit is in hip extensor (HE), it is manifested through pathologies such as patellofemoral dysfunction [10] and Achilles tendinopathy [11].
For muscle strength assessment, isokinetic dynamometers are the gold standard [12], they allow us to establish agonist/antagonist ratios through different angular velocities and are also considered an effective means to rehabilitate and condition muscle function [13]. Isokinetic dynamometry arrived in the late 1960s with the first Cybex I and since then, a great deal of research has been generated in the field of rehabilitation and sports performance [14] with the knee joint being the most studied and to a lesser extent the hip [15].
Additionally, as isokinetic dynamometry is widely used, it should be noted that its usefulness depends on how reproducible its measurements are [16]. It is known that isokinetic measurements are reproducible in the shoulder [17], knee [18] and spine joints [19]; however, studies on the hip joint are scarce and sometimes contradictory [20]. Therefore, it is important to know the reliability of isokinetic evaluations at the hip level that allow us to determine the strength levels in this joint.
The reliability of hip flexion and extension force measurement with an isokinetic dynamometer has not been thoroughly investigated; therefore, the objective of this systematic review was to (I) examine the reliability of isokinetic flexion and extension force measurements in healthy subjects and athletes; (II) determine which position is the most valid and reliable for force measurement; and (III) select the most reliable velocity for assessing hip flexor and extensor strength.

2. Methods

2.1. Study Design

A systematic revision and meta-analysis were carried out on 14 July 2021 to summarize the current knowledge regarding test–retest reliability (as measured by the ICC) of isokinetic hip strength testing in physically active and/or athletic adults and adolescents. We included quantitative and qualitative summaries: (1) a quantitative meta-analysis to estimate the reliability of present tests for hip flexion-extension and (2) a qualitative review of factors influencing reliability. Before starting the review, a protocol was registered in the International Prospective Register of Systematic Reviews (PROSPERO) registration number CRD42020199520. This systematic review’s reporting flow diagram was based on the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) guidelines [21] (Figure 1).

2.2. Search Strategy

Original quantitative research studies were identified through searching the five principal electronic databases: Web of Science, SCOPUS, MedLine and PubMed. The bibliographic search was carried out by combining the different Medical Subject Headings (MeSH) terms with the following keywords: “Isokinetic”, “Dynamometer”, “Hip”, “Reliability” and “Reproducibility”. These search terms were combined with two Boolean operators AND, OR. The bibliographies of other previous related reviews and the studies finally selected were examined to search for new studies. Other possible scientific evidence related to the subject was identified by contacting authors of the published articles through email.
Two authors examined the articles’ title/summary found in the databases. After the initial selection, they analyzed each study with the inclusion criteria. Each criterion was evaluated as yes/no. If discrepancies existed between the authors, the articles’ ratings were shared and discussed until a consensus was reached. The authors were familiar with the existing literature and did not have a different bias with any of the studies selected for inclusion in the review.

2.3. Eligibility Criteria

Original quantitative research was eligible for inclusion in the quantitative meta-analysis if (1) studies were in English or Spanish language; (2) the subjects were healthy, physically active adults and athletes; (3) isokinetic test of hip flexion/extension were evaluated; (4) mean ICC values, as well as a number of subjects and test (used for estimating variance), could be readily determined from the text. The articles that met the inclusion criteria were identified and their full-text versions were obtained. For studies where multiple ICC results were presented, a typical value was sought for the quantitative analysis (i.e., 10–15 min duration, inter-session interval >1 day and <1 month, median result from multiple pipelines).

2.4. Evaluation of the Methodological Quality of the Studies Included

The methodological quality of the selected studies was evaluated using a critical appraisal tool (CAT) [22] and through the quality assessment of reliability studies (QAREL) [23].
The CAT scale contains items of validity and reliability to evaluate the methodological quality of the studies. There are 13 evaluation points, of which four points refer to validity and nine points to the reliability, the latter being used for the review. A column was added that evaluated each study’s final result in percentage (%), a maximum rating of 90% is considered the highest methodological quality and a score over 45% is considered a high-quality study [22].
The QAREL scale contains 11 points. Points 1 and 2 consider the bias of the sample and the representativeness of the subjects and qualifiers, points 3 to 7 correspond to the blinding of the qualifiers, point 8 refers to the order in which the subjects were evaluated, point 9 considers the time interval in which the subjects were evaluated, point 10 evaluates whether the test was applied and interpreted adequately and point 11 refers to the statistical analysis [23]. A column was added that evaluated each study’s final result as a percentage (%), with a maximum of 110% considered the highest methodological quality.

2.5. Data Collection Process

R was used for all statistical analyses [24]. Excel data was extracted with the read.xls function in gdata [25]. The metafor package was used to perform the analysis meta-analysis results [26]. The rma.mv function was used to compute a meta-analytic estimate of the population ICC with studies nested by authors; random effects were specified for the dataset and the resulting model was fit using restricted maximum likelihood estimation. This procedure has been documented by Noble et al. [27] for formal meta-analysis of ICC values. Therefore, two assumptions were made to conduct an ICC-based meta-analysis. First, a meta-analysis was performed using the raw ICC values with the assumption that these were distributed normally. While not exact, this assumption is often made in the similar case of meta-analysis with Pearson’s correlation coefficient and tends to be less skewed when values are far from one. Second, we assumed that each study’s ICC variance could be approximated as Donner, 1986; via Shoukri et al. [28] established.
Forest plots of all studies included in the meta-analysis were created with the forest function. A funnel plot showing the relationship between ICC coefficients and their estimated standard errors was created with the funnel function. Heterogeneity was assessed with Cochrane’s Q and publication bias was assessed by estimating funnel plot asymmetry via the ranked regression test (rank test function).
The researchers’ data extraction included: number of subjects, gender, type of subject, unilateral or bilateral hip evaluation, and the time between the re-test and the dynamometer used (Table 1).

3. Results

3.1. Study Selection

A total of 1760 studies were identified through an electronic database search (PubMed, n = 146, Web of Science, n = 286, Medline, n = 145, Scopus, n = 189), of which 486 duplicate articles were identified and eliminated. After reading the title and the abstract, 1245 articles were eliminated, leaving 29 studies for full reading, but 17 were eliminated for not meeting inclusion criteria, leaving 12 articles, of which three were eliminated for not having a test-retest. Therefore, a total of nine studies on hip isokinetic assessment were included in this systematic review (Figure 1).

3.2. Characteristics of the Studies

From each manuscript selected for review, the following information was considered: number of subjects, gender, type of subjects (healthy and/or athletic), unilateral or bilateral hip assessment, the time between test–retest and the dynamometer used during the assessment (Table 1). The sample size of the selected studies was between 10 and 52 subjects, aged between 6 and 45 years, all healthy and/or physically active. Selected studies used the following isokinetic dynamometers; Cybex II [29], Biodex Medical System [30,31], biodex system 3 [32], Cybex 340 [33], Cybex Norm [34], CON-TREX [35], biodex [36] and Biodex System 3 Pro [37]. The reliability data extracted included: author, year, movement, position, velocity (°/s), intraclass correlation coefficient (ICC, 95%CI), standard error of measurement (SEM, Nm) for concentric flexion (Table 2), concentric extension (Table 3) and eccentric flexion/extension (Table 4).

3.3. Risk of Bias in Studies

The quality of studies evaluated through the CAT scale obtained a score between 56% and 78%, of which eight articles had a high-quality evaluation (Table 5). The quality of the studies evaluated through QAREL obtained a score between 40% and 60% (Table 6).

3.4. Anatomical Plane and Motion

Seven studies [29,30,32,33,34,36,37] assessed hip strength using two planes: sagittal plane (flexion/extension movements) and frontal plane (abduction/adduction movements). Two studies [31,35] evaluated hip strength only in the sagittal plane, performing flexion and extension.

3.5. Muscle Contraction

Three studies [30,33,35] presented concentric and eccentric strength measurements for the hip, two studies [31,36] evaluated concentrically and isometrically and four studies [29,32,34,37] only evaluated in concentric mode.

3.6. Position

Hip flexion and extension were evaluated in two different positions: supine and standing. The most used position was supine [29,31,33,34,35,36] and, to a lesser extent, the bipedal position [30,32,37].

3.7. Velocity

The velocity used for the concentric phase was between 30°/s and 180°/s and the velocity used for the eccentric phase was between 30°/s and 90°/s. For the concentric phase, two studies [29,32] used a speed of 30°/s, six studies [30,33,34,35,36,37] used a speed of 60°/s, one study [29] used a speed of 90°/s, three studies [31,33,36] used a speed of 120°/s, one study [32] used a speed of 150°/s and two studies [35,37] used a speed of 180°/s. For the eccentric phase, two studies [30,33] used a speed of 60°/s and one study [35] used a speed of 30 and 90°/s.

3.8. Reliability

For this review, we suggest that ICC values below 0.5 indicate low reliability, values between 0.5 and 0.75 indicate moderate reliability, values between 0.75 and 0.9 indicate good reliability and values above 0.90 indicate excellent reliability [38].
When all selected studies were combined, the estimated mean reliability of hip flexion was found to be moderate (ICC = 0.73; 95% CI: 0.51–0.94), in contrast to hip extension which was found to be good (ICC = 0.88; 95% CI: 0.83–0.93) (Figure 2).
In case other variables are considered that may cause a bias, such as the position of the subject during the tests, we note that the two most used positions for hip strength assessment are supine and standing. Hip flexion shows moderate reliability in the supine position (ICC = 0.72; 95% CI: 0.46–0.99) and good reliability in the standing position (ICC = 0.79; 95% CI: 0.54–1.04). Hip extension shows excellent reliability in the supine position (ICC = 0.90; 95% CI: 0.85–0.96) and moderate reliability in the standing position (ICC = 0.72; 95% CI: 0.48–0.96) (Figure 3).
As mentioned above, another of the variables considered most relevant to the reliability of the tests is the speed of the repetitions performed. There is a wide variety of velocities used for hip strength assessment. These range from 30°/s to 180°/s, with multiple others in between. To understand a little about how this variable affects the variety, we can see that the most reliable speeds are intermediate and high speeds.
The 120°/s and 180°/s flexion showed excellent reliability (ICC = 0.93; 95% CI: 0.85–1.00), (ICC = 0.96; 95% CI: 0.92–1.01). The 60°/s and 120°/s extension presented good reliability (ICC = 0.90; 95% CI: 0.82–0.98), (ICC = 0.87; 95% CI: 0.75–0.99). The 180°/s extension presented excellent reliability (ICC = 0.93; 95% CI: 0.82–1.03) (Figure 4).

4. Discussion

This systematic review aimed to (I) examine the reliability of isokinetic flexion and extension strength measurements in healthy subjects and athletes; (II) determine which position is the most valid and reliable for strength measurement; and (III) select the most reliable velocity for assessing hip flexor and extensor strength. The main finding of the present study revealed (I) The reliability of isokinetic strength measurement is moderate in flexion (ICC = 0.73) and good in extension (ICC = 0.88); (II) Standing position presents good reliability in hip flexion (ICC = 0.79) and moderate reliability in hip extension (ICC = 0.72); (III) Supine position presents moderate reliability in hip flexion (ICC = 0.72) and excellent reliability in hip extension (ICC = 0.90); (IV) in flexion the velocity of 120°/s and 180°/s have excellent reliability (ICC = 0.93), (ICC = 0.96) and (V) in extension the velocity of 180°/s show excellent reliability (ICC = 0.93). Isokinetic evaluations of hip flexion and extension movements have moderate to good reliability depending on the position and velocity at which they are performed.

4.1. Movement and Position

Hip flexion and extension movements performed in the sagittal plane using an isokinetic device can be performed in two positions: supine position and standing position, with the supine position being the most frequently used [29,31,33,34,35,36] and the one with the highest reliability during hip extension movement (ICC = 0.90) according to the results of the meta-analysis. Other authors, such as Abdelmohsen et al. [39] also used the supine position to compare the isokinetic strength of the hip flexor and extensor muscles of the dominant side versus the non-dominant side, finding no significant differences between the two sides. Sugiura et al. [40] measured hip extensor, knee extensor and knee flexor strength in elite sprinters to establish a relationship between strength deficits and hamstring injury by measuring hip extension in a standing position, simulating the sprinting motion, finding an association between hamstring injury and the ratio of eccentric hamstring strength to concentric hip extensor strength. Ambegaonkar et al. [41] studied the relationship between core endurance, hip strength and balance in female college athletes using a hand-held dynamometer to measure hip strength in the three planes of motion, evaluating seated hip flexion and bipedal hip extension, based on the positions recommended by manual muscle testing [42], finding that bilateral hip flexion and extension were positively correlated with anterior balance scores (anterior SEBT).

4.2. Velocity

The velocity used for the concentric phase was between 30°/s and 180°/s and for the eccentric phase was between 30°/s and 90°/s, with the most used velocity being 60°/s for both phases [30,33,34,35,36,37]. The results of the meta-analysis show that for hip flexion, the velocities with the highest reliability are 120°/s (ICC = 0.93) and 180°/s (ICC = 0.96) while for hip extension, the velocities with the highest reliability correspond to 60°/s (ICC = 0.90), 120°/s (ICC = 0.87) and 180°/s (ICC = 0.93). Other authors, such as Calmels et al. [43] studied the relationship between flexion/extension torque in hip, knee and ankle of healthy subjects, using concentric velocity of 60°/s, 120°/s and 240°/s and eccentric velocity of 60°/s and 120°/s for all joints, finding no significant differences between the left/right side flexion-extension torque ratios for hip and knee at all angular velocities in concentric and eccentric mode. Arokoski et al. [44] studied muscle strength and the cross-sectional area in men with and without hip osteoarthritis, evaluated abduction, adduction, flexion and extension isometrically and isokinetically (60°/s and 120°/s) in supine position, finding good reliability for flexion at 60°/s (ICC = 0. 70) and 120°/s (ICC = 0.89), excellent reliability for extension at 60°/s (ICC 0.90) and good reliability for extension at 120°/s (0.84) in healthy subjects. Subjects with osteoarthritis also had good reliability for flexion at 60°/s (ICC = 0.84), 120°/s (ICC = 0.89) and extension at 60°/s (ICC = 0.87) and 120°/s (ICC = 0.86). Eng et al. [45], in their reliability study of lower extremity strength measurements in people with chronic stroke, measured isokinetic hip, knee and ankle flexion and extension strength of the hemiparetic side and the healthy side at 60°/s in semi-reclined position, finding high ICCs for peak torque and average torque (0. 95–0.98 for peak torque and 0.88–0.96 for average torque) in hip, knee and ankle flexion and extension movements on the healthy side and high ICC for the hemiparetic side (0.97–0.99 for peak torque and 0.96–0.98 for average torque) during the same movements.
The limitations of these studies have to do with the variability of the population, ranging from children to adults, trained and untrained subjects, with and without experience in the evaluation, which influences the overall or total reliability of the meta-analysis. However, despite the heterogeneity and methodological quality of the studies, we were able to identify positions and velocities with good and excellent reliability, which guides the health and sport professional to make decisions and standardize processes during the evaluation.

5. Conclusions

The reliability of isokinetic hip assessments is determined by factors such as subject position, speed of movement, muscle contraction and pelvic stability [29,30,31,32,33,34,35,36,37]; however, according to the results of the meta-analysis there are measurements with higher reliability:
  • Hip flexion shows good reliability in the standing position (ICC = 0.79; 95% CI: 0.54–1.04).
  • Hip extension shows excellent reliability in supine position (ICC = 0.90; 95% CI: 0.85–0.96).
  • Hip flexion at 120°/s and 180°/s shows excellent reliability (ICC = 0.93; 95% CI: 0.85–1.00), (ICC = 0.96; 95% CI: 0.92–1.01).
  • Hip extension at 60°/s and 120°/s show good reliability (ICC = 0.90; 95% CI: 0.82–0.98), (ICC = 0.87; 95% CI: 0.75–0.99) and excellent reliability at 180°/sond (ICC = 0.93; 95% CI: 0.82–1.03).
Despite these results, it is necessary to carry out more studies with higher methodological quality in different populations, of different ages and sexes, with different modalities of muscle contraction and in both extremities, which will allow us to confirm our results and reproduce them systematically.

Author Contributions

Conceptualization, G.C.-D. and D.J.-M.; methodology, G.C.-D. and D.J.-M.; formal analysis, G.C.-D., D.J.-M., L.I. and D.M.-G.; writing—original draft preparation G.C.-D. and D.J.-M.; writing—review and editing, G.C.-D., D.J.-M., L.I., L.J.C.-R., I.C.-R. and D.M.-G.; visualization, G.C.-D., D.J.-M., L.I., L.J.C.-R., I.C.-R. and D.M.-G.; supervision, G.C.-D., D.J.-M., L.I., L.J.C.-R., I.C.-R. and D.M.-G. All authors have read and agreed to the published version of the manuscript.

Funding

Universidad de Los Lagos, Concurso de Investigación en Ciencias de la Salud y Actividad Física, N° S10/20.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Acknowledgments

This paper will be part of Guido Contreras-Diaz Doctoral Thesis performed in the Biomedicine Doctorate Program of the University of Granada.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Lorenz, D.S.; Reiman, M.P.; Walker, J.C. Periodization: Current review and suggested implementation for athletic rehabilitation. Sports Health 2010, 2, 509–518. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  2. Teng, H.L.; Powers, C.M. Hip-extensor strength, trunk posture, and use of the knee-extensor muscles during running. J. Athl. Train. 2016, 51, 519–524. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  3. Kotsifaki, A.; Korakakis, V.; Graham-Smith, P.; Sideris, V.; Whiteley, R. Vertical and Horizontal Hop Performance: Contributions of the Hip, Knee, and Ankle. Sports Health 2021, 13, 128–135. [Google Scholar] [CrossRef]
  4. Tottori, N.; Suga, T.; Miyake, Y.; Tsuchikane, R.; Otsuka, M.; Nagano, A.; Fujita, S.; Isaka, T. Hip flexor and knee extensor muscularity are associated with sprint performance in sprint-trained preadolescent boys. Pediatr. Exerc. Sci. 2018, 30, 117–125. [Google Scholar] [CrossRef]
  5. Tottori, N.; Suga, T.; Miyake, Y.; Tsuchikane, R.; Tanaka, T.; Terada, M.; Otsuka, M.; Nagano, A.; Fujita, S.; Isaka, T. Trunk and lower limb muscularity in sprinters: What are the specific muscles for superior sprint performance? BMC Res. Notes 2021, 14, 10–15. [Google Scholar] [CrossRef]
  6. Ema, R.; Sakaguchi, M.; Kawakami, Y. Thigh and Psoas Major Muscularity and Its Relation to Running Mechanics in Sprinters. Med. Sci. Sports Exerc. 2018, 50, 2085–2091. [Google Scholar] [CrossRef] [PubMed]
  7. Nadler, S.F.; Malanga, G.A.; Deprince, M.; Stitik, T.P.; Feinberg, J.H. The Relationship between Lower Extremity Injury, Low Back Pain, and Hip Muscle Strength in Male and Female Collegiate Athletes. Clin. J. Sport Med. 2000, 10, 89–97. [Google Scholar] [CrossRef]
  8. Casartelli, N.C.; Maf, N.A.; Staehli, S.; Bizzini, M.; Impellizzeri, F.M.; Leunig, M. Hip muscle weakness in patients with symptomatic femoroacetabular impingement. Osteoarthr. Cartil. 2011, 19, 816–821. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  9. Mouzopoulos, G.; Siebold, R.; Tzurbakis, M. Hip flexion strength remains decreased in anterior cruciate ligament reconstructed patients at one-year follow up compared to healthy controls. Int. Orthop. 2015, 39, 1427–1432. [Google Scholar] [CrossRef]
  10. Van Cant, J.; Pineux, C.; Pitance, L.; Feipel, V. Hip muscle strength and endurance in females with patellofemoral pain: A systematic review with meta-analysis. Int. J. Sports Phys. Ther. 2014, 9, 564–582. [Google Scholar] [PubMed]
  11. Habets, B.; Smits, H.W.; Backx, F.J.G.; van Cingel, R.E.H.; Huisstede, B.M.A. Hip muscle strength is decreased in middle-aged recreational male athletes with midportion Achilles tendinopathy: A cross-sectional study. Phys. Ther. Sport 2017, 25, 55–61. [Google Scholar] [CrossRef] [PubMed]
  12. Brown, L.E.; Stone, M.H. Isokinetic Exercise and Human Performance. Strength Cond. J. 2000, 22, 53. [Google Scholar] [CrossRef]
  13. Dvir, Z.; Müller, S. Multiple-Joint Isokinetic Dynamometry: A Critical Review. J. Strenght Cond. Res. 2019, 00, 1–15. [Google Scholar] [CrossRef] [PubMed]
  14. Busby, K.E.; Malone, T.R. Sports Injury Management. In Evaluation of Isokinetic Equipment; Nr. 1 March 1988; Williams & Wilkins: Philadelphia, PA, USA, 1988; Volume 1, ISBN 9780683078602. [Google Scholar]
  15. Caruso, J.F.; Brown, L.E.; Tufano, J.J. The reproducibility of isokinetic dynamometry data. Isokinet. Exerc. Sci. 2012, 20, 239–253. [Google Scholar] [CrossRef]
  16. Drouin, J.M.; Valovich-McLeod, T.C.; Shultz, S.J.; Gansneder, B.M.; Perrin, D.H. Reliability and validity of the Biodex system 3 pro isokinetic dynamometer velocity, torque and position measurements. Eur. J. Appl. Physiol. 2004, 91, 22–29. [Google Scholar] [CrossRef] [PubMed]
  17. Plotnikoff, N.A.; MacIntyre, D.L. Test-retest reliability of glenohumeral internal and external rotator strength. Clin. J. Sport Med. 2002, 12, 367–372. [Google Scholar] [CrossRef]
  18. Duarte, J.P.; Valente-dos-Santos, J.; Coelho-e-Silva, M.J.; Couto, P.; Costa, D.; Martinho, D.; Seabra, A.; Cyrino, E.S.; Conde, J.; Rosado, J.; et al. Reproducibility of isokinetic strength assessment of knee muscle actions in adult athletes: Torques and antagonist-agonist ratios derived at the same angle position. PLoS ONE 2018, 13, e0202261. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  19. Guilhem, G.; Giroux, C.; Couturier, A.; Maffiuletti, N.A. Validity of trunk extensor and flexor torque measurements using isokinetic dynamometry. J. Electromyogr. Kinesiol. 2014, 14, 986–993. [Google Scholar] [CrossRef]
  20. Dvir, Z. Clinical applicability of isokinetics: A review. Clin. Biomech. 1991, 6, 133–144. [Google Scholar] [CrossRef]
  21. Shamseer, L.; Moher, D.; Clarke, M.; Ghersi, D.; Liberati, A.; Petticrew, M.; Shekelle, P.; Stewart, L.A.; Altman, D.G.; Booth, A.; et al. Preferred reporting items for systematic review and meta-analysis protocols (prisma-p) 2015: Elaboration and explanation. BMJ 2015, 349, 1–25. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  22. Brink, Y.; Louw, Q.A. Clinical instruments: Reliability and validity critical appraisal. J. Eval. Clin. Pract. 2012, 18, 1126–1132. [Google Scholar] [CrossRef] [PubMed]
  23. Lucas, N.P.; Macaskill, P.; Irwig, L.; Bogduk, N. The development of a quality appraisal tool for studies of diagnostic reliability (QAREL). J. Clin. Epidemiol. 2010, 63, 854–861. [Google Scholar] [CrossRef]
  24. R Core Team. R: A Language and Environmentfor Statistical Computing; R Foundation for Statistical Computing: Vienna, Austria, 2017. [Google Scholar]
  25. Warnes, G.R.; Bolker, B.; Gorjanc, G.; Grothendieck, G.; Korosec, A.; Lumley, T.; MacQueen, D.; Magnusson, A.; Rogers, J. Gdata: Various R Programming Tools for Data Manipulation. R Packag. Version 2.13.3. 2014. Available online: https://cran.r-project.org/web/packages/gdata/ (accessed on 23 October 2021).
  26. Viechtbauer, W. Conducting meta-analyses in R with the metafor. J. Stat. Softw. 2010, 36, 1–48. [Google Scholar] [CrossRef] [Green Version]
  27. Noble, S.; Scheinost, D.; Constable, R.T. A decade of test-retest reliability of functional connectivity: A systematic review and meta-analysis. Neuroimage 2019, 203, 116157. [Google Scholar] [CrossRef]
  28. Shoukri, M.M.; Al-Hassan, T.; DeNiro, M.; El Dali, A.; Al-Mohanna, F. Bias and Mean Square Error of Reliability Estimators under the One and Two Random Effects Models: The Effect of Non-Normality. Open J. Stat. 2016, 6, 254–273. [Google Scholar] [CrossRef] [Green Version]
  29. Burnett, C.N.; Filusch Betts, E.; King, W.M. Reliability of isokinetic measurements of hip muscle torque in young boys. Phys. Ther. 1990, 70, 244–249. [Google Scholar] [CrossRef]
  30. Claiborne, T.L.; Timmons, M.K.; Pincivero, D.M. Test–Retest reliability of cardinal plane isokinetic hip torque and EMG. J. Electromyogr. Kinesiol. 2009, 19, e345–e352. [Google Scholar] [CrossRef]
  31. Krantz, M.M.; Åström, M.; Drake, A.M. Strength and Fatigue Measurements of the Hip Flexor and Hip Extensor Muscles: Test-Retest Reliability and Limb Dominance Effect. Int. J. Sports Phys. Ther. 2020, 15, 967–976. [Google Scholar] [CrossRef] [PubMed]
  32. Santos Andrade, M.; Mascarin, N.C.; Benedito-Silva, A.A.; Carderelli Minozzo, F.; Vancini, R.L.; Barbosa De Lira, C.A. Assessment of isokinetic peak torque reliability of the hip flexor, extensor, adductors and abductors muscles in female soccer players from 14 to 25 years old. J. Sports Med. Phys. Fit. 2016, 56, 843–848. [Google Scholar]
  33. Dugailly, P.M.; Brassinne, E.; Pirotte, E.; Mouraux, D.; Feipel, V.; Klein, P. Isokinetic assessment of hip muscle concentric strength in normal subjects: A reproducibility study. Isokinet. Exerc. Sci. 2005, 13, 129–137. [Google Scholar] [CrossRef]
  34. Emery, C.A.; Maitland, M.E.; Meeuwisse, W.H. Test-retest reliability of isokinetic hip adductor and flexor muscle strength. Clin. J. Sport Med. 1999, 9, 79–85. [Google Scholar] [CrossRef] [PubMed]
  35. Julia, M.; Dupeyron, A.; Laffont, I.; Lemoine, F.; He, C. Reproducibility of isokinetic peak torque assessments of the hip flexor and extensor muscles. Ann. Phys. Rehabil. Med. 2010, 53, 293–305. [Google Scholar] [CrossRef] [Green Version]
  36. Meyer, C.; Corten, K.; Wesseling, M.; Peers, K.; Simon, J.P.; Jonkers, I.; Desloovere, K. Test-retest reliability of innovated strength tests for hip muscles. PLoS ONE 2013, 8, e81149. [Google Scholar] [CrossRef] [Green Version]
  37. Parsons, J.L.; Porter, M.M. Reliability of Measuring Hip and Knee Power and Movement Velocity in Active Youth. Pediatric Phys. Ther. 2015, 27, 82–89. [Google Scholar] [CrossRef] [PubMed]
  38. Koo, T.K.; Li, M.Y. A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. J. Chiropr. Med. 2016, 15, 155–163. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  39. Abdelmohsen, A.M. Leg Dominance Effect on Isokinetic Muscle Strength of Hip Joint. J. Chiropr. Med. 2019, 18, 27–32. [Google Scholar] [CrossRef] [PubMed]
  40. Sugiura, Y.; Saito, T.; Sakuraba, K.; Sakuma, K.; Suzuki, E. Strength deficits identified with concentric action of the hip extensors and eccentric action of the hamstrings predispose to hamstring injury in elite sprinters. J. Orthop. Sports Phys. Ther. 2008, 38, 457–464. [Google Scholar] [CrossRef] [Green Version]
  41. Ambegaonkar, J.P.; Mettinger, L.M.; Caswell, S.V.; Burtt, A.; Cortes, N. Relationships between core endurance, hip strength, and balance in collegiate female athletes. Int. J. Sports Phys. Ther. 2014, 9, 604–616. [Google Scholar]
  42. Scott, D.A.; Bond, E.Q.; Sisto, S.A.; Nadler, S.F. The intra- and interrater reliability of hip muscle strength assessments using a handheld versus a portable dynamometer anchoring station. Arch. Phys. Med. Rehabil. 2004, 85, 598–603. [Google Scholar] [CrossRef]
  43. Calmels, P.M.; Nellen, M.; van der Borne, I.; Jourdin, P.; Minaire, P. Concentric and eccentric isokinetic assessment of flexor-extensor torque ratios at the hip, knee, and ankle in a sample population of healthy subjects. Arch. Phys. Med. Rehabil. 1997, 78, 1224–1230. [Google Scholar] [CrossRef]
  44. Arokoski, M.H.; Arokoski, J.P.A.; Haara, M.; Kankaanpää, M.; Vesterinen, M.; Niemitukia, L.H.; Helminen, H.J. Hip Muscle Strength and Muscle Cross Sectional Area in Men with and without Hip Osteoarthritis. J. Rheumatol. 2002, 29, 2185–2195. [Google Scholar] [PubMed]
  45. Eng, J.J.; Kim, C.M.; MacIntyre, D.L. Reliability of lower extremity strength measures in persons with chronic stroke. Arch. Phys. Med. Rehabil. 2002, 83, 322–328. [Google Scholar] [CrossRef] [PubMed] [Green Version]
Ijerph 18 11326 g001 550
Figure 1. PRISMA flow diagram.
Figure 1. PRISMA flow diagram.
Ijerph 18 11326 g001
Ijerph 18 11326 g002 550
Figure 2. Forest plot, 95% confidence interval (CI95%) for reliability hip flexion (a) and reliability hip extension (b).
Figure 2. Forest plot, 95% confidence interval (CI95%) for reliability hip flexion (a) and reliability hip extension (b).
Ijerph 18 11326 g002
Ijerph 18 11326 g003 550
Figure 3. Forest plot, 95% confidence interval (CI95%) for reliability hip flexion in supine position (a), reliability hip extension in supine position (b), reliability hip flexion in standing position (c) and reliability hip extension in standing position (d).
Figure 3. Forest plot, 95% confidence interval (CI95%) for reliability hip flexion in supine position (a), reliability hip extension in supine position (b), reliability hip flexion in standing position (c) and reliability hip extension in standing position (d).
Ijerph 18 11326 g003
Ijerph 18 11326 g004 550
Figure 4. Forest plot, 95% confidence interval (CI95%) for reliability hip flexion 120°/s (a), reliability hip flexion 180°/s (b), reliability hip extension 60°/s (c) reliability hip extension 120°/s (d) and reliability hip extension 180°/s (e).
Figure 4. Forest plot, 95% confidence interval (CI95%) for reliability hip flexion 120°/s (a), reliability hip flexion 180°/s (b), reliability hip extension 60°/s (c) reliability hip extension 120°/s (d) and reliability hip extension 180°/s (e).
Ijerph 18 11326 g004
Table
Table 1. Characteristics of the participant.
Table 1. Characteristics of the participant.
StudyNGenderType SubjectsBilateralTime RestDynamometer
Burnett et al. [29]29BoysHealthyNo7–14 daysCybex II
Claiborne et al. [30]13Boys/GirlsHealthyYes7 daysBiodex Medical System
Krantz et al. [31]30Men/WomanHealthyYes7 daysBiodex Medical System
Dos Santos Andrade et al. [32]17WomanSoccer PlayersYes7 daysBiodex System 3
Dugailly et al. [33]28Boys/GirlsSports-activeYes10 minCybex 340
Emery et al. [34] 19MalesHealthyYes7 daysCybex Norm
Julia et al. [35]10Males/WomanHealthyYes7 daysCON-TREX
Meyer et al. [36]10Males/WomanHealthyNo7 daysBiodex
Parsons et al. [37]52Boys/GirlsHealthyNo7 daysBiodex System 3 Pro
Table
Table 2. Reliability and absolute reliability of concentric flexion hip.
Table 2. Reliability and absolute reliability of concentric flexion hip.
StudyHip Action EvaluatedPostureSpeed (°/s)Mean/(SD) 1° TestMean/(SD) 2° TestMean/(SD) 3° TestICC (95% CI)Limits ICCSEM (Nm)SEM (%)SRD (Nm)SRD (%)
Burnett et al. [29]FlexionSupine306.26.0(-) 0.63(-)(-)(-) (-) (-)
FlexionSupine905.15.2(-) 0.75(-)(-)(-) (-) (-)
Claiborne et al. [30]Flexion LeftBiped6031.3734.05(-) 0.820.80–0.9013.92(-) (-) (-)
Flexion RightBiped6037.4925.89(-)0.830.80–0.9013.16(-) (-) (-)
Dos Santos Andrade et al. [32]Flexion RightBiped30218.9208.5(-)0.25(-) 0.26–0.65(-) (-) (-) (-)
Flexion LeftBiped30225.3217.4(-) 0.38(-) 0.13–0.73(-) (-) (-) (-)
Flexion RightBiped150208.7203.2(-)0.610.18–0.84(-) (-) (-) (-)
Flexion LeftBiped150197.7190.5(-)0.760.44–0.91(-) (-) (-) (-)
Dugailly et al. [33]Flexion WomenSupine60(-)(-)(-)0.75(-)3.7(-)(-)(-)
Flexion WomenSupine120(-)(-)(-)0.79(-)3.6(-)(-)(-)
Flexion MenSupine60(-)(-)(-)0.75(-)3.7(-)(-)(-)
Flexion MenSupine120(-)(-)(-)0.79(-)3.6(-)(-)(-)
Emery et al. [34]Flexion RightSupine6054.262.951.00.370.06–0.64(-)(-)(-)(-)
Flexion LeftSupine6052.458.343.10.040–0.35(-)(-)(-)(-)
Julia et al. [35] Flexion LeftSupine60(-)(-)(-)0.90(-)(-)(-) (-) (-)
Flexion LeftSupine180(-)(-)(-)0.94(-)(-)(-) (-) (-)
Flexion RightSupine60(-)(-)(-)0.94(-)(-)(-) (-) (-)
Flexion RightSupine180(-)(-)(-)0.96(-)(-)(-) (-) (-)
Krantz et al. [31]FlexionSupine12028.2928.67(-)0.830.89–0.971.766.2(-) (-)
Meyer et al. [36]FlexionSupine60120.7124.7(-) 0.920.80–0.9710.518.5729.1323.75
FlexionSupine120103.3107.1(-) 0.930.82–0.978.418.0023.3222.17
Parsons et al. [37]FlexionBiped6074.8 (28.5)70.1 (25.4)(-)0.960.91–0.987.1(-) (-) (-)
FlexionBiped180194.1 (74.6)189.0 (72.6)(-)0.970.95–0.9916.7(-) (-) (-)
ICC = intraclass correlation coefficient (95% confidence interval); SEM = standard error of measurement; SRD = smallest real difference; (-) = not available.
Table
Table 3. Relative and absolute reliability of concentric extension hip.
Table 3. Relative and absolute reliability of concentric extension hip.
StudyHip Action EvaluatedPostureSpeed (°/s)Mean/(SD) 1° TestMean/(SD) 2° TestMean/(SD) 3° TestICC (95% CI)Limits ICCSEM (Nm)SEM (%)SRD (Nm)SRD (%)
Burnett et al. [29]ExtensionSupine3010.312(-)0.68(-)(-) (-) (-) (-)
ExtensionSupine9010.112.3(-)0.84(-) (-) (-) (-) (-)
Claiborne et al. [30]Extension LeftBiped6032.2234.55(-)0.800.80–0.9014.84(-) (-) (-)
Extension RightBiped6033.7831.77(-)0.900.80–0.9010.40(-) (-) (-)
Dos Santos Andrade et al. [32]Extension RightBiped 30239.0227.4(-)0.16(-) 0.35–0.59(-)(-)(-)(-)
Extension LeftBiped30258.8250.4(-)0.490.01–0.79(-)(-)(-)(-)
Extension RightBiped150228.8227.7(-)0.550.09–0.82(-)(-)(-)(-)
Extension LeftBiped150237.5235.4(-) 0.760.44–0.91(-) (-) (-) (-)
Dugailly et al. [33]Extension WomenSupine60(-) (-) (-) 0.94(-)9.5(-) (-) (-)
Extension WomenSupine120(-) (-) (-) 0.89(-)8.2(-) (-) (-)
Extension MenSupine60(-) (-) (-) 0.94(-) 9.5(-) (-) (-)
Extension MenSupine120(-) (-) (-) 0.89(-) 8.2(-) (-) (-)
Julia et al. [35]Extension LeftSupine60(-)(-)(-) 0.80(-) (-)(-) (-) (-)
Extension RightSupine60(-)(-)(-) 0.62(-) (-)(-) (-) (-)
Extension LeftSupine180(-)(-)(-) 0.94(-) (-)(-) (-) (-)
Extension RightSupine180(-)(-)(-) 0.83(-) (-)(-) (-) (-)
Krantz et al. [31]ExtensionSupine12051.5454.74(-)0.830.67–0.927.2213.6(-)(-)
Meyer et al. [36]ExtensionSupine60120.2140.7(-) 0.840.61–0.9312.669.7035.1026.90
ExtensionSupine120114.9132.1(-) 0.800.55–0.9216.1113.0644.6536.31
Parsons et al. [37]ExtensionBiped6052.2 (25.2)50.6 (25.3)(-) 0.790.63–0.8815.1(-) (-) (-)
ICC = intraclass correlation coefficient (95% confidence interval); SEM = standard error of measurement; SRD = smallest real difference; (-) = not available.
Table
Table 4. Relative and absolute reliability of eccentric flexion and extension hip.
Table 4. Relative and absolute reliability of eccentric flexion and extension hip.
StudyHip Action EvaluatedPostureSpeed (°/s)Mean/(SD) 1° TestMean/(SD) 2° TestMean/(SD) 3° TestICC (95% CI)Limits ICCSEM (Nm)SEM (%)SRD (Nm)SRD (%)
Flexion Eccentric
Claiborne et al. [30]Flexion LeftBiped6035.3834.89(-) 0.740.62–0.7918.06(-) (-) (-)
Flexion RightBiped6028.3434.90(-) 0.910.80–0.919.42(-) (-) (-)
Dugailly et al. [33]Flexion RightSupine6079.568.360.5 0.280.09–0.66(-)(-) (-) (-)
Flexion LeftSupine6063.273.653.3 0.350.04–0.62(-)(-) (-) (-)
Extension Eccentric
Claiborne et al. [30]Extension LeftBiped6030.9135.44(-) 0.800.80–0.9114.68(-) (-) (-)
Extension RightBiped6045.5034.26(-) 0.760.62–0.7919.49(-) (-) (-)
Julia et al. [35]Extension LeftSupine30(-)(-)(-) 0.68(-) (-)(-) (-) (-)
Extension RightSupine30(-)(-)(-) 0.80(-) (-)(-) (-) (-)
Extension LeftSupine90(-)(-)(-) 0.75(-) (-)(-) (-) (-)
Extension RightSupine90(-)(-)(-) 0.78(-) (-)(-) (-) (-)
ICC = intraclass correlation coefficient (95% Confidence Interval); SEM = standard error of measurement; SRD = smallest real difference; (-) = Not available.
Table
Table 5. Evaluation of the quality of studies with the critical evaluation tool (CAT).
Table 5. Evaluation of the quality of studies with the critical evaluation tool (CAT).
Study123456789%
Burnett et al. [29]YNNNNYYYY56
Claiborne et al. [30]YNNNNYYYY56
Dos Santos Andrade et al. [32] YYNNNYYYY67
Dugailly et al. [33]YNNNNYYYY56
Emery et al. [34] YNNNNYYYY56
Julia et al. [35]YNNNNYYYY56
Krantz et al. [31]YYNNNYYYY67
Meyer et al. [36] YNNNYYYYY67
Parsons et al. [37]YNYYNYYYY78
Y = Yes; N = No. 1. If human subjects were used, did the authors give a detailed description of the sample of subjects used to perform the test? 2. Did the authors clarify the qualification, or competence of the rater(s) who performed the test? 3. If interrater reliability was tested, were raters blinded to the findings of other raters? 4. If intra-rater reliability was tested, were raters blinded to their own prior findings of the test under evaluation? 5. Was the order of examination varied? 6. Was the stability (or theoretical stability) of the variable being measured taken into account when determining the suitability of the time interval between repeated measures? 7. Was the execution of the test described in sufficient detail to permit replication of the test? 8. Were withdrawals from the study explained? 9. Were the statistical methods appropriate for the purpose of the study? %: final percentage of reliability (Items “yes” × 100)/9.
Table
Table 6. Evaluation of the quality of studies with the QAREL scale.
Table 6. Evaluation of the quality of studies with the QAREL scale.
StudyP1P2P3P4P5P6P7P8P9P10P11%
Burnett et al. [29]YYYUCNAUCUCUCYYY60
Claiborne et al. [30]YYUCUCNAUCUCNYYY50
Dos Santos Andrade et al. [32]YYUCUCNAUCUCNYYY50
Dugailly et al. [33] YYUCUCNAUCUCUCUCYY40
Emery et al. [34]YYUCUCNAUCUCUCYYY50
Julia et al. [35]YYNUCNAUCUCNYYY50
Krantz et al. [31]YYNNNAUCUCNYYY50
Meyer et al. [36]YYUCUCNAUCUCNYYY50
Parsons et al. [37]YYNYNAUCUCNYYY60
P = Question on the QAREL scale; Y = Yes, complies; N = No, does not comply; UC = Unclear; NA = Not applicable. 1. Was the test evaluated in a sample of subjects who were representative of those to whom the authors intended the results to be applied? 2. Was the test performed by raters who were representative of those to whom the authors intended the results to be applied? 3. Were raters blinded to the findings of other raters during the study? 4. Were raters blinded to their own prior findings of the test under evaluation? 5. Were raters blinded to the results of the reference standard for the target disorder (or variable) being evaluated? 6. Were raters blinded to clinical information that was not intended to be provided as part of the testing procedure or study design? 7. Were raters blinded to additional cues that were not part of the test? 8. Was the order of examination varied? 9. Was the time interval between repeated measurements compatible with the stability (or theoretical stability) of the variable being measured? 10. Was the test applied correctly and interpreted appropriately? 11. Were appropriate statistical measures of agreement used? %: final percentage of reliability (Items “yes” × 100)/11.
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Contreras-Díaz, G.; Chirosa-Ríos, L.J.; Chirosa-Ríos, I.; Intelangelo, L.; Jerez-Mayorga, D.; Martinez-Garcia, D. Reliability of Isokinetic Hip Flexor and Extensor Strength Measurements in Healthy Subjects and Athletes: A Systematic Review and Meta-Analysis. Int. J. Environ. Res. Public Health 2021, 18, 11326. https://doi.org/10.3390/ijerph182111326

AMA Style

Contreras-Díaz G, Chirosa-Ríos LJ, Chirosa-Ríos I, Intelangelo L, Jerez-Mayorga D, Martinez-Garcia D. Reliability of Isokinetic Hip Flexor and Extensor Strength Measurements in Healthy Subjects and Athletes: A Systematic Review and Meta-Analysis. International Journal of Environmental Research and Public Health. 2021; 18(21):11326. https://doi.org/10.3390/ijerph182111326

Chicago/Turabian Style

Contreras-Díaz, Guido, Luis Javier Chirosa-Ríos, Ignacio Chirosa-Ríos, Leonardo Intelangelo, Daniel Jerez-Mayorga, and Darío Martinez-Garcia. 2021. "Reliability of Isokinetic Hip Flexor and Extensor Strength Measurements in Healthy Subjects and Athletes: A Systematic Review and Meta-Analysis" International Journal of Environmental Research and Public Health 18, no. 21: 11326. https://doi.org/10.3390/ijerph182111326

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop