The Child-Focused Injury Risk Screening Tool (ChildFIRST) Demonstrates Greater Reliability When Using a Dichotomous Scale vs. a Seven-Point Likert Scale, and Is Preferred by Raters

Abstract

The Child-Focused Injury Risk Screening Tool (ChildFIRST) assesses movement competence in children and currently uses a dichotomous scoring scale, which, while simple and practical, may lack the precision needed for nuanced movement skill analysis. This study compared the inter- and intra-rater reliability of the ChildFIRST when scored using a dichotomous scale versus a seven-point Likert scale. Fourteen trained raters evaluated video recordings of eight children performing ten standardized movement tasks using both scales across two sessions. Reliability was assessed using intraclass correlation coefficients (ICCs). The dichotomous scale demonstrated moderate to excellent inter-rater reliability (ICC = 0.50–1.00) and good to excellent intra-rater reliability (ICC = 0.75–1.00). The seven-point scale showed similar inter-rater reliability but generally lower intra-rater reliability (ICC = 0.50–1.00). In addition, raters preferred the dichotomous scale in terms of practicality (91.6%), feasibility (75%), and overall usability (66.6%). These findings suggest that while both scales provide comparable inter-rater agreement, the dichotomous format offers greater consistency across repeated ratings and is more favorably received by users. The dichotomous scoring system is therefore recommended for continued use in field-based screening and future applications of the ChildFIRST.

1. Introduction

Over the last two decades, there has been a growing trend of increased physical inactivity and sedentary behavior in children [1]. Globally, in 2016, physical inactivity rates for children aged 11 to 17 was at 81.0% [1]. Prolonged physical inactivity is associated with an increased risk of musculoskeletal injury [2]. Physical inactivity in children is also referred to as the pediatric inactivity triad [3]. This triad includes physical illiteracy, which refers to the lack of confidence, competence, and motivation to engage in meaningful physical activities with interest and enthusiasm [3]. Children with physical illiteracy lack movement competence, foundational strength, endurance, and neuromuscular control and are at a greater risk of underuse-type injuries when participating in sports and recreational activities [4,5,6,7]. Efforts to improve physical activity levels are using the concept of physical literacy [8]. According to the 2017 consensus from the International Physical Literacy Association, physical literacy is defined as the motivation, confidence, physical competence, knowledge, and understanding to value and take responsibility for engagement in physical activities for life. The physical competence component of physical literacy has a close relationship to injury prevention in youth [9]. Physical competence is broken down into different categories: fitness, movement competence, locomotion, stability and balance, and object manipulation [8].

Movement competence assessment tools can be product-based or process-based [10]. Product-based assessment tools focus on the objective result of a movement test, whereas process-based assessment tools focus on the subjective quality of a movement test. Both product-based and process-based assessment tools are needed to have a comprehensive assessment of movement. However, process-based assessment tools are better suited for movement analysis since they measure movement quality [10]. Process-based assessment tools can be used to assess movement technique errors that predispose children to injury. A lack of movement competence in children 8–12 years old is associated with injury risk [11], so screening children in their developing years is important to have a better understanding of injury risk factors and to implement better prevention strategies. There are few valid and reliable process-based assessment tools that measure physical competence in children.

The Test of Gross Motor Development 3 [12] and the POLYGON-FMS [13] assess only a portion of physical competence, largely neglecting the balance component, and are not specific to injury risk screening and injury prevention [9]. The Child-Focused Injury Risk Screening Tool (ChildFIRST) (

Table A1. ChildFIRST movement skills and evaluation criteria [14].

Movement Skill	Criteria 1	Criteria 2	Criteria 3	Criteria 4
Bodyweight Squat	Push hips back and bend knees until the thighs are approximately parallel with the ground	Hips, knees, and ankles aligned	Knees do not go too far in front of the toes	Keep the heels down all the time
Single Leg-Hop	Hips, knees, and ankles aligned	Take off from one foot, land on same foot	Knees and hips bend to land softly in a controlled fashion	Swing arms to assist the movement
Running	Upper body straight and eyes focused on the direction traveled	Swing bent arms in opposition to legs	Knee drives upwards and forward to lift the foot off the ground	Knee and hip bend slightly to land softly
Vertical Jump	Swing arms to assist the movement	Knees and hips bend to land softly in a controlled fashion	Land on both feet at the same time	Hips, knees, and ankles aligned
Horizontal Jump	Swing arms to assist the movement	Knees and hips bend to land softly in a controlled fashion	Land on both feet at the same time	Hips, knees, and ankles aligned
Walking Lunge	Hips, knees, and ankles aligned	Upper body straight and eyes focused on direction of travel	Front knee does not go too far in front of the toes	No twisting or back bending
Two to One-Foot Hop and Hold	Knees and hips bend to land softly in a controlled fashion	Toes pointing forward	Foot flat on the floor	Hips, knees, and ankles aligned
Single-Leg Sideways Hop and Hold	Knees and hips bend to land softly in a controlled fashion	Hips, knees, and ankles aligned	Foot flat on the floor	Stands up straight within three seconds after landing
Leaping	Take off from one foot, land on the opposite foot	Knees and hips bend to land softly in a controlled	Hips, knees, and ankles aligned	Swing bent arms in opposition to legs
90-Degree Hop and Hold	Knees and hips bend to land softly in a controlled fashion	Hips, knees, and ankles aligned	Whole body turns together	Toes pointing forward

) is a process-based physical literacy assessment tool used to measure physical competence that was developed to screen for high-risk technique error that predispose children aged 8–12 to lower limb injury [14]. The ChildFIRST consists of ten movement skills: bodyweight squat, single-leg hop, running, vertical jump, horizontal jump, walking lunge, two to one-foot hop and hold, single-leg sideways hop and hold, leaping, and 90-degree hop and hold. Each movement skill has four evaluation criteria, for a sum composite score of up to 40 points (see Table A1). The goal of the ChildFIRST is to measure movement competence and the risk of lower limb injury in children aged 8–12 and is designed to be used in a field setting. See

Table 1. ChildFIRST movement skills description [14].

Movement Skill	Description
Bodyweight Squat	Squatting involves flexing the knees and pushing the hips back to lower the center of gravity. The feet are shoulder-width apart and the hands are placed either crossed on the chest or extended out in front of the body. The movement should be smooth.
Single-Leg Hop	Single-leg hop is performed by taking off from one foot and landing on the same foot. The movement should be smooth and performed equally on both sides.
Running	Running is faster than walking, but it is not sprinting. It will present the gait pattern (heel strike–midfoot–forefoot) and a flight phase. The ChildFIRST does not intend to measure how fast the child runs but the quality of the movement. The movement should be smooth.
Vertical Jump	Vertical jump is the action of propelling the body vertically into the air from the ground using both legs and landing with both feet. The movement should be smooth.
Horizontal Jump	Horizontal jump is the action of propelling the body horizontally into the air from the ground using both legs and landing with both feet. The movement should be smooth.
Walking Lunge	A lunge can refer to any position of the human body where one leg is positioned forward with the knee bent and foot flat on the ground while the other leg is positioned behind. The movement should be smooth and performed equally on both sides.
Two to One-Foot Hop and Hold	Two to one-foot hop and hold is a balance test in which the child starts with feet in a comfortable distance apart, hops forward, and lands on one foot. The child tries to recover and keep balance after landing.
Single-Leg Sideways Hop and Hold	Single-leg sideways hop and hold is a balance test in which the child tries to recover and keep balance after landing. The child starts by standing on one leg, jumps to the side of the free leg, lands with the free leg, and holds the position for three seconds.
Leaping	Leaping is the action of propelling the body forward and is performed by taking off on one foot and landing on the other foot. The movement should be smooth and performed equally on both sides
90-Degree Hop and Hold	The 90-degree hop and hold is a balance test in which the child stands on the right leg, hops and turns 90 degrees to the right, and lands on the right foot. The child tries to recover and keep balance after landing. The 90-degree hop and hold is repeated using the left leg.

for a description of each skill in the ChildFIRST [14].

The current version of the ChildFIRST uses a dichotomous scale to score the four evaluation criteria of each movement. The ChildFIRST was tested for face and content validity using a modified Delphi method [14,15]. While the reliability of the ChildFIRST was established, the tool could benefit from further refinement [11]. While a dichotomous approach provides ease of use and minimizes rater burden, it may limit the instrument’s sensitivity to detect partial achievement in evaluation criteria. In contrast, Likert-type scales can improve score precision by measuring more nuanced gradings in performance [16,17,18]. This is especially important in tools like the ChildFIRST, where binary scoring may mask variability in movement competence. Comparing these scales is important, as the trade-off between precision and reliability is important when considering the utility of an instrument. Likert-type scales allow for more sensitive measurements when compared with dichotomous scales, offering a continuum of responses that may better reflect nuanced differences in movement quality [16]. However, this increased level of detail may also create ambiguity in interpretation if raters lack precision or if scale intervals are not clearly defined [19]. While Likert scales could enhance sensitivity, they may also introduce rater subjectivity and reduce feasibility in field settings. Recent studies have shown Likert scales to be susceptible to middle and extreme response bias, which can reduce inter-rater reliability [20], and these elements need to be considered when choosing an evaluation method. The need for efficient and accurate assessment tools to identify movement deficits is critical to evaluate not only what is being assessed but how it is scored. Evaluating rater performance and preference across both scoring formats provides important insight into the optimal scoring strategy for field deployment of the ChildFIRST. Therefore, evaluating both scoring systems within the same cohort allows us to determine whether theoretical improvements in scale precision translate into practical gains in reliability, usability, and user preference for injury screening.

Therefore, the primary objective of this study is to compare the intra-rater and inter-rater reliability of the ChildFIRST using a dichotomous and a seven-point Likert scale (

Table A2. Seven-point Likert scale.

1	2	3	4	5	6	7
Strongly Disagree	Agree	Somewhat agree	Undecided	Somewhat disagree	Disagree	Strongly Agree

). The secondary objective of this study is to determine the user preference between the two scales. We hypothesized that each movement skill would demonstrate good to excellent ICCs (ICC > 0.75) and improved intra-rater and inter-rater reliability when using a seven-point scale.

2. Materials and Methods

2.1. Study Design

The existing version of the ChildFIRST uses a dichotomous scale to score the evaluation criteria. For comparison, we used a seven-point Likert scale for each evaluation criteria. We conducted two testing sessions during which raters watched recordings of the ChildFIRST movement skills being performed. The raters scored the same videos using the dichotomous and seven-point scale in each session. We conducted the testing sessions seven days apart to decrease the likelihood of the raters remembering their scores yet remaining familiar with the ChildFIRST criteria. For reference, participants refer to the children recorded performing the ChildFIRST movement skills, and raters refer to those assessing the videos to establish the reliability of the tools. Following the institutional ethics approval, the children in the recordings provided assent and their guardians provided informed consent prior to using their videos in the study. Raters provided informed consent prior to participation. The raters had no access to the names or other demographic information of the children. The videos were shown on a screen, and the videos were not provided to the raters.

2.2. Participants and Videos

Participants were eligible to participate in the study if they were between 8 and 12 years old. We recruited eight participants aged 8–12 years from a local Taekwondo club through convenience sampling, so they were an active population. We recorded them performing the 10 movement skills of the ChildFIRST, following the procedure of a previous study using the ChildFIRST [14]. We demonstrated each movement skill without giving the participants instructions that are directly related to the evaluation criteria to avoid influencing participants’ movements. We recorded the participants performing each skill in the frontal and sagittal plane to give raters enough information to accurately assess each evaluation criteria. The ChildFIRST focuses on determining lower limb injury risk, which is closely linked to dynamic lower limb alignment [15]. Therefore, filming in both the frontal and sagittal planes is important for raters to have all the information they need to assess the evaluation criteria related to lower limb alignment. We selected 4 videos for each movement skill for a total of 40 videos to present to the raters. We created two presentations with the videos in different orders to be used in the testing sessions: one presentation for the dichotomous scoring and another for the seven-point scoring of the ChildFIRST. The identity of the participants in the recordings remained anonymous and all procedures were approved by the institutional human research ethics committee.

2.3. Raters

We recruited fifteen raters using convenience sampling from a health, kinesiology, and exercise-oriented university program. All raters successfully completed university-level courses in musculoskeletal and systemic anatomy, physiology, and strength and conditioning, giving them a good understanding of fundamental movement skills. Having a larger number of raters could have improved our results.

2.4. Procedures

On the first day of testing, each rater filled out a demographic questionnaire. We then gave the raters a 45 min ChildFIRST training session. During these sessions, the raters were exposed to a variety of skill achievements to anchor a child’s success on the evaluation criteria. We explained each movement skill and how to score each evaluation criteria for the dichotomous and seven-point versions of the ChildFIRST. For the dichotomous scale, a score of 0 is given when a criterion is not observed and a score of 1 when a criterion is observed. The seven-point scale Likert scale version of the ChildFIRST used the following scale: 1—Strongly Agree; 2—Agree; 3—Somewhat Agree; 4—Undecided; 5—Somewhat Disagree; 6—Disagree; 7—Strongly Disagree. We instructed raters to score each evaluation criteria using the seven-point scale based on their agreement with the level of achievement of each evaluation criteria.

On day 1, raters watched the videos of participants performing the movements and scored them using the dichotomous scale. We played each video twice with a 20 s pause between recordings to allow raters enough time to score the movement skills. After a five-minute rest period, the raters watched the same videos in a counterbalanced order and scored the movement skills with the seven-point scale. The first day of testing lasted 2.5 h in total.

On the second day, the raters participated in a refresher course of the ChildFIRST skills, evaluation criteria, and scoring options. The raters then scored the movement skills first using the seven-point scale, followed by the dichotomous scale. The two scoring sessions were separated by a 5 min break. The second testing session lasted for approximately 2 h. The raters also completed a feedback survey. The survey included questions soliciting the raters’ opinions of each scale, with the following questions: “What scoring system is more practical? What scoring system is more feasible? What scoring system do you prefer?”. These questions were answered via an electronic survey form.

2.5. Statistical Analysis

We used Python 3.11 (Python Software Foundation, https://www.python.org/, accessed on 10 April 2022) for all statistical analyses and compared the total scores of each movement skill for each rating scale between all raters on the first and second days to establish inter-rater reliability for each given day. The total scores for each movement skill for the dichotomous version range from 0 to 4 and the total scores for each movement skill for the seven-point version range from 0 to 28. To determine the intra-rater reliability of each scale, we compared the total scores for each movement skill for the first and second viewing sessions. We followed recommended guidelines for our statistical analysis which follows the model used by the authors of the first ChildFIRST reliability study to allow for comparison [11,21]. We calculated the intraclass correlation coefficients (ICCs) and absolute agreement using multiple raters (ICC 2, k), and we used a 95% confidence interval (CI) to establish inter- and intra-rater reliability for the total score for each movement. The strength of agreement is as follows: ICC = 0.00–0.50, poor; ICC = 0.50–0.75, moderate; ICC = 0.75–0.90, good; ICC = 0.90 to 1.00, excellent [21]. General guidelines recommend an ICC ˃ 0.75 to establish acceptable reliability [21]. While generalizability theory and item response theory offer the advanced modeling of rater error and scale properties, ICC remains a widely accepted, interpretable, and replicable method for assessing inter-rater and intra-rater reliability in observational tools [21]. Our decision to use the number of raters and ICC procedures aligns with the prior validation of the ChildFIRST tool, allowing direct comparison with earlier findings while maintaining methodological consistency [11].

3. Results

3.1. Rater Demographics

A total of 15 raters were initially recruited. One rater dropped out and fourteen raters completed the study. The mean age of the raters was 24.92 ± 4.02 years old. Nine of the raters were seniors; one rater was a sophomore; one rater was a junior; two raters were MSc candidates; and one rater was a PhD student. The mean years of experience working with children was 3.71 (range 0 to 7.82 years).

3.2. Dichotomous ChildFIRST Scoring Scale

On the first day of testing, the dichotomous scale demonstrates excellent inter-rater reliability (

Table 2. Dichotomous scale: inter-rater and intra-rater reliability total scores for each movement skill.

	Inter-Rater Reliability Day 1		Inter-Rater Reliability Day 2		Intra-Rater Reliability
Movement Skills (N = 14)	ICC (2, k)	95% CI	ICC (2, k)	95% CI	ICC (2, k)	95% CI
Walking Lunge	0.86	[0.55–0.99]	0.75	[0.21–0.98]	0.91	[0.72–0.99]
Bodyweight Squat	0.9	[0.68–0.99]	0.93	[0.77–0.99]	0.96	[0.86–1.00]
Vertical Jump	0.44	[−0.39–0.95]	0.75	[0.26–0.98]	0.78	[0.36–0.98]
Horizontal Jump	0.91	[0.71–0.99]	0.91	[0.69–0.99]	0.95	[0.85–1.00]
Running	0.92	[0.86–0.96]	0.91	[0.84–0.96]	0.96	[0.94–0.98]
Leaping	0.81	[0.65–0.92]	0.76	[0.56–0.9]	0.88	[0.78–0.95]
Two to One-Foot Hop and Hold	0.95	[0.83–1.00]	0.92	[0.74–0.99]	0.97	[0.89–1.00]
Single-Leg Sideways Hop and Hold	0.45	[−0.14–0.95]	0.81	[0.41–0.99]	0.83	[0.52–0.99]
Single-Leg Hop	0.72	[0.17–0.98]	0.72	[0.22–0.98]	0.86	[0.57–0.99]
90-degree Hop and Hold	0.97	[0.86–1.00]	0.97	[0.89–1.00]	0.98	[0.94–1.00]
All movement skills EC	0.92	[0.90–0.94]	0.9	[0.88–0.92]	0.95	[0.94–0.96]
All movement skills composite for each skill	0.91	[0.86–0.95]	0.9	[0.85–0.94]	0.9	[0.85–0.94]

N, sample size; ICC, intraclass correlation; CI, confidence interval; EC, evaluation criteria.

; ICC ˃ 0.90) in five movement skills, bodyweight squat, horizontal jump, running, two to one-foot hop and hold, and 90-degree hop and hold, and good inter-rater reliability (ICC ˃ 0.75) in the walking lunge and leaping, but poor inter-rater reliability (ICC < 0.50) in the vertical jump and the single-leg sideways hop and hold. On the second day of testing, the dichotomous scale demonstrates excellent inter-rater reliability (ICC ˃ 0.90) in five movement skills, bodyweight squat, horizontal jump, running, two to one-foot hop and hold, and 90-degree hop and hold; good inter-rater reliability (ICC ˃ 0.75) in four movement skills, the walking lunge, vertical jump, leaping, and single-leg sideways hop and hold; and moderate inter-rater reliability (ICC = 0.72) in the single-leg hop. The dichotomous scale demonstrates excellent intra-rater reliability (ICC ˃ 0.90) in six movement skills, walking lunge, bodyweight squat, horizontal jump, running, two to one-foot hop and hold, and 90-degree hop and hold, and good intra-rater reliability (ICC ˃ 0.75) in the remaining movement skills: vertical jump, single leg-hop, leaping, and single-leg sideways hop and hold.

3.3. Seven-Point ChildFIRST Scoring Scale

On the first day of testing, the seven-point scale demonstrates good inter-rater reliability (

Table 3. Seven-point scale: inter-rater and intra-rater reliability total scores for each movement skill.

	Inter-Rater Reliability Day 1		Inter-Rater Reliability Day 2		Intra-Rater Reliability
Movement Skills (N = 14)	ICC (2, k)	95% CI	ICC (2, k)	95% CI	ICC (2, k)	95% CI
Walking Lunge	0.52	[0.11–0.95]	0.71	[0.15–0.98]	0.74	[0.36–0.98]
Bodyweight Squat	0.89	[0.66–0.99]	0.99	[0.95–1.00]	0.97	[0.92–1.00]
Vertical Jump	0.53	[0.15–0.95]	−0.32	[−2.05–0.88]	0.48	[0.06–0.94]
Horizontal Jump	0.74	[0.36–0.98]	0.89	[0.66–0.99]	0.89	[0.67–0.99]
Running	0.76	[0.55–0.89]	0.92	[0.86–0.96]	0.92	[0.77–0.99]
Leaping	0.63	[0.37–0.83]	0.87	[0.75–0.95]	0.79	[0.48–0.98]
Two to One-Foot Hop and Hold	0.77	[0.4–0.98]	0.89	[0.63–0.99]	0.9	[0.7–0.99]
Single-Leg Sideways Hop and Hold	0.65	[0.24–0.97]	0.84	[0.51–0.99]	0.85	[0.59–0.99]
Single-Leg Hop	0.8	[0.46–0.98]	0.95	[0.84–1.00]	0.94	[0.82–1.00]
90-degree Hop and Hold	0.85	[0.56–0.99]	0.96	[0.85–1.00]	0.95	[0.84–1.00]
All movement skills EC	0.85	[0.78–0.9]	0.93	[0.92–0.95]	0.95	[0.93–0.96]
All movement skills composite for each skill	0.74	[0.54–0.85]	0.92	[0.87–0.95]	0.92	[0.87–0.95]

N, sample size; ICC, intraclass correlation; CI, confidence interval; EC, evaluation criteria.

; ICC ˃ 0.75) in five movement skills, bodyweight squat, running, 90-degree hop and hold, two to one-foot hop and hold, and single-leg hop, and moderate inter-rater reliability (ICC ˃ 0.50) in the remaining movement skills: walking lunge, vertical jump, horizontal jump, leaping, and single-leg sideways hop and hold. On the second day of testing, the seven-point scale demonstrates excellent inter-rater reliability (ICC ˃ 0.90) in the bodyweight squat, running, single-leg hop, and 90-degree hop and hold; good inter-rater reliability (ICC ˃ 0.75) in the horizontal jump, leaping, two to one-foot hop, and hold and single-leg sideways hop; moderate inter-rater reliability (ICC = 0.71) in the walking lunge; and poor inter-rater reliability (ICC = −0.32) in the vertical jump. The seven-point scale demonstrates excellent intra-rater reliability (ICC ˃ 0.90) in five movement skills, bodyweight squat, running, two to one-foot hop and hold, single-leg hop, and 90-degree; good intra-rater (ICC ˃ 0.75) reliability in leaping and single-leg sideways hop and hold; moderate intra-rater reliability (ICC = 0.74) for the walking lunge; and poor intra-rater reliability (ICC = 0.48) in the vertical jump.

3.4. Feedback Survey

Twelve raters completed the feedback survey (

Table 4. Rater satisfaction survey.

Survey Question (N = 12)	Dichotomous Scale	Seven-Point Scale
What scoring system is more practical?	11 (91.6%)	1 (8.4%)
What scoring system is more feasible?	9 (75%)	3 (25%)
What scoring system do you prefer?	8 (66%)	4 (34%)

N = sample size.

). The results showed that 91.6% of raters think the dichotomous scale is more practical, 75% of raters think the dichotomous scale is more feasible, and 66% of raters prefer to use the dichotomous scale.

4. Discussion

The primary objective of this study was to compare the reliability of the dichotomous scale to the seven-point scale. Both scales had similar results with slightly greater inter-rater and intra-rater reliability with the dichotomous scale. The dichotomous scale showed greater overall inter-rater reliability scores for each movement skill when looking at the composite scores for both days of testing. On the first day of testing, only three movements scored poorly for the dichotomous scale: the single-leg sideways hop and hold, the single-leg hop, and the vertical jump. However, these movement skills demonstrated better inter-rater reliability on the second day of testing, indicating that raters scored more consistently with more experience. The seven-point scale only scored better for inter-rater reliability than the dichotomous scale in both testing sessions for the single-leg hop and the single-leg sideways hop and hold. These results are consistent with a previous reliability study [11] except for a few movement skills. For instance, in the previous reliability study the 90-degree hop and hold had exceptionally poor inter-rater reliability, whereas in our study both scales demonstrated good reliability. Overall, the intra-rater reliability results from our study are an improvement compared with the intra-rater reliability from the previous reliability study. These improvements could be due to the addition of a training session and the better preparation of the raters.

The single-leg hop and the single-leg sideways hop and hold had greater reliability with the seven-point scale. The single-leg hop has the following evaluation criteria: hips, knees, and ankles aligned; take off from one foot, land on the same foot; knee and hip bend to land softly in a controlled fashion; and swing arms to assist the movement. The swing arms to assist the movement evaluation criterion has multiple components to observe simultaneously. The child needs to be swinging their arms, but the swinging must also be assisting the movement to drive their momentum forward. Raters must therefore determine if the swinging of the arms is present and whether it is sufficiently contributing to propelling their movement forward to consider the criteria observed. The single-leg sideways hop and hold has the following evaluation criteria: knees and hips bend to land softly in a controlled fashion; hips, knees, and ankles aligned; foot flat on the floor; and stand up straight within 3 s after landing. Quickly after performing the single-leg sideways hop and hold, children frequently put their non-weight-bearing foot down. The evaluation of the single-leg sideways hop and hold skill must depend on the instant prior to the children putting their foot down. The children likely put their foot down to maintain their balance or to prepare for the next repetition. The difference can be subtle, but it is key when evaluating the movement skill and increases the difficulty of rating the single-leg sideways hop and hold. Subtleties in the evaluation criteria could benefit from the nuance of the seven-point scale.

The primary objective of the ChildFIRST is to screen for high-risk movements and the risk of lower limb injury. Having a reliable measurement of lower limb alignment during dynamic movements is important to screen for injury risk. Eight out of ten movement skills with the hips, knees, and ankles aligned evaluation criteria showed greater overall reliability with the dichotomous scale, which is an important consideration when comparing both scales. One of the main mechanisms of injury for anterior cruciate ligament (ACL) tears is landing with the tibia in internal/external rotation and/or knee valgus [22,23]. The 90-degree hop and hold movement skill is likely to recreate a mechanism of injury for ACL injury [11]. Therefore, children who perform poorly in this movement, leading to a loss of lower limb alignment, would increase their risk of suffering an ACL injury. The 90-degree hop and hold achieved excellent inter-rater and intra-rater reliability scores using the dichotomous scale which supports the use of the dichotomous version of the ChildFIRST. Its use could be helpful as a screening tool in ACL injury prevention, but long-term data would aid this perspective.

Caution should be taken when interpreting ChildFIRST movement skill reliability scores. The interpretation of movement skill reliability scores with different evaluation criteria depends on the focus of the evaluation criteria. The ChildFIRST has two objectives, measuring physical competence and screening for high-risk movements. To meet these objectives, each movement skill includes evaluation criteria that are focused on movement competence and evaluation criteria focused on screening for technique errors. For instance, one of the evaluation criteria of the single-leg hop is swing arms to assist the movement. Disagreement among raters for the swing arms to assist the movement evaluation criteria could lead to lower reliability scores for the single-leg hop. However, swinging of the arms is an evaluation criterion with an emphasis on assessing movement competence, not injury risk. With movement skills having evaluation criteria focused on both factors, the interpretation of each movement skill reliability score and their relative importance will vary depending on the primary objective of the evaluation criteria and of the movement skill. It is important to note that the ChildFIRST is not yet considered to have predictive capacity.

Both scales showed improvements in inter-rater reliability during the second testing session. This trend suggests that the reliability of the tool improves when raters gain experience and are more accustomed to it. Considering that the ChildFIRST requires adequate training to be used optimally [11] the improvement in reliability on the second day is important to note. The seven-point scale demonstrates weaker scores in both the first and second testing sessions compared with the dichotomous scale. The lower reliability of the seven-point scale could be due to its complexity. The seven-point scale is a Likert scale with a wide range of possible answers which can cause the magnitude between values to become unclear [24]. As the magnitude becomes unclear, there is greater variability in answers [24]. Especially at the high and low ends of the scale, Likert scales can lead raters to confound the intensity of their agreement leading to the under-reporting of the highest levels of agreement or disagreement. The under-reporting of the high and low ends of the scale could be due to the type of data common for a Likert scale. The scale in our study should be considered ordinal data because the increments between each level cannot accurately be deemed equal. Due to this interpretation, there may be differences between raters’ interpretation of the seven-point scale [25], which may explain the lower inter-rater reliability. Each level on the seven-point scale is difficult to define in terms of what observable behavior we would expect to see for each of the seven scores. For instance, one rater’s five may be rated differently by another rater’s (scoring something other than five) which will lead to consistent discrepancies in scoring.

The ChildFIRST is designed to be used in a physical education or sports team setting and administered simultaneously to a large group of children. The feedback survey showed overall better rater satisfaction of the dichotomous scale. Raters overwhelmingly preferred using the dichotomous scale; they found it more practical and feasible to use. We must consider rater preference because a higher preference, practicality, and feasibility should improve the acceptance of the ChildFIRST across a population of raters with different levels of expertise. Even discounting the raters’ opinions, the dichotomous version demonstrates better inter-rater and intra-rater reliability than the seven-point version.

Limitations

We did not collect any clinical data during this study, which may limit our ability to demonstrate the utility of the ChildFIRST. Additionally, while children were of a variety of skill levels, all were active. We did not collect the race or ethnicity data of our participants, which may reduce the generalizability. Increasing the pool of participants in future studies to include less-skilled or elite athletes would aid the generalizability of the tool.

There are some additional limitations in our study that may reduce the value of our interpretations. Some video subjects were wearing pants which made it more difficult to see lower limb alignment in the videos and could negatively affect the ability of the raters to properly score some of the criteria, specifically the hip, knees, and ankles aligned evaluation criterion. The vertical jump demonstrated weaker reliability than the horizontal jump, despite having the same evaluation criteria. It is difficult to determine if the poor reliability scores of the vertical jump are a function of the seven-point scale, as the inter-rater reliability on day 1 was also poor using the dichotomous scale. However, the day 2 inter-rater values, as well as the intra-rater values using the dichotomous scale, were moderate. These values give rise to the limitation of our videos. The vertical jump videos are shorter in length since participants do not need to reset between repetitions. The shorter time between repetitions shortens the recording, giving raters less time to analyze the movement and each evaluation criteria.

Most of the viewing sessions were conducted in person; however three participants were tested using remote viewing sessions via video conference for reasons out of our control. One participant was present in-person for the first viewing session but had to do the second viewing session remotely. While we have successfully evaluated participants remotely, these discrepancies may have affected the results. The poor reliability on the first day of testing for the seven-point scale could be in part due to rater fatigue and decreased attention. The first testing session included a training session, and the seven-point scale was used second after the dichotomous scale. The attention span in university students decreases steadily after 20 min of exposure to the same subject [26]. The second day of testing had no training, only a refresher, which reduced the total time of the session. Therefore, the length of the session may not have affected the dichotomous scale as much even though it was in the second part of the session. Finally, our analysis did not include the agreement at the level of each evaluation criteria. Future studies should build on these results by incorporating generalizability theory to better understand sources of rater variance or item response theory to model item-level performance across different rater backgrounds.

5. Conclusions

The dichotomous scale demonstrated good to excellent inter-rater reliability (ICC = 0.75–1.00) in seven of ten movement skills on the first day of testing and nine of ten movement skills on the second day of testing, and good to excellent intra-rater reliability in all movement skills. This was marginally better than the inter-rater reliability and intra-rater reliability of the seven-point scale, which demonstrated good to excellent inter-rater reliability in five of ten movement skills on the first day of testing and eight of ten movement skills on the second day of testing, and good to excellent intra-rater reliability in eight of ten movement skills. The dichotomous scale also demonstrated higher satisfaction among raters. The practical implications of our results suggest that coaches, educators, or health professionals should retain the dichotomous scale when using the ChildFIRST in real-world settings and future studies. We believe that the preference of raters for using the dichotomous scale is an important consideration for determining the scale used, but our results do not lead us to recommend a change.