Review Reports - Intra- and Inter-Rater Reliability of a Systematic Video Analysis of ACL Injuries in Elite Men’s Football

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

I would like to begin by congratulating the authors for this well-structured and clearly written manuscript. The study addresses a highly relevant topic regarding the intra- and inter-rater reliability of systematic video analysis for ACL injuries in elite men’s football, and it has potential value for improving epidemiological data collection and observational research in sports injury mechanisms. However, several aspects of the manuscript require clarification and revision.

The Statistical Analysis section and the corresponding Results in Tables 1 and 2 present Cohen’s kappa and Intraclass Correlation Coefficients (ICC) to demonstrate reliability. However, confidence intervals (95% CI) for both ICC and kappa coefficients are not provided. This limits the precision and interpretability of the reliability estimates, particularly considering the relatively small subsample used for the reliability analysis. In reliability studies, reporting only point estimates is insufficient, as the uncertainty around the estimates is essential for judging the robustness of the findings. The authors should include 95% CIs for all reliability metrics in both the text and the tables.
The authors state that a subsample of 25 injury cases was randomly selected from the initial cohort of 119 injuries to perform the reliability analysis. Although this sample size may be consistent with previous methodological recommendations, the justification should be strengthened. Given that the checklist includes several categorical variables with multiple response categories, some kappa estimates may be unstable when category frequencies are low or unbalanced. The authors should provide a clearer rationale for the adequacy of this sample size for all assessed variables and, ideally, report the distribution of responses across categories to allow readers to assess the robustness of the reliability estimates.
The Methods section indicates that a “sub-sample of 25 injury cases was randomly selected” from the initial cohort of 119 injuries. However, the randomization procedure is not described in sufficient detail. Although the Acknowledgments section mentions that Gemini was used to randomly select the twenty-five injuries, this information should be included in the Methods section rather than only in the acknowledgments. To ensure reproducibility and transparency, the authors should describe the randomization procedure in detail, including the tool or software used, whether a random seed was applied, and how the second subsample for the neurocognitive variables was selected.
There appears to be an inconsistency regarding the precipitation variable. The Methods section states that no statistical analysis could be performed for precipitation due to the total absence of variance in both intra- and inter-rater responses. However, the Results section later reports perfect agreement for precipitation. This creates a statistical reporting inconsistency. The authors should clarify whether Cohen’s kappa was actually calculated for this variable. If the absence of variance prevented the estimation of kappa, precipitation should not be reported as κ = 1.000 or interpreted as “perfect agreement.” Instead, the authors should state that reliability could not be estimated due to lack of variability.
In the Ethical Considerations section, the authors state that formal approval from an institutional ethics committee was not required because the video footage was publicly available and player data were anonymized. While this is a common rationale in studies using publicly available sports footage, a self-declaration by the authors may not be sufficient to fully align with MDPI ethical standards. The authors should clarify whether an Institutional Review Board or local ethics committee formally granted a waiver or exemption for this study. Additionally, the manuscript should explicitly state whether the study was conducted in accordance with the Declaration of Helsinki.
The inclusion of attentional inhibition and motor response inhibition is interesting and innovative. However, these variables involve inference about internal cognitive processes based only on external video footage. The authors correctly acknowledge that these variables showed lower agreement than contextual and biomechanical variables. Nevertheless, the manuscript should further discuss the limitations of inferring neurocognitive processes from 2D broadcast videos. The authors should clarify whether raters received specific training or operational definitions for these variables and whether any examples or calibration procedures were used before the formal analysis.
The finding that visual speed estimation was unreliable is important and clinically relevant. However, the manuscript should more clearly distinguish between variables that can be reliably assessed using standard video footage and variables that require objective tracking technologies. Since horizontal and vertical speed showed poor-to-moderate reliability, these variables should not be treated with the same level of confidence as contextual variables such as injury side, playing phase, or situational pattern. The authors should consider explicitly recommending that visual speed estimation be removed or substantially modified in future versions of the checklist unless objective tracking data are available.
The Conclusion summarizes the primary findings adequately but could be strengthened by adopting a more cautious tone. While the checklist is shown to be reliable for several contextual and situational variables, the statement that video analysis can inform “better decision making” and “a more effective ACL injury prevention strategy” extends slightly beyond the scope of a reliability study. The manuscript does not directly test injury prevention strategies or clinical decision-making outcomes. I recommend revising the conclusion to avoid overly broad or intervention-oriented language and to align more closely with the specific psychometric findings of the study.
The manuscript reports that statistical analyses were performed using JASP statistics software. However, the version is reported as “v. 0.95.4,” which should be checked carefully, as this version number may not correspond to the current standard versioning format of JASP. Please verify and report the exact software version used.
The manuscript uses terms such as “match minute,” “individual minutes played,” and “effective match minute.” Please ensure that these terms are used consistently throughout the Methods, Results, and Tables. If these variables refer to different constructs, their definitions should be clearly stated.
The manuscript states that two independent observers evaluated the video footage according to a predetermined checklist. However, it is not fully clear whether the observers underwent standardized training or calibration before data collection. Please clarify whether any training session, pilot coding, consensus meeting, or familiarization process was performed prior to the formal reliability analysis.
The authors mention that multiple camera angles and slow-motion replays were added when available. However, it would be useful to clarify whether all raters had access to the same video clips, camera angles, and replay conditions. This is important because differences in video availability could influence inter-rater agreement.

Author Response

Response 1:

We thank the reviewer for pointing this out. We have now included the 95% Confidence Intervals (95% CI) for all Intraclass Correlation Coefficients (ICC) and Cohen’s kappa metrics in the Results section, and Tables 1 and 2. Additionally, we have included a reference to the use of 95% CI in the Statistical Analysis subsection of the Methods:

“…with corresponding 95% Confidence Intervals (95% CI).” and “…reported alongside their 95% CIs.”.

Response 2:

We agree with the insightful observation that variables with multiple response categories are susceptible to kappa instability when frequencies are low or unbalanced. Our subsample of 25 cases represents ~21% of the entire study cohort (n = 119). We believe that this is a robust and highly representative fraction for reliability assessments in sports medicine video analysis. Furthermore, statistical guidelines (Walter et al., 1998; Koo & Li, 2016) indicate that 20 to 30 cases provide adequate statistical power when the expected reliability is high, as it was hypothesized for our macro-contextual variables. However, to ensure complete transparency, we have created a new supplementary table with the exact distribution of responses across all categories for the assessed variables (Supplemental Table B1 and B2). The Video Analysis Procedures subsection has also been updated to reflect this rationale and to reference the new supplemental table:

“From this initial cohort of 119 included for analysis, a sub-sample of 25 injury cases was randomly selected to perform the reliability analysis. This represents approximately 21% of the total cohort, providing a highly representative fraction for evaluating observational agreement. This sample size is consistent with established methodological guidelines for reliability studies involving two raters and expecting high reliability coefficients, which suggest that 20 to 30 heterogeneous cases are sufficient to achieve adequate statistical power when high reliability coefficients are expected (Walter et al., 1998; Koo & Li, 2016). However, because the checklist includes several categorical variables with multiple response options, we acknowledge that kappa estimates can be sensitive to low or unbalanced category frequencies. Therefore, the exact distribution of intra- and inter-rater responses across all categories are reported in Supplemental Table B1 and B2.”.

Response 3:

Thank you for this suggestion. We have added the information regarding the use of the Gemini AI model for randomization into the Methods section. We also clarified that the randomization for the second sample of neurocognitive variables was performed using the same method. We added a sentence and adjusted the following in the “Video Analysis Procedures” subsection to be clearer. It now states:

“The randomization procedure was performed using the Google generative artificial intelligence assistant, Gemini. Concerning neurocognitive variables, another subsample of 25 injuries from the 89 non-contact and indirect contact injuries was randomly selected using the same method, …”

Response 4:

We apologize for this inconsistency. We have corrected the Results section by removing the statement indicating perfect agreement (κ = 1.000) for precipitation. We added a sentence to each of the intra-rater and inter-rater reliability sections to clarify that Cohen’s could not be estimated for this variable due to a total lack of variability:

“As previously stated, reliability for precipitation could not be estimated due to a total lack of variability.”

“Similarly, the inter-rater reliability for precipitation could not be estimated due to a lack of variance.”

Response 5:

We appreciate the reviewer's attention to this important detail. We must clarify that a formal waiver or exemption from an Institutional Review Board (IRB) or local ethics committee was not sought. Because our study relied exclusively on the secondary use of publicly available broadcast footage and anonymized data from a commercial platform, it does not meet the standard regulatory definition of human subjects research that requires formal committee review. However, we agree with the need to respect international ethical standards. We have revised the Ethical Considerations section to provide a more robust justification for the absence of formal ethical approval, and we have explicitly added that the study was conducted in accordance with the principles of the Declaration of Helsinki.

“Nevertheless, the study was conducted in strict accordance with the ethical principles outlined in the Declaration of Helsinki.”.

Response 6:

This is a highly relevant point. The assessment of neurocognitive variables presented an additional layer of complexity to the observational analysis. We have expanded the Discussion to further address the inherent limitations of inferring internal cognitive processes from external 2D broadcast videos:

“… Despite the consensus meeting and familiarization with the checklist, the evaluation of internal cognitive processes based solely on 2D broadcast video footage remains inherently subjective, as raters are required to infer internal neurocognitive processes from contextual behavioral cues rather than from directly measurable features. This discrepancy is likely to account for the lower consistency observed for these variables compared to macro-contextual ones. Future research should explore the potential of more detailed decision trees or enhanced rater training to further enhance agreement for these constructs.”

Additionally, the familiarization process with the software and checklist has also been detailed in the "Video Analysis Procedures" section:

“The evaluation team members have diverse professional backgrounds, including biomedical engineering (SO), sports science (KS), and epidemiology, biostatistics, and research in health (MC, currently a medical student). Before data collection, the raters underwent a familiarization process at a dedicated consensus meeting to review the checklist, align operational definitions, and establish standardized evaluation criteria.”

Response 7:

We agree with this clinical perspective. We have revised the Discussion section to explicitly recommend that visual speed estimation be removed from the checklist in future systematic video analyses, unless objective tracking technologies (e.g., optical tracking or GPS) are available:

“Consequently, it is strongly recommended that visual speed estimation be removed or substantially modified in future versions of systematic video analysis checklists, as it cannot be treated with the same level of confidence as contextual variables. Researchers should replace subjective visual assessment with objective data driven from synchronized optical tracking technologies, GPS wearable microtechnology, or specialized computer vision algorithms.”.

Nevertheless, we believe that the inclusion of horizontal and vertical speed variables in this reliability study, as well as the poor results obtained, was highly important in drawing these conclusions.

Response 8:

Thank you for the suggestion. We have toned down the Conclusion section by removing the broader and intervention-oriented statements regarding “better decision making” and “more effective ACL injury prevention strategy”. The Conclusion is now strictly aligned with the specific psychometric and methodological findings of the reliability study:

“The validation of these observational tools is a crucial step in ensuring high-quality, precise, and reproducible systematic video analysis data, which can robustly support future epidemiological and biomechanical research in sports medicine and injury prevention.”

Response 9:

We apologize for the error regarding the version numbering format. In addition, the software version mentioned was incorrect. We have already updated this in the Statistical Analysis” section to “v0.97.1”.

Response 10:

We have reviewed the entire manuscript and standardized the terminology. The term "effective match minute" has been completely replaced with "individual minutes played" throughout the text to ensure consistency.

Response 11:

We have clarified in the Video Analysis Procedures subsection of the Methods section how the observers were prepared for data collection. While a formal pilot coding session was not performed, all observers underwent a familiarization process that consisted of a dedicated consensus meeting where the raters thoroughly reviewed the checklist, aligned operational definitions, and established standardized evaluation criteria:

“Before data collection, the raters underwent a familiarization process at a dedicated consensus meeting to review the checklist, align operational definitions, and establish standardized evaluation criteria.”

Response 12:

We have added a statement in the Video Analysis Procedures subsection of the Methods section confirming that all raters had access to the exact same video clips, camera angles, and slow-motion replays on the platform to ensure equal observational conditions for inter-rater reliability.

“During the formal evaluations, all raters had access to the exact same video clips, camera angles, and slow-motion replay conditions on the platform to ensure standardized observational conditions.“

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

This is a reliability study validating the consistency and degree of agreement from video analysis of soccer matches. The authors acknowledge the limitations of their study which is a plus for the review report. However, the importance and impact of their work is not efficiently communicated and it is suggested they work to improve that aspect. In that context, the Discussion could be improved by linking the impact of correctly estimating an ACL injury from a video analysis to the next steps, for instance providing feedback to both coaches and athletes about risk factors, technique correction and perhaps early detection of other risky situations.

Specific other comments and observations follow below:

Abstract: no comments

Introduction: no comments

Methods:

L58: a symbol indicating trademark properties or copyright ones for the platform should be included here.

L76: "consistent with previous literature, was estimated at 40 ms following initial contact (IC)...": ": the authors should report the studies that provided evidence about the suggested time period of the 40ms to justify its use. Why wasn't this 40ms time period used in all analyses? L72-73 report a shorter by almost 50% time period for analysis, which creates a serious discrepancy in how the videos were analyzed.

L77-78: This needs to be clarified more. If understood correctly, the majority of the video footage on which the reliability analysis was based on, came from 2D data, i.e. single camera angle? if so, the authors must inform readers about their methodological procedures in more detail.

L82-95: it is suggested to present the investigated variables in a more organized way; divide them in contextual and biomechanical and present them accordingly.

L114-115: the authors need to define the professional background of the raters and if they received any level of training with the software or the checklist.

L118: are there references to support the minimum of 30 days interval?

Results:

L139: The authors do not report the outcome of the normality assessment analysis and if normality was violated, what measures did they take.

L141-143: a similar statement is provided in the Discussion. To avoid repetitions, it is recommended that it is removed from this section.

L152-153: "body location" is an anatomical detail rather a biomechanical one; further, the word location is a general word and if the authors desire to use appropriate terminology, then perhaps they consider options like body part, affected area or affected body part, or anatomical area.

L156: In my opinion, including visual speed estimation in the video analysis checklist is erroneous. Subjective visual perception is highly unreliable and since the observer and the phase he/she observes are not moving in the same speed, this is problematic. Present results confirm the unsuitability of this parameter.

Discussion:

L179: the study has a single purpose, so the adjective primary is unnecessary.

L238-242: This suggestion goes without saying. However, integrating qualitative video analysis with numerical data, changes the nature of this assessment method and transforms it into something else. Perhaps the authors could suggest the assessment of acceleration, which might be deduced by the change in body posture and the respective frames. So, perhaps by combining two pieces of information, two observers might agree better with each other. Of course, this scenario remains to be validated.

Author Response

Response to general comment:

We agree that the practical implications of our work were not sufficiently emphasized. We have now added a paragraph in the Discussion detailing how reliable systematic video analysis can directly impact clinical and coaching practices, specifically by providing practical feedback to coaches and athletes regarding risk factors, technique correction, and the early detection of risk movement patterns.

“Establishing this high reliability is not merely a methodological exercise, it has significant clinical and practical implications. Consistent and precise video analysis allows sports medicine professionals to accurately identify risky movement patterns and contextual triggers that precede ACL injuries. This robust data can then be translated into practical feedback for coaches and athletes, who can implement targeted technique corrections during training, adjust tactical behaviors that expose players to vulnerable positions, and ultimately improve the early detection of other risk situations, creating a more effective feedback loop for injury mitigation strategies.”.

Methods

L58:

Thank you for this observation. We have added the registered trademark symbol (®) to the Noisefeed platform in the Methods section.

L76:

We agree with the need to include references to support this point. Therefore, we have added the appropriate references [Della Villa, et al. (2020); Buckthorpe, et al. (2024)] to justify the 40-ms timeframe for the visible tissue failure following initial contact. Regarding the reported discrepancy, we respectfully clarify that two distinct time metrics are involved here. The "12-15 seconds before to 3-5 seconds after" refers to the total duration of the video clip assessed to evaluate the contextual, situational, biomechanical, and anatomical variables. Conversely, the "40 ms" refers to the specific micro-timeframe (milliseconds) within that clip where the actual biomechanical failure of the ligament occurs.

L77-78:

We have added more details to the “Video Acquisition and Processing” subsection. We clarified that all video footage was based on standard 2D broadcast data, from the main camera angle, which was subsequently supplemented by slow-motion or regular-speed replays from alternative angles, only when these were provided by the television broadcaster.

“Video footage relied on standard 2D broadcast data from the main broadcast camera angle. When available, multiple 2D camera angles and slow-motion replays were added to the sequence.”.

L82-95:

This is a very helpful suggestion for easier reading. We have reorganized the Observational Checklist and Variables subsection, clearly dividing and presenting the variables under categories: “Contextual and situational variables” and “Biomechanical and anatomical variables”.

L114-115:

We thank the reviewer for pointing this out. We have updated the Methods section to explicitly detail the diverse multidisciplinary professional backgrounds of our raters (SO has a background in biomedical engineering; KS in sports science; and MC in epidemiology, biostatistics, and health research). The familiarization process with the software and checklist has also been detailed in the "Video Analysis Procedures" section.

L118:

We appreciate the reviewer's comment. The 30-day interval (approximately 4 weeks) was selected as a conservative "washout" period to completely mitigate recall bias (memory effect). Sim & Wright (2005) report ed that an interval of 2 to 14 days is usual, but that this depends on the variable being measured. In this case, as this is an observational reliability study using video analysis, it was considered that an interval of 2 to 4 weeks would be sufficient to ensure raters do not simply remember their previous assessment. We have updated the manuscript to clarify this rationale and added a methodological reference (Sim & Wright, 2005) that highlights the need for an adequate time interval to minimize memory effects in intra-rater reliability studies.

Results:

L139:

We have added a sentence at the beginning of both subsections of the Results to report that the preliminary assumptions of normality (Shapiro-Wilk test) were met for the quantitative variables.

“For quantitative variables, preliminary assessment confirmed that the data met the assumptions of normality (Shapiro-Wilk test, p > 0.05).” and “…again after confirming the assumption of normality (Shapiro-Wilk test, p > 0.05).”

L141-143:

We agree with the reviewer. To avoid repetition with the Discussion, we have removed the referenced sentence from the Results section.

“Overall, the systematic video analysis checklist demonstrated high reliability, though the visual estimation of player speed proved to be a significant limitation in both assessment settings.”

L152-153:

We would like to thank the reviewer for this terminological correction. We have replaced the general term "body location" with the more precise term "anatomical area" throughout the manuscript.

L156:

We agree with this perspective. Subjective visual perception of speed is highly problematic, as our findings confirmed. As also suggested by another reviewer, we have explicitly stated in the Discussion section that this parameter is unsuitable for visual estimation and recommended its removal or replacement with objective tracking data in future checklist.

Discussion:

L179:

We have removed the word "primary", as suggested.

L238-242:

We thank the reviewer for this excellent and insightful methodological suggestion. We agree that, in the absence of objective tracking data, deducing acceleration or deceleration indirectly through qualitative changes in body posture across consecutive frames could be an alternative to the highly unreliable simple visual speed estimation. We have integrated this possibility into the Discussion section, but we made sure to mention that it requires full validation.

“Alternatively, in the absence of objective tracking data, future observational methodologies could explore deducing acceleration or deceleration indirectly by evaluating qualitative changes in body posture across consecutive frames. However, whether combining these pieces of observational information improves intra- and inter-rater agreement remains a hypothesis and requires formal validation.”

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

Thank you for the revised version of the manuscript and for the detailed responses to the previous review comments. The authors have addressed most of the methodological and reporting issues raised in the first round of review. In particular, the inclusion of 95% confidence intervals for reliability estimates, the clarification regarding the precipitation variable, the expanded description of observer familiarization, and the more cautious interpretation of visual speed estimation have improved the manuscript.

However, a few minor issues should still be corrected before the manuscript can be considered ready for publication.

The random selection procedure is now described as having been performed using Gemini. However, this still raises concerns regarding reproducibility. Please provide more detail on the procedure used to randomly select the cases, including whether a numbered list was used, whether any prompt or seed was retained, and how the process could be replicated. If the procedure cannot be fully reproduced, this should be acknowledged as a limitation.
The Ethical Considerations section has improved by explicitly referring to the Declaration of Helsinki. However, the manuscript should ensure that the Institutional Review Board Statement and Informed Consent Statement are fully aligned with the journal’s requirements.

Author Response

Comment 1: The random selection procedure is now described as having been performed using Gemini. However, this still raises concerns regarding reproducibility. Please provide more detail on the procedure used to randomly select the cases, including whether a numbered list was used, whether any prompt or seed was retained, and how the process could be replicated. If the procedure cannot be fully reproduced, this should be acknowledged as a limitation.

Response 1:

We thank the reviewer for pointing out this important methodological detail regarding reproducibility. Accordingly, we have updated the Video Analysis Procedures section to explicitly describe the process: we introduced the complete list of unique injury identification numbers (IDs) from the full cohort and used a specific prompt asking the Gemini model to randomly select 25 distinct IDs without replacement. The Video Analysis Procedures section was revised as follows:

“The randomization procedure was performed using the Google generative artificial intelligence assistant, Gemini. Specifically, the complete list of unique injury identification numbers (IDs) corresponding to the initial cohort was provided to the AI model, along with a prompt instruction to randomly select 25 distinct IDs. Concerning neurocognitive variables, another subsample of 25 injuries was randomly selected using the same method, but this time from among the 89 non-contact and indirect contact injuries, as the 25 previously selected for the remaining variables included 125 direct contact ACL injuries, for which no neurocognitive variable assessment was performed.”

Furthermore, as noted by the reviewer, because standard generative AI conversational interfaces do not utilize a fixed mathematical seed like traditional statistical software, the exact algorithmic replication of this specific random draw is not possible. As such, we have acknowledged this as a minor methodological limitation in the Discussion section:

“Finally, although the random selection of injury cases was conducted using an AI model to avoid human selection bias, the lack of a fixed mathematical seed in the conversational interface means the exact algorithmic replication of the specific draw cannot be guaranteed, representing a minor methodological limitation regarding perfect reproducibility.”

Comment 2: The Ethical Considerations section has improved by explicitly referring to the Declaration of Helsinki. However, the manuscript should ensure that the Institutional Review Board Statement and Informed Consent Statement are fully aligned with the journal’s requirements.

Response 2:

We appreciate this guidance. We have reviewed the specific formatting requirements for MDPI Sports. To ensure full compliance, we have adapted the “Institutional Review Board Statement” and “Informed Consent” sections at the end of the manuscript to align with MDPI Sports requirements and to clearly indicate the rationale for the waivers based on the use of publicly available and anonymous broadcast images:

“INSTITUTIONAL REVIEW BOARD STATEMENT: Ethical review and approval were waived for this study. Because the research involved only the secondary use of publicly available television broadcast footage, without direct human interaction, clinical intervention, or the processing of identifiable private health information, it did not meet the standard regulatory definition of human subjects research requiring formal institutional ethics committee review. Nevertheless, the ethical principles of the Declaration of Helsinki were considered where relevant to the conduct of the research.”

“INFORMED CONSENT STATEMENT: Patient consent was waived due to the non-interventional and observational nature of the study. The research used only publicly available broadcast footage of professional football matches. Since there was no direct interaction with participants and all data were fully anonymized before analysis, ethical permission and informed consent were not required.”

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

Dear authors,

Your revised version has improved your work that now reads more clear. Video analysis is an acceptable method for team sports analysis and the use of validated checklists is undoubtedly needed. Please find below some comments to further enhance the quality of your manuscript.

Abstract -L3: it is written ACL injuries and then injury mechanisms; something needs to be corrected here.

Introduction

L26: The adjective new suggests that there were already injuries. So, it is not clear to what the authors refer, that is recurrent injuries with regards to long-term consequences?

L37: here it is reported as ACL tears and before as ACL injuries. it is suggested to keep the terminology consistent.

L38: "...large consecutive number of ACL injuries". Consecutive following what?

L40-41: this statement needs a bit more clarification to make its message straight-forward.

L45-46: "...reduce the reliability of evaluative decisions". The authors talk about inconsistency within and between observers, so I am not sure if it is a reliability and not a validity issue. They do mention validity in L42-43, so perhaps some evidence how these structured observational lists become validated by the observers high (intra- and inter-rater) reliability is required first.

Methods

L57: the validity of the checklist used in the study needs to be provided.

L93-95: duplicate lines.

L104-105: I do not understand how this parameter could be evaluated by a video analysis. Certainly there is no way that this information could be derived from a single frame pertaining to the time point of injury. How could the authors examine that the player's attention was focusing on the task-relevant situation? I hypothesize that by "task-relevant situation" they refer to the player's intended action? This parameters as well as the next one, the motor response inhibition (L106-109) need to be more thoroughly elaborated since they involve motor and cognitive mechanisms for which one should be very careful about.

L126: in relation to my previous comment, I think it is far-stretched to talk about neurocognitive analysis from videotaped matches.

L147-148: which were? No informed consent was obtained and since data were anonymized, what were the ethical principles applied in the study?

Discussion

L218-223: too long a sentence; please rephrase it.

L249: correct "this" to "these robust data..."

L266-278: As already commented, I have serious concerns about assessing these parameters from video analysis data. However, if the authors reported data that could be derived from video analysis and relate to a player's decision making process, this would improve this weak point in their study. For instance, I would imagine there exist previous studies reporting on the time it takes before an action occurs, which might be an indirect proxy for attentional inhibition. These data could be calculated from the frame rate of the video footage. I would prioritize this suggestion over having more detailed decision trees and rater training. With no doubt, the assessment of those variables via video analysis should also be acknowledged as one out of two main limitations in this study (the 2nd being the visual assessment of speed) in the last paragraph of the Discussion.

Conclusions: L315: it is suggested to remove both parentheses; not really needed here.

Author Response

Abstract

Comment 1: L3: it is written ACL injuries and then injury mechanisms; something needs to be corrected here.

Response 1:

We thank the reviewer for spotting this redundancy. We have corrected the text by removing the duplicate word:

“While systematic video analysis is widely used to understand anterior cruciate ligament (ACL) injury mechanisms and contexts…”

Introduction

Comment 2: L26: The adjective new suggests that there were already injuries. So, it is not clear to what the authors refer, that is recurrent injuries with regards to long-term consequences?

Response 2:

We agree that the term "new" is somewhat ambiguous in this context. We have replaced it with "subsequent" to clarify that we are referring to recurrent injuries or new injuries on the contralateral limb as long-term consequences of the initial injury:

“…and a significant risk of long-term clinical consequences for athletes, such as subsequent ipsilateral and contralateral injuries…”

Comment 3: L37: here it is reported as ACL tears and before as ACL injuries. it is suggested to keep the terminology consistent.

Response 3:

Thank you for pointing out this inconsistency. We have standardized the terminology throughout the manuscript, replacing "ACL tears" with "ACL injuries" to maintain uniformity.

Comment 4: L38: "...large consecutive number of ACL injuries". Consecutive following what?

Response 4:

We agree that the sentence is confusing. . We have removed the word "consecutive" and rephrased the sentence to improve clarity:

“Previous studies using identical observation checklist methodologies detailed a large number of ACL injuries, injury mechanisms, situational patterns, and the kinematics in Italian [7], Spanish [22], and English [17] professional football.”

Comment 5: L40-41: this statement needs a bit more clarification to make its message straightforward.

Response 5:

We have rephrased this statement to make the message more straightforward, explicitly clarifying that it remains unknown if findings from specific national leagues apply to others:

“While these single-country studies offer important insights, it remains unclear whether their findings regarding injury mechanisms and situational patterns can be generalized across different countries and leagues.”

Comment 6: L45-46: "...reduce the reliability of evaluative decisions". The authors talk about inconsistency within and between observers, so I am not sure if it is a reliability and not a validity issue. They do mention validity in L42-43, so perhaps some evidence how these structured observational lists become validated by the observers high (intra- and inter-rater) reliability is required first.

Response 6:

The reviewer raises a very valid methodological point distinguishing between validity and reliability. We have adjusted the phrasing to focus on the "robustness" and "reliability" of the methodology, rather than its validity, since this study evaluates observer consistency:

“Despite the widespread adoption and clear clinical value of systematic video analysis, the robustness of this methodology inherently relies on the accuracy and consistency of human observers.”

Methods

Comment 7: L57: the validity of the checklist used in the study needs to be provided.

Response 7:

We appreciate this comment. We have added a brief statement clarifying that the variables included in the checklist are based on previously validated and widely published mechanisms in the sports medicine literature regarding ACL injuries:

“A retrospective observational study was conducted to evaluate the reliability of a previously used systematic video analysis checklist for ACL injuries. The variables included in this checklist are based on well-established and previously validated injury mechanisms and situational patterns widely described in the scientific literature.”

Comment 8: L93-95: duplicate lines.

Response 8:

We have carefully reviewed the document and removed any duplicate lines or formatting errors in that section.

Comment 9: L104-105: I do not understand how this parameter could be evaluated by a video analysis. Certainly there is no way that this information could be derived from a single frame pertaining to the time point of injury. How could the authors examine that the player's attention was focusing on the task-relevant situation? I hypothesize that by "task-relevant situation" they refer to the player's intended action? This parameters as well as the next one, the motor response inhibition (L106-109) need to be more thoroughly elaborated since they involve motor and cognitive mechanisms for which one should be very careful about.

Response 9:

The reviewer raises an excellent point. We have expanded this section to briefly explain how these cognitive concepts were operationally defined as observable indicators during the video analysis:

“… For the purpose of video analysis, these neurocognitive concepts were evaluated using observable physical indicators. For example, attentional inhibition was inferred by evaluating whether or not the player’s visual focus and head position changed immediately before the injury. On the other hand, motor response inhibition was inferred by observing evident physical attempts to alter their movement trajectory or to interrupt an action.”

Comment 10: L126: in relation to my previous comment, I think it is far-stretched to talk about neurocognitive analysis from videotaped matches.

Response 10:

We agree that the term “neurocognitive analysis” seems a bit far-stretched for a video-based study. We have softened the terminology to “assessment of neurocognitive variables”:

“…as the 25 previously selected for the remaining variables included 125 direct contact ACL injuries, for which no neurocognitive variable assessment was performed.”

Comment 11: L147-148: which were? No informed consent was obtained and since data were anonymized, what were the ethical principles applied in the study?

Response 11:

The reviewer raises a highly pertinent and accurate point. We completely agree that stating "in strict accordance with the ethical principles outlined in the Declaration of Helsinki" was paradoxical, given that obtaining informed consent and approval from the ethics committee are fundamental pillars of that Declaration. We have adapted this contradictory statement from the manuscript. Instead, we have clarified that, since our study involves no clinical interventions and relies only on the secondary analysis of publicly available broadcast footage, it does not fall in the regulatory scope of medical research requiring formal ethical approval and informed consent. The ethical principles applied were strictly limited to data protection, confidentiality, and the rigorous anonymization of the players’ identities during the analysis. The "Ethical Considerations", "Institutional Review Board Statement", and "Informed Consent Statement" sections have been updated to reflect this rationale:

“ETHICAL CONSIDERATIONS: This study utilized publicly available video footage from televised professional football matches and data from a commercial subscription platform accessible for analysis purposes. No direct interaction with human participants occurred, no clinical interventions were performed, and no private health records were accessed. All player data were strictly anonymized during the analysis phase. Accordingly, since the research involved only the secondary analysis of publicly broadcast video records, ethical approval and informed consent were not required. However, the ethical principles of the Declaration of Helsinki were considered where relevant to the conduct of the research. Moreover, general ethical principles regarding research integrity, data protection, and confidentiality were strictly respected throughout the study.”

“INFORMED CONSENT STATEMENT: Patient consent was waived due to the non-interventional and observational nature of the study. The research used only publicly available broadcast footage of professional football matches. Since there was no direct interaction with participants and all data were fully anonymized before analysis, ethical permission and informed consent were not required.“

Discussion

Comment 12: L218-223: too long a sentence; please rephrase it.

Response 12:

We agree that the sentence is too lengthy and complex. We have split it into two clearer sentences:

“Systematic video-analysis methodologies and observational reliability assessments have been widely and successfully explored within the sports medicine and biomechanics literature. However, to the best of our knowledge, this is the first study to assess the intra- and inter-rater reliability of this specific comprehensive observational checklist for assessing ACL injury mechanisms, situational patterns, and contextual factors through video analysis in elite men’s football.”

Comment 13: L249: correct "this" to "these robust data..."

Response 13:

Thank you for this grammatical correction. We have changed "This robust data" to "These robust data".

Comment 14: L266-278: As already commented, I have serious concerns about assessing these parameters from video analysis data. However, if the authors reported data that could be derived from video analysis and relate to a player's decision making process, this would improve this weak point in their study. For instance, I would imagine there exist previous studies reporting on the time it takes before an action occurs, which might be an indirect proxy for attentional inhibition. These data could be calculated from the frame rate of the video footage. I would prioritize this suggestion over having more detailed decision trees and rater training. With no doubt, the assessment of those variables via video analysis should also be acknowledged as one out of two main limitations in this study (the 2nd being the visual assessment of speed) in the last paragraph of the Discussion.

Response 14:

Thank you for this insightful feedback. We have incorporated the suggestion regarding frame rate-derived time metrics as an indirect proxy for attentional inhibition:

“To improve the robustness of these parameters, future studies could explore indirect quantitative indicators, such as calculating the time elapsed until an action occurs, based on the video frame rate, which could serve as a more objective measure of attentional inhibition. Additionally, the potential of more detailed decision trees or enhanced rater training might be explored to further enhance agreement for these concepts.”

Conclusions:

Comment 15: L315: it is suggested to remove both parentheses; not really needed here.

Response 15:

We have removed the parentheses in this sentence to improve the flow of the text, as suggested:

“The findings demonstrate a high degree of consistency, both over time and between independent raters.”

Author Response File: Author Response.pdf