Abstract
Accident analysis frameworks such as Human Factors Analysis and Classification System (HFACS) are widely used in high-risk industries to determine risk mitigation strategies. In comparison, equestrianism which is classified high-risk due to human-horse interactions at work, sport, and social activities, rarely utilizes accident analysis. This study developed and tested the validity and inter-rater reliability of an equestrian-specific accident analysis framework, that included elements of human error, horse risk factors, and environmental factors. The study involved three coders who independently classified 10 simulated horse-related human accident reports with the novel Human Factors Analysis and Classification System-Equestrianism (HFACS-Eq) framework. The results demonstrated that the HFACS-Eq framework achieved moderately reliable to reliable coding percentage agreement. In addition, substantial to reliable agreement was achieved for HFACS-Eq nominal category and nano-codes levels. This study is the first step towards an equestrian industry-specific, accident analysis framework to improve industry safety. Elimination of possible bias and validation with real incident data are required before the wider application of the framework can be recommended. The study highlights organizational and procedural failures, segregating the horse as a contributing factor as well as the environment in which the human acts or makes decisions informing risk.
1. Introduction
Humans are frequently injured or die during human-horse interactions in both work and non-work environments, with horse-related activities being rated as high risk [1]. Horse-related activities can be dangerous [2], with riders citing hyperactive horse behavior as a major cause of injury [3]. Other human-horse related risks include the horses’ size and weight, speed capabilities, unpredictable or reactive behaviors and its herd-animal freeze, fight, or flight instincts [4,5].
Equestrianism refers to the use of equids during leisure or sport activities and includes horses as working animals, used for transportation, therapeutic services, artistic or cultural exercises [6]. Equestrianism is also classified as a high-risk industry in Australia [7], along with mining [8,9,10], aviation [11,12,13], construction [14], and rail [15]. Most high-risk industries have regulatory frameworks, implement work safety management systems, report incidents and complete accident analysis to mitigate risks [8,9,10,16]. In Australia, equestrian work and non-work environments, have few such regulations or processes and appear to lack insight into the value of exploring lessons that might be learned, from analyzing incidents, accidents and fatalities [17].
Accident analysis is the process of investigating potential accident causes, post event. Analytical tools, based on theoretical foundations can be applied to determine risk, understand failures, and identify prevention strategies [18]. Following an increase in aviation accidents during the mid-1990s, two behavioral scientists Wiegmann and Shappell developed the accident analysis framework Human Factors Analysis and Classification System (HFACS) [19]. The HFACS framework is well-known for its adaptability and hybrid-modality, being a reliable tool for accident analysis and validated by multiple high-risk industries worldwide [20,21,22]. The HFACS framework also provides analysts with taxonomies of failures modes across four levels of Human Factors (HF), as follows:
- (1)
- Unsafe acts which occur close to or directly result from the event and are usually identifiable as resulting from errors or violations.
- (2)
- Preconditions for unsafe acts being (a) the environment, surroundings, or technology, contributing if human decisions result in errors or unsafe conditions; (b) human conditions either physical or psychological and (c) personal readiness affecting individual practices, conditions, or actions.
- (3)
- Unsafe supervision occurs when an individual lacks direction, training, or guidance to safely achieve a required task. Resulting from (a) inadequate or limited assistance; (b) planned inappropriate activities during an emergency or normal operations; (c) failure to correct a problem when identified or (d) a supervisor willfully disregarding a correct procedure.
- (4)
- Organizational influences include (a) resource management and decisions about skills, staff, equipment needs or funding requirements; (b) organizational work climate and culture and (c) operational processes, structure, decisions, and rules that support the organization’s vision.
Across these four levels, the original HFACS framework contains 19 categories each of which contains a set of subcategories described as nano-codes. HFACS has demonstrated interchangeability, with the capability for expansion from its four main categories to additional sub-categories and replacing nano-code descriptors. HFACS provides sound cause classifications using a top-down approach starting with organizational failures that cascade below to immediate accident causation factors, assisting the analyst in classifying errors using selected taxonomies. The HFACS framework is a practical accident analysis tool; it is convenient, captures multiple cases at one time, can merged with other accident frameworks [23]. The HFACS hybrid-modality accommodates industry-specific error extensions which advances HFACS as a best-fit model [22] for the development of an equestrian framework. A key benefit of HFACS is its ability to investigate system and process failures, thus targeting training, preventative measures, and mitigating risk.
Equestrianism with its unique triad of human-horse-environment potential accident causal factors, requires an accident framework that can interpret all three-accident causal factors independent contributions, their relationships, and any other relevant influences. All human and horse errors, behaviors, responses, and acts, are equally important when determining equestrianism accident causation. Furthermore, by identifying risk-mitigation actions humans can adopt to reduce some unpredictable horse behaviors, an industry specific accident analysis framework is likely to promote proactive management of human-horse interactions [3,5].
The horse’s association with humans and its capability of independent unsafe acts, needed capturing due to its potential contribution of harm, along with active or latent causal factors during accident analysis. The purpose of this preliminary study was to develop and validate a suitable accident analysis framework for equestrianism based on a current high-risk industry accident analysis framework and test its ease of use, validity, and inter-rater reliability. Therefore, the authors developed the framework Human Factors Analysis and Classification System-Equestrianism (HFACS-Eq). Having an industry specific accident analysis framework is likely to assist equestrianism in identifying accident causes as a valuable prerequisite for risk mitigation and guidance for future preventative safety management during human-horse interactions [24].
2. Methods
The fundamental determinants for a successful accident analysis framework, such as HFACS-Eq or other industry models, is validity and its ability to replicate consistent outcomes with multiple users, being inter-rater reliability [25,26,27]. This study set out to design a valid and reliable framework for accident analysis in the context of equestrianism, drawing on the existing framework of the HFACS system.
2.1. Stage One: Designing a Valid HFACS Framework for Equestrianism
Ensuring a valid adaptation of the basic HFACS framework for accident analysis for use in the equestrian context was the first stage of this research. Broadly, this process involved review of the existing framework, identifying elements that were not relevant to equestrianism, and drawing on expertise in equestrian safety to identify elements that were known contributors to incidents and accidents in equestrianism that were absent from the original HFACS framework.
First, it was determined that all four original levels of HFACS should be retained for use in HFACS-Eq. The level of ‘organizational influences’ was retained (but slightly re-worded to ‘unsafe work-related influences’) despite equestrianism having less regulatory influences and management structures when compared to other high-risk industries. This was to enable the capture of horse-related accidents that occur in both work and non-work environments during accident analysis. This level and its classifications including organization, culture and operational processes was redesigned to include single management identities, with taxonomies depicted by large-scale and complex organizational operations being removed. The three remaining levels being (1) ‘unsafe supervision’; (2) ‘preconditions for unsafe acts’; and (3) ‘unsafe acts’ themselves (errors or violations) of human intervention were subsequently reviewed for their relevance to equestrianism.
Once the initial HFACS-Eq level review was completed the research group determined categories and generated new nano-codes from known factors associated with accidents in equestrianism. The five equine-specific nano-codes were included within the category ‘preconditions for unsafe acts’, which considered (1) the horse’s physiological composition; (2) the horse’s behaviors; (3) the horse’s general health and (4) the level of horse training and welfare status. Refer to Table 1.
Table 1.
HFACS-Eq summary of final data set levels, categories, and nan-codes.
Following the initial development of the HFACS-Eq framework, its validity was assessed against a set of horse-related accident scenarios to establish initial content and construct validity. This process highlighted several nano-codes that required re-working due to possible overlap between constructs. This led to the ultimate creation of a new category specifically related to the horse. This category enabled the grouping of nano-codes for horse-related characteristics, behaviors, and conditions as potential factors in accident causation. Following additional piloting of the HFACS-Eq framework, it was identified that some of the pre-existing categories and nano-codes presented overly industrial and engineering constructs, that did not relate well to equestrian incidents and accidents. This led to the further refinement of HFACS-Eq and the development of explanations and examples of human-horse related activities that described each nano-code, applicable to either work or non-work environments, to further demonstrate construct validity. Refer to Table 2.
Table 2.
HFACS-Eq summary of final data set of levels, categories, and nano-codes with explanations.
In summary, stage one of the research ensured initial construct and content validity of HFACS-Eq through the adaptation of the original HFACS framework. The inclusion of a range of factors known to be present in horse-related incidents and accidents, and the removal of factors that were only relevant to industrial accident causation. The final HFACS-Eq framework is provided in Figure 1.
Figure 1.
Human factors analysis and classification system with modification to include equestrian industry specific components (HFACS-Eq).
2.2. Stage Two: Ensuring Reliability of HFACS Framework for Equestrianism
Once content and construct validity was established for HFACS-Eq, the second stage of the research involved an assessment of the reliability of the new framework. To examine the reliability of the equestrian industry-specific accident analysis HFACS-Eq framework a set of 10 simulated accident scenarios were created by the first author (MC) who was experienced in WHS and an accomplished equestrian. Accident scenarios were based on real-life experiences of the authors, as well as knowledge of previous horse-related human incidents, and examples from coronial reports.. Each scenario used fictitious characters, including a variety of locations and circumstances, from different equestrianism activities such as, riding, leading, training, caring for and loading a horse for transportation.
Each of the 10 accident scenarios (vignettes) captured work or non-work environments where humans were in close contact with a horse, resulting in the human injury or fatality. The use of simulated scenarios provided a rigorous approach and allowed for better standardisation in the material presented to the coders that otherwise would not be possible using highly variable accident reports from police or coronial records. This approach also facilitated the inclusion of a wide cross-section of HFACS-Eq categories and nano-codes required to facilitate robust assessment of inter-rater reliability [28,29]. Scenarios were also standardised for word length and content style to facilitate comparative analysis.
The 10 accident scenarios were then provided to the remaining authors for coding (KF-senior equestrian safety academic, and MT-senior human factors academic) who remained blind to the specific mapping of scenarios against HFACS-Eq framework. All coders were to read each accident scenario and use the HFACS-Eq framework to identify specific human error or system failures by placing a yes along-side each of the nano-codes that best described and classified factors present in the accident scenario. This process was repeated for each of the 10 scenarios without any discussion between the coders. All three authors were the coders for the study; therefore, no ethics approval was considered necessary.
At the completion of the accident analysis classification process for all 10 simulated accident scenarios the coded data were subjected to reliability analysis. All data were analyzed in SPSS (V28) with inter-rater reliability established using first an evaluation of raw percent agreement between coders at both the category and nano-code levels. The second and more stringent assessment used for reliability for each category and nano-code that accounts for chance was Fleiss Kappa (KF), a measure used for multiple rater level of agreement [30]. Statistics were interpreted in line with Landis and Koch [31] with strength of reliability using the following threshold values: 0.00–0.20 slight; 0.21–0.40 fair; 0.41–0.60 moderate; 0.61–0.80 substantial; and 0.81–1.0 almost perfect agreement.
3. Results
The study results identified percent agreement was greater than 68% across all scenarios at the category and nano-code level, except for one scenario. Fleiss’ Kappa statistics demonstrated moderate to substantial (KF = 0.47–0.77) levels of agreement across all but two scenarios at the category level, and fair to moderate (KF = 0.25–0.58) levels of agreement across all but one scenario at the nano-code level.
A summary of inter-rater reliability results using the HFACS-Eq model to analyze all 10 accident scenarios is provided below for both the category levels (Table 3) and nano-code levels (Table 4).
Table 3.
Inter-rater reliability for HFACS-Eq Categories (19 categories).
Table 4.
Inter-rater reliability for HFACS-Eq Nano-Codes (60 nano-codes).
4. Discussion
The purpose of this study was to design and examine the value of a modified HFACS accident analysis framework, similar to those adapted in high-risk industries. HFACS-Eq was used to determine if an equestrian specific accident analysis framework may identify horse-related human accident causal factors. The authors used the study to assess the HFACS-Eq model for its usefulness, limitations, and areas for future development as a suitable accident analysis framework to assist in risk mitigation in equestrianism.
4.1. Safety and Risk Mitigation Application of the Equestrianism HFACS Framework
Using the well-researched, accident analysis framework HFACS, provided a solid foundation to build the hybrid HFACS-Eq model accommodating human, horse, and environmental elements. The multi-level review of categories and nano-codes facilitated adaptation for equestrianism’s two unique elements being: (1) work and non-work environments and (2) the addition of the horses as a potential hazard.
During the development phase of HFACS-Eq it was apparent to the research group that the higher levels of organisational factors would not be fit-for-purpose in equestrianism. Significant adjustments were required to capture sole traders or otherwise referred to as a sole proprietor and small business with minimal personnel or single operators. This category represents all work-related horse activities where the human is receiving payment for interacting with the horse. It is important to note that without this adjustment HFACS-Eq usability for the equestrian industry would limit its purpose and meaning for future users.
Equestrianism is somewhat unfamiliar with high-risk safety and risk-mitigation systems and processes. Therefore, during the initial trial of HFACS-Eq, it was necessary to create equestrian-specific descriptive analysis language, terminology, and definitions. On further review it was highlighted that all nano-codes required frequent adaptation and simplification and the study’s results demonstrated the importance of having a clear, concise industry-specific framework. This reinforces the importance of industry-specific fit-for-purpose applications. Ongoing reviews, modifications are also likely to improve users’ knowledge and support new learnings from accident analysis and ultimately risk mitigation through improved practices and safety management, especially in equestrianism.
4.2. The Horse as a Living Animal and Its Association with Human Interaction-Error
With the view of a horse having distinctive independent factors, a sentient being, along with known dangerous actions and behaviors, it was critical for the horse to have a single category within HFACS-Eq. Additional nano-codes explicit to the horse and its potential to contribute to human harm were included. Analysis of the 10 accidents scenarios identified the horse as a 60% contributor for incident causation during human-horse interactions. This would indicate that human acts or omissions, supervision, organisational factors, and the environment contributed 40%. This finding may have been overlooked and its significance for equestrianism if the original HFACS was used for the purpose of this study. However, due to the small sample of accidents analysed further exploration of multiple human horse-related accident may be beneficial.
4.3. HFACS-Eq Accident Analysis
On reviewing the HFACS-Eq raw-data two or more coders consistently had similar agreement with the category levels particularly in the areas of human error and unsafe acts. However, the two categories referring to supervisory skills created coder confusion, resulting in lower levels of agreement. This may be attributed to similar nano-codes in HFACS-Eq describing supervisory activities and coders choosing one over the other. Of interest the category named preconditions for unsafe acts: horse, highlighted two-coders with experience in accident analysis had similar coding’s compared to two of the three coders having more horse-related experience. This finding may indicate coders with extensive knowledge of horse-related and unsafe behaviors, may be indecisive or overthink analysis, instead of applying simple accident facts to relevant nano-codes.
It is important to mention the degree of coder agreement was higher in categories compared to nano-codes. With respect to individual interpretations of accident information, this may support the importance of coders having a thorough understanding of HFACS-Eq categories and nano-codes. Of interest was coders were able to easily identify accident analysis facts within the 19 HFACS-Eq categories, compared to less agreement of the 60 nano-codes. This is not a common finding as HFACS models in other industries have also been shown to have better inter-rater reliability at the broader category level, than at the level of individual nano-codes [32,33]. The value of explanatory notes and examples relating to the horse especially for coders with limited horse-related knowledge should not be underestimated. Furthermore, to improve accident analysis outcomes, training and guidance in HF analysis methods may be beneficial [34].
Many of the HFACS-Eq accident scenario findings were located within levels (2) Unsafe Supervision; (3) Preconditions for Unsafe Acts and (4) Human Unsafe Acts. This may indicate the accident scenarios used for this study had limited detail in the areas of management, culture, safety processes and supervisory skills. Therefore, by sourcing relevant and sufficient accident information such as police reports, witness statements, incident reports, toxicology, autopsy, operational and communication records, and environmental conditions may improve future HFACS-Eq analysis outcomes.
4.4. HFACS-Eq Associations and Areas for Improvement
The HFACS-Eq framework emphasised the horse as a contributing factor in most accident scenarios. This was accompanied by a strong association with (1) supervisory skill or capability and (2) the physical environment and surroundings or both. Furthermore, the categories regarding the level of supervision and planning for safer human-horse interactions were associated as frequent accident causal factors. This supports the use of a systematic, integrated framework for accident analysis to extract organisational deficits that may contribute to causality of human factor errors or omissions. The HFACS-Eq framework is likely to assist future investigators to locate active system failures that lead to an accident, highlighting area to target training, support, and improvements to mitigate future accidents in equestrianism.
4.5. HFACS-Eq Limitations and Future Research
Initial HFACS-Eq modifications and the study in general identified that some minor reviews of the framework would be beneficial. Further refinements in the areas of supervision and the addition of a descriptive horse-related nano-code for an unfavourable horse impact, for example when a horse falls on a human would improve HFACS-Eq outcomes. This simple and ongoing nano-code review of HFACS-Eq may improve inter-rater reliability and outcomes for future users.
This study only reviewed 10 simulated accident scenarios each crafted to supply specific information to enable classification against the HFACS-Eq framework. Further research using a larger sample of actual incident and accident data will be necessary to further establish the validity and reliability of the HFACS-Eq framework. A larger study is underway, which uses the HFACS-Eq framework for the classification of a large coronial data set of horse-related accidents and incidents resulting in fatalities to riders/handlers.
The provision of HFACS-Eq model training and clarity of accident analysis processes was limited for this study’s research group. Future HFACS-Eq users are more likely find value in a factsheet explaining the accident analysis process and coaching on nano-code explanations. To further enhance HFACS-Eq, users would benefit from having experience in equestrianism and a clear understanding, even formal education in accident and HF analysis [35,36,37].
Finally, the development of an equestrian industry specific accident data repository across organisations and independent operators, with a detailed and consistent incident reporting process may improve accident analysis using HFACS-Eq. Ongoing improvements in human and horse causal factor accident analysis for work and non-work equestrian stakeholders, policy makers and government authorities is likely to drive future best-practice safety and risk management practices. Longitudinal data collection using the HFACS-Eq will improve human safety and horse welfare across the equestrian industry.
5. Conclusions
This study is the first step towards designing and assessing an accident analysis framework for equestrianism. The hybrid framework HFACS-Eq was examined for initial construct validity and inter-rater reliability. Results showed a reasonable level of agreement (>65%) across all categories and nano-codes within the HFACS-Eq framework. Further statistical analysis confirmed substantial category and fair to moderate nano-code levels of agreement across most of the simulated scenarios. This study highlights the importance of organizational and procedural failures, segregating the horse as a contributing factor as well as the environment in which the human acts or makes decisions informing risk. Further research is needed using the HFACS-Eq framework to evaluate real incident data and eliminate possible bias before the wider application of the framework can be recommended to improve safety in equestrianism.
Author Contributions
A summary of author individual contributions have been outlined. Authors for conceptualization, M.C.; methodology, K.F. and M.J.W.T.; formal analysis, M.J.W.T.; investigation, M.C., K.F. and M.J.W.T.; data curation, M.C. and K.F.; writing—original draft preparation, M.C.; writing—review and editing, K.F. and M.J.W.T.; supervision, M.J.W.T. All authors have read and agreed to the published version of the manuscript.
Funding
This research received no external funding.
Institutional Review Board Statement
Not applicable.
Informed Consent Statement
Not applicable.
Data Availability Statement
Not applicable.
Conflicts of Interest
The authors declare no conflict of interest.
References
- Rowland, G.; Franken, R.; Harrison, K. Sensation seeking and participation in sporting activities. J. Sport Exerc. Psychol. 1986, 8, 212–220. [Google Scholar] [CrossRef]
- Cripps, R. Horse-Related Injury in Australia; AIHW National Injury Surveillance Unit Flinders University: Adelaide, Australia, 2000. [Google Scholar]
- Luke, K.; McAdie, T.; Smith, B.; Warren-Smith, A. New insights into ridden horse behaviour, horse welfare and horse-related safety. Appl. Anim. Behav. Sci. 2021, 246, 105539. [Google Scholar] [CrossRef]
- McGreevy, P.; McLean, A. The Domestic Horse: The Origins, Development and Management of Its Behaviour; Cambridge University Press: Cambridge, UK, 2005; pp. 196–211. [Google Scholar]
- Williams, F.; Ashby, K. Horse Related Injuries; 23; Monash University Accident Research Centre: Clayton, Australia, 1995. [Google Scholar]
- Dashper, K. Human-Animal Relationships in Equestrian Sport and Leisure; Routledge: London, UK, 2016. [Google Scholar]
- Safework NSW. High-Risk Workplaces Strategy; Safework, NSW: Gosford, NSW, Australia, 2018; pp. 1–8.
- DPI. Risk Management for the Mining Industry. Available online: https://mmstbpi.weebly.com/uploads/4/7/2/5/4725854/5363_mdg-1010-risk-mgt-handbook-290806-website.pdf (accessed on 30 May 2021).
- Amponsah-Tawiah, K.; Mensah, J. Occupational health and safety and organizational commitment: Evidence from the Ghanaian mining industry. Saf. Health Work 2016, 7, 225–230. [Google Scholar] [CrossRef]
- Joe-Asare, T.; Amegbey, N.; Stemn, E. Human Factor Analysis Framework for Ghana’s Mining Industry. Ghana Min. J. 2020, 20, 60–76. [Google Scholar] [CrossRef]
- Edkins, G. A review of the benefits of aviation human factors training. Hum. Factors Aerosp. Saf. 2002, 2, 201–216. [Google Scholar]
- Salmon, P.; Cornelissen, M.; Trotter, M. Systems-based accident analysis methods: A comparison of Accimap, HFACS, and STAMP. Saf. Sci. 2012, 50, 1158–1170. [Google Scholar] [CrossRef]
- Fuller, J.; Hook, L. Understanding General Aviation Accidents in Terms of Safety Systems. In Proceedings of the 2020 AIAA/IEEE 39th Digital Avionics Systems Conference (DASC), San Antonio, TX, USA, 11–15 October 2020; pp. 1–9. [Google Scholar]
- Forsythe, P. Proactive construction safety systems and the human factor. Proc. Inst. Civ. Eng. -Manag. Procure. Law 2014, 167, 242–252. [Google Scholar] [CrossRef]
- Elms, D. Rail safety. Reliab. Eng. Syst. Saf. 2001, 74, 291–297. [Google Scholar] [CrossRef]
- Thomas, M. A Systematic Review of the Effectiveness of Safety Management Systems; Australian Transport Safety Bureau: Canberra, Australia, 2012.
- Chapman, M.; Thompson, K. Preventing and investigating horse-related human injury and fatality in work and non-work equestrian environments: A consideration of the workplace health and safety framework. Animals 2016, 6, 33. [Google Scholar] [CrossRef]
- Li, W.; Zhang, L.; Liang, W. An Accident Causation Analysis and Taxonomy (ACAT) model of complex industrial system from both system safety and control theory perspectives. Saf. Sci. 2017, 92, 94–103. [Google Scholar] [CrossRef]
- Shappell, S.; Detwiler, C.; Holcomb, K.; Hackworth, C.; Boquet, A.; Wiegmann, D. Human Error and Commercial Aviation Accidents: A Comprehensive, Fine-Grained Analysis Using HFACS; Federal Aviation Administration, Office of Aviation Medicine Washington: Washington, DC, USA, 2006.
- Gong, L.; Zhang, S.; Tang, P.; Lu, Y. An integrated graphic–taxonomic–associative approach to analyze human factors in aviation accidents. Chin. J. Aeronaut. 2014, 27, 226–240. [Google Scholar] [CrossRef]
- Li, W.; Harris, D. HFACS Analysis of ROC Air Force Aviation Accidents: Reliability analysis and cross-cultural comparison. Int. J. Appl. Aviat. Stud. 2005, 5, 65–81. [Google Scholar]
- Liu, D.; Nickens, T.; Hardy, L.; Boquet, A. Effect of HFACS and non-HFACS-related factors on fatalities in general aviation accidents using neural networks. Int. J. Aviat. Psychol. 2013, 23, 153–168. [Google Scholar] [CrossRef]
- Øien, K. A framework for the establishment of organizational risk indicators. Reliab. Eng. Syst. Saf. 2001, 74, 147–167. [Google Scholar] [CrossRef]
- Kouabenan, D. Beliefs and the perception of risks and accidents. Risk Anal. 1998, 18, 243–252. [Google Scholar] [CrossRef]
- Drost, E. Validity and reliability in social science research. Educ. Res. Perspect. 2011, 38, 105–123. [Google Scholar]
- Goncalves Filho, A.; Waterson, P.; Jun, G. Improving accident analysis in construction–Development of a contributing factor classification framework and evaluation of its validity and reliability. Saf. Sci. 2021, 140, 105303. [Google Scholar] [CrossRef]
- O’Leary-Kelly, S.; Vokurka, R. The empirical assessment of construct validity. J. Oper. Manag. 1998, 16, 387–405. [Google Scholar] [CrossRef]
- Barth, J.; de Boer, W.; Busse, J.; Hoving, J.; Kedzia, S.; Couban, R.; Fischer, K.; Von Allmen, D.; Spanjer, J.; Kunz, R. Inter-rater agreement in evaluation of disability: Systematic review of reproducibility studies. BMJ 2017, 356, j14. [Google Scholar] [CrossRef]
- McGregor, M.; Cambron, J.; Jedlicka, J.; Gudavalli, M. Clinical trial variability: Quality control in a randomized clinical trial. Contemp. Clin. Trials 2009, 30, 20–23. [Google Scholar] [CrossRef]
- Fleiss, J. Measuring nominal scale agreement among many raters. Psychol. Bull. 1971, 76, 378. [Google Scholar] [CrossRef]
- Landis, J.; Koch, G. The measurement of observer agreement for categorical data. Biometrics 1977, 33, 159–174. [Google Scholar] [CrossRef] [PubMed]
- Yesilbas, V. The Relationship Among HFACS Levels and Analysis of Human Factors in Unmanned and Manned Air Vehicles, in Engineering Management and Systems Engineering; Old Dominion University: Norfolk, VA, USA, 2014. [Google Scholar]
- O’Connor, P.; Walliser, J.; Philips, E. Evaluation of a human factors analysis and classification system used by trained raters. Aviat. Space Environ. Med. 2010, 81, 957–960. [Google Scholar] [CrossRef] [PubMed]
- Underwood, P.; Waterson, P. Systemic accident analysis: Examining the gap between research and practice. Accid. Anal. Prev. 2013, 55, 154–164. [Google Scholar] [CrossRef] [PubMed]
- Hollnagel, E.; Speziali, J. Study on Developments in Accident Investigation Methods: A Survey of the ‘State-of-the-Art’; Swedish Nuclear Power Inspectorate: Stockholm, Sweden, 2008. [Google Scholar]
- Johansson, B.; Lindgren, M. A quick and dirty evaluation of resilience enhancing properties in safety critical systems. In Proceedings of the Third Symposium on Resilience Engineering, École des mines de Paris, Antibes-Juan-les-Pins, France, 28–30 October 2008; pp. 28–30. [Google Scholar]
- Sklet, S. Comparison of some selected methods for accident investigation. J. Hazard. Mater. 2004, 111, 29–37. [Google Scholar] [CrossRef] [PubMed]
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).