Assessment of the Intra- and Inter-Observer Reliability of Beef Cattle Mobility Scoring Performed by UK Veterinarians and Beef Farmers

Simple Summary

Cattle lameness is a critical issue facing the livestock industry, as it has negative effects on both welfare and production. Lameness prevalence in beef herds is reportedly lower than in dairy herds, but it is still vital that lame cattle are identified and treated promptly. Mobility scoring assigns a numerical score to an animal correlating to its lameness severity and is the most common way to systemically identify lame cattle within the herd. To investigate how reliably beef farmers and veterinary surgeons can mobility score beef cattle, forty video clips of beef cattle were watched and scored online by 39 beef farmers and 42 vets. The agreement on scores between the respondents against each other was fair, although poorer than previously reported. Vets were significantly more likely to agree with the scores of other vets. The level of agreement within themselves when they scored the same video a second time ranged from only slight to almost perfect, depending on which clip it was. This apparent unreliability of mobility scoring beef cattle could be a barrier in identifying lameness and the subsequent consequences of this.

Abstract

Background: Lameness in cattle negatively affects welfare and productivity. Early identification of lameness allows for prompt treatment, and mobility scoring allows for herd-level prevalence data to be monitored. The reliability of a four-point mobility scoring system was investigated when used by beef farmers and veterinary surgeons. Methods: An online questionnaire that contained forty video clips of beef cattle was created for mobility scoring performed by farmers and vets. Results: The Fleiss kappa coefficient for inter-observer agreement across all 81 respondents and all videos was 0.34, which showed fair agreement. Beef farmers generally had lower agreement than vets (0.29 vs. 0.38). Vets had significantly higher inter-observer reliability compared to beef farmers (p = 0.035). Overall, Cohen’s kappa coefficient for intra-observer agreement across all respondents varied from 0.085 (slight agreement) to 0.871 (almost perfect agreement). Limitations: The survey was only available online, which may have limited distribution and engagement. The recruitment of participants was not specific to differing levels of previous experience in mobility scoring. The mobility scoring was not performed in person, which could be more reflective of clinical application. Conclusions: The application of a four-point mobility scoring system for beef cattle had fair inter-observer reliability and a wide range of intra-observer reliability, but this is poorer than previously reported. This presents a challenge for the identification of lame beef cattle at both the individual and herd levels.

1. Introduction

Lameness is a significant welfare concern within the cattle industry, with substantial investment from stakeholders to reduce both the severity and prevalence of lameness cases across the country and worldwide. There are multiple predisposing and causal factors of lameness, with gait and behaviour altered because of lameness. Lameness affects each of the five domains included in the ‘Five Freedoms Framework’, which is frequently used to evaluate the welfare state of an animal [1]. Unlike dairy systems, where the opportunities to observe cattle to identify lameness are more available due to milking routines, the opportunities to observe a herd of beef cattle walking on a suitable surface are infrequent. Therefore, when they do arise, suitably reliable methods of lameness detection must be performed to identify lame animals who would benefit from treatment. In addition to the negative welfare implications, lameness is important due to its impact on production efficiency, with a reduction of 240 g in average daily live weight gain if the individual was lame once during the finishing period in the UK [2]. A negative return of CAD −286 was reported when feedlot cattle in Canada were diagnosed with joint infections, with a larger negative return of CAD −701 if a lameness with no visible swelling was diagnosed [3].

The lameness prevalence of beef cattle in the United Kingdom (UK) is reported at 8.3% (range 2.0 to 21.2%) in finishing cattle, increasing to 14.2% (range 0.0 to 43.2%) in suckler cows [4]. This figure varies depending on geographical location and production system, with lower lameness levels reported in feedlot cattle in North America (2.1% to 4.5%) [5,6]. When a mobility scoring system was implemented in 10 North American auctions evaluating 9249 beef animals, 15.1% of beef cows and 15.4% of beef bulls were classed as lame [7]. In a small Norwegian study, the reported prevalence was even lower at 1.1% [8].

A vital step in reducing the severity of individual lameness cases is early detection of lameness itself, followed by prompt treatment [9]. A delay in treatment will result in a poorer response and invariably affect individual animal welfare for longer [9,10,11]. To facilitate early detection, various methods exist, with observer-based mobility scoring being most commonly used for beef animals. Automated lameness-detection methods such as computer vision gait analysis [12], infrared thermography [13], and accelerometers [14] are also available, but these methods have predominantly been trialled in dairy systems and have variable reported validity and reliability.

There are multiple different observer-based mobility scoring systems worldwide, some of which are briefly summarised in

Table 1. Table that includes some of the different cattle locomotion scoring systems that have been used in research or in industry.

Name of System	Cattle Type	Scale
Agriculture and Horticulture Development Board (AHDB) system [15] 2007	Dairy cattle	Four-point (0–3)
Sprecher locomotion scoring system [16] 1997	Dairy cattle	Five-point (1–5)
Terrell locomotion scoring system [17] 2006	Feedlot/Finishing cattle	Four-point (0–3)
Zinpro Corporation Step-Up system [18] 2014	Feedlot/Finishing cattle	Four-point (0–3)
North American Meat Institute Animal Welfare Committee system [19] 2015	Feedlot/Finishing cattle	Five-point (1–5)
Tunstall locomotion scoring system [4,20] 2020	Beef suckler cattle	Five-point (0–4) or four-point (0–3)
Fjieldaas locomotion scoring system [8] 2007	Beef suckler cattle	Three-point (0–2)
Simon locomotion scoring system [21] 2016	Beef cattle (cow–calf operation)	Three-point (1–3)

. In the UK dairy industry, the AHDB system is recognized as the industry standard. There is a self-regulatory organization, the Register of Mobility Scorers (RoMS), which provides training and accreditation to individuals to perform mobility scoring for dairy farms using the AHDB scoring method. Accredited status must be upheld by an individual by payment of membership fees, refresher training, and re-examination using online videos of dairy cattle. Some milk suppliers mandate mobility scoring to be performed by a RoMS-accredited scorer as part of the contract between the farmer and supplier.

Regarding the reliability of scoring systems used for beef feedlot cattle, the Terrell system had an inter-observer agreement mean kappa value of 0.52 and an intra-observer agreement mean kappa value of 0.64 when 50 feedlot employees and agricultural students scored video footage of 16 individual feedlot cattle [17]. In the UK, feedlots are uncommon and are not present at the scale seen in American systems. Scoring systems more suited to UK suckler systems and beef cattle herds include the Tunstall four-point system, which was an adaptation of the AHDB system and the Sprecher system. The most recent work by Tunstall et al. enrolled 24 scorers, comprised of 8 researchers, 8 clinicians and 8 veterinary students, to score 40 video clips of individual animals using a 4-point scale shown in

Table 2. Tunstall four-point locomotion scoring system [20], adapted from the AHDB and Sprecher systems [15,16], presented to respondents in the training video for use during the survey.

Score	Locomotion Score Description
0	Normal	Even weight-bearing and rhythm on all four feet. The back is level.
1	Imperfect	Uneven steps or shortened strides, but affected limb not identifiable. The back may show minimal arching while walking.
2	Impaired	Uneven weight-bearing or shortened strides. Affected limb is identifiable (unless multiple limbs are affected). The back may show arching while walking.
3	Severely Impaired	Slower pace—unable to keep up with the healthy herd. Affected limbs are easily identifiable (unless multiple limbs are affected). An arched back may be noted while standing and walking.

. They reported a weighted Gwet’s Agreement coefficient value for inter-observer agreement of 0.70, 0.69 and 0.64 and weighted mean kappa values of 0.84, 0.81 and 0.84 for each group, respectively [20]. Tunstall also described a five-point scoring system (0–4), where a score of 4 indicated an animal with severely impaired locomotion, including non-weight-bearing limb (s). This increased resolution suited its use for research purposes [4]. It was decided that the Tunstall four-point system would be used for the present study due to its previous application on beef cattle showing acceptable reliability results when used by a different demographic of users.

The objective of this study was to assess the intra- and inter-reliability of a four-point mobility scoring system when used by both beef farmers and veterinary surgeons on commercial beef cattle in the UK. The hypothesis was that both beef farmers and veterinary surgeons would have a reliability of over 0.70, with veterinary surgeons having higher intra- and inter-reliability of their scores than beef farmers. This is due to reports of farmers underestimating lameness prevalence by 7% in their herds [22], with this pattern also reported in the dairy industry [23,24]. The definition of acceptable inter and intra-observer reliability in this study was 0.70 [25], which is generally selected as an indicator of acceptable agreement. The reliability of mobility scoring, when performed by beef farmers in the UK, has, to the authors’ knowledge, not yet been reported, despite the importance of the role of farmers in the early detection of lameness within their herds.

2. Materials and Methods

Ethical approval for this study was granted by the Royal Veterinary College’s Clinical Research Ethical Review Board, reference number URN M2023-0182.

2.1. Video Selection

For practical reasons, video clips of cattle were used for this research rather than viewing live animals. Using high-quality videos has been reported to be an acceptable method of reviewing straight-line lameness workups in equines [26] and is common practice in cattle mobility scoring research and RoMS-accreditation examinations [27,28].

To generate the video clips used in the survey, one farm in the West Midlands of the UK was recruited, with approximately 150 suckler cows and a further 240 followers up to finishing weight. All the animals in the herd were filmed from the side view whilst walking on concrete after exiting a crush following a routine handling procedure. The lameness prevalence (scores 2 and 3) in this herd was 26.6%.

R software kappSize package [29] was used to determine the minimum number of clips needed for the online survey. Aiming for 95% certainty that agreement between observers would not be down to chance (25% probability, as a four-point scale was used), a minimum of 29 clips were required.

The footage was screened and edited into clips of individual animals. The resulting 61 clips were scored utilizing the Tunstall four-point (0–3) scoring system [20]. Three veterinary researchers with existing interest and knowledge of cattle lameness assessed the video clips independently. If, on first viewing, all three researchers recorded the same mobility score, the video was classified as suitable for use. However, clips were excluded if there was any disagreement on either score or video suitability between the researchers. For example, videos were excluded if the farmers’ identity could not be protected in the clip or the animal stumbled when exiting the handling system or moved with a changeable gait. Score agreement was obtained on 41/61 clips (10 scored 0, 11 scored 1, 10 scored 2, and 10 scored 3). If there was more than one animal in the video, a visual description of the cow to be identified for scoring was included in the question text, which was necessary on six individual clips.

Participant training video: within each score category, one video was selected for training purposes. Video clips used for training purposes were accompanied by both text and audio descriptions of the scoring system and could be watched as many times as the participant felt necessary. Within the training session, each score-specific video was shown twice and repeated straight away. A transcript of the audio description for the training video can be found in Appendix A.

Training was followed by the presentation of 32 clips to be scored by the participants (8 from each score category). The order of the clips was jumbled with respect to score category, but in the same viewing order for the respondents. These clips included 22 suckler cows, 9 replacement or store heifers and 1 breeding bull, and a range of beef breeds (7 Limousins, 7 Limousin crosses, 5 Herefords, 1 Hereford cross, 8 British Blue crosses, 3 Charolais crosses, and 1 Simmental). Furthermore, within each score category, two video clips were randomly selected using a random number generator to be repeated. This was to gather data for intra-observer reliability analysis. Therefore, participants were presented with a total of 40 video clips to score (32 unique clips, 8 per score category, plus 8 repeated clips, 2 per score category). The clips shown for scoring were able to be watched as many times as the participant felt necessary.

2.2. Survey

An online survey was designed and created using a survey tool (Microsoft Forms, Office 365) to collect data from participants. This meant no manual data entry was required. An initial set of qualitative questions were asked to gather participant information. This was followed by a mobility score training video; then, the forty video clips of cattle were shown. Survey participation was optional, with informed consent collected from each respondent prior to completion of the survey. All results were anonymised.

The first question asked respondents to identify themselves as either a beef farmer or a veterinary surgeon. If ‘farmer’ was selected, a further eleven short text and multiple choice demographic questions were asked about handling facilities, use of footbaths, register of mobility scorer (RoMS) status, and the development and confidence in their lameness knowledgebase. If ‘vet’ was selected, a further eight short text and multiple choice demographic questions were asked about RoMS status, beef client handling facilities, and engagement with lameness. The participants then watched and scored the forty video clips. Participants could watch the videos an unlimited number of times whilst they undertook the survey.

2.3. Distribution

The target audience for the survey was qualified veterinary surgeons (defined as a Member of the Royal College of Veterinary Surgeons) working in the field of large animal medicine within the UK and beef farmers working in the UK. The survey was only available online and was distributed via social media, contact to veterinary practices, advertisement at in-person meetings for beef farmer groups, and advertisement at the British Cattle Veterinary Association congress held in October 2023.

2.4. Statistical Analysis

This survey used a convenience sample. Results of the survey were analysed using Microsoft Excel (Microsoft Excel Office 365) and using IBM Statistical Package for Social Sciences (SPSS) Data Editor and R software version 4.4.1.

Descriptive statistics were employed to summarise the qualitative data gathered. To analyse intra-observer agreement Cohen’s kappa values were calculated, and to analyse inter-observer agreement, Fleiss kappa values were calculated. Landis and Koch level of agreement criteria were applied as follows: κ ≤ 0.00 was poor, κ = 0.001–0.20 was slight, κ = 0.21–0.40 was fair, κ = 0.41–0.60 was moderate, κ = 0.61–0.80 was substantial, and κ = 0.81–1.00 was almost perfect agreement [30].

The data were evaluated for the assumption of normality using a Shapiro–Wilk test and found to be not normally distributed (p < 0.01). Therefore, a non-parametric Kruskal–Wallis test was performed to test the hypothesis that veterinary surgeons would have higher intra- and inter-reliability in mobility scoring than beef farmers.

To investigate if condensing a four-point scale to a binary scale would improve inter- or intra-observer reliability of the scale, scores of 0 and 1 were combined to represent ‘non lame’ cattle, and scores of 2 and 3 were combined to represent ‘lame’ cattle. This reclassification was selected as, although score 1 cattle on the Tunstall scale may have shortened strides or uneven steps (which could represent a lameness problem), identification of an affected limb was not possible and would likely represent a less severe lameness than cattle assigned scores of 2 and 3.

3. Results

A total of 81 respondents completed the mobility scoring survey, with a median time for survey completion of 21 min 41 s, which included the time spent viewing the training video, which was 4 min 4 s in length. The survey was open for two and a half months. Each respondent provided an answer to each mobility scoring video, allowing for analysis of all responses.

3.1. Beef Farmer Demographic

A total of 39 respondents identified themselves as a beef farmer, with a mean time of working in the industry of 20.8 years (range of 0.1 to 50 years). A mean of 188 cattle (range 1 to 950) were managed by each farmer. Of the farmers that responded, two were fully RoMS-accredited, and one had received training but was not accredited.

A range of beef production systems was represented, with almost half farming suckler cows (18/39, 46%). Some farmed a combination of both suckler and either stored or finished cattle (11/39, 28%). Others solely produced store cattle or finishers (6/39, 15%). A small number of reared calves went through to store or finishing (3/39, 8%), and one (1/39, 3%) did not clarify their system.

3.2. Veterinary Surgeon Demographic

A total of 42 respondents identified themselves as a veterinary surgeon, with a mean time working as a qualified vet within a large animal practice of 13.2 years (range of 0.3 to 38). Of the vets that responded, 57% held further postgraduate qualifications. Some of these further qualifications included nine holders of a Certificate of Advanced Veterinary Practice, five speciated to cattle or sheep, four holders of a Postgraduate Diploma in Veterinary Clinical Practice, a holder of a Diploma of Bovine Reproduction, and three Diplomats of the European College of Bovine Health and Management. When asked to estimate what proportion of their large animal work was with beef cattle, the median percentage was 20% (range 5% to 60%). Ten vets were fully accredited RoMS scorers, with a further seven having received the training but were not currently accredited.

3.3. Researcher Inter-Observer Reliability

The overall Fleiss kappa coefficient (κ) across the three researchers when establishing which video clips to include in the survey was κ = 0.70, indicating a substantial level of agreement. For score 0 animals, κ = 0.68; κ = 0.71 for score 1; κ = 0.87 for score 2; and κ = 0.97 for score 3 categories. Clips were not included in the final selection for the survey if there was no fully independent agreement on the score or suitability.

3.4. Inter-Observer Reliability

Figure 1 displays the proportion of responses of each score for each video included in the survey. The overall Fleiss kappa coefficient (κ) across all respondents and all videos was κ = 0.34, which shows fair agreement between observers (

Table 3. Table showing the Fleiss kappa coefficient values with related Asymptotic 95% Confidence Intervals, p values, and Landis and Koch level of agreement when all responses, farmer-only, and vet-only responses were analysed. κ ≤ 0.00 was poor, κ = 0.001–0.20 was slight, κ= 0.21–0.40 was fair, κ = 0.41–0.60 was moderate, κ = 0.61–0.80 was substantial, and κ = 0.81–1.00 was almost perfect agreement.

Rating Category	All Respondents			Farmer Respondents			Vet Respondents
	Fleiss Kappa Coefficient (95% CI)	p-Value	Level of Agreement	Fleiss Kappa Coefficient (95% CI)	p-Value	Level of Agreement	Fleiss Kappa Coefficient (95% CI)	p-Value	Level of Agreement
Overall	0.339 (0.336–0.342)	<0.001	Fair	0.294 (0.287–0.300)	<0.001	Fair	0.381 (0.375–0.387)	<0.001	Fair
Score 0	0.423 (0.417–0.428)	<0.001	Moderate	0.380 (0.369–0.392)	<0.001	Fair	0.464 (0.452–0.474)	<0.001	Moderate
Score 1	0.187 (0.182–0.193)	<0.001	Slight	0.166 (0.155–0.177)	<0.001	Slight	0.208 (0.197–0.218)	<0.001	Fair
Score 2	0.198 (0.193–0.203)	<0.001	Slight	0.155 (0.144–0.167)	<0.001	Slight	0.236 (0.226–0.247)	<0.001	Fair
Score 3	0.591 (0.585–0.596)	<0.001	Moderate	0.515 (0.503–0.526)	<0.001	Moderate	0.660 (0.649–0.670)	<0.001	Substantial

). Both beef farmers and veterinary surgeons had an overall fair agreement (κ = 0.29 and κ = 0.38, respectively). Of the 20 respondents who were RoMS-accredited or who had received training (17 vets and 3 beef farmers), the overall agreement was fair, κ = 0.37, with κ = 0.43 for score 0, κ = 0.19 for score 1, κ = 0.25 for score 2, and κ = 0.65 for score 3 categories.

3.5. Intra-Observer Reliability

Eight duplicate videos were included in the survey to investigate intra-observer reliability. Cohen’s kappa coefficient was calculated for each pair (

Table 4. Table to show the Cohen’s kappa coefficient values with related asymptotic standard error values, calculated lower and upper 95% Confidence Intervals, p values, and Landis and Koch level of agreement when all respondents, farmer-only, and vet-only intra-observer reliability was analysed. κ ≤ 0.00 was poor, κ = 0.001–0.20 was slight, κ= 0.21–0.40 was fair, κ = 0.41–0.60 was moderate, κ = 0.61–0.80 was substantial, and κ = 0.81–1.00 was almost perfect agreement.

Duplicate Clip Numbers	Research Score	Observer	Intra-Observer Reliability
			Mean Assigned Score		Cohen’s Kappa (κ) and Aysmp. SE	95% CI	p Value	Level of Agreement
			1st View	2nd View
4/14	0	Overall	0.22	0.22	0.56 (0.14)	0.29–0.83	<0.001	Moderate
		Farmer	0.36	0.26	0.63 (0.17)	0.46–0.80	<0.001	Substantial
		Vet	0.10	0.19	0.47 (0.23)	0.24–0.69	<0.001	Moderate
1/32	0	Overall	1.01	0.75	0.22 (0.08)	0.14–0.30	0.004	Fair
		Farmer	1.03	0.67	0.37 (0.12)	0.25–0.48	<0.001	Fair
		Vet	1.00	0.83	0.07 (0.11)	−0.10–0.18	0.504	Slight
3/18	1	Overall	0.88	0.79	0.32 (0.09)	0.23–0.41	<0.001	Fair
		Farmer	1.00	0.87	0.31 (0.13)	0.18–0.44	0.004	Fair
		Vet	0.76	0.71	0.33 (0.13)	0.20–0.45	0.008	Fair
9/40	1	Overall	1.14	0.78	0.09 (0.08)	0.005–0.17	0.240	Slight
		Farmer	1.10	0.74	0.03 (0.12)	−0.09–0.15	0.779	Slight
		Vet	1.17	0.81	0.14 (0.11)	0.03–0.24	0.154	Slight
26/36	2	Overall	1.46	1.44	0.51 (0.08)	0.43–0.58	<0.001	Moderate
		Farmer	1.44	1.39	0.48 (0.11)	0.37–0.59	<0.001	Moderate
		Vet	1.48	1.50	0.54 (0.10)	0.44–0.64	<0.001	Moderate
17/20	2	Overall	1.69	1.48	0.43 (0.09)	0.34–0.51	<0.001	Moderate
		Farmer	1.62	1.44	0.31 (0.13)	0.19–0.44	0.019	Fair
		Vet	1.76	1.52	0.51 (0.12)	0.40–0.63	<0.001	Moderate
31/39	3	Overall	2.90	2.91	0.87 (0.11)	0.76–0.98	<0.001	Almost Perfect
		Farmer	2.80	2.82	0.87 (0.12)	0.75–0.98	<0.001	Almost Perfect
		Vet	3.00	3.00	1.00 (0.00)	1.00–1.00	<0.001	Almost Perfect
11/15	3	Overall	2.49	2.44	0.48 (0.09)	0.40–0.57	<0.001	Moderate
		Farmer	2.39	2.26	0.53 (0.12)	0.41–0.64	<0.001	Moderate
		Vet	2.60	2.62	0.41 (0.14)	0.27–0.55	0.006	Moderate

). There was a large range of overall intra-reliability, with video pair 9/40 showing only slight agreement with κ = 0.085, compared to video pair 31/39, showing almost perfect intra-observer agreement with κ = 0.87.

A comparison of inter-observer reliability between farmers and vets was tested, which returned a significant value of p = 0.035. The mean Cohen’s kappa value for farmers was 0.33, which is fair agreement, and for the vets, this was 0.40, which is moderate agreement. This tells us that veterinary surgeons are significantly more reliable at the application of a four-point mobility scoring system than beef farmers; however, both groups still demonstrate poorer reliability than the accepted level of 0.70.

3.6. Non-Lame vs. Lame Analysis

The inter-observer reliability when using a binary non-lame/lame scale was κ = 0.44 for all scorers combined, κ = 0.37 for beef farmers, and κ = 0.51 for vets, as shown in

Table 5. Table to show the inter-observer Fleiss kappa coefficient with related asymptotic 95% CI, p values, and Landis and Koch level of agreement when the four-point mobility scoring system is grouped to a two-point scoring system of ‘Not Lame’ (scores 0 and 1) and ‘Lame’ (scores 2 and 3). κ ≤ 0.00 was poor, κ = 0.001–0.20 was slight, κ = 0.21–0.40 was fair, κ = 0.41–0.60 was moderate, κ = 0.61–0.80 was substantial, and κ = 0.81–1.00 was almost perfect agreement.

Not Lame (0&1) vs. Lame (2&3) System Inter-Reliability	Overall Fleiss Kappa Coefficient	Asymptotic 95% Confidence Interval (CI)		p Value	Level of Agreement
		Lower 95% Asymptotic CI Bound	Upper 95% Asymptotic CI Bound
All	0.441	0.436	0.447	<0.001	Moderate
Beef farmer	0.372	0.361	0.383	<0.001	Fair
Vet	0.512	0.502	0.523	<0.001	Moderate

. The intra-observer reliability for a binary scale is shown in

Table 6. Table showing the intra-observer Cohen’s kappa coefficient with related asymptotic 95% CI, p values, and Landis and Koch level of agreement when the four-point mobility scoring system is grouped to a two-point scoring system of ‘Not Lame’ (scores 0 and 1) and ‘Lame’ (scores 2 and 3). κ ≤ 0.00 was poor, κ = 0.001–0.20 was slight, κ = 0.21–0.40 was fair, κ = 0.41–0.60 was moderate, κ = 0.61–0.80 was substantial, and κ = 0.81–1.00 was almost perfect agreement.

Duplicate Clip Numbers	Binary Research Score	Observer	Binary Scale Intra-Observer Reliability
			Cohen’s Kappa (κ) and Aysmp. SE	95% CI	p Value	Level of Agreement
4/14	0/Not lame	Overall	0.71 (0.16)	0.55–0.87	<0.001	Substantial
		Farmer	0.72 (0.18)	0.54–0.90	<0.001	Substantial
		Vet	0.66 (0.32)	0.34–0.98	<0.001	Substantial
1/32	0/Not lame	Overall	0.37 (0.13)	0.24–0.50	<0.001	Fair
		Farmer	0.66 (0.15)	0.51–0.81	<0.001	Substantial
		Vet	0.11 (0.17)	−0.06–0.28	0.482	Slight
3/18	0/Not lame	Overall	0.32 (0.14)	0.18–0.46	0.002	Fair
		Farmer	0.38 (0.17)	0.21–0.55	0.012	Fair
		Vet	0.18 (0.21)	−0.03–0.39	0.234	Slight
9/40	0/Not lame	Overall	0.21 (0.11)	0.10–0.32	0.240	Fair
		Farmer	0.33 (0.17)	0.16–0.50	0.017	Fair
		Vet	0.11 (0.14)	−0.03–0.25	0.386	Slight
26/36	1/Lame	Overall	0.67 (0.08)	0.59–0.75	<0.001	Substantial
		Farmer	0.58 (0.13)	0.45–0.71	<0.001	Moderate
		Vet	0.76 (0.10)	0.66–0.86	<0.001	Substantial
17/20	1/Lame	Overall	0.48 (0.10)	0.38–0.58	<0.001	Moderate
		Farmer	0.39 (0.14)	0.25–0.53	0.013	Fair
		Vet	0.56 (0.13)	0.43–0.69	<0.001	Moderate
31/39	1/Lame	Overall	0.79 (0.20)	0.59–0.99	<0.001	Substantial
		Farmer	0.79 (0.21)	0.58–1.00	<0.001	Substantial
		Vet	1.00 (0.00)	1.00–1.00	<0.001	Almost Perfect
11/15	1/Lame	Overall	0.37 (0.20)	0.17–0.57	<0.001	Fair
		Farmer	0.38 (0.22)	0.16–0.60	0.010	Fair
		Vet	1.00 (0.00)	1.00–1.00	<0.001	Almost Perfect

, with varying levels of intra-observer reliability depending on which pair of clips has been scored.

4. Discussion

The aim of this study was to assess the intra- and inter-reliability of a four-point mobility scoring system between both beef farmers and veterinary surgeons who applied it to a beef herd during a routine handling procedure. Reliable mobility scoring allows for herd-level lameness monitoring and identification of individual cattle who would benefit from treatment. The overall results from all observers (beef farmers and veterinary surgeons) were poorer than previously reported [17,20]. Of particular interest is the fair level of agreement between the beef farmers, as the owners of the cattle and the people who will see them daily. Farmers are at the frontline of lameness detection and have a greater opportunity to detect lameness as it presents in the individual animal, compared with veterinary surgeons who have a more restricted involvement in herds. The inter-observer agreement of both farmers and vets was higher at the extremities of the scale, with fair to moderate agreement at a score of 0, slight to fair agreement at both scores of 1 and 2, and moderate to substantial agreement at a score of 3. Higher agreement at scale extremities when mobility scoring has previously been reported, suggesting that detection of slight changes in gait is more challenging to identify confidently [31]. The fair to moderate agreement at a score of 0, which represents a truly sound animal, may suggest that animal factors such as breed conformation, age, and sex, alongside environmental factors such as walking surface, can affect the ability to identify a sound animal. Conversely, walking on a more forgiving surface, such as deep straw bedding, and also considering the aforementioned animal factors, may make identifying imperfect or slightly impaired mobility (score 1 or 2) more challenging.

Although still only at a fair level of inter-observer agreement overall, veterinary surgeons were significantly more reliable at mobility scoring than beef farmers. This could be multifactorial; veterinary surgeons will have had exposure to identifying lameness in multiple species as part of their undergraduate training, they may mobility score herds as part of their work, or have undertaken RoMS training. In this study, being RoMS-accredited or trained did not alter the overall category of inter-observer agreement. Although various agricultural qualifications are available that would cover lameness identification, including RoMS training, they are not mandatory for farmers. Improvements in inter-observer reliability following the training of inexperienced and experienced scorers have been previously reported when scoring dairy cattle [31,32]. However, the impact of the short training video viewed by the participants in this study cannot be quantified as there was no control group; without the training video at the start of the survey, the inter-rater reliability could have been poorer than reported. The impact of the number of clips scored on the inter- and intra-observer reliability cannot be established, as improved reliability was reported when both lower (16 clips) and higher (40 clips) numbers of clips were scored for beef cattle [17,20].

Further research across a larger pool of respondents and demographics likely to be involved in mobility scoring would be beneficial to quantify the magnitude of the difference in the reliability of different demographics of scorers. This may further help identify contributing factors to the difference in reliability. With inter-observer reliability of κ = 0.70 between the initial researchers, this shows it is very possible to achieve acceptable levels of reliability when mobility scoring beef animals.

The subjectivity of ‘gold standard’ scores for the 32 videos used for scoring is open to scrutiny, although all reasonable steps were taken to try and ensure objectivity. The substantial level of agreement between the researchers is in line with the inter-observer agreement previously reported when beef cattle are mobility scored by researchers involved in livestock researchers [20], and any disagreement in score from the researchers led to the exclusion of the clip. The higher level of agreement could be linked to a better understanding of the specific scoring system and experience with its application on beef cattle compared with the exposure to the short training video that survey respondents had.

The intra-observer reliability for scoring duplicate videos was very variable, ranging from κ = 0.09 to κ = 0.87. This wide range was observed in both the beef farmers and the vets. This shows poor intra-observer reliability, which presents a challenge in reducing the subjectivity of mobility scoring even when performed by the same scorer. Similarly to the inter-observer reliability discussed above, the intra-observer reliability was improved at the extremities of the scale. This could be because respondents were comfortable with their own detection of moderate to severely lame cattle or ones with good mobility but were less sure of themselves for those with only slightly affected gaits or subtle signs of lameness.

The adaptation from a four-point system to a binary not lame/lame system improved the overall inter-observer agreement from a fair level of agreement (κ = 0.34) to a moderate level of agreement (κ = 0.44) in this study, although it did not increase the inter-observer agreement to an acceptable level of 0.70 or greater. A change of inter-observer reliability to an acceptable level was reported when a five-point scale used on dairy cattle was merged into a two-point scale, although by doing so, the resolution of individual scores within the five-point scale was lost [31]. However, for on-farm use to detect lame cattle for treatment, a binary scale may have a use.

Selection bias could have affected the results due to exclusive online availability and geographical limitation of ‘in person’ information about the survey [33]. This study used a convenience sample; therefore, selection for participants with a balanced range of experience with mobility scoring was not possible. As the training for the survey was delivered online, this may have also affected the understanding of the mobility-scoring system, as respondents did not receive an opportunity to compare their scoring with others as would have been possible if the training was delivered ‘in person’.

The results of this study indicate that inter- and intra-observer reliability, when applied by beef farmers and veterinary surgeons in this context, is fair overall but not of sufficient reliability. This does not detract from the importance of monitoring cattle specifically for signs of lameness. Opportunities for beef herds to be mobility scored by veterinarians, who in this study had increased reliability, should be facilitated to allow for lameness monitoring as a starting point for lameness reduction. Quarterly whole-herd mobility scoring of beef herds would be in line with numerous dairy contract requirements and would allow more reflective lameness data to be recorded on annual health and welfare reviews. This indicates that further research into other methods of lameness detection on beef cattle, for example, thermal imaging or automated camera technology investigating dairy cattle [34,35], would be valuable. Additionally, research into the effect of more formal, detailed knowledge sharing about mobility scoring and cattle lameness detection of beef cattle would also be important.

5. Conclusions

The application of a four-point mobility scoring system to beef cattle, when performed by beef farmers and veterinary surgeons, had lower inter- and intra-reliability than previously reported; however, veterinary surgeons are significantly more reliable at utilising a four-point mobility score on beef cattle compared with beef farmers. This presents a barrier to the early and reliable diagnosis of lameness, which has negative consequences for both welfare and efficiency. Further research into increasing the reliability of beef lameness detection with operator-based scoring would be beneficial, as would research into the application of automated lameness-detection technologies to beef cattle. Specificity and sensitivity studies of mobility scoring and the subsequent diagnosis of a lameness-causing disease would also be valuable.