Machine Learning Applied to NHS Electronic Staff Records Identifies Key Areas of Focus for Staff Retention

Abstract

Background: In this work, we examine determinants of staff departure rates in the NHS, a critical issue for workforce stability and continuity of care. High turnover, particularly among clinical staff, undermines service delivery and incurs substantial replacement costs. Methods: Here, we analyse a unique dataset derived from Electronic Staff Records at Ashford and St. Peter’s NHS Foundation Trust, using a machine learning approach to move beyond traditional survey-based methods, to assess propensity to leave. Results: In addition to established predictors such as salary and length of service, we identify drivers of increased risks of staff exits, including the distance between home and workplace and, especially for medical staff, cost centre vacancy rates. Conclusions: These findings highlight the multifactorial nature of staff retention and suggest the potential of local administrative data to improve workforce planning, for example, through hyperlocal recruitment strategies. Whilst further work will be required to assess the generalisability of our findings beyond a single Trust, our analysis offers insights for NHS managers seeking to stabilise staffing levels and reduce attrition through targeted interventions beyond pay and tenure.

1. Introduction

Retention planning is a critical component of healthcare workforce management, recognised globally as a key prerequisite for high-quality patient care. In the UK, the NHS faces challenges in maintaining a stable workforce (Moscelli et al., 2024), with staff and resource shortages, job demands, work-related stress, the inability to deliver care according to professional standards, discrimination, and poor psychological health identified as some of the key factors contributing to turnover (Leary et al., 2024; A. Weyman et al., 2023; A. K. Weyman et al., 2019). High turnover rates among healthcare professionals lead to disruptions in care delivery, increased workloads for remaining staff, and low staff morale. These are all challenges currently faced by the NHS, exacerbated by the ageing workforce. This has the potential to lead to the loss of experienced personnel, eroding institutional knowledge, compromising team dynamics, and weakening the culture of safety that is essential in healthcare environments (McHugh et al., 2021; Needleman et al., 2002).

Retention is a key focus of the NHS Long Term Workforce Plan (NHS England, 2023), with an aim to drive improvements in organisational culture, leadership, and staff wellbeing, building on the principles set out in the NHS People Promise (NHS England, 2020). Effective retention strategies are essential for NHS organisations, both to reduce recruitment and onboarding costs and also to support continuity of care, improve patient outcomes, and maintain organisational morale (Tikhonovsky et al., 2023). But such strategies need to be monitored and adjusted, with some factors being national in nature and some being local and organisational (Kelly et al., 2022). In addition, understanding what factors constitute early warning signals for individual staff members’ intention to leave can help local management retain staff via more targeted interventions or support than can be delivered by a ‘one-size-fits-all’ retention strategy.

Two sets of organisational factors influencings employee job satisfaction have previously been proposed: extrinsic “hygiene factors” (salary, working conditions, etc.), which need to be adequately addressed to prevent discontentment, and intrinsic “motivation factors” (development, recognition, etc.), which increase long-term satisfaction and motivation (Herzberg, 2005; Sachau, 2007). Much work has been done on understanding key factors relating to turnover and retention amongst healthcare staff (Ahmed et al., 2022; Bimpong et al., 2020; Leary et al., 2024; Moscelli et al., 2025; A. Weyman et al., 2023; A. K. Weyman et al., 2019), and this body of literature indicates that there is an interplay of a range of individual, job-related, interpersonal, and organisational factors, with specific findings differing depending on research methodology, NHS staff groups included, and causes taken into consideration. Many publications have been based on staff surveys and therefore will suffer from subjectivity, survey fatigue, low completion rates, and recency bias (Bogner & Landrock, 2016; Byrne, 2022; McClendon, 1991), with declared intention to leave or stay in post as an outcome variable rather than actual departures. Other approaches to the analysis and prediction of staff departures have also been proposed (Bolt et al., 2022), for example, the use of fluctuation theory to understand and predict turnover. Such an approach can incorporate a full range of measures and indicators, including compensation, job satisfaction, external factors (including the job market), and life events, and model how these might feed into ‘staff departures’ as a stochastic process. These more data-driven approaches can also allow for tipping-point thresholds, such as when initial departures lead to a later surge in turnover (Nyberg & Ployhart, 2013).

The potential linkage of turnover to key data often already included in staff records is underpinned by extensive prior research in the field of electronic Human Resource Management (e-HRM) (Gardner et al., 2003; Strohmeier, 2007). In this work, we take a data-driven approach to identifying factors that influence retention, using Electronic Staff Record (ESR) data from Ashford and St. Peter’s NHS Foundation Trust (ASPH). Our aim is to explore how organisational-level data, which includes all staff groups, data relating to individual, job-related, and organisational factors, and information on whether an individual stayed in the organisation or left, could inform retention policies and whether such data can yield insights into which employees may be most at risk of departure. By demonstrating in a small pilot study that data-driven approaches can yield insight and contribute to an e-HRM-supported understanding of staff turnover, we seek to contribute to a framework for improving retention strategies in the NHS.

2. Materials and Methods

2.1. Dataset and Comparator Data

Staff data were retrieved from ASPH’s ESR and anonymised. ASPH is based in the southeast of England and operates across two main sites—St Peter’s Hospital in Chertsey and Ashford Hospital in Ashford. It is a medium-sized general trust serving a population of around 410,000 people. Records of all staff employed between 31 March 2018 and 31 March 2024 were included, with records excluded for (1) staff who ceased employment due to redundancy, death, or end of non-executive contracts for board members, and (2) rotational resident doctors, as these cycle through placements every 6–12 months. Staff were characterised as Medical, Infrastructure (estates and facilities staff, admin and clerical, and managers), Nursing/Midwives, Qualified STT (scientific, therapeutic and technical, covering all clinical staff other than doctors or nurses) and Clinical Support (including inter alia healthcare assistants, therapy assistants, phlebotomists and similar roles). The dependent variable was defined as whether or not a staff member left the Trust within the following 12 months, using a 31 March start date in each year. Only data that would have been available ‘at the time’ were included in the model, to avoid data contamination with future information, and the model was run once for each year. Employee retention is a ‘repeated measures’ problem so cumulative years in service (effectively, the total number of years of prior successful retention) were included as a covariate in the model. The variables captured are summarised in

Table 1. Variables included in the dataset.

Variable	Measure
Age (last birthday)	Years
Distance to work	Miles
Point of departure	12-month period (April to March)
Contract type	Full-time or part-time
Salary	Pounds
Line manager years in position	Years
Sickness/absence in last 12 months	Days
Cost centre vacancies	Proportion over/under budget
Length of service	Years
Line manager training	Days

; the source for all data was the NHS Trust ESR.

In total 8753 individual records were retrieved for the dataset. Of these, 1330 were medical staff, 2478 were NHS Infrastructure Staff, 2655 were Registered Nurses & Midwives, and 2290 were in other categories (including Qualified STT and Clinical Support). In aggregate, across the series of annual models, there were 48,293 observations.

Comparator data for Ashford and St. Peter’s Hospitals NHS Foundation Trust were taken from NHS Workforce Statistics, published monthly by NHS Digital England (NHS England, 2024).

2.2. Machine Learning Methodology: Supervised Binary Classification

Following data extraction from ASPH Electronic Staff Records, a data matrix was constructed using anonymised staff-in-post identifiers (rows) versus predictor variables (columns) for machine learning. All data were de-identified before analysis to preserve anonymity. An XGBoost (eXtreme Gradient Boosting) binary classifier was chosen, due to its advantages in terms of model interpretability, for example, easily understandable feature importance and its ability to deal with multicollinearity and non-linear relationships. A training/testing split of 80:20 was used to facilitate model training and evaluation, with the random seed set to 42. Grid search cross-validation was employed to systematically search the hyperparameter space and identify the best combination of hyperparameters. Model performance was evaluated using overall accuracy, sensitivity, specificity, and the area under the receiver operating characteristic curve (AUC). Accuracy represents the proportion of correct predictions across all instances, providing an overall measure of model correctness. Sensitivity (also known as recall) quantifies the model’s ability to correctly identify individuals who departed, reflecting its effectiveness in detecting true positive cases. Specificity measures the ability of the model to correctly identify individuals who remained employed, indicating its capacity to minimise false positives. The AUC provides a summary measure of the model’s discriminative ability across all classification thresholds, with higher values indicating better overall performance in distinguishing between staff who resigned and those who did not. All measures were reported as proportions from 0 to 1. To analyse the individual features’ impact on the likelihood of a staff member leaving, feature importance within the model was assessed using SHAP values. SHAP (SHapley Additive exPlanations) values quantify each feature’s contribution to a specific model prediction by distributing the prediction difference (from a baseline). A positive SHAP value indicates that the feature pushes the model output toward a higher risk of departure, whereas a negative SHAP value suggests that it reduces the predicted risk. The SHAP values were then visualised in a beeswarm plot, providing an assessment of both overall feature importance and the directionality of each feature’s effect. Finally, partial dependence plots were generated. A partial dependence plot shows how predicted outcomes change as one input feature is varied, while holding all other features constant. This provides an interpretable curve that reveals the isolated effect of that feature on the model’s output.

All data were analysed in the Python programming language (version 3.14), with the XGBoost algorithm and partial dependence plots implemented using the sk-learn library (version 1.6.1) (Pedregosa et al., 2011).

3. Results

3.1. Overall Retention and Departure Data

Ashford and St. Peter’s NHS Foundation Trust had an average leaver rate of 11% over the period from 2018 to 2024. Whilst detailed comparator data were not available from NHS Workforce Statistics for the complete timeframe, the Trust leaving rates from 2021 to 2024 were close to the average of its ten closest ‘recommended peers’ (using the NHS Model Health System), with the most similar attributes using variables/metrics from, inter alia, finances, estates, workforces, populations, and clinical factors, as summarised in Figure 1.

3.2. Machine Learning Classification Results

An XGBoost model was used to identify the most important factors and breakpoints in predicting the likelihood of a staff member terminating employment in the Trust (12 month prospectively). Key metrics for the XGBoost classification model are shown in

Table 2. Performance metrics for XGB model using the test set.

Metric	Score
Accuracy	0.70
Specificity	0.73
Sensitivity	0.50
AUC score	0.65

, in predicting whether an individual staff member would leave over the next twelve months, for the held-out test set of ESRs.

Overall, the model showed only a modest ability to predict departures, with an area under the curve (AUC) score of 0.65, where a score of 1.00 would be a perfect prediction and a score of 0.50 would represent random chance.

3.3. Key Predictors

The ten variables used as predictors of a staff member leaving the Trust are ranked by feature importance in Figure 2. The single most significant predictor was length of service, followed by age. Distance (home to work) was also notably significant.

As feature importance does not fully explain the complexity of some relationships, partial dependence plots for each predictor variable are shown in Figure 3 for the overall population of ASPH staff. A partial dependence plot fixes other variables and shows how, in isolation, the measured variable influences the probability of departure over the next 12 months.

A number of clear relationships related to individual members of staff emerged, which were largely consistent between staff types. Length of service tended to reduce the likelihood of departure. Age showed a U-shaped relationship with the probability of departure, with higher probabilities of leaving for both younger and older employees. The number of days off work due to sickness in the previous year also showed a relationship with leaving probability, albeit modest, with an increase noticeable at 8 days. Salary also played a role, with a declining probability of leaving as remuneration (and probably seniority) increased. Finally, the distance from home to work showed a notably strong relationship with the probability of leaving, with distances over five miles showing a step-change in the probability of leaving.

Turning to environmental characteristics, cost centre vacancies exhibited a weak overall relationship with leaver probability. Line manager length in post showed a weak relationship with leaver probability, with very short tenures linked to a slight increase in departures, but little effect beyond this point. The year of departure had a modest impact, with reduced rates of leaving in 2019-2020 followed by an increase, likely reflecting pandemic-related issues. Finally, increased days of training undertaken by line managers were positively associated with retention (and conversely, line managers not taking up training was associated with increased departure risk).

Data were also analysed for subsets of the population, specifically Medical staff (n = 1330), Nursing & Midwifery staff (n = 2478), Qualified STT/Clinical Support Staff (n = 2290) and Infrastructure staff (n = 2478). For some variables the trends were similar between employee groups. There were, however, more substantial differences between categories of staff related to (1) cost centre vacancies and (2) distance to work, shown here in Figure 4 and Figure 5. The weak overall relationship between vacancy rates and the probability of departure masked variation across staff groups. For Infrastructure and Nursing and Midwifery staff, vacancy rates had little impact on leaving probability, whereas Medical staff saw a significant rise in the probability of leaving as cost centre vacancies increased. For the distance to work, again, the overall relationship masked larger differences, with medical staff being relatively less sensitive to the distance to work.

Additional partial dependence plots showed less variation between staff groups but are provided for completeness in the Supplementary Materials.

4. Discussion

In this study, we show which factors contribute most to increased probabilities of staff departures. The work is not intended to suggest that machine learning can be used to predict individual reasons for staff leaving. In this sense, the algorithm is similar to car insurance models: it can identify factors that increase the likelihood or risk of an event but will not perform well in forecasting outcomes for specific individuals. The overall model showed modest predictive capabilities, with an AUC value of 0.65. Nonetheless, whilst there are too many individual specific factors that cannot practicably be included in a model, the machine learning approach utilised in this work did show which variables tended to increase the risks of departure.

Length of service showed the strongest feature importance with regard to staff retention, with newly qualified/new-in-post staff being particularly at risk. Age also played a significant role, showing a U-shaped relationship, with younger staff likely more mobile and less settled and older staff possibly facing challenges and/or issues relating to pension arrangements. These relationships are already well-described in the literature (Moscelli et al., 2024; Raman et al., 2024; Taylor et al., 2024). The distance from home to work showed a notably strong relationship with the probability of leaving and was the third most important variable in the model, with a step-change increase for the Trust’s staff at five miles. This cut off point is likely to be related to the geographical location of the Trust just outside the administrative boundaries of Greater London and the available transport links and transport costs, therefore would presumably be different for other Trusts. Medical staff were the only group that was relatively insensitive to this variable, possibly due to higher salaries and therefore smaller influence of transport costs. Nonetheless, this result indicates that Trusts could increase their focus on this as a retention factor, by targeting ‘hyperlocal’ recruitment, especially for non-Medical staff groupings. Some NHS Trusts in the UK are actively pursuing hyperlocal recruitment, with Leeds Community Healthcare NHS Trust successfully filling vacancies through a local recruitment campaign, though the impact on retention has not been reported; however, other benefits such as exceptionally high-quality applications and a positive impact on the local community by offering good work have been noted (NHS Employers, 2023). It is also well-understood that commuting times are an important contributor to work–life balance, which in turn influences employees’ commitment to the organisation and their wellbeing (Moscelli et al., 2025). Issues such as these can receive less attention than headline pay levels but can nonetheless contribute to cumulative dissatisfaction. Healthcare organisations based in rural areas are known to have particular difficulties with recruiting and retaining staff, and in these settings, recruiting locally increases retention (Abelsen et al., 2020). Conversely, Trusts operating in urban and suburban areas may be able to implement measures to address staff issues around commuting. The NHS 10 Year Health Plan (Department of Health and Social Care, 2025) recommends greater focus on recruiting staff in local communities, and our findings corroborate this as a key measure for improving retention.

The number of days off work due to sickness in the previous year also showed a relationship with leaving probability, albeit modest, with an initial increase with any sick leave and a further increase noticeable at 8 days. The relationship between sickness absences and the probability of leaving is well-evidenced (Taylor et al., 2022), with another study indicating that 3 days of absence within a month relating to mental health reasons increased the likelihood of nurses and midwives leaving the acute sector by 27% and consultants by 58%, compared to peers who did not take time off work (Kelly et al., 2022). Presenteeism may also play a role in illness or stress being poorly measured by the actual days of sick leave taken (Taylor et al., 2022). ASPH targets 10 days of sickness absence over a rolling 12-month period as a trigger for formal sickness management, typical within NHS Trusts. The data analysed here indicate that the increased probability of departure occurs sooner than this point, consistent with the fluctuation theory stance on departures not being single decisions, but an accumulation of micro-decisions driven by stress over time. This highlights the significance of supporting staff psychological health and suggests that HR managers should consider providing support for staff taking even a few days of sick leave, and that existing approaches targeting a one-size-fits-all threshold should be adapted by staff type.

Unlike other professional groups, medical staff saw a significant rise in the probability of leaving as the proportion of vacancies in their clinical area increased, indicating that overall staffing levels within their area of practice have a greater impact on retention risks for doctors than for others, potentially due to their overall clinical responsibility being more demanding. An association between senior doctor retention and the retention of co-worker nurses, particularly at pay bands 6–7, which are in operational roles, has been previously reported (Moscelli et al., 2025). High levels of vacancies are often remedied by temporary staffing which can be disruptive for permanent staff and increase their propensity to leave (Bajorek & Guest, 2019; Oliveira et al., 2023). It may also reflect tighter job markets for medical staff, making it easier for medical staff to find new roles and leave when cost centre vacancy rates are high (under-resourcing itself creates a job opportunity as gaps need to be filled, a zero-sum game for NHS Trusts overall). Cost centre vacancy rates are, of course, already measured by NHS managers, but these results illustrate the potential for the differential treatment of specific staff groups to improve retention, as well as the importance of not becoming reliant on temporary staffing.

The probability of leaving decreased with line manager tenure and the number of days of management training that an employee’s line manager had in the preceding 36 months. Although the measured line manager impact is modest compared to other factors, it is a manageable variable. Leadership in acute healthcare settings is complex and requires a multitude of competencies—investing in management training and supporting staff to develop as leaders could improve working relationships and conditions. A UK-wide study with emergency department staff found that leaders did not have protected time for performing this role and were not supported to attend leadership training as part of professional development (Daniels et al., 2024). The Messenger Review of Health and Care Leadership in England (Department of Health and Social Care, 2022) identified gaps in support for leaders and recommended focusing on more consistent and substantive career development pathways for them. This study’s findings support this as a strategy that could help support staff retention.

The work described here does have a number of limitations. First, data were provided on an anonymised and processed basis, with each year treated as an independent set of records rather than being freely available within a Secure Data Environment, which limited the methods that could be applied. Second, whilst clear associations were shown, it is inherent to staff retention that data will be noisy and that some staff will leave for reasons that cannot be measured or predicted through organisational datasets, such as sudden changes in personal circumstances or adverse team cultures. Multi-method approaches to data collection such as the integration of staff surveys or other sources could improve model performance via a comprehensive e-HRM approach (incorporating greater use of fluctuation theory, for example, to model the contribution from changes in environmental instability) but were not possible here. Model performance showed a result that was better than chance but tended to over-predict leaving rates, had poor AUC overall, and would not be suitable for individual staff member predictions (albeit it is also possible that the model predicts leavers who were ultimately persuaded to stay by management actions). The innate variability of staff retention reduces the statistical power of any analysis or algorithm and likely would have been made worse during the 2020–2022 COVID-19 pandemic period, in spite of efforts to manage these strains (Coster et al., 2022; Zasada et al., 2024). Second, due to only having data from a single Foundation Trust, it was not possible to construct well-powered datasets for smaller staff groups, such as therapists or technical staff. The use of single-trust data also limits the generalisability of the conclusions. Third, this work focused on easily accessible data from ESRs and so the list of possible variables was constrained. Whilst this choice was deliberate in order to provide an analysis framework that could be replicated by any Foundation Trust without complex data integration, a broader analysis might have revealed other factors driving retention and allowed for more modelling of interaction effects. A larger study encompassing multiple trusts would be well-placed to address these limitations, as well as other analyses, such as whether the influence of independent variables (predictors) changed over time.

5. Conclusions

Here we have shown that analysis of electronic staff records can yield insight into predictors of staff retention rates and represents an important additional source of information compared to the existing use of survey data, which by its nature tends to be backward looking. Whilst this work represents a pilot study for a single NHS Foundation Trust, it is likely that the results derived herein will reflect common issues and problems. Nonetheless, future work across different NHS Foundation Trusts will be essential to ascertain the generalisability of these findings. An expanded work should also target longitudinal analysis of Trust staff retention including additional sources of data. For example, it may be possible to detect the influence of policy changes over time or achieve better granularity of analysis with regard to specific staff categories (paramedics, nurses, support staff, and others) or measure tipping points and interaction effects. Nonetheless, the findings presented here provide an initial demonstration of how data-driven methods, including machine learning, can yield insight for health planners.