What Are Restorative Justice Services Recording? Qualitative Analysis of Six Restorative Justice Reporting Templates for Offices of the Police and Crime Commissioner in England

Abstract

This paper is a qualitative documentary analysis of six restorative justice reporting templates used by the Offices of the Police and Crime Commissioner in England for collecting restorative justice service data. Findings identify differences in the following areas: general presentation and format of templates; types of data recorded; areas of interest; definitions and use of descriptive language; methods and timing for counting data; and interpretation of restorative justice processes and outcomes. Conclusions highlight the need to standardise definitions and methods, outlining potential pitfalls when using data to draw further conclusions when equivalency is problematic, and further research avenues that could illuminate the use of data to evidence effectiveness, efficiency, impact and success.

1. Introduction

The concept of a “black box” in restorative work is not new (Umbreit et al. 2002; Bolitho 2017; Lanterman 2021), but never has the metaphor been more literal or appropriate when considering the confidential and potentially commercially sensitive data captured through routine recording and monitoring processes undertaken by restorative services working with service users. The historic focus of the sector, and rightly so, on professional best practice in undertaking restorative processes with service users, and the wide variety of outcomes that can be evidenced as a result, has left this black box of paperwork and data largely untouched until recently.

The work of the All-Party Parliamentary Group (APPG) on Restorative Justice (RJ) has been pivotal in beginning to crack this black box. The APPG on RJ is chaired by Elliott Colburn, MP, and was formed to drive forward cross-party conversations on RJ and raise awareness about its principles. An advisory board was formed to support the parliamentarians, consisting of key stakeholders within RJ in the UK, and is currently chaired by Jim Simon, Chief Executive of the Restorative Justice Council (RJC). The APPG on RJ has three goals: to examine the use of RJ principles within the UK justice system and beyond; to raise the profile of RJ principles within Parliament; and provide opportunities for policy discussion and consultation (All-Party Parliamentary Group on Restorative Justice 2021). The APPG on RJ initially focused on the criminal justice sector but quickly expanded to include the broader spectrum of restorative practice (RP). An inquiry was commenced in 2021 and a call for evidence received submissions from in-house statutory services, externally commissioned private and third-sector providers, practitioners, trainers, managers, academics, activists, those who have been harmed and those who have committed harm. Many of these identities intersect, such as Ray and Vi Donovan who were harmed by the impact of crime, and have become activists, practitioners, and trainers who were instrumental in the birth of the APPG on RJ.

The initial inquiry report highlights the evidence of RJ outcomes, including systemic cost savings, victim/perpetrator satisfaction, and decreased recidivism. However, contributors highlighted several concerns that formed the nine key recommendations from the report (All-Party Parliamentary Group on Restorative Justice 2021, p. 19):

• Registration of commissioned services;
• Standardise the sharing of information;
• Improving quality through effective monitoring and evaluation;
• Publication of a new Action Plan;
• Reviewing ring-fenced funding for restorative justice practice;
• Explore automatic rights for victims through the Victim’s Law;
• End to blanket bans;
• More and better communications;
• Government minister with specific responsibility for restorative justice.

Four workstreams have been organised to address these recommendations and additional evidence has been gathered through virtual and in-person events throughout 2022. The workstreams are responsible for producing a briefing paper for the APPG on RJ, with three of those briefing papers published at the time of writing (All-Party Parliamentary Group on Restorative Justice 2022).

This paper is the first attempt within the field of RJ to directly analyse the recording and monitoring processes of service providers and contributes to an analysis of evidence in respect of recommendations two and three. This paper will provide context to the qualitative analysis of templates used by six different Offices of the Police and Crime Commissioner (PCC) in England by outlining the key literature associated with recording and monitoring in restorative work. The documentary analysis identifies several key themes across the templates and a variety of headings and data points which demonstrate broad use of terminology and count types that require standardisation to enable meaningful analysis of data collected.

1.1. Restorative Practice and Restorative Justice Defined

The lack of a single definition of RP or RJ is compounded by the complex socio-political background from which contemporary methods have developed. With at least 11 definitions of RJ outlined by Paul and Borton (2017), each with a different focus or key feature, the lack of a single definition reflects the multiple ways of working that have developed internationally, at different times, with their own unique culture and legal frameworks that confer institutional legitimacy (Boyes-Watson 2018). The International Institute for Restorative Practice (IIRP) define RP as a social science and the umbrella term of which RJ is a subset, noting that RJ is often reactionary, whereas RP can be preventative, build social capital and relationships, and involve processes that precede wrongdoing (McCold and Wachtel 2003). Marshall (1999) and Zehr (2002) agree that RJ is a process for stakeholders in an offence, but vary in using the language of resolution, dealing with aftermath and future implications (Marshall 1999) and putting things right through a focus on harms, healing, needs and obligations (Zehr 2002). Zehr appears more person- and relationship-centered, whereas Marshall is focused on the offence itself. It could be argued that the emphasis on the crime as opposed to the person is why Marshall’s definition is used by the Crown Prosecution Service (2019) of the UK, which in turn reflects the focus of that institution. The Ministry of Justice (MoJ) notes stakeholder communication for repairing the harm and finding a positive way forward (Ministry of Justice 2014), whilst Calkin (2021) takes this a step further by emphasising the role of restorative questions (Wachtel 2016) and communication as being essential to the definition.

Presser and Van Voorhis (2002) and Doolin (2007) highlight processes and outcomes as driving forces behind the type of definition used, but Bazemore and Green (2007, p. 295) have argued that process-based definitions of RP are insufficient as no practice or process is inherently restorative. Rather, there exist multiple types of processes, with a generally accepted continuum model (Wachtel 2016, p. 4) that outlines the range of informal and formal processes available. Specific outcomes such as victim satisfaction are complex and multiple factors may be at play which no single theory can explain outright (Van Camp and Wemmers 2013). Common principles, general guidelines, and commitments underpinned by values have developed separate RJ processes and outcomes from the criminal justice system (Doolin 2007; Anfara et al. 2013; Van Ness and Strong 2015). Research has identified that practitioners understand the work they are doing encompasses both practice and values (Anfara et al. 2013), of which encounter, amends, reintegration, and inclusion are considered ‘cornerstone’ values (Van Ness and Strong 2015, p. 49).

Challenges to definition have occurred when terminology and branding are used to describe practices that may not utilise or embody specific features (Wood and Suzuki 2016). Distinctions have been made about more and less restorative processes through typology and continuum, with emotional exchange as a core component that cannot occur if only one stakeholder participates (McCold and Wachtel 2003). In prisons, victim and offender terminology has been found to be unhelpful (Calkin 2021). In cases of drug use, the victim and perpetrator are the same person, and the ‘ecosystem’ is disembodied in environmental crime (Pali and Maglione 2021). As processes adapted during the COVID-19 pandemic, stakeholders were not physically present during virtual meetings (Velez et al. 2021). Analysis of international law suggests RJ is being used as a bureaucratic instrument, criminal justice option, and product-service used to enrich the ‘toolbox’ of traditional criminal justice, as opposed to a community-based, informal, and radical justice practice (Pali and Maglione 2021).

1.2. Recording, Monitoring and Accountability

Recording and monitoring serves a number of purposes in restorative work and there exist long-standing opinions that have shaped contemporary practice. Roche (2003, p. 182) cites communication to an internet discussion group “Family Group Conference Interest” from McCold in 2001 who promoted minimal monitoring of individual cases (case and agreement nature, preparation time, meeting length and participant numbers, reason why a case did not proceed), programme-level monitoring (numbers or referrals and sources, number of meetings agreements reached and complied with), and participant surveys around satisfaction, fairness, and referral to other interested persons. Twenty years later, interviews undertaken by the All-Party Parliamentary Group on Restorative Justice (2021) highlight the vital role that service data plays negotiating or reviewing service contracts. Evidence is required by commissioners that demonstrates effectiveness, impact, efficiency, and value for money of restorative approaches, and can affect future work and funding opportunities which can impact staff and service users (All-Party Parliamentary Group on Restorative Justice 2021).

Research in the US has highlighted training for those who wish to implement or employ restorative approaches (Lanterman 2020) as an area where information about recording and monitoring in the sector is problematic. Some trainers may not have been involved in delivering restorative processes themselves, and whilst the RJC training provider code of practice (Restorative Justice Council 2020c) states “trainers must be experienced restorative practitioners”, this is not regulated in the UK. Lanterman (2020) notes a lack of emphasis on quality assurance of practitioners post training and outlines the potential for further research with agencies who have operational quality assurance processes to examine the relationship between staff compliance and competency with RJ conference models and participant outcomes. This suggests a lack of consistency in how services and practitioners develop, and the potential omission of any quality assurance processes in some services. The role of trainers in developing recording and monitoring methods may be important, particularly if an organisation buys into a package with the intention of fully adopting restorative approaches in their work.

The Code of Practice for Victims of Crime (“Code”) contains a number of rights that enable the possibility for recording and monitoring related to RJ that takes places prior to a service accepting a referral and undertaking any formal work. The right to information about RJ and the right to a referral to a RJ service has been interpreted differently by police services because of the use of qualifying language that includes it as a component of a suite of victim services and the lack of ring-fenced funding (Clamp 2022). The promotion of RJ to victims is undertaken broadly via websites, but also directly through informational leaflets, written letter, and verbally in person or over the phone. Who undertakes this promotion and how it is undertaken also varies, and with the offer being available throughout the criminal justice process, it is unclear if each attempt is recorded individually, whether prompts used in forms are actioned, or whether automated processes are checked for completion. Clamp (2022) evidences the role of computer systems in data collection processes, and the cited ease at which PCCs can access this information, but also the problems with data sharing between agencies that can impact on these processes.

Compliance with the rights under the Code continues to be the only requirement related to the recording and monitoring of RJ in the criminal justice sector, but restorative services do collect other data, and this is evidenced by the six reports published by the organisation Why Me? (2016, 2017a, 2017b, 2020, 2021, 2022) using PCC data gathered by use of the Freedom of Information (FOI) Act. The initial paper (Why Me? 2016) focused on how funding was used, with the further research papers moving into greater detail on monitoring and evaluation criteria that emphasised significant variations in terminology and reporting styles which call into question the reliability of any data comparison (Why Me? 2017b). Wider data around outputs and outcomes were explored (Why Me? 2020), with concerns about data quality and breadth in terms of questions and potential answers explored (Why Me? 2021) and a simple business model of restorative services as a flow diagram of inputs, outputs, and outcomes presented (Why Me? 2022). Corroboration of data occurred beyond the FOI data, as direct feedback from a number of PCC’s and the use of case studies were provided for qualitative context (Why Me? 2017b, 2021). The most recent report (2022) reinforces the previous findings highlighting significant concerns about what data is collected and issues with accuracy that impact meaningful comparisons, making recommendations for the inclusion of other data sources such as probation, the public reporting of data, and collaboration between the MoJ and Association of PCCs to ensure accurate spending reporting, clarification of the data return for victims supported by RJ, and redefining outcome measures. Across the six reports, the following types of reported data (other than financial information) were identified:

• Number of referrals;
• Number of new cases;
• Case referral outcomes;
• RJ outputs;
• Victims supported;
• Outcomes related to impact on victims;
• Satisfaction.

These reports offer a valuable insight into the types of data collected for PCCs, highlighting that poor data hinders RJ service development and accountability, and demonstrating what happens to data collected in the types of templates analysed for this research paper. However, these reports include data reported by the MoJ and may not include all the data that is reported by services, let alone that which is collected and not reported to the MoJ.

FOI requests provide secondary data, are not voluntary, and may not be an accurate or complete representation of information from the PCC. The previously cited work of Clamp (2022) used a survey of PCCs, achieving an 88% response rate, and demonstrated the role of compliance monitoring in relation to RJ and the Victims Code directly from PCCs. Contributing PCCs outlined how performance monitoring took place using service user data, aggregate statistics and dip sampling. The majority of respondents described these processes as well developed, although those that stated development was on-going highlighted issues with procurement of new software, implementation of a new contract or uncertainty around monitoring the code and its implications. The research demonstrated how compliance monitoring is varied across respondents using several different methods that include general contract reporting, measuring of performance against organisational action plans, internal working groups, and multi-agency compliance groups. In terms of insight afford by this research about the types of data collected, differences were noted in types of referrals that are accepted, with some restrictions in terms of offence type, and which party initiates a referral.

The recording and monitoring of the formal restorative process by a service starts around the time of a referral. Research has demonstrated that referrals can be impacted by the perception of practitioners (Banwell-Moore 2022, p. 15), with notions of an “ideal restorative justice victim” affecting these. The making of a referral can be linked to information on offence type or victim characteristics, and some forces may only accept a referral if initiated directly by the victim. This work offers an insight into the types of data that may be captured by services at an early stage, such as demographics and referral source. Linked to this area, further research has offered an in-depth analysis of informational materials from PCCs to explore how RJ service providers explain the offer of RJ to victims (Clamp et al. 2022). There is limited research around the documents that RJ services use, and this paper is an example of important work in a developing area of the field. Both of these pieces of research (Banwell-Moore 2022; Clamp et al. 2022), along with that of Clamp (2022), suggest that services may monitor the work practitioners undertake before a referral proceeds, but how this is recorded and monitored is problematic as the service itself may not be the one initially offering information, and furthermore, it is unclear if this time is factored in to calculations of work undertaken in terms of funding received by commissioners. Both studies are exploring components of “the offer” of RJ in different ways and suggest that the recording and monitoring of services could happen earlier in the timeline of a case, but that this early part of the timeline is harder to track due to the potential involvement of multiple agencies that may have different reporting requirements.

Service evaluations and systematic research reviews provide valuable insight into the types of data captured by RJ services. These include, but are not limited to, the previously mentioned longitudinal study of three RJ services for the MoJ (Shapland et al. 2004, 2006, 2007, 2008), the Thames Valley Police restorative cautioning initiative (Hoyle et al. 2002), the Northern Ireland Youth Conferencing Service (Campbell et al. 2006), the Sacro RJ (Diversion) Services across Scottish local authorities (Kearney et al. 2007), and the systematic review of ten RJ conference control trials across three continents (Strang et al. 2013). Despite many historical evaluations of services, data collection methods and templates used by services have not been analysed directly. Aggregation of service data in academic analyses presents issues in relation to understanding outcomes measured in specific services, and broad definitions may not capture individual service nuances and how these differ in use (Strang et al. 2013, p. 35). Shapland et al. (2004) provide clear outlines of numerical data from the services evaluated in the form of counts and percentages concerning case flow and case type data, demonstrating the collection of referral numbers and sources, general demographic data, offender data (offence, sentence, and custody data), progression of cases for offenders and victims, time period for closed cases, interval time period between different parts of the process, general outcomes and individual items included in outcome agreements. Minor variations or omissions are observed between the different services in some of the data categories. Feedback from participants (Shapland et al. 2007) is another data source that contributes to the recording and monitoring of services. Evaluations are usually controlled and standardised to a significant degree for experimental purposes and may not be an accurate representation of the types of data collected and measures used by services more generally. The variations in data types collected by the services (Shapland et al. 2004) suggests that even in experimental conditions this degree of control is not always possible in restorative work. It is also difficult to separate out the types of data collected purely for the purpose of evaluation from that collected normally by services, or whether services continue to collect the full range of data presented in the evaluation after this involvement ceases. For services generally, it is unclear if evaluations shape the data services collected, or if the services shape the data evaluations analysed. Indeed, Shapland (2014) states that the purpose of RJ service evaluation is not simply measuring satisfaction or impact on recidivism, but must consider the outcomes desired, if they have been achieved, and potentially the value for money, and that to evaluate a service on an outcome it did not seek to attain would be unfair (Shapland 2022). It is not possible to know the inner machinations of negotiation for every evaluation undertaken or the outcomes desired, but the literature highlights the distinctions between self and external evaluation and the potential challenges of a service choosing either option (Hamad et al. 2020, p. 19), and that those responsible for setting up services should consider professional appraisals of the dimensions the evaluation is looking at, such as practitioner, referrer, and other support service views (Hamad et al. 2020, p. 17).

Professor Joanna Shapland’s contribution to the RJ literature was cited frequently by contributors to the All-Party Parliamentary Group on Restorative Justice (2021) inquiry, and her work has made important contributions to how the evaluation of services should be undertaken (Shapland et al. 2011), how RJ services should be implemented, and the role of mitigation and risk in RJ for the Scottish Government (Hamad et al. 2020; Shapland et al. 2022). She has noted the important connection between monitoring and evaluation but clearly distinguishes between them as something that all services and practitioners should be doing, and the specialised skills and knowledge required for further evaluation (Shapland 2014), noting that evaluation enables economic analysis, it can attract more service funding, increase facilitator confidence and build reflective practitioners, and enable the documentation and evidencing of restorative processes (Shapland 2022). RJ service evaluations have explored effects and their congruence with the original service aims and have historically lacked transparency in terms of organisational set up, which is troubling given the need to understand measures to evidence effects (Shapland et al. 2011, p. 43). Challenges related to monitoring tasks varied, with volunteering organisations having more difficulty than those supported by statutory criminal justice organisations, but all services required a full-time administrator to track cases, undertake monitoring tasks, and deal with service queries (Shapland et al. 2011, p. 45). Monitoring is vital for effective service evaluation but is also essential to ensure the accountability of services participants, staff, referrers, and funders (Shapland 2014, 2022).

Ample guidance exists globally and publicly from organisations and government agencies about developing restorative services within specific contexts, but specific information on what and how to record and monitor is mixed (O’Dwyer 2014; Chicago Public Schools 2017; Scottish Government 2017; Gregory and Evans 2020; Lustica et al. 2020). The Criminal Justice System (2004) stressed the need for clear referral processes, formal written assessments, and clarifications of administrative tasks in co-working cases. The Sussex Police (2020) are an example of a force with data protection and information sharing requirements linked directly to electronic systems used to input RJ data.

In the UK, the RJC has played a key role in the production of documents that touch on this area, but they are limited in both prescription and detail by design to enable freedom in professional judgement and practice (Restorative Justice Council 2020b). These include the National Occupational Standards for Restorative Practice (Skills Sector Council 2013), restorative service standards (Restorative Justice Council 2016a), and the service providers handbook (Restorative Justice Council 2016b). In 2020 the RJC replaced the Restorative Service Quality Mark with a new Registered Service Provider status and published the service provider registration framework (Restorative Justice Council 2020b) and the service provider code of practice (Restorative Justice Council 2020a). These documents are used primarily by practitioners and services in the criminal justice sector, and emphasize the importance placed on ensuring quality of practice at the service level.

The evaluations of Leeds City Council’s Children Services (Mason et al. 2017; Harris et al. 2020) provide some of the best insight and practice into what restorative services are recording. The project theory of change (Harris et al. 2020, p. 56) highlights the extensive outputs and outcomes that can be achieved when multiple agencies (schools, early help, targeted support services and social work) adopt a single practice model and consistent approach to RP. The implementation of outcomes-based accountability enabled the service to monitor and track performance using the data it was collecting about itself on electronic systems. This suggests that services may track, use, and report data at different points in time across the life of a case, and the development and use of bespoke software can effectively enable systemic analysis of restorative service data sets.

Outside of general service performance and data about services users and cases, recording and monitoring can have other purposes. Council of Europe Recommendation CM/Rec(2018)8 (The 2018 Council of Europe Recommendation CM/Rec 2018) states RJ services should regularly monitor the work of their facilitators to ensure adherence to standards and safe, effective delivery (Pali and Maglione 2021). How practitioners and services undertake their restorative work can be categorized through these methods, and it is possible that best practice can be identified that can improve the field. However, the potential for managerial reach and control of practitioners could be called into question, which previous research has identified the role of financial reports for performance measurements and funding, which could impact on job security, managerialism, and the ‘McDonaldization’ of services (Umbreit 1999) which could affect consistency of reporting requirements (Roche 2003). It could be suggested that there is a potentially coercive impact on some practitioners from recording and monitoring processes.

The publicly available transcripts of submissions to the All-Party Parliamentary Group on Restorative Justice (2021) inquiry highlight key challenges for recording and monitoring with individual PCCs setting their own data recording requirements, an over-reliance on count data that neglects the breadth of restorative work, and a focus on cost and efficiency of service rather than effectiveness, resulting in geographical inconsistencies that prevent meaningful data comparison. Contributors felt that this was further compounded by a lack of understanding by commissioners about what RJ is, with decisions about what to record based on other measures linked to experience of other services, and mixed definitions of RJ and the type of work it constitutes that directly links to funding. Maturity and experience of service is a factor in the type of measures used, where a new service may focus on referrals and service promotion, whereas a longer-standing service may be interested in diversifying the case types. A lack of ategorizedon in recording allows these types of variations to exist. Contributors frequently offered their support for the implementation of a national recording template for RJ that would eliminate these variations and provide a national sample for effective quality assurance, analysis, and comparison that avoids the need for FOI requests if it were published directly by the MoJ.

Theoretical approaches, such as the agency-accountability framework, based on empowerment theory (O’Mahony and Doak 2017, p. 65) and relational theory (Llewellyn et al. 2013, p. 283), offer pathways for improving recording and monitoring by exploring how RP might be measured and what any indicators of success would be.

Doak and O’Mahony (2018) argue that there is no common view of what success is in RP, and discourse around success and what it means needs to be ‘finely attuned’ to the nuances of outcomes and practices within restorative approaches. RJ can be mapped against ‘empowering values’, ‘empowering processes’, and ‘empowering outcomes’ (O’Mahony and Doak 2017, p. 65) via an agency-accountability framework, based on empowerment theory, that moves past the problem-focus of delivering RJ in criminal justice to articulate how and why RJ works at individual and collective levels. The framework envisions a justice system with the fundamental normative goal of re-empowerment, with victims and offenders more likely to achieve this through mechanisms that enable agency and accountability to be exercised (O’Mahony and Doak 2017, p. 197). Crime disempowers victims, offenders, and communities, whilst RJ can be used to re-empower. They outline five common benchmarks:

• Participation;
• Engagement;
• Satisfaction;
• Restoration;
• Reducing recidivism.

The agency-accountability framework is an overarching framework of values housed within those two concepts that help inform the benchmarks in evaluating and understanding success. Wood (2020) built on the work of O’Mahony and Doak (2017) in considering accountability as a core element of RP, but stressing the lack of consensus in definition and noting that research actually highlights two forms: temporal and object accountability. Accountability can be considered in terms of the accountability between participants, as well as accountability of services.

Llewellyn et al. (2013) propose different forms of measurement within RP that focuses more on the values of the process, and measures linked to those such as relational impact, as opposed to measures linked to financial efficiency or counting number of processes completed. The concerns these authors have is that rather than RJ defining these for itself, they have been a response and critique to the justice system and success is in reference to those measures rather than anything that has developed from the RJ field. They propose that RJ should be measured and subsequently evaluated in terms of the way people relate to each other and their communities, leading success to take on new forms beyond the limits of value and measurement as currently understood that reflect the normativity of RJ. They argue that relational theory explores the needs and capacities of human beings their systems, and as such supports the expansion of RP into other domains such as education, workplaces, and communities. Justice must take account of connectedness to others, and the multiple intersecting relationships that exist among them (Ministry of Justice 2014, p. 297). The principles that emerge from a relational theory of restorative approaches are (Ministry of Justice 2014, pp. 301–4):

• Relationship focused;
• Comprehensive/holistic;
• Contextual/flexible;
• Subsidiarity, inclusion, and participation;
• Dialogical or communicative;
• Democratic/deliberative;
• Forward-focused, solution-focused, and remedial.

They note the significant challenge this theory presents for academics, particularly in the field of evidence-based practice and policy development. Measures of success would need to be developed that were convincing enough that resources would be put in place to support further research. Their critique of evaluation as the current method of measuring success notes that often the ‘goal’ of a program is also the ‘outcome’, meaning the change mechanism is often not explored, ergo RP goals, if they are inherently different from the criminal justice system, are not explored either.

The literature has highlighted a number of important areas related to recording and monitoring in RP. These include guidance, standards, and best practice; the types of data that could be collected; reasons why data is collected, whom it is reported to, and the purpose it serves; and theoretical approaches that outline areas of restorative work that can potentially be measured. What becomes apparent is that we do not understand how services are recording their data, or the language and categories they choose to use to capture data about different components of restorative processes. By outlining these in more detail, this paper intends to provide an initial insight into the question: what are RJ services recording?

2. Method

Document analysis is a systematic procedure for reviewing or evaluating an extensive array of document types (Bowen 2009, pp. 27–29), with increased complexity when using hardcopy and digital formats (Padgett 2016) by leveraging raw primary data sources such as policy documents and ategorizedo literature for collection and analysis (Gorsky and Mold 2020). Documents are ‘social facts’ because of the manner in which they are produced, shared, and used (Atkinson and Coffey 2004). This requires the researcher to become an ‘active constructor’ of meaning (Carter 2005, p. 373), leveraging interpretative analysis that explores context, language, and meaning of textual discourses that give insight into how people construct reality, and access to the ‘official’ language used by organisations (Gorsky and Mold 2020). Organisations may have their own terminology or acronyms and timescales related to record keeping, with different sources or rationales which may impact on the way documents are used by practitioners. Conversely, there may be language that practitioners are dissuaded from using, manager or commissioner preferences, or research-driven frameworks. Documents are ‘social facts’ and have the capacity to shape views, perspectives, attitudes, and behavior by the way they are produced, shared, and used (Gorsky and Mold 2020, p. 85). This requires the researcher to treat them as contextual, complex, and potentially problematic (Carter 2005), requiring corroboration before making broad statements (Gorsky and Mold 2020).

Six digital templates were analysed via qualitative documentary analysis methods, supported by the Nvivo 12 software to understand what data RJ services are collecting and reporting to PCCs. Recording and monitoring methods constitute raw primary data sources for systematic review and evaluation via documentary analysis (Bowen 2009).

For this paper, the researcher has been commissioned by the University of Gloucestershire, supporting Workstream 4 of the advisory board for the APPG on RJ, to provide evidence on recording and monitoring within restorative services in the UK. The blank templates were provided by six PCCs as part of their responses to the survey undertaken by the APPG on RJ, which was circulated to all commissioners and police forces in England and Wales. The original draft of this report was reviewed by members of Workstream 4 for clarity, and the service provider responsible for using an individual template provided comment on the initial report to give further insight on some of the findings, or provide clarity on items that were unclear. Further agreement and consent were then obtained directly from the commissioners for that report to be formally submitted for peer review and publication. No personal identifying data regarding commissioners, police forces, geographical areas, participants or practitioners was shared with the researcher prior to analysis, and full anonymity has been maintained for all parties in this report. Elements of the data are considered commercially sensitive and, as such, permission has not been given to share the templates publicly.

Initial observations were made about the variations in presentation and formatting of templates, the general prevalence of quantitative and qualitative data types, and the overarching categories and areas of data requested. Coding was then undertaken to identify all of the headings across templates. Those headings were then explored to identify to what extent different terminologies were being used synonymously. Further coding was undertaken to explore in greater depth the types of data requested, the types of language used to describe these, and the potential implications for ategorizedon of data collection templates across restorative services. Finally, using the headings and groups of data, five themes were identified to broadly describe the data observed across all templates:

• Timeline of case progression;
• Outputs and outcomes;
• Demographics;
• Service accountability;
• The language of crime.

3. Analysis

3.1. Overview of Templates

An initial analysis of the templates (N = 6) highlighted differences in their general structure, including presentation style, general layout of the template, and choice of digital format. Further analysis identified differences in the broader contents and types of data that were requested. These will now be explored in more detail.

3.1.1. Presentation and Format

Five of the six templates analysed used a spreadsheet (Microsoft Excel) format (T1; T2; T3; T4; T5) to facilitate input and the remaining template (T6) was in document (Microsoft Word) format. Immediately, this presents challenges for consolidation of data across services when different types of computer applications are used to collect and report data. Of the five using the spreadsheet format, divergent choices in layout have been made. The most significant outlier amongst these is T3, which has used a horizontal landscape-style format, ordering data categories from left to right using columns, whereas the other spreadsheets (T1; T2; T4: T5) have used a vertical portrait style, ordering their categories from top to bottom using rows. This makes a simple visual comparison of data challenging, but also suggests that basic compatibility across data sets could be problematic due to general presentation and formatting differences. T5 used four sheets within their reporting template to separate key areas of data (referral, outcomes, demographics, and workforce/finance), which supported ease of use when looking for specific information that matched those categories.

Use of spreadsheet software enables robust and automated collection and analysis of quantitative data; however, the power of such software has not been leveraged for these templates. Basic count data requires users to directly calculate totals themselves or select these options within the software. Only T5 is set up to automatically count and total this data, although the additional requirement to provide percentages for outcome measures still requires the person completing this to calculate that themselves. The document format used by T6 makes this particularly problematic as the software lacks these types of statistical functions. Furthermore, T6 asks the individual reporting to undertake further breakdowns of collected count data using a self-referential coding system which requires the individual to scroll back through the document and check the code itself to ensure the correct proportion of that sum is calculated accurately. This method could potentially lead to basic errors, as the software is not able to check this, and requires additional due diligence by the reporter.

Variations in reporting periods are evident across the templates. T5 offers the most comprehensive collection schedule, with data requested monthly for the period of one year, presented April to March, presumably for ease of cataloguing data across the financial year. Only T4 captures a broader range of data over time, requesting data for two years, although this is broken down into intervals of quarters over the standard calendar year, providing eight data points per category (four per year) compared to 12 with T5. T2 and T6 both collect data for the current reporting period and ask the inputter to provide figures related to the previous reporting period. However, T2 requests count data for the current quarter and inclusion of the same count data for the previous quarter to presumably enable quick visual comparison, whilst T6 collects count data for “this period” which is not defined in the blank document analysed, but the service provider confirmed is quarterly, and an additional column for “+/− PRP (previous reporting period)” which requires the inputter to calculate whether the figure for the current quarter is more or less than the previous, rather than simply reporting the previous period figure. The reporting periods for T1 are also not defined, and the majority of the template asks for data only from the current reporting period, except for 10 data points included at the start of the template that constitute a summary of the RJ Service Performance Report.

These data suggest variations in purpose and use of templates. Considering these reporting period variations, it could be assumed that T4 and T5 are longitudinal master documents that enable the continuous collection of service data in a single location, whereas the remaining templates (T1; T2; T3; T6) are individual reports that may or may not be collated into another document at a later time. Given the breadth of work that PCCs undertake, and the range of data they collect, it is highly likely that all of these reports will be used and data from them collated into further documents and reports for varying formats. As individual reporting documents, T2 and T6 ask for previous data which would require additional work of the inputter to locate and identify previous reports and log (in the case of T6) or make additional calculations (T2) in a new report. This feels like unnecessary additional work. For anyone reviewing the data of a single service over a longitudinal period, they would need to review multiple documents to understand patterns and trends. For that function, T4 and T5 offer simpler approaches for quick use of data.

3.1.2. Contents and Data Types

The content and number of potential data inputs varies between templates. Variations in reporting periods make a basic count comparison of data points across templates difficult. For the spreadsheets, these vary from 30 rows to 250. It is not apparent how labour intensive the templates are for the practitioner or service to complete. A template that requires around 60 inputs every month would amount to over 700 points of entry, whereas a template that requires 250 inputs every quarter would accrue 1000 inputs over the year. As noted, formatting and presentation may have as much of an impact on labour intensity, but this study has not ascertained that information at this time. Whether documents that practitioners use can filter digitally into these templates, or require copy and paste by the inputter, or transposing from handwritten documents is not clear but would also impact on workload. Difference in input quantity is not indicative of template quality, as there is clearly a balance between rich detail for analytical purposes and concise data recording.

What is key in terms of general content is the differences between all templates in the way they order and group their contents, and the variations in language and terminology used. These will be explored in more depth throughout this paper. Other than templates which request a summary of key data across categories, the majority of templates begin with requests for data about referrals. T6 requests data on “proactive contact” before referrals, which explores data about potential victims who could be contacted by the service and next steps regarding contact with a RJ facilitator. T1 explores “contact” in the “participation” section, and T2 explores this in the “RJ activity” section, but the sub-headings used across templates are not entirely synonymous and it is unclear if all of these instances of “contact” are pre-referral or post-referral.

All templates request quantitative data predominantly in the form of basic counts, with T3, T4, and T5 requesting some input of percentages. For T5, percentages are required for all data related to outcomes, T4 for five types of conversion rate, whereas T3 asks for them across the majority of sections, including referrals, assessment and action plans, interventions and planned exits.

Qualitative data could be submitted in response to case study and example requests (T1; T2; T5; T6), with differing degrees of opportunity afforded across templates. The opportunity to add “any relevant comments” (T6) for the majority of sections, and two columns consisting of “definition/measure/narrative” and “supporting notes” (T4) for all measures, contrasts most significantly with T3 which uniquely does not request any form of qualitative data in the template itself. All restorative services, other than that of T6, provide additional narrative reports and case studies, and the templates themselves are positioned as tools to collect performance data.

3.2. Timeline of Case Progression

Most data in the templates tracked the progression of cases at a variety of different points in the life of that case work.

Table 1. Timeline of case progression.

Referral	Progress	Case Status	Intervention
Number of referrals	Contact	Active/ On-going	Intervention
Number of new referrals	Engagement/ Participation	Closed	Activity
Number of new referrals accepted	Attrition/ Conversion	Completed	Process

outlines four of these areas of data collection where variations in language were observed that described similar features, or where measurement varied.

3.2.1. Referrals

Methods and language used for counting referral data varied across templates, and this is presented in

Table 2. Referrals.

Area of Language Variation	Example(s)
Declined/Rejected	Variation in input options, including open text options for stating reasons for “other declined referrals”, and restriction to pre-defined choices that do not include “other”. Lack of clarity about what constitutes declined or rejected, and whether there is a variation in the specific term selected by a service.
Referral source	37 unique referral sources identified (Range of 0–18 sources). Use of “Other” varied, including the opportunity to elaborate on that further.
Offence/Crime type	Referral by offence type where victim engaged (new referral only). Offence type where victim engaged. Accepted referral by crime type.

. These measures will inform different types of data depending on if referrals themselves are counted regardless of whether any further engagement with service users occurs, and specifically victims. The headings “accepted referrals by crime type” and “referrals by offence type where victim engaged (new victims only)” are similar data types, but technically distinct due to the additional caveats around “accepted”, “where victims engaged”, and “new victims only” which could lead to different interpretations for data. A total of 37 unique referral sources were identified across all six templates.

3.2.2. Contact, Engagement, Participation, Conversion and Attrition

These five concepts overlapped in use, but variations were observed in how they were grouped.

Table 3. Contact, engagement, participation, conversion, and attrition.

Area of Language Variation	Example(s)
Participation	Used as a heading to capture different data types, with data included varying between templates that used it; Heading for capturing all data regarding referrals, contact, and conversion (T1); Heading to track data about offenders and victims who took part in process.
Contact/Engagement	Terms appear to be used synonymously across templates to describe a professional making contact or engaging with a service user in the early life of a case; Proactive contact (T6) was victim specific, and broken down into numbers identified, numbers agreeing to explore the process, and those declining.
Conversion	“Any participation by both parties to direct activity involving victim and offender”; “From referral to first contact with victim”; “Conversation rate %” (T4) included victim initiated accepted referral to contact made with victim/restorative conversation with victim; Contact established with both parties to indirect/direct restorative process; Offender initiated accepted to initial contact made with victim.
Attrition	Reasons for victims declining RJ; Reasons for offenders’ non-engagement.

highlights the examples of these. Use of the terms “conversion” and “attrition” varied, appearing to reference the move from one stage of a process to another, offering an interesting insight on the general focus of the data. The depth of data in this area varies, with the “attrition” focus primarily connected to process completion when service users do not participate. Attrition language use suggests that offender (non-engagement) and victim (decline) involvement may not be considered equal, with further clarity from service providers highlighting the role of procedural factors such as a service only accepting a victim-initiated referral, or external services such as prisons or probation restricting direct contact with offenders, which highlights wider systemic issues in ensuring equity in the restorative process between victim and offender.

3.2.3. Case Status

Language related to case status is outlined in

Table 4. Case status.

Area of Language Variation	Example(s)
Active/On-going	Synonymous language across all templates, referral to open, active, on-going case work.
Closed/Completed	Distinct terms, case closure referring to the work of the service provider itself, and a case completion linked to processes on the commissioner’s system after the provider has closed the case.

. T6 requests the majority of its data about closed and completed cases. This is distinct from other templates (T1; T2; T5) where the majority of data about processes is about active and on-going cases. Whilst all the data would eventually be collected, there are implications for reporting, particularly if a case takes a long time to be closed or completed, so accuracy is problematic in this template.

It is unclear from the templates whether the amount of additional work that takes place post intervention is fully captured, such as whether cases are subject to further reviews or on-going follow-up, additional paperwork and time taken in relation to this, and where this other work might stand in terms of “completion” or “closure”.

3.2.4. Intervention, Process, and Activity

Language choice, outlined in

Table 5. Intervention, process, and activity.

Area of Language Variation	Example(s)
Intervention	VOC or indirect intervention; “Total referrals ending in restorative conversation” (T3).
Process	Processes included as components of “participation” (T1); “Direct or indirect RJ processes or mediations facilitated” (T6).
Activity	“Indirect RJ” (T1; T2; T5).; Indirect separated between “shuttle” and “letter” (T2); “RJ Activity” (T2); Number of restorative conversations, distinct from direct and indirect activities (T4).

, may provide insight into the view services have about what RJ is. Each template applies a different structure and order to the types of data they collect in terms of processes: T2 describes “RJ Activity” whereas T1 describes “Participation” for overall headers that capture a number of different types of data linked to RJ processes.

3.2.5. Signposting and Onward Referrals

The terms “signposting” (T1; T2; T4) and “onward referral” (T6) are used by templates in reference to service user interaction with other services. These service providers clarified that “onward referral” is a formal process that requires consent, whereas signposting is distinct from this. It is unclear what constitutes the act of “signposting”, and whether this is ategorized in method (such as verbal, complemented with leaflets, direct contact details, and other methods) or counted in the same way.

3.3. Outcomes and Outputs

Feedback, satisfaction, and outcomes are used interchangeably across the templates, covering a variety of different measures, and were grouped differently across the templates. An overview of the differences is contained in

Table 6. Outcomes and outputs.

Area of Language Variation	Example(s)
Feedback and Satisfaction (cases closed in period only)/Victim Outcomes (T1)	Number of surveys given and completed counted, and numbers satisfied or dissatisfied, victim and offenders counted in same section; Improved health and well-being; Better able to cope with aspects of everyday life; Increased feelings of safety; Better informed and empowered to act.
Feedback and Satisfaction/Victim Outcomes Distance Travelled/RJ Outcomes this period for closed cases (T2)	“Coping everyday life post support”; “Satisfaction with outcome of intervention”; “Satisfaction with RJ service”; Total number of process types undertaken.
Feedback and Outcomes (T3)	Separate sections for victims and offenders; “Victims recommending service”; “Victims viewing service in positive light”.
RJ Activity/Victim Outcome/Feedback and Satisfaction (T4)	Distinction in headings for measuring different counts and percentages; Feedback and satisfaction included victims and offenders separately; Count of surveys given, completed, number satisfied, number dissatisfied.
RJ Outcome Measures (T5)	Separate sections for victims and offenders; Number of new and existing offender referrals and longitudinal reoffending count data; “Satisfaction” and “Outcomes for victims” separate; Direct report victim measures and practitioner opinion of victims’ report include the measures: “Coping everyday life post support”; “Satisfaction with outcome of intervention”; “Satisfaction with RJ service”.
RJ Process (T6)	Number of discussions between victim and practitioner. Number of direct restorative process (face to face); Number of indirect restorative process (Shuttle RJ); Number of indirect restorative communication letter—apology or explanation); Improved health and well-being; Better able to cope with aspects of everyday life; Increased feelings of safety; Better informed and empowered to act; Satisfaction in process/dissatisfaction in process.

. Many of the templates use counts of process, activity, or intervention as evidence of service outcomes. For example, T2 uses the header “RJ outcomes this period for closed cases” to cover total number of different process types that are undertaken, whereas T5 includes victim satisfaction measures in their “outcomes”. This could infer that the completion of the process is an outcome for some RJ services, which is distinct from outcomes for the victim or offender.

It is not possible to ascertain how feedback from service users is measured, as it is not obvious from the templates what options are given, such as a Likert scale or a binary yes or no, to allow participants to rate services. Service providers use surveys with participants and their results are reported in the templates. Clarification was provided by some providers that these feedback processes are formally agreed upon with the commissioners. The templates themselves do not provide further detail about how these surveys are undertaken (in-person, sent electronically, sent hard copy), and whether they are all distributed and completed at the same time. For example, T2 header “Feedback and Satisfaction” (which mentions a feedback survey) is distinct from the header “Victims Outcomes Distance Travelled” that notes an “Outcomes survey”.

Service providers clarified that practitioner opinion is captured to support the commissioner to distinguish between service user and practitioner interpretation, and that in practice victim survey response is generally low. Tracking of outcomes for victims varied. The reporting of the measure “victims better able to recover and cope with aspects of everyday life” varied in terms of those that reported this outcome. Victims were able to self-report if this had “improved”, “no change”, or “deteriorated”, and some templates allowed practitioners to report their opinion of the impact on the victim. Data from T6 has been included here as it is exploring what other templates have categorised as “outcomes”, but T6 does not explicitly use the word “outcomes” to describe the questions regarding victim reports.

3.4. Demographics

Half of the templates request data on demographics (T2; T5; T6), but the range of data requested is not uniform across templates. T5 does not request data on religion or belief, and T6 cross references age and gender into a single table for the inputter to break down more specifically, which is not requested in either of the other templates. T2 and T6 have a wider range of ages than T5, which only asks for “less than 18”, offering “12 and under”, and “12–17”. Information on sexual orientation varied, with the options of “gay” or “lesbian” omitted entirely on T5, but “heterosexual/straight”, “bisexual”, “other”, and “not stated” included on all three templates.

The positioning of these data on the templates varies, coming at the end of the template for victims who engaged in a RJ process (T6), towards the start, and for new and existing referrals (T2) and on a separate sheet for new referrals only (T5). Demographic data for offenders is not collected on any template.

3.5. Service Accountability

Terminology that describes the service itself and not necessarily the work it was undertaking included “workforce”, “staffing”, “training” and “service”. Data inputs for “staffing and training” (T2) and “workforce” (T5) were primarily quantitative, but T6 included qualitative sections for “Lessons Learned” and “Current/Future Focus for RJ Service”. Data about the service featured in all templates but was not always placed into such explicit categories. Inputs such as “service satisfaction” were classed as outcomes, and feedback indicating if service users “do not trust RJ service” were given as a reason for declining a referral. T3 tracked participants who would recommend or view the service in positive light. Only two services formally tracked complaints within their templates (T2; T5), and “workforce” combined complaints with staff numbers. All service providers clarified that further information is provided to commissioners in narrative reports which are separate from these templates, but not all services include these details in the template documents.

The approach to tracking staff numbers and hours worked varied, with only T1 and T6 requesting a count of practitioner contact hours. T2 asks for “current and optimum numbers” of staff, which adds a subjective judgment which may incorporate data on current referral numbers and staff caseloads, which could be guided by an “optimum” ratio. T5 asks for vacancies, but it is unclear if this is linked to staff attrition, increases in funding for new staff, or other reasons. T5 requested the most comprehensive pre-populated breakdown of all staff numbers (with six distinct staff types), T2 requested a breakdown of current and new staff and volunteers with further details about the training undertaken. T4 requested detail on current volunteer numbers, including numbers recruited, trained, and how many left the service. T1 only requested volunteer practitioner hours, and required count data on “total number of hours working on cases” and “direct contact hours”, compared to T6 which asks for “hours spent in direct communication” and “hours spent facilitating indirect communication”, but does not track data about additional work undertaken by practitioners.

The explicit reporting of financial data is referred to in T5 alone, with reference to the requirement to attach a quarterly contract finance report in the service workforce and finance Table. It is unclear from the other templates whether a separate financial report is required of them, and the detail required in T5 is not apparent. Questions are asked about numbers of hours worked, and staff and volunteer numbers in other templates which would have a clearer relationship with financial information than other data points. All service providers clarified that they undertake direct financial reporting to commissioners, with the majority undertaking this quarterly, reporting on actual spending and variances against budgets across multiple cost lines.

3.6. The Language of Crime

A common theme around language linked to crime was identified, with terminology such as “victims”, “offenders”, “crime”, and “offences” used frequently across templates. The terms “participants” and “service users” are used across the templates to broadly describe those involved in restorative processes related to crime. The term “victim” is used consistently across all templates, but “offender”, “perpetrator”, and “harmer” are used interchangeably across templates and on two occasions (T3; T6) interchangeably within a template. Data collected about victims is in greater quantities and suggests a general focus by services on the victim component of the restorative process. The majority of data about offenders concerns the specific offence and sentence stage. Demographic information is not collected for offenders in any of the templates analysed. Victim outcomes are requested on all templates, but the language used around offender outcomes varies, with some templates capturing them specifically (T3; T5), and others asking for “offender satisfaction”. T5 and T6 take different approaches to reoffending data as T6 asks for count data on how many “harmers” have or have not reoffended, and T5 asks for the number and percentage of offenders who did not reoffend within 6, 12, and 24 months of the RJ intervention.

Some templates are designed with parallel columns for collecting both victim and offender data, such as number of referrals received and number of feedback surveys given and returned (T1), and RJ activity which includes contacts attempted, case progress, attrition, feedback, and satisfaction (T2). The combining of data occurs in several areas, making it difficult to differentiate between victim and offender, such as safeguarding and complaints (T2), how many people improved (I), saw no change (NC), or deteriorated (D) as a result of the service (T6).

“Offence” and “crime” are used interchangeably across templates, and positioning of offence type varies between templates where those data are requested (T2; T4; T5; T6), located at accepted referral (T2), new accepted referral (T4), referral where victim engaged (T5), and closed/completed cases (T6). Pre-populated options vary with some using breakdowns of certain offence types (e.g., injury/non-injury), and some asking for “other”. All templates combine “Arson” and “Criminal Damage” into a single reporting domain, so it is not possible to statistically separate the two. T6 includes “manslaughter” and “death by dangerous driving” which are not used in any other template. This may be indicative of the crimes where RJ has been used in this geographic area.

T2 requests “accepted referrals by crime type” but states that there should be “one main crime type only for each referral”. T5 does not state that one main crime should be added and is a potential for double or multiple counts if more crimes are involved. A “main crime” could be subjective and depends on who decides this. If at a given time a certain crime is seen as more or less important than another type of crime due to local or institutional priorities, the reliability and validity of data could be called into question.

The counting of offense type is positioned in different ways across templates. T5 describes “referrals by offence type”, and T6 requests “offence type” for closed and completed cases only. These will generate a different type of count of offences depending on if they are linked to the referral, or casework that carries on whether closed or completed. What happens to a referral, whether it becomes a case or not, could impact on reporting as services may get a broader range of offences at the referral stage than the closed or completion stage as not all cases will go beyond referral. Hypothetically, this count could also be affected if services do not work with certain types of offences, which begs questions about whether an inquiry or formal referral about a case with that offence type would be considered a referral if it were rejected.

4. Discussion

An analysis of six templates for reporting data from RJ services commissioned by PCCs in England highlights common themes in recording and monitoring, but significant differences between templates. The differences identified were in general format and presentation, types of data requested, basic content, and use of data collection categories. For data to be used simply and meaningfully, these variations can be overcome by the adoption of a single system for reporting which enables and promotes effective data analysis, as called for in evidence submitted to the All-Party Parliamentary Group on Restorative Justice (2021). The documents themselves do not provide information on author, origin or source, their specific purpose, or the underpinning research and background (Carter 2005; Gorsky and Mold 2020). Whilst we know they are used for reporting to PCCs, how or when they were created and the thinking behind how these developed requires further research to provide context and enable a more holistic analysis that would prevent repetition of previously tried and abandoned methods for recording and monitoring within a universal template.

The bulk of the templates were concerned with data that captured the timeline of case progression, with emphasis placed on different points of the process across the documents. Differences in language and wording were highlighted frequently across templates, but for reasons of volume alone case progression is where the highest number of variations were observed, even where commonality existed in the broad area of language linked to crime. Issues of synonymity and homonymity were observed frequently across the templates. Where language could be presumed synonymous, caveats were frequently added to measures that meant data collected and compared would not be equivalent (offences measured at referral or at case closure; interventions, which include or exclude certain types of restorative work). Different terms were used frequently where meaning should be identical. Take the language used to describe the same type of restorative processes, or the language used to describe individuals who have committed a crime; the different terminology used to describe the same person adds value judgements, and gives potential insight into the thinking of the commissioner and/or service that created their template and their relationship to these individuals.

The analysis highlighted the disproportionate focus on data about victims when compared to offenders, greater variations in terminology used to describe offenders, and differences in how and what data about them are captured. The greater consistency of language and higher number of data entry points for victims is likely due to the positioning of funding that commissions these services, with money coming from the Victims Service grant. Further research would benefit from comparing these findings with the data that the MoJ requests from PCCs, as the All-Party Parliamentary Group on Restorative Justice (2021) highlighted that the MoJ has frequently changed the types of data requested, which would provide further understanding about how these templates have developed and how they go about complying with the data demand. The literature highlighted the variations in language when defining restorative approaches (Paul and Borton 2017; Marshall 1999; Zehr 2002; Crown Prosecution Service 2019; The Ministry of Justice 2014) which it could be suggested has filtered into the processes for recording and monitoring, resulting in the lack of standardised terminology to enable effective use of the data captured by services. Whilst the RJC service provider registration framework (2020) made reference to attempts not to hamper creativity or constrain practice, by avoiding over-prescription and allowing flexibility, significant variations in recording and monitoring have been able to develop that limit purposeful use of data on a systemic level.

The data identified in the theme “outcomes and outputs” suggests that the results of restorative work are complex and contested. Differences in which components of a process and what type of restorative intervention are counted make comparisons difficult. Indeed, many counts of the process itself could be considered an output, such as number of referrals received or accepted, but are also classed as outcome in some templates. The placing of different process components within different interchangeable overarching terms, such as “activity”, “intervention”, “engagement” and “participation”, demonstrates a lack of clarity across the templates and potentially the sector. The intervention itself, whether part of the process or an output, or an outcome, or all of these requires further clarification and more detailed analysis. Many outputs are simple counts which lack qualitative detail that could highlight differences in impact. Limited information about the process of gaining victim feedback and satisfaction would call into question the reliability of any comparison of data sets, particularly if the analysis was not using primary data and clarification of the processes was undertaken with the service itself. The addition of subjective components, where practitioners are asked to give an opinion on victim outcomes, which is potentially aligned with professional judgement, adds an interesting factor. Whilst the reporting of the practitioner judgement of victim’s satisfaction is clearly separated out in the templates analysed here, it is unclear how that is processed once it is reported to the MoJ, and has implications for data about services that are collected by the PCC, reported to the MoJ, and could then be obtained in some form by an FOI request, such as those undertaken by Why Me?. For example, when processed, is that data then deemed equivalent to the self-report of the victims by anyone looking at it without the appropriate context, or is the data used as separate counts of individual cases that could suggest a higher level of satisfaction than achieved. This could therefore call into question the validity of any report that uses these data. Further information is required about the data journey, and how data are used in practice.

Measuring recidivism in restorative services is complex, but not impossible, as highlighted and elaborated on by previous research (Shapland et al. 2008; Strang et al. 2013; Sherman et al. 2015). The general absence of recidivism data, and mixed approach when included, suggests that recidivism can be tracked at the service level but more often than not requires additional data sources and further analysis to extrapolate conclusions. The templates themselves do not use the term “recidivism”, rather referring to “reoffending”. There may be variations in service access to data that enables judgements about recidivism to be made. Service positioning, in-house, externally commissioned, or another form, and the professional relationships between services may impact upon access to wider data that the restorative service itself is not responsible for collecting and managing, therefore limiting potential for analysis and judgement without the need for expensive commissioning of external evaluators to explore data that a service could collect by themselves.

Demographic data should be expanded to both victims and offenders and must be entirely consistent. It is not currently possible to draw wider conclusions about the availability, effectiveness, or impact of RJ within specific communities from templates alone. It is unclear how the templates relate to the methods for collecting data from service users, and whether the categories highlighted in the templates are representative of what service users are given. It is not possible to draw a conclusion about the omission of “gay” and “lesbian” from one template, which could be perceived as not being inclusive. Other sexual orientations, such as pansexual and asexual, are omitted entirely. This would necessitate the use of “other”, but it is unclear if the opportunity for elaboration is afforded to the service user. For a sector that focuses on relationships, this has the potential to negatively affect the one between service users and the service itself, particularly for those who may be more broadly discriminated against in society. Improvement in tracking demographics has the potential to further detail the societal make-up of “ideal victims” (Banwell-Moore 2022), and whether protected characteristics play a role in professional decision making that can affect participation and may highlight discrimination that results in exclusion of persons from restorative processes.

Differences were observed in data collected about services, workforce, and staffing. Hours and types of work undertaken by practitioners, training undertaken or delivered, and approaches to complaints requires standardisation of collection. Drawing wider conclusions about the size, experience, reach and status of the current workforce, their average case load, or numbers of practitioners working geographical areas is not possible with the current templates. This may be due to significant differences in how RJ services are formulated, which is an avenue for further research in itself. Financial input was difficult to navigate from the templates, as the way practitioner hours and associated costs are tracked varied, and financial reports where requested were to be added in separate documents that were not available for this research. From these templates it is unclear how services differ, but the work of Clamp (2022) offers more insight that could be used for triangulation in future work.

Areas of narrow focus, such as focusing on single victims or only enabling the recording of single crimes, draws back to Wood (2020) and the ideas of temporal and object accountability. Accountability towards only one victim does not recognize the impact on wider society or groups of victims, or an inability to link an offender to multiple victims through poor data capture—mixed capture over time—but also aspects of success over time. The many ways that accountability has been considered in restorative work (Wood 2020, pp. 168–73) are predominantly in terms of the offender. Accountability goes beyond just the process and those service users that participate (O’Mahony and Doak 2017; Wood 2020) and includes the accountability of the service itself to funders and the general public. Accountability in restorative work and the ability of the service to demonstrate this requires significant improvements to recording and monitoring. Reporting requirements for PCCs and subsequently the MoJ are a component of service accountability in terms of performance and whilst individual services may be able to report to better degrees than others, minimal reporting may be more satisfactory for some individual commissioners and even in accordance with earlier thinkers’ positions on data use in restorative work (Roche 2003), but on a systemic level it is difficult to gauge with the templates explored here. Training undertaken by staff, and training that the service delivered to others was a reporting requirement in a number of templates. The quality of either cannot be ascertained via the templates alone, even if evaluations are completed, as results are not reported here (Lanterman 2020; Restorative Justice Council 2020c).

Whilst proformas for data capture are not offered in the literature in terms of alternative methods to measure RP (Llewellyn et al. 2013; O’Mahony and Doak 2017), parallels between these templates and the five common benchmarks offered by O’Mahony and Doak (2017) are more apparent than the seven principles from Llewellyn et al. (2013). The benchmarks may appear to be more concrete and easier to measure, but this paper highlights how measuring constructs such as participation, engagement, and satisfaction is still a complex task where meaning can be interpreted differently.

The five themes in this work may be transferable to templates or monitoring processes where RP is used in other sectors, with the “language of crime” being adapted to the chosen professional language of that sector, such as education or social work. However, where service providers undertake restorative work across sectors, language may be observed that is more specifically “restorative” and not aligned with a specific work sector. It is possible that these types of templates may be more closely aligned with the relational theory of Llewellyn et al. (2013).

Given the highlighted differences between templates, it could be argued that the full range and scale of restorative work is not being captured by services. Each offers components, some overlap, but further research is required to understand how these templates relate to practice in the field, and whether templates can more effectively capture the depth of success in practice and outcomes. Leveraging the power of technology in any universal template would make recording and reporting easier in practice, whilst ensuring that data could be used in a meaningful way that provides oversight, insight, and the opportunity to reasonably draw conclusions about effectiveness, efficiency, and impact of restorative services. Gorsky and Mold (2020) highlight the importance of corroboration before making broad statements about findings from document analysis, and this is true of this research. Further research should include a wider range of templates, and questionnaires or interviews with practitioners, managers, commissioners and others who have involvement in the collection of data to enable further elaboration on the initial findings of this document analysis. A further research avenue linked to corroboration would explore what is offered to victims, particularly in terms of the “mechanics of the process” and “outcomes of participation” (Clamp et al. 2022) and what services are recording. Further research is also required to understand how templates are designed, and who decides their content.

5. Conclusions

On top of navigating complex relationships and working with service users towards achieving restoration and repair, administrative tasks that may seem trivial in comparison to practice are drawing out important data. Even the most well-meaning service must be accountable for its actions when working with potentially vulnerable individuals, and when funds are coming from the public purse at a time of continued austerity and strained public resources. This analysis of six templates highlights the urgent need for measures used in restorative service data to be clarified and standardised to ensure that any claims of equivalency are accurate and meaningful. Areas that require particular focus are those of general language used and types of measures employed. Key decisions to improve these should include as many as stakeholders as possible and an agreement should be reached that is used by all services moving forward.

Language should be carefully considered if any attempt is made to standardise data categories, collection, and reporting schedules. Standardisation should enable effective analysis. Examples where this should occur include referral details, the types of intervention offered to ensure equivalence, the language used to describe those who have committed harm, and clear distinctions for feedback, satisfaction, and outcome that support effective evidencing of success. Any standardisation must be accompanied by clear guidance to support practice and ensure clarity. If a template enables the explicit measurement of “manslaughter”, but another would require this to be recorded as “other” crime, the perception of the potential impact of restorative processes on higher profile and more serious crimes could be diminished, or considered non-existent, which in turn would not provide evidence of the effectiveness or impact of restorative work with the most serious crimes.

Timescales for capture and reporting need to be carefully considered and agreed upon, particularly in terms of when specific categories of data are recorded such as types of crimes at point of referral or point of case closure, so any independent analysis of service data is an accurate snapshot at the time it is undertaken. An example of this is when comparing the data from a template that tracks data monthly and then reports, compared to a template that tracks quarterly and then reports, which demonstrates how accuracy can be impacted. Broader, infrequent counts may offer a simpler equivalence, but ignoring opportunities for detailed analysis when data exist limits opportunities for more informed and meaningful data use. It is likely that requesting data about only closed cases will produce data with lower numbers in general about progress, participant demographics, offence type, and sentence stage than those capturing active cases alone. If a system could be used that could accurately isolate and track active cases distinct from closed cases, more effective data reporting could be undertaken.

It is a testament to any service that is willing to be transparent and open its processes up to analysis and potential scrutiny by an outside party. For that, even under anonymity, these services should be applauded. The templates themselves are singular documents which touch upon the vast amount of important work undertaken by practitioners and cannot capture the rich, transformative, and emotional detail of restorative work, nor that of those tasked with compiling these templates. In this sense, quantitative measures of success can only say so much in isolation, and require further reporting by managers to give context, including the use of case studies as specific examples. Whilst each template is different, and has its own merits, it is not the aim of this analysis to ascribe value judgements on specific templates as the restorative community at large must be responsible for deciding which of the components identified should be taken forward and which language and definitions should be used universally. It is interesting that templates have diverged so much from one another, and a further avenue of research would explore how this happened and elaborate on the reasons behind this.

The understanding of ideal measures that could be used would benefit from further research. There are implications for the evaluation of RJ services, systematic reviews, and meta-analyses where the researcher is using secondary or tertiary data. Analyses across international borders are potentially even more problematic if they do not factor in cultural context and use of language which could result in greater variation in definition. This does not negate the significant findings of effectiveness, efficiency, and impact within individual service evaluations, but highlights the need for assurances that any quantifiable claims to broader systemic success are accurate and based on valid and reliable data, and that terminology used is synonymous when drawing conclusions. These findings identify some of the practical challenges related to templates themselves, and the added layers of complexity within them that affect the analysis of data collected. Service or national level analysis drawn from these templates could result in significant challenges related to ambiguity of equivalence and complexity of task, resulting in issues of validity of findings from comparing and contrasting the majority of the collected data.

The simple answer to the question, what are RJ services recording?, is broad sets of different data in a few general areas that do not appear to have a shared language or meaning for terminology and measurement criteria. To call back to the black box analogy, if each service were an airplane, the respective flight recorders would differ in how they recorded speed by miles per hour, kilometers per hour, or knots per hour, and the flight departure times would vary between the point the plane physically moved from the terminal, when it began to accelerate down the runway, and the point of physical take off. Much like these templates, comparison of data gathered would not be impossible but it is unnecessarily complicated. The frequent omissions and data variations in comparable template categories is a lost opportunity to understand the true impact of restorative work and the effectiveness and efficiency of services within and across multiple domains. Further research is required to provide a more nuanced understanding of success in restorative services, and the wide range of success types that occur and could be measured in different ways at different points on the restorative journey for service users, practitioners, services and ultimately society at large.