Evaluating Innovation in European Rural Development Programmes: Application of the Social Return on Investment (SROI) Method

Abstract

The quest for innovation lies at the heart of European rural development policy and is integral to the Europe 2020 strategy. While social innovation has become a cornerstone of increased competitiveness and the rural situation legitimizes public intervention to encourage innovation, the challenges of its effective evaluation are compounded by the higher ‘failure’ rate implied by many traditional performance measures. Social Return on Investment (SROI) is employed to assess the social innovation outcomes arising from implementation of Axes 1 and 3 of the 2007-13 Rural Development Programme for England (RDPE). Analysis of primary data gathered through structured face-to-face interviews from a weighted sample of 196 beneficiaries reveal that social innovation outcomes generate a total of £170.02 million of benefits from Axis 1 support measures, compared to £238.1 million of benefits generated from innovation outcomes from Axis 3 measures. Benefits are generated through four social innovation outcome categories: individual, operational, relational, and system; and range from changes in attitudes and behaviour to institutional change and new ways of structuring social relations. The paper calls for more comprehensive evaluation approaches that can capture, and value, the multiple benefits arising from social innovation, and further bespoke applications of SROI to help develop and legitimise innovation indicators that will enable stronger linkages back into the policy process.

1. Introduction

The quest for innovation lies at the heart of European rural development policy. It is a central theme of the European Agricultural Fund for Rural Development (EAFRD) and wider European development policies, including the Cohesion policy programmes for 2007-13, the smart specialisation approach to regional development [1], and is integral to the Europe 2020 Strategy and Innovation Union Initiative [2]. Not only is innovation the cornerstone of increased competitiveness, it also represents a cross-cutting theme for a number of socio-economic activities in rural areas, where SMEs dominate and communities strive for endogenous, bottom-up development [3]. However, whilst the rural situation legitimises public intervention to encourage innovation, its effective evaluation represents a problem [4,5]. Evaluative frameworks administered by the European Commission commonly require ex-ante, mid-term, and ex-post evaluation of rural development programmes (RDPs) based on a common set of criteria and quantitative indicators of change described in the Common Monitoring and Evaluation Framework (CMEF) [6]. The indicators focus on easy to measure outputs (for example, changes to employment and Gross Value Added, number of people who have undertaken training) and in most situations, the indicators are too crude to identify any but the largest changes. The myriad subtle improvements to local economies, or quality of life, and the local ‘innovations’ to improve processes and production in rural areas, are seldom identified or assessed, while changes to human or social capital resulting from innovative activity delivered through RDPs are often missed entirely [7].

Innovation outcomes are usually integrated with wider causal factors of change and are difficult to separate out and evaluate as part of a specific funding stream. At the same time, a central concern of programme implementation agencies is to avoid enhancing the visibility of any negative socio-economic effects, an attitude that does not support innovative activity [8]. More importantly, current approaches to evaluation can actually deter political support for innovation actions due to the higher risk and the difficulty of predicting positive outcomes from expenditure of public money [8]. Innovative activity implies a higher failure rate, at least according to many traditional performance measures such as GDP and employment, but where this is accompanied by the fear that any evidence of failure might discourage future funding, there is no incentive to be either innovative, or to look at the outcomes very carefully.

In addition, the evaluation process itself can also fail to capture important incremental gains due to the time lags involved, and the unpredictable nature of an innovative development trajectory. While this represents an important methodological gap in rural development evaluation, in a wider policy context it results in a failure to capture the full range of outcomes flowing from ‘innovative’ actions. Addressing such a gap requires consideration of conceptual shortcomings which fail to reflect the nuances of how innovation processes occur, and of both the nature and scale of innovation outcomes for a diversity of beneficiaries.

Subject to an adequate re-conceptualisation of innovation in a rural development context, the UK government has recognised that a Social Return on Investment (SROI) framework [9] provides one possible methodological approach to capturing and evaluating a range of innovation outcomes. SROI is a tool that can measure social change in ways that are relevant to the people or organisations that experience or contribute towards that change [10]. In the UK, it has been promoted as a means of enabling social enterprises to quantify the value of their work and increase understanding of how they make a difference [11,12]. SROI has also been developed as a way of assessing the outcomes from more innovative-types of action and is therefore potentially useful in evaluating the complex and increasingly fragmented policy and social contexts in which rural development programmes take place [13]. Through its focus on stakeholder engagement, the principles of an SROI evaluative framework also complement those of contemporary rural development itself, offering opportunities to foster innovation through an improved ability to identify and track relevant outcomes arising from projects. But while the methodology has been encouraged as a tool for measuring social values created through community-focused activities [12], its application has remained small scale, with little consideration of its potential to improve evaluation at a wider ‘meso’ scale, which is the crucial arena for transformative social innovation [14].

Rather than looking directly at the sustainability of territory, the paper examines the potential for producing innovative action among rural stakeholders. This will, of course, have indirect consequences for sustainability in that innovation may lead to more environmental and socially responsible production methods, and social innovation itself can be understood as the effective and sustainable application of a new product, service and/or business model having positive implications at a broader social level [15]. Indeed, SROI itself is also a triple bottom line framework, with direct relevance to sustainability in that economic, environmental, and social outcomes can be dealt with simultaneously. The research described here, therefore, has both methodological and empirical relevance to the wider sustainability framework and debate.

Specifically, the paper describes the application of SROI to assess the social innovation outcomes arising from implementation of Axes 1 and 3 (focusing respectively on ‘modernisation of agriculture’, and ‘improving rural services and the quality of life’) of the 2007-13 Rural Development Programme for England (RDPE). Application of this outcomes-focused and stakeholder-driven method required the collection of primary data from a weighted sample of 196 beneficiaries through structured face-to-face interviews involving Likert-type scales to assess perceived outcome change and underpin the generation of innovation indicators. Following a brief overview of the context around the evaluation of rural development programmes, a re-conceptualisation of social innovation in a rural development context is presented. This provides a more nuanced framework that accounts for the varied nature of social innovation activity, the scale at which the related outcomes occur, and the interactions between stakeholders across various scales. The SROI model parameters pertaining to the innovation outcomes are then provided and a range of Benefit-Investment ratios reported. The subsequent discussion focuses primarily on the methodological lessons learned from the study, including the effectiveness of an SROI approach for evaluating social innovation in rural development, the utilisation of the evaluation findings for policy formulation, and the implications of specific findings to the design of future rural development programmes.

2. Literature Review

2.1. Re-Framing Social Innovation in a Rural Development Context

In many instances, innovation within rural development becomes translated as no more than ‘the application of new technology’ to current situations in attempts to make rural enterprises more resource efficient, more economically productive, and thus more financially resilient. This approach to “optimising performance under neo-classical conditions” [16], however, results in a rather narrow view of ‘innovation’ activity. Innovation consists of more than improvements to cost-efficiency and the concept can also be applied to the wider social and economic processes and outcomes arising from government-driven interventions; for example, the EU adoption of ‘smart specialisation’ as a strategic policy approach to achieving sustainable industrial development and modernisation at the regional scale being one recent example [1,17,18].

Social innovation can be conceived as action that brings about process change, which then influences attitudes, behaviours, and structures. There are multiple definitions of ‘social innovation’ based on a range of perspectives and theoretical foundations [19]. Manzini [20], for example, describes social innovation broadly as “a new idea that works in meeting social goals”, while Preskill and Beer [21] have defined it as:

…‘a novel solution to a social problem that is more effective, efficient, sustainable, or just, than present solutions and for which the value created accrues primarily to society as a whole rather than private individuals.’ Innovation can take the form of new programs, products, laws, institutions, ideas, relationships or patterns of interaction, and it is often a mix of many of these. But perhaps more importantly, the term also describes the process of generating, testing, and adapting these novel solutions, which is inherently exploratory and uncertain.

This definition picks up on two key issues: first, the fact that innovation is the result of a mix of different types of activity, and second, that it is a process of testing and trying out new approaches with all the associated risks. This suggests some commonality with the principles of endogenous [22] and neo-endogenous [23] development, where the aim is to understand how local socio-economic and environmental conditions are improved. In this sense, social innovation can be linked to notions of improvements in social capital [20,24] and the development of local assets [25]. To be considered ‘socially’ innovative, an action requires collective outcomes, critical mass, and social networks [18,26] and may or may not originate at the local level [27,28]. Social innovation can apply to individuals, groups, communities, or entire societies based on the underlying notion that it is the application of activities that are perceived to be ‘new’ or ‘improved’ by the participants in a particular context [29,30], or the development of skills, competencies, networks, and social relations, that results in innovation and improvement [31,32].

Early thinking on the subject [33,34], posits that social innovation is concerned with developing society’s knowledge and knowhow. This places social innovation alongside technical innovation in terms of their need for both the generation and receipt of ‘new’ knowledge, but it also implies knowledge about how a project or programme may be utilized to achieve local stakeholder aims and objectives. Others [5,19] even suggest that a lack of social innovation is often the main barrier to progress in a number of areas. One example being socially sustainable farming systems [35], which develop at the level of co-operative entrepreneurial action rather than through technical intensification. Similar observations have been made about local food initiatives [36].

Two opposing theoretical approaches to social innovation [37] can be identified—agency and structuration, one of which posits that innovation originates in action undertaken by individual ‘agents’, the other that changes in the larger social system initiate opportunities for changes in behaviour, which then lead to innovation. This paper follows Neumeier [19] who suggests both elements are involved, and defines social innovation as:

…changes of attitudes, behaviour or perceptions of a group of people joined in a network of aligned interests that in relation to the group’s horizon of experiences lead to new and improved ways of collaborative action within the group and beyond.

In our view, the collective activity that underpins innovation, whether initiated externally by changes in social structures or by local ‘agent’ behaviour, depends initially on individual action. For example, increased collaboration along a supply chain will only happen if the individuals involved have sufficient—or increased—confidence or skills to work together. To capture the outcomes of social innovation, it is therefore necessary to understand how individual behaviour changes, as well as how social relations and institutional arrangements are structured or altered to constrain or provide opportunities for action. Thus, in a rural development context, social innovation can be considered as any deliberate action that creates new or altered relationships between individuals, or relationships between or within organisations or communities resulting in: new or enhanced forms of collaborative or cooperative activity; changes in the effectiveness or efficiency, of activities undertaken by identified individuals, groups, organisations, and/or communities; or, alterations to the quality of life and wellbeing of individuals, groups, organisations and/or communities.

A change sparked by an innovative activity enables a group of individuals (or community) to achieve some joint goal or valued common purpose, which would not have occurred without the action being implemented. While the explicit recognition of ‘social’ innovation alongside economic and technical aspects of innovation takes us a step closer to capturing and demonstrating the impact of innovative activities in a rural development context, it fails to reflect the nuances of how such activities can be understood and captured to allow for their systematic evaluation. The lack of a credible framework [37,38] for facilitating meaningful evaluation of social innovation practices and outcomes requires a re-conceptualisation of innovation in a rural development context that takes account of the following: changes in the capacity of individuals to engage in innovative actions; the extent to which individuals or groups are empowered/disempowered [14]; alterations in the institutional arrangements creating opportunities for change; the scale at which activities occur (from the individual to the system); the change in linkages between actors across the scales; and, the nature of activities generating innovation outcomes in terms of the relationships between entities, actors and institutional structures.

One approach to re-framing social innovation in a rural development context is set out in the following section. A central challenge is to understand how the changes initiated through a deliberate external intervention (e.g., a government funded programme) influence relationships, resulting in outcomes based on alterations to individual, group or organisational attitudes, values, or behaviour. Figure 1 illustrates how rural development programme action might stimulate socially innovative processes through changing relationships, producing outcomes at multiple levels from the individual upwards. This is similar in some ways to the “shades” of evolution and change conceptualised by Avelino et al. as underpinning “transformative” social innovation [14]. It also acknowledges the need for networks of “enablers” and “brokers” in order to embed innovation and change in wider society [13,39]. Thus, the institutional structures and the wider cultural context in which they operate, are all significant factors influencing what might be considered ‘socially’ innovative. In this way, we can start to think about social innovation in a rural development context as actions that initiate some form of lasting change leading to improvement, based on individual or group behaviour operating within the socio-ecological system structure in which they function [14]. While Figure 1 illustrates the potential for innovation resulting from interactions at the socio-ecological system (SES) level (through which programmes are developed and delivered), there is also an implied territorial aspect. Innovation occurring at individual, social group, and community scales are more likely to be focused on a relatively small spatial area, while regional and national scale impacts will take in multiple communities, sectoral impacts, and wider society.

What initiates change is more difficult to answer. It may be the result of a different way of thinking, a new way of looking at the world based on acquired knowledge/skills, an exploitation of a perceived opportunity within the institutional constraints, a change in the surrounding social structures, or a mix of some or all of these. At this point, the wider capacity of society for change [1] and the issue of ‘legitimacy’ comes into play [37], influencing action beyond the individual. ‘Legitimacy’ is the perceived value of a changed practice, suggesting that an individual will only interact with and seek to influence others within a group or community if the proposed change is accepted as a valid means of improving personal and/or communal welfare, within the current socio-cultural context.

2.2. Evaluating ‘Social Innovation’ Processes within Rural Development

Social innovation originates from behavioural change and social interaction (Figure 1) that alters the way in which individuals or organisations carry out their activities, leading to a change in interpersonal relationships and/or creation of new relationships [19,40]. Innovation behaviour may also result from policy change that alters institutional arrangements to create opportunities for, and/or barriers to, certain forms of activity. Measuring success, however, is difficult due to the integrated nature of both the factors influencing social innovation and the benefits, which do not necessarily produce tangible (material) outcomes [38]. Social innovation has both process and outcome dimensions [19,38] where ‘success’ is linked to the scale of adoption, and requires satisfying four specific criteria: it is innovative with regard to the user, context or application; it meets needs more effectively than pre-existing alternatives; it provides long-term solutions; and it is adopted beyond the initial group/network that developed it.

The first two criteria arguably make sense with regard to rural development but the latter two might be more restrictive in a rapidly changing local context or policy environment where creating the capacity for innovation might be more important than meeting interim targets. This also suggests that the factors affecting the success of social innovation fall into three categories: factors that determine the ‘room to manoeuvre’ (including: funding, organisational structure, public administration support/obstruction); factors influencing the participation process (including: networks, active participation, education, collaboration, individual abilities); and factors important for the success of the overall innovation process (including: relative advantage, compatibility with existing values and experiences, complexity, trialability, observability) [38,41].

This approach to ‘success’, with its emphasis on scale and the numbers adopting a new innovation in order to reach a ‘tipping point’ [38] is more applicable to large scale changes. What we are often dealing with in policy-driven rural development is change resulting from a limited set of policy instruments within specific local contexts over relatively short time periods. In such situations, benefits can be difficult to identify, partial, or incomplete at the point of evaluation. Additional ‘uncaptured’ benefits might relate more to developing the ‘capacity’ to try something different that results in a small-scale improvement, rather than with persuading large numbers to adopt a new approach. The focus of programme evaluation then, should be on capturing the potential benefits of change rather than final outcomes of a process that is only just beginning or incomplete at the end of a programme cycle. In order to capture the more subtle changes of programme activity within the rural development arena, we conceptualise social innovation processes as arising from a limited range of policy instruments in four broad areas:

• Enterprise support: through improving the capacity of individuals in business and operational management to the point where they will be more receptive to change, and more comfortable with initiating change within their organisations;
• Technological change: through technological improvements that open up new business opportunities; capacity building through skills training;
• Service delivery: establishing new ways of organising (individuals, organisations, communities, society) or undertaking familiar activities; developing alternative approaches to making decisions and problem solving;
• Operational processes: initiating new ways of thinking about the relationship with the environment (for example, through concepts of sustainability and resilience) that cause systemic (or system-wide) change in attitudes, behaviour, and processes.

The approach, like the EU Smart Specialisation activity developed to support regional industrial development, suggests a need to focus not only on technological change but also on capacity building and institutional change [42,43]. It also recognises, however, that social innovation involves more complex and evolutionary changes than just creating networks of stakeholders to identify and build on the comparative advantage of regions [1,17]. In order to evaluate social innovation outcomes, we need to measure changes in the processes identified above. This must be accomplished in a systematic manner, and where possible, the magnitude and significance of the changes quantified. In a context where continuation of support (funding, advice and skills development, administrative support) for social innovation depends on persuading policy makers that tangible benefits are produced, there is a need for evaluation tools that can assess the subtle and often ‘immaterial’ changes that occur [38] along with the ‘potential’ to generate future benefits.

The approach applied in the study reported here explores social innovation within five evaluation outcome categories: individual, operational, relational, catalytic, and system. Figure 2 illustrates the linkages between the types of innovation outcome. Outcomes at the individual level might come from an improvement in skills, changed attitudes, or behaviour, for example, that increase confidence enabling an individual to make an investment or adopt new practices. At the operational level, innovation outcomes might arise from an alteration of business management practices or ‘ways of doing’, to improve efficiency, reduce costs, or add value to product development. At the system level, outcomes may derive from changes in the institutional structures within which individuals and groups operate, for example, by providing support for marketing of products, or changing the rules regarding quality standards. Cutting across the different levels from the individual to the system are ‘relational’ outcomes from activities designed to encourage or create opportunities for development of new links, or enhanced levels of cooperation and collaboration. In addition, some activities may have wider impacts or ‘catalytic’ outcome effects that alter perceptions and attitudes of groups of people within a given area or socio-ecological system creating ‘knock-on’ or unintended effects. Catalytic effects can occur at local as well as at wider regional or societal levels. As Figure 2 illustrates, the system itself is situated in a wider socio-cultural context, which will constrain and create opportunities that influence outcomes. Examples include climate change, the weather, market prices, and technological change, all of which will affect attitudes and behaviour of individuals, and may alter the systems in which they operate in unpredictable ways.

The focus of the paper is, thus, on developing an approach to evaluating the capacity for national scale rural development programmes to stimulate social innovation. The RDP for 2007-13 provided a suitable test-bed by enabling application of a social return on investment model (SROI) across four regions of England and collection of relevant data from a wide range of rural communities, and from business enterprises at different stages in the supply chain (i.e., not just primary producers). The high-level objectives for Axes 1 and 3 of the RDPE focused on: improving the competitiveness of farming and modernising agriculture (Axis 1); and, increasing quality of life in rural areas through improvement in services and increasing employment opportunities (Axis 3) [44]. The overall RDP objectives were delivered through a number of Measures under each Axis, with some flexibility allowing member states to select which ones to apply based on the national (or regional) context. Appendix A [44] lists the Measures selected for implementation in England along with the funding schemes utilised or specially created to deliver the RDP objectives.

The nuances of the innovation outcome types outlined above are unpacked further in

Table 1. Conceptual framework for the evaluation of social innovation outcomes (based on an application to Axes 1 and 3 of the 2007-13 RDPE).

Programme Focus	Description of Social Innovation Outcomes at Various Scales
	Individual and Operational	Relational	System
Enterprise support Technical and advisory support for operation and management Improving business skills Support for new products, services and adding value Support for financial services	Improvements in business confidence • Improved financial security • Efficiencies in management • Enhanced capacity to invest and take risks • Alter ‘traditional’ ways of doing	• Increased collaboration Catalytic—‘knock-on’ effects on other community activities Organisational learning Enhanced capacity to invest and take risks	• Increased collaboration Catalytic effects across area/region More efficient value chains Management efficiencies Enhanced capacity to invest Enhanced entrepreneurial skills
Technological change Supporting investment in production techniques Increasing technical skills Increasing environmental awareness to achieve resource efficiencies and reduced emissions	• Improved knowledge and skills • Efficiencies in energy and materials resource use • Cost reduction and higher productivity • New product development • Alter ‘traditional’ ways of doing	• Access to new technologies creates new opportunities Alter ‘traditional’ ways of doing Catalytic—‘knock-on’ effects across sector/community Improved environmental quality	• More highly skilled/productive workforce Efficiency improvements Catalytic effects across area/region Improved environmental quality
Service delivery Improving service delivery Targeting hard to reach sectors of society Addressing gaps in service delivery Increasing collaborative action Changing attitudes and behavior	• Enhanced confidence and well-being • New relationships and/or networks of activity • Increased levels of trust • Enhanced capacity to invest and take risks • Alter ‘traditional’ ways of doing	• Enhanced confidence and well-being Improved quality of life Organisational learning catalytic—‘knock-on’ effects across sector/community New relationships/networks of activity Increased levels of trust	• New relationships/networks Increased levels of trust More comprehensive service delivery Catalytic effects across area/region Community cohesion and increased participation
Operational processes Improving decision-making capabilities Improving delivery mechanisms Reducing implementation costs Enhancing adaptability and resilience	• Application of new techniques Enhanced capacity to take risks Increased levels of trust Wider utilisation of new technology Alter ‘traditional’ ways of doing	• Increased confidence in government support • Improved project selection (lower deadweight/displacement) Lower project failure rate Organisational learning (how to do things)	• Improved project selection (lower deadweight/displacement) Lower project failure rate Organisational learning Enhanced local and regional outcomes Reduced delivery costs

Note: Environmental ‘innovations’ are subsumed into the other categories—the model assumes that environmental goals can only be reached through behavioural changes in economic management, technological improvements, and social action.

, where they are considered in relation to issues of process and scale (Figure 1). Table 1 describes the targeting of activity using the measures and schemes developed within the 2007-13 RDPE, and summarises a set of likely outcomes from ‘innovation action’ at three different scales (the individual and individual business/organisation, the social group, and the wider system/economic sector/society). Two points worth noting are the range of outcomes occurring across the scales, and the emphasis on the role of the individual. Table 1 constitutes a conceptual framework for the systematic evaluation of innovation outcomes in the 2007-13 RDPE, which is described in the subsequent sections of this paper. Although the focus of Table 1 is on programmatic evaluation, the outcomes can also be analysed at different spatial scales in order to explore sustainability performance across territories of interest.

3. Materials and Methods

Given its focus on outcomes from deliberative actions to bring about change, the use of SROI to evaluate innovation activity would seem particularly apt. Through expressing social outcomes as a monetised value, SROI serves as an accounting tool to provide policy makers and funding bodies with a summary of the extent to which expectations about the outcomes of an intervention have been met or not [45]. In addition, it aspires to engage stakeholders, including beneficiaries, in capturing outcomes and impact beyond those that have been pre-defined [46], and has been widely encouraged as a tool for measuring social value created through community-focused activities [12].

In addition to SROI, there are four other main approaches to social impact measurement that are commonly used in policy and programme evaluation or that feature in UK or EU government guidance on policy evaluation. These are: Cost Benefit Analysis (CBA), Cost-utility Analysis (CUA), Cost-effectiveness Analysis (CEA) and Multi-criteria Analysis (MCA). CEA was not considered to provide a broad enough framework to capture the varied nature of outcomes revealed through an initial pilot study, and MCA was discounted due to the complexity of weighting factors necessary to understand societal benefits and the heavy reliance on subjective judgments in scoring and weighting which may not be sufficiently informed. This left CBA and CUA, which could have both provided a platform on which to develop an analytical framework. While CBA has a long history of research across a number of disciplines and is an accepted methodology based on economic and econometric theory, issues around the valuation of market and non-market goods such as health and quality of life, which would be restricted to contingent valuation methods, was deemed to be both methodologically problematic and too resource heavy for the present study. Moreover, whilst the quality-adjusted life year (QALY) approach of CUA helps to get around this problem, its narrow focus on health outcomes was not deemed appropriate to allow a much broader range of social outcomes to be explored, captured and measured.

SROI was therefore deemed the most appropriate framework on which to base development of an outcomes-based framework for the ERDP, and more particularly innovation. Most importantly, it provided a clear role for stakeholders who could tell the story of how change is created for beneficiaries through implementation of a theory of change, and even more importantly, take ownership of this story. Monetisation of outcomes, which allows benefits to be compared against costs on the same metric, was also deemed attractive, providing that a robust programme theory could be developed that would provide a conceptual basis for outcome identification.

Evaluation of EU rural development programmes is driven by the Common Monitoring and Evaluation Framework (CMEF) [47]. A central issue raised in the Common Evaluation Questions (CEQs are set by the European Commission and are a legal requirement of the ex-post evaluation. They establish the overall structure of the evaluation report and are generic across EU Member States). A CEQ set by the European commission to be addressed by the ex-post evaluations for the 2007-13 programme cycle was: to what extent has the RDP contributed to the introduction of innovative approaches? This question (along with others) influenced the design of a Social Return on Investment (SROI) model to evaluate the 2007-13 England RDP, underpinned by a previously developed conceptual framework. SROI has been traditionally employed to assess the social, economic and environmental (triple-bottom line) outcomes at project level and for enabling social enterprises to quantify the value of their impacts to help understand how they make a difference [11,12].

The SROI approach reported here was designed for use at the programme level, to assess the socio-economic outcomes from implementation of Axes 1 and 3 of the 2007-13 RDP for England. The evaluation was conducted over a 12-month period in 2015-16 (the study was commissioned by the UK’s Department for Environment Food and Rural Affairs (Defra).) Axes 1 and 3 of the RDP incorporated a number of Measures, designed and funded at CAP level, which were combined in different ways to create national level schemes targeting specific rural development objectives (Table 1).

3.1. Programme Theory and Outcome Selection

An initial pilot study (A pilot study commissioned by Defra in 2012-13) involving over 60 exploratory interviews with RDPE grant holders (carried out in 2012) revealed SROI to have credible application at the rural development programme level, where it can provide an alternative view of area-wide or regional effects. Other studies have also demonstrated the capacity for the approach to capture a wider range of outcomes [48,49]. In relation to the study presented here, a scoping analysis produced a detailed ‘programme theory’ to identify all of the intended outcomes from each Axis (1 and 3) of the RDPE, and map them within a chain of events to articulate the links between activities and outcomes. This is akin to using Grounded Theory to inform the development of indicators through which the significance and magnitude of programme-induced changes might be measured [50,51,52]. In summary, this approach ensures that outcomes are grounded in the realities of those who experience and implement them, and provides researchers with the means of developing explanatory models of phenomena grounded in empirical data. The programme theory is based on generating detailed knowledge capable of explaining the perceived changes that have occurred (or are occurring) as a consequence of programme actions. This knowledge is then used to inform the development of indicators through which the significance and magnitude of changes might be measured. A distinctive feature of the process is the focus on generating a theory of change, which is used to articulate the links between programme activities and in the SROI impact map [12,52]. Outcomes were identified through analysis of relevant RDPE documents and face-to-face interviews with regional and central policy design and implementation personnel. The resulting programme theory identified a comprehensive set of outcomes arising from the full range of Axis 1 and 3 activities. The conceptual framework developed earlier in the paper (See Table 1) was then utilised to draw out a set of social innovation outcomes for analysis.

3.2. Data Collection and Sampling

In accordance with UK government endorsed SROI guidance [8], primary data was gathered to evidence change, or perceived change, in the identified outcomes to feed into the SROI model. Data were collected from a sample of 196 beneficiaries of the programme. A weighted sample of beneficiaries were selected from each Measure. The sample was drawn from the Defra database of those who had received grants under the schemes funded through Axes 1 and 3 of the RDP in three regions of England (North-west, South-west, East Midlands).

Data were collected by a team of seven interviewers. The interviews were complex, collecting data not only on project outcomes to populate the SROI analysis but also on impacts on employment and productivity of the businesses (or social enterprises) that had received grant awards. Interviewers underwent a full day of training before starting to conduct interviews.

Grant beneficiaries were initially contacted with a letter from Defra informing them of the survey and given the opportunity to opt out of the sample being drawn up from the RPA database. A small proportion opted out, and a reduced database, which also excluded grant beneficiaries that had been interviewed in the Ekos 2013 SROI survey, was provided from which to draw a sample. A weighted sample was drawn for each Axis based on the proportion of grant beneficiaries within each Measure under each Axis. The following additional factors were also taken into account in drawing the sample:

• Year project ended
• Project size (based on size of grant award)
• Location (geographic spread)
• Type of project (selected from the range within each Measure).

A sample of approximately 1200 grant beneficiaries was drawn from across Axes 1 and 3 for the interviewers to contact by phone (Measure 111 is not represented in the sample as it was not possible to identify individuals who had benefited directly from a training programme. Grant awards under M111 were made through contracts with regional or national providers who then subcontracted the delivery of training and skills programmes to suitable local providers (such as agricultural colleges, specialised training companies, and other organisations (e.g., the Cumbria Farmer Network). Interviewers initially contacted grant beneficiaries by phone to request an interview and establish a time and place to meet. A total of 43.8% of respondents in the final sample are businesses/organisations in the lowest category (turnover of <£250,000 per year) while 12.4% of the sample have turnover in excess of £1.6 million per year. Overall, almost two-thirds (62.3%) of the sample are businesses/organisations with a turnover of <£500,000 per year.

Measuring outcomes and model development

All data were collected through structured face-to-face interviews lasting between 1.5 and 2 h. (All subjects gave their informed consent for inclusion before they participated in the study. The study was conducted in accordance with the Declaration of Helsinki, and the protocol was approved by the Ethics Committee of the University of Gloucestershire). Outcomes for each individual beneficiary were measured through creation of 1–5 Likert-type scales assessing perceived levels of change in relation to the impacts of the relevant scheme or Measure under consideration. Scale data was transformed into an appropriate functional range of 0–1, whereby scaled variables were transformed in the form (X-min[X]/(max[X]–min[X]). This produced a transformation of the ordinal codes 1 through 5 (i.e., Strongly Disagree through Strongly Agree): 1 = 0; 2 = 0.25; 3 = 0.50; 4 = 0.75; 5 = 1.0. Outcome measures derived for wider system and programme delivery changes were validated through exploration with regional programme implementation personnel. Beneficiary measures were utilised in the SROI model by creating a set of composite outcome change scores derived from amalgamation of two or more questions from the structured interview process. These outcome scores, measuring the perceived level of change resulting from implementation of the same underlying action, were then converted into ‘indicators of change’ for use in the model. Indicators of change scores were multiplied by the relevant number of programme beneficiaries for each identified outcome, and subsequently by a ‘financial approximation’ to provide a monetary value for each outcome (as described further below).

Accounting for deadweight, attribution and displacement is an important element of the SROI methodology. Deadweight relates to the extent to which outcomes would have happened anyway, without the programme measures, while Attribution refers to the extent to which observed and anticipated outcomes can be attributed to the programme measures as opposed to other projects, activities or initiatives. Both measures are represented as proportions in the SROI model and were informed through the collection of data in the interviews and cross-checked against equivalent social and environmental trends identified through secondary data sources to take account of similar changes or trends that may have occurred for society as a whole over the same time period. The initial pilot study and the programme theory activities had indicated that displacement (the extent to which programme activities had displaced other activities or benefits in the local area) was likely to be minimal, and often not relevant. However, to adhere to the principle of not over-claiming, displacement of impacts was estimated to be 10% for the majority of outcomes.

It was also important for the SROI ratios to account for diminishing impacts of the project over time, and for the value of money to change over time by the inclusion of estimates for drop-off and discount rate. Over time, the amount or significance of an outcome is likely to reduce, or if it remains constant, is more likely to be influenced by other factors, meaning that the attribution of the outcome to the respective programme measure(s) is lower. A drop-off rate (calculated by deducting a fixed percentage from the remaining level of outcome at the end of each year. For example, an outcome of 100 that lasts for 3 years but drops off by 10% per annum would be 100, 90 and 81 in years 1, 2 and 3 respectively) was used to account for this, and was calculated for those outcomes deemed to last more than one year.

A drop-off coefficient of 25% was applied to all outcomes where the benefit period was longer than one year. An annual discount rate of 3.5% was applied to all outcomes in accordance with the HM Treasury recommendations.

Central to the SROI methodology is the monetisation of outcomes, in order that they can be measured in a consistent way using a common currency. This allows computation of a ratio of benefits to costs as the measure of impact which, expressed in monetary terms, can be set against the initial financial investment. Monetisation under this approach, however, represents more than a primary currency facilitating a cost-benefit analysis, and the process of monetisation should not be viewed as purely reductionist in the sense that powerful, often context-specific, outcomes are simply ‘reduced’ to a monetary unit for the purposes of financial and economic accounting. The process undertaken in measuring impacts and selecting financial proxies is more a form of social accounting, within which monetisation allows the ‘significance’ of outcomes to be compared in a consistent way.

The process of monetising the relevant outcomes involves identifying financial proxies for each separate outcome. In other words, approximations of value were sought for each outcome, which in some cases may not be wholly representative of the specific outcome in question. They are instead the ‘best approximation’ (or one of the best) available through which to assess the significance of the outcome to society or the state, and thus allow comparison with other (monetised) outcomes. Two main types of approximation, or valuation, methods, were used in this process: equivalent cost or income that would produce a similar outcome; and potential cost savings to an agency or the state as a result of a negative outcome being partially mitigated. In limited cases, Revealed Preference (the inference of valuations from the prices of market-related goods) or Stated Preference (Willingness to Pay) techniques were incorporated, although under-reliance on such indirect valuation methods is deemed to be a strength of the SROI model [9,46,48,49].

4. Results

4.1. Social Innovation Benefit Estimates from the SROI Model

Table 2. Axis 1 Summative Innovation Impact Map.

Innovation Category	Outcome	Change Score (after Accounting for Deadweight and Attribution)	Financial Proxy	Proxy Value (£) per Unit/Year	Present Value (PV) *
Individual	Increased confidence to apply for grants	0.0758	Percentage change in income required to enter/exit dairy industry (per person)	1325	£8,900,589
	Increased business confidence	0.0678	Cost of self-esteem course (per person)	215	£535,949
	Enhanced capacity to resolve issues	0.1019	Cost of training course to improve business performance (per business)	545	£19,747,605
	Generation of new business ideas	0.1078	Earnings differential realised by completing an HND/HNC qualification (per person)	1950	£10,603,391
	Changes to soil and land management practices	0.0342	Estimated cost of soil erosion (per ha)	2250	£5,664,808
Operational	Improved competitiveness (livestock)	0.0395	Average agricultural gross margin for livestock farms (per farm)	4617	£7,354,250
	Improved wood product value	0.0704	Cost of agricultural consultant advice on business management (per business)	1800	£2,181,109
	Improved business efficiency (woodland)	0.0352	Annual value of wood fuel from 1 ha of woodland (+30% premium for quality biomass)	878.336	£476,961
	More effective woodland management	0.0363	Annual value of wood fuel from 1 ha of woodland (+30% premium for quality biomass) (per business)	878.336	£1,005,373
	Improved viability of farm/business through increased scale and/or capacity	0.0926	Value of increased and safeguarded sales for agriculture/forestry through LEADER (per business)	1243	£4,129,867
	More efficient management of on-farm resources	0.0793	Total input (variable) costs per farm in England (per business)	9494	£17,355,741
	Improvement of farm product quality	0.1113	Added value from investing in precision agriculture (per business)	1100	£9,053,799
	Reduced disease costs and improved animal performance	0.0298	Average cost of a Bovine Tuberculosis (bTB) breakdown borne by the farm (per business)	14,000	£5,198,123
	Increase in farm action to reduce water pollution	0.0481	Average grant for tackling diffuse pollution on farms (per business)	7300	£6,962,889
Relational	Farm benefits from partnership building	0.0184	DfT estimation of business time savings (per business)	7352.64	£26,333,184
	Increased level of engagement across farming community	0.0303	Improvement in knowledge and skills from taking a part-time course (per person)	847	£1,736,779
	Improved wood fuel supply chain capacity	0.0234	Annual value of wood fuel from 1 ha of woodland (per business)	777.004	£119,183
	Opening up of new markets	0.1038	Cost of membership to CLA (per business)	437	£5,263,580
System	Woodland owners better informed	0.0513	Cost of agricultural consultant advice on farm management (per woodland owner)	1800	£3,398,616
	Improved biodiversity and management	0.0469	Household WTP for biodiversity value of woodland (per ha improved management)	45	£34,004,952
Total					£170,026,748

* Discounted to 3.5% following UK HM Treasury guidance.

and

Table 3. Axis 3 Summative Innovation Impact Map.

Innovation Category	Outcome	Change Score (after Accounting for Deadweight and Attribution)	Financial Proxy	Proxy Value (£) per Unit/Year	Present Value (PV) *
Individual	Creation/growth of new micro-enterprises	0.1539	Average cost of young person not in education, employment or training (per business)	561.62	£7,074,836.80
	Improved business capacity to resolve issues	0.1557	Cost of training course to improve business performance (per person)	545	£10,703,487.02
	Improved well-being through development of cultural and recreational facilities	0.1092	WTP for keeping the body and mind active from taking a part-time course (per person)	693	£50,459,131.57
	Increased skills and confidence of local leaders	0.0541	Cost of leadership management training course (per person)	780	£1,416,838.86
Operational	Improved viability of farm/business	0.1189	Value of increased sales (agriculture + forestry) through LEADER (per business)	1243	£8,629,471.90
	Increase in farm incomes through diversification	0.1518	Value of increased sales from diversification (per business)	1099	£4,529,684.62
	Improvement in tourism service provision	0.0821	DfT estimation of business time savings (per tourism provider)	7352.64	£40,873,587.88
	Improved performance of business including resource efficiency	0.0803	Utility bill savings through increased resource efficiency (per business)	138	£861,337.31
Relational	Increased collaboration between tourism providers	0.1192	Value of increased sales arising from tourism development through LEADER (per tourism provider)	17,274	£56,506,314.21
	Increase in collaborative and networking enterprises	0.0949	Improvement in knowledge and skills from taking a part-time course (per business)	847	£4,315,204.53
	Increase in the creation and development of rural social enterprises	0.1447	Cost of leadership management training course (per social enterprise)	780	£3,728,856.17
	Improved social capital, community ties and strengthened civic engagement	0.1816	Average volunteer hourly rate for England (per person)	2891.2	£29,324,584.51
	Increased cross-community development and regeneration through integrated village initiatives	0.0912	Average spend on social activities (per person)	167	£16,150,834.97
System	Improved capacity for local solutions to local problems	0.1293	Cost of leadership management training course (per person)	780	£3,049,790.77
	Improved links between tourism businesses and local environmental and cultural assets (including food and drink)	0.1433	Tourism value of heritage	34.8	£450,072.74
Total					£238,074,034

* Discounted to 3.5% following HM Treasury guidance.

summarise the RDPE Axis 1 and 3 outcomes identified as ‘socially innovative’, under the four categories described earlier (Individual, Operational, Relational, System). In accordance with the SROI method described above, the tables also provide information on the following:

• the ‘change score’, or indicator, used to assess the magnitude of impact of the programme on the outcome, modified by deadweight and attribution estimates that were determined through beneficiary and wider stakeholder interviews;
• financial approximations (or proxies) used to determine the value of change in the outcome;
• the present value of the outcome over the 5-year time horizon, informed by the financial proxy and the number of programme beneficiaries relevant to that outcome.

Social Innovation Outcomes

The largest number of ‘socially innovative’ outcomes under Axis 1 are categorised as ‘operational’. These are mainly linked to business or organisational management activities that lead to improved efficiency. This might include the adoption of modern technology, or a new way of operating a business enterprise, local authority, or civil society organisation. Innovation relates to changes in practice and ‘ways of doing’, and may therefore involve changes in management, adding value, or creation of a new business, product, or activity (e.g., opening up of new markets). Table 2 indicates that a significant number of outcomes arise from changes in management practices, and improvements in efficiency (changing ‘ways of doing’).

Under Axis 1, the highest ‘change’ scores were found for ‘Improvement of farm product quality’, ‘Opening up of new markets’, ‘Enhanced capacity to resolve issues’, and ‘Generation of new business ideas’, suggesting that changes to human capital are significant outcomes from innovation activity, none of which are captured by current evaluation methods. In terms of the monetised value of outcomes, the highest value (£34.004 million) was generated by ‘improved biodiversity and management’, based on a household willingness-to-pay (WTP) value applied to the number of households estimated to benefit across the whole of England. The second largest outcome value stemmed from ‘Farm benefits from partnership building’ (£26.333 million) indicating the significance of ‘relational’ social innovation outcomes.

Under Axis 3, the outcomes with the highest change scores (Table 3) were: ‘Improved social capital, community ties and strengthened civic engagement’, ‘creation/growth of new micro-enterprises’, and, ‘increase in farm incomes through diversification’, indicating the focus of Axis 3 on encouraging improved community services, farm diversification, and wider support for micro-enterprise development. Highest monetary values were for tourism and well-being related outcomes: ‘Increased collaboration between tourism providers’ (£56.50 million) and ‘Improvement in tourism service provision’ (£40.873 million) reflect the considerable value of tourism and large number of potential beneficiaries across rural areas, while ‘Improved well-being through development of cultural and recreational facilities’ (£50.45 million), reflects the potentially high number of beneficiaries across rural England with improved access to local services.

A larger number of ‘relational’ outcomes were identified for Axis 3 than for Axis 1, which is not surprising given the nature of the programme objectives targeting improvements in quality of life and services through community collaboration. ‘Relational’ outcomes identified under Axis 1 refer mainly to the enhanced capacity for networking and engaging more widely with other supply chain stakeholders across the sector. Examples include the strengthened relationships between stakeholders in the livestock sector, and between those in the wood fuel supply chain. Other examples occur on a smaller scale, such as improved relations with advisers, more interaction with neighbouring farmers (e.g., over actions to reduce water pollution), and through engaging in cooperative arrangements (e.g., for grain storage). ‘Relational’ outcomes under Axis 3 focus on increased collaboration between service providers (e.g., tourism), both within and between local communities. The monetary value of the ‘Improved social capital, community ties, and strengthened civic engagement’ outcome (£29.324 million), and ‘Increased cross-community development and regeneration through integrated village initiatives’ (£16.150 million), for example, suggest significant benefits to local communities from engaging in new forms of activity together.

Relatively few ‘system’ outcomes were identified under either Axis. ‘System’ outcomes refer to processes creating outcomes across a wider area or across an economic or social sector of activity. System outcomes might appear at local and regional levels, for example, in the shape of enhanced capacity of programme delivery personnel for solving problems, or for creating and implementing local strategies. Under Axis 3, interviews with programme delivery personnel identified additional capacity for problem solving, while local authority personnel indicated increased skills and confidence, with both outcomes arising from engaging in innovative activity. The monetary value of these outcomes tends to be low due to the relatively small number of beneficiaries. System-wide outcomes might also arise as a result of multiple actions taken at individual and/or operational levels in an area. Under Axis 1 for example, the policy approach taken to enhance woodland management for wood fuel production created secondary benefits of improved biodiversity, helping to achieve some of the wider environmental objectives of the RDPE (largely delivered through Axis 2). Whether these benefits should be included as valid outcomes from innovation is open to discussion, but they have been included here to illustrate some of the potential indirect benefits that might arise from programme- driven innovation activity.

4.2. SROI Model Outputs

The full SROI model outcomes revealed total benefits of £368 million for Axis 1 and £426 million for Axis 3 over the programme period. As previously described, outcomes were valued over a five-year time-frame incorporating a drop-off rate for benefit generation and application of a 3.5% discount rate (

Table 4. Innovation and total Axis 1 and 3 benefits measured by the SROI model.

Category	Axis 1	Axis 3
Total programme benefits (Years 1–5)	£368,078,857	£426,257,211
Total social innovation benefits (Years 1–5)	£170,026,748	£238,074,034

). It is important to note that the SROI model was only measuring ‘social outcomes’ and not the additional improvements in income arising from improved productivity and value, added as a result of investments. The data presented in this paper is a subset of all evaluation data collected, consisting only of the outcomes identified as contributing to social innovation and arising from RDP actions. It indicates that the total value of social innovation outcomes was lower for Axis 1, with a total of £170.02 million of benefits generated, compared to £238.1 million from Axis 3.

Other applications of SROI to rural development programmes have yielded results of a similar order of magnitude.

Table 5. Comparison of SROI estimates applied to rural development.

Title of SROI Study	Scope/Type of Project	SROI Ratio	Comment
Social Return on Investment: a new way to measure FLAG results. Cornwall and Isles of Scilly FLAG. 2016 https://webgate.ec.europa.eu/fpfis/cms/farnet/social-return-investment-new-way-measure-flag-results	To measure the impact of animation activities of a Fisheries LAG in Cornwall 2012 - 15	1: 5.45 euro (1: £4.87)	Limited information available on methodology.
Solway Borders and Eden LAG, Cumbria (Rose Regeneration evaluation)	Application across all SBE LAG Axis 3 projects	1: 5.34	No information on how SROI was applied; benefits appear to be measured over 5 years. Mostly focused on jobs and use of community centre. Application of project. SROI across all Axis 3 of the SBE Programme. 15% ‘leakage’ applied to overall benefits.
North York Moors Coasts and Hills LAG (Rose Regeneration, 2014)	SROI of 3 capital projects	Capital projects average across the LAG: 1: 9.86	SROI analysis of three projects; also ‘suggests’ the average return rate for capital projects is estimated to be around £6.00.
Social Return on Investment (SROI) Analysis of the Greenlink. Central Scotland Forest Trust (CSFT) (Ea O’Neill, greenspace, Scotland, 2009)	The Greenlink - a 7 km cycle path creating a direct route from Strathclyde Country Park to Motherwell Town Centre.	SROI ratio based on total investment: 1: 7.63	Programme of woodland management, conservation and community events are part of the project, developed in partnership with the communities along the route. Variable deadweight/attribution measures applied. NPV over 5 years; standard rate of drop-off = 15%, and discounted at 3.5%.
Evaluation of the impact and economic and social return on investment of Axis 1 and Axis 3 activities, RDPE 2007-13 (Ekos Final Report June 2015)	A partial (‘cut-down’) SROI model developed based on 32 projects from three Axis 3 measures: (M321), village renewal (M322), and conservation of rural heritage (M323). 28 of the projects delivered through LEADER. SROI values presented in 8 thematic groups.	Thematic group ratios range from 1: 1.21 (Cultural and heritage improvements) to 1: 15.01 (Broadband) Community halls 1: 3.40 Natural asset improvements 1: 8.13 Overall Axis 3 SROI ratios: 1: 5.85 for RDPE investment 1: 6.65 for total public + private investment.	Based on data collected from beneficiaries in workshop settings (153 beneficiaries attended, approx. 5 people from each project) where benefits, attribution, deadweight, and displacement were all agreed by participants. Each beneficiary limited to 2 outcomes. NPV values calculated over 2–3 years. (n + 2).

below provides a range of SROI scores for comparative purposes. In many instances, the values in the table are significantly higher than the values presented in this report. This is due to a number of factors including different ways in which models have been constructed, variable time periods over which benefits are calculated, and variability in estimates of deadweight and attribution (the latter are not always included as part of the SROI model). The SROI values derived for Axes 1 and 3 in the current study can thus be regarded as conservative values that have carefully considered and taken into account the following:

• Attribution of outcomes to the grant award
• Deadweight (what would have occurred without the programme grant award)
• Displacement (depending on the type of outcome)
• Annual drop-off in value (based on type of outcome)
• Careful selection of financial proxy values.

Table 6. Proportion of Axis 1 and 3 benefits attributed to ‘social innovation’ outcome categories.

Category of Social Innovation Outcome	Social Innovation Outcome Categories as a Proportion of Social Innovation Benefits (%)		Social Innovation Outcomes as a Proportion of Total Benefits (%)
	Axis 1	Axis 3	Axis 1	Axis 3
Individual	0.267	0.293	0.123	0.163
Operational	0.316	0.231	0.146	0.129
Relational	0.197	0.462	0.091	0.258
System	0.220	0.015	0.102	0.008
Total	100.00	100.00	0.461	0.558

identifies the proportion of benefits which can be attributed to the four categories of social innovation outcomes. A lower proportion of Axis 1 outcome benefits (46.1% of the total) can be attributed to social innovation outcomes compared to Axis 3 (55.8%). This is not surprising given the focus of Axis 1 funding on modernising agriculture and increasing the competitiveness of farm businesses, compared to Axis 3, which was focused on improving the quality of life in rural areas through improvement in service provision. The highest proportion of benefits under Axis 1 are generated through ‘operational’ outcomes (14.6% of total Axis 1 benefits but 31.6% of the total value of social innovation outcomes). These are defined as outcomes that alter the way individuals operate their businesses or are new ways of carrying out activities as a result of programme activities to enhance innovation (e.g., training, skills improvement, advice). It may also include investment in new technology as a result of receiving a mix of investment support, training and advice.

For Axis 3, the highest proportion of innovation outcome benefits are generated through ‘relational’ outcomes (25.8% of the value of overall Axis 3 outcomes, and 46.2% of social innovation value). These are activities focused on changing behaviour and attitudes, and supporting creation of linkages among groups, organisations, and communities. The proportion of social innovation benefits arising from outcomes associated with operational activities is less than for Axis 1 (at 12.9%, or 23.1% of the total social innovation outcome value) and are generated largely through improvements in tourism service provision, thus benefitting potentially large numbers of businesses and communities across the country. The value of individual social innovation outcomes is higher than for Axis 1 (16.3% of total Axis benefits), driven in part by the large number of micro-enterprises benefitting, and the improved well-being for large numbers of residents of rural communities able to access new or improved service provision.

A higher level of social innovation value would be expected from Axis 3 given its focus on cooperation, collaboration, and improving the quality of life in rural areas. A significant number of Axis 3 ‘social’ outcomes were not included as part of this analysis, however, as they did not fall within the definition of ‘social innovation’ as conceptualised in this paper. These included, for example: changes in public transport; level of local retail use; improvements in recreational infrastructure; reported increase in ‘the development of new ideas’; increased restoration of historic buildings, and, increased levels of training. Benefits flowing from these outcomes, while significant in the evaluation of Axis 3 impacts, either do not flow from ‘social innovation’ activities as defined in this study, or cannot be considered as ‘innovations’. For example, an increased use of public transport is not the result of a ‘social innovation’, and ‘development of new ideas’ alone do not generate benefits, it requires application through further action to create some defined outcome that can then be measured. In a similar manner, an ‘increased level of training’ does not automatically produce improvement, training has to be applied in some way in order to generate benefits. It is only where additional training can be identified as contributing to social innovation outcomes that it is included within the accounting framework.

5. Discussion

This paper has focused on the application of SROI as a tool for evaluating social innovation within European rural development programmes. The key question of interest is the extent to which SROI offers a means for improving the evaluation of social innovation. If innovation lies at the heart of development as previously suggested [19,23,38,53], then a more comprehensive approach to evaluation is required that can capture, and value, the multiple benefits arising from social innovation support measures. In this paper, we have taken a small step towards exploring the sources and outcomes of innovation. We have identified: causal pathways leading to the occurrence of innovation outcomes; how outcomes vary spatially, sectorally, and across time; the extent to which programme outcomes can be attributed to ‘social innovation’ activities; and the ‘value’ of those outcomes over time.

The review of the literature earlier in this paper revealed that the long-standing discussion over the effectiveness of top-down or bottom-up approaches to rural/regional development continue to occupy academics and policy makers [4,5,36]. Differing strands of the literature suggest the need for top-down centralized support for innovation [1,14,18], and for bottom-up processes that allow local capacity to develop in more flexible ways [54]. There is general agreement in the more recent literature, however, on the importance of the ‘meso’ level of activity—the area where policy and industry actors must meet and implement policy objectives through programmatic activity [1,14]. A strong role is also advocated for ‘enablers’ and ‘brokers’ at this level [38,54]; those who will build the networks and develop communications to overcome barriers to make the programmes and projects operate on the ground. The literature is clear that there is a role for government, and for the individual (whether a single person, firm, or other organisation), and to achieve ‘transformative’ innovation at the wider society level, the role of accepted norms, incentives, and capacity for change also become significant factors.

RDPs implemented under Pillar 2 of the CAP focus on improving both productivity of rural businesses and quality of life in rural areas through improved service delivery and community support. Examination of the RDPE programme documents and interviews with delivery personnel revealed a wide range of desired outcomes, some of which can be characterised as ‘socially innovative’ where the aim of a proposed action is to encourage stakeholders to adopt new ‘ways of doing’. The study described here suggests the efficacy of an approach to assessing social innovation based on identifying programme activities that contribute to four types of outcome: individual, operational, relational, and system, using an SROI model to measure the value of benefits created. Two additional areas of activity affecting social innovations were identified but not included in the analysis: outcomes arising from catalytic change, and those from changes in the wider context. The analysis has identified a number of issues that need to be addressed in order to develop a credible social innovation evaluation tool using SROI. These are discussed below. The SROI approach applied here adopted a broad definition of social innovation: an action was assumed to be innovative if it was new to a person, a group, or a place, and outcomes were identified as relating to the individual, organisational operation, relational, or the larger system in which individuals were located. The SROI model was able to value (monetise) a range of outcomes that flow from expenditure on activities identified as enhancing ‘social innovation’. The analysis demonstrated that outcomes do not always fit neatly into a specific category (for example, an ‘increase in confidence’ which leads to more collaborative working). In an SROI framework, such issues can be addressed through development of a clear theory of change model which clearly articulates the source of outcomes. However, a number of potential limitations remain which should be considered in future comparable studies employing the SROI method. Three pertinent issues that should be taken into account for future application of SROI in this area are identified below.

First of all, programme impact is assessed at the individual level, based on beneficiary perceptions of the level of each outcome. Sampling and aggregation issues are central concerns in relation to the validity of reported values and benefit-to-investment ratios when presenting regional or national estimates of programme impact. Indeed, given the limitations of sampling, it is not possible to generalise the conclusions to the wider population of ERDP beneficiaries. In order to achieve this more accurately, estimates of beneficiary numbers are required, and sampling must be designed to represent the full range and scale of those benefitting from measure support. Where potentially large numbers of indirect beneficiaries are present, such as those experiencing well-being benefits from access to improved services, more detailed analysis is required to determine level of use, rather than use of area population data as a measure of impact (as the CEMF currently does).

A second issue relates to the assessment of indirect effects. Two classes of social innovation outcome related to the wider spread of impacts were not comprehensively assessed in the study reported in this paper: Firstly, changes in relationships (for example, increased collaboration or cooperation); and secondly, the ‘catalytic’ effects of some projects across a wider community, area, or economic sector. In the first case, the SROI included several outcomes identified through the programme logic model focusing on delivery of enhanced collaboration or partnership-type working (at individual, organisation, and community levels). The scale of those partnerships or collaborative actions, and the numbers of beneficiaries (some of whom might lie outside of the beneficiaries identified through programme monitoring) were not assessed. In the second case, the SROI framework could not measure the additional ‘multiplier’ impacts from what might be called catalytic (or ‘game-changing’) projects, i.e., those that had impacts far beyond the expectations of the initial project investment. One example was investment in a floating jetty on Lake Windermere, which not only benefitted businesses in the immediate vicinity, but set in motion changes that affected the wider local economy, and the whole manner in which local politicians and officials perceived local economic development strategies. On a smaller scale, a £50,000 capital grant supporting investment in provision of a fishing pond had ramifications across a large area of southern Lancashire for people with disabilities, improving the availability of recreational activities as well as providing for different forms of social interaction. While the SROI could identify approximate numbers impacted, it was not possible to explore the complete level and nature of the indirect outcomes arising from ‘catalytic’ projects. A useful addition to the evaluative framework would be integration with some form of path analysis and multipliers capable of assessing the level of indirect benefits arising from changes in social relationships and wider impacts arising from a specific funded action.

A third issue relates to the nature of the data available for assessing beneficiary numbers within specific Measures and schemes. In this study, it was not possible to undertake full analysis of outcomes at RDPE Measure level due to the integration and/or division of Measure funding into different schemes (one example is the linking of funding from several Measures to provide training, advice, and grant support under integrated livestock schemes in the North-west and South-west regions. In most cases, funding beneficiaries were only aware of the scheme itself, not the way in which funding streams were combined in the background. This can create problems for policy personnel who need to report back evaluation results to central government or the EU at Measure level. A solution might include modifications to an SROI approach in terms of tracing causal pathways in order to attribute outcomes in situations where ‘entangled’ delivery makes it difficult to allocate outcome values to specific funding streams at any scale below Axis level.

A common issue in any evaluation is deciding on the time period over which outcomes should be measured [55]. Some outcomes are immediate, others may take longer (for example, the effects of knowledge acquisition, advice, and skills training), some might be temporally ‘fleeting’ while others continue to have an impact over longer time periods or build sequentially in combination with other changes. The timing of the evaluation itself can have implications for the capture of benefits that only tend to occur, or become fully realised, after a significant time lag. With its ability to assess impacts over defined periods (five years in the present study) combined with the ability to look forwards (forecast SROI), backwards (evaluative SROI) or combine the two, the SROI framework does provide an element of flexibility in this respect. Nevertheless, the ideal scenario would be to undertake a clear theory of change and forecast SROI at the very start of a programme, and collect ongoing distance travelled data to measure the level of change in outcomes and periodically update the SROI model using ‘live’ data as it progresses. Time lag effects would then be taken care of automatically, at the same time as a meaningful and temporally relevant monitoring and evaluation of programme activities was being implemented.

The analysis presented here demonstrates the capacity for the SROI method to capture a range of innovation outcomes. While the information provided by the model is useful in identifying the relative value of social innovation outcomes, it does need to be underpinned by an understanding of the policy delivery and wider socio-cultural contexts, which will influence programme effectiveness. Certain outcomes identified in this study such as improvements in business confidence, and improved capacity to resolve issues suggest the potential for enhanced resilience among individuals and organisations to withstand and adapt to change. The analysis can also provide insights into where and how a particular activity is contributing to innovation outcomes through capacity building [24,32], and provide indications of where broad policy objectives may be achieved. The extent to which social innovation contributes to broader rural development, however, is more difficult to ascertain [22,23], requiring additional information on the numbers of beneficiaries of social innovation outcomes within particular rural areas, and whether the scale of outcomes are sufficient to deliver sustainable and long-term change [19].

Application of SROI to the evaluation of social innovation offers a means to develop deeper understanding of the role of social innovation in the development of a rural area, or within a specific community, at the level of the individual, organisation or local system. The method can provide information on the origins or source of development, it can begin to tell us about the scale or extent of changes through measuring outcomes, and the extent of collaborative activity in relation to a programme measure. It can also provide information about legitimacy through improved understanding of how an outcome change influences stakeholder thinking and institutional structures [37] within a system. In this regard, we can identify some potential avenues for future research that will enable further progress towards a legitimate analytical and policy framework to help foster innovation in rural development, and, in turn, inclusivity. These encompass three inter-related research directions. First, to ensure the potential for improved understanding is incorporated into rural policy (Figure 1) requires stronger linkages back into the policy process. In turn, this first requires further research into how innovation outcomes and the pathways that underpin them vary by sector and across space and time. Second, the indicators utilised and developed in such studies the need to be legitimated [37] into a recognized framework in order to avoid the insights to be gained from understanding how social innovation contributes to rural development being underplayed in successive rural development policy cycles. Conceptual and empirical studies to further develop and pilot innovation indicators with large data sets would be especially useful. Addressing the sampling issues previously highlighted as part of this work should then enable derived findings to be generalised to the wider population. However, this approach should not deter the implementation of bespoke SROI approaches, such as the one described here; and the methods, advantages and limitations described in this paper should assist in honing successful tailored studies, and importantly help to encourage further application of SROI at programme level. This in itself will contribute greatly to the emergence of a robust set of tested and adaptable indicators.

6. Conclusions

SROI appears to offer one way forward to increasing understanding of the role of innovation in rural development, and the potential to improve rural policy evaluation in the post-2020 programming period. Our understanding of social innovation, which places a significant role on the individual (person, firm, or organisation), is cognisant of the main principles drawn from past experience and the theoretical developments described above. Where our approach differs is in identifying different forms of social innovation, suggesting, for example, that relational or even catalytic outcomes, stem from individual action, and can occur at the local as well as wider system levels. Whether or not those outcomes occur, and at what scale and magnitude is often (but not always) dependent on getting the right type and level of support at local, regional, or wider socio-ecological system levels. Support may be needed in relation to finance (not necessarily grants), skills development, or advice, and may or may not come from government. What we have tried to show in this paper is that when evaluating outcomes from innovative action, understanding the negative as well as the positive, is essential to developing improved packages of support that policy actors can apply to different forms of social innovation.

SROI offers a means to improve understanding of the types of outcomes arising from social innovation, and who is benefitting. It can also highlight the potential for failure, provide indications of where support is most effective (through measures of attribution and deadweight), and provide indications of value added to enable comparison of alternative strategies over variable time periods. Design of strategic policy to enhance innovative rural development is essential to achieving sustainable growth, but so are evaluation techniques that provide sufficient information to illuminate the outcomes of innovative action, thereby helping policy actors to recognise, and nurture, the seeds of change.