Ambiguity in the Attribution of Social Impact: A Study of the Difficulties of Calculating Filter Coefficients in the SROI Method

Abstract

In order to analyse, manage, and compare social projects we need, among other things, to be able to measure their impact. One of the methodologies currently used to measure and manage social impact is Social Return on Investment (SROI). However, not all the results calculated by the SROI method are directly attributable to the project, and, therefore, to determine the real impact it is necessary to filter out the changes that the project has not produced. Filter coefficients perform this function. However, the theoretical logic on which the chain is constructed that converts the outputs into impacts is ambiguous. In this study, we will analyse twenty-five real cases where SROI was used to measure social projects. We will identify the difficulties of isolating and measuring impacts by performing a comparative study of the procedures that entities develop to calculate the filters. This allows us to calculate the impacts from the outputs. We will then propose the improvements needed to overcome these shortcomings.

1. Introduction

In recent years, Third Sector of Social Action organisations are facing ever-increasing demands for information about the social and environmental impact they generate. The various groups affected by the activity of these organisations, including investors, consumers, funders, employees and the public administration, are aware that in order to assess entities in the Third Sector of Social Action (TSSA from now on), they need information that provides a holistic picture of their activity that is not limited to financial indicators. These interested parties have started to put pressure on the entities to disclose information about their social and environmental performance [1,2].

The various interest groups of TSSA organisations can exercise this control in a formal and direct way, through evaluation requirements included in contracts, or indirectly, through rules that individuals and organisations must follow in order to gain approval and the verification that their activity meets the established requirements [3]. Therefore, TSSA organisations are conditioned by the influence of the logics and rules of other groups [4].

These demands have led to an increase in sustainability reports by TSSA organisations. Therefore, does this increase mean an improvement in the transparency of TSSA entities? Previous research has concluded that the content and depth of sustainability reports varies significantly between entities [5]. There are even studies that reflect a lack of quality in these reports [6,7,8,9].

There is a need for a standardised methodology for monitoring and evaluation that integrates, in plain and easily understood language, the internal information needs for learning and development in the organisation, with the information demanded by external interest groups to account for their performance [10,11,12].

One of the external groups that applies the most pressure is the funders [13], meaning that in order to obtain the necessary resources to develop their activity, TSSA organisations need to communicate and account for what they do and how they do it [14,15]. It is not enough to show information that is favourable to the organisation or project; the institutional pressures exerted on the third sector mean that the demand for transparency is very high [15,16,17].

In parallel, and at an internal level, these entities, in addition to meeting the economic, social and environmental objectives required of them, must prepare, process and communicate a large quantity of information that meets specific requirements, both to practise effective management, and to control their results and possible deviations [2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18].

This explains the relevance of the dependence of TSSA entities on their main financial backers, and how this dependence entails providing information to backers on their activities and results in a particular format. Increasingly, funders require formal methodologies for measuring social impact that generate objective metrics with an empirical basis, with the aim of improving the standardisation, verifiability and accountability of these metrics [19,20,21,22].

However, the measurement of social impact is an extensively questioned practice, owing to, among other reasons, the ambiguity that methodologies for measuring impact suffer from. A frequent object of criticism is the lack of logic of the underlying assumption within most methodologies of a causal chain or logic model through which the activities of the entity are translated into results, changes and impacts for each of the stakeholders [2,23]. In contrast, practitioners with experience in implementing such formal methodologies often stress the causal ambiguity of this chain; they contend that impacts are difficult to understand with precision, much less calculate [2,24,25].

One of the methodologies currently used to measure and manage social impact is the Social Return on Investment method, or SROI, which, based on traditional cost-benefit analysis, quantifies social value through indicators associated with the results achieved. SROI is a process of understanding, measurement, management and communication of the social, environmental and economic values generated by an organisation. Its purpose is to examine, qualitatively and quantitatively, the project’s process of generating social value.

Where impact is caused by a range of factors, it is difficult to determine the amount of social value that can be attributed to the activity of the project that is being measured. To cope with this and determine the real impact of the project, filter coefficients are introduced. However, their estimation has some shortcomings [4,26,27]. Even though the filters are perfectly defined within the SROI methodology, when we fix them at a certain percentage, we cannot find a method or reference that enables these filters to be calculated with precision and ease [28,29,30]. We can see that these coefficients have a different accuracy depending on the person who conducts the study of measurement, their interests and ability to access the information. Since the function of these filter indexes is the correction of the result to reach the impact, their effect on the impact is key. This is why improving the construction of the coefficients that filter the outcomes to calculate the impact is essential for accuracy in the measurement of social value.

In this study we investigate the following research questions: what are the difficulties in calculating the filter coefficients of the SROI and how to develop the necessary improvements to overcome these difficulties. We explore these questions through a comparative case of real SROI analysis of organisations from different sectors and geographical areas.

This article critiques the legitimacy of the SROI method to measure the social impact created by organisations, calling into question the underlying theoretical logic between outputs and impacts. While outputs can be managed by organisations, exogenous factors that escape their control influence results and impacts. Therefore, the attribution of results to a particular actor can be very difficult [2,24,31]. Through the analysis of the reports of SROI measurement of twenty-three social projects, and the identification of the gaps in the underlying theoretical logic between outputs and impacts, we will study the calculation of the filter coefficients of social impact and identify their shortcomings.

We will consider an impact to be those results that can be attributed directly to the organisation or project in question. To arrive at this concept, it is necessary to filter the changes, taking away that which has not been produced by this project, so that the organisation only claims what it has created itself as a social impact. To do this, we must introduce these elements: deadweight, displacement, attribution and drop-off [27] (p. 54).

Deadweight deals with quantifying whether the project has really influenced society or whether, on the contrary, the result we see is a product of other external interventions. In other words, this coefficient responds to the question of the extent to which the result would have been achieved if the project being evaluated had not taken place [27] (p. 56). Displacement studies what percentage of the change achieved has displaced other changes. If the project consisted in helping people at risk of social exclusion to get a job, displacement would seek to quantify whether these jobs were obtained to the detriment of another worker. If so, the quantification of this circumstance has to be removed, which will diminish the impact of our initiative [27] (p. 57). Attribution measures the percentage of changes that is not attributable to carrying out the project. It indicates the need to deduct, in the calculation of the impact, the percentage of the result that has been caused by the contribution of other organisations or people [27] (p. 59). Drop-off is the decline in the results of a project over time, which could be compared to the depreciation on a fixed asset due to use. In other words, this coefficient analyses how long the results have lasted [27] (p. 61). It is logical to think that in the years after completing the project, the quantity of the result obtained will diminish, and therefore the attribution to this project will diminish [18].

The next section outlines the methodology that was followed throughout the study. The third section is a comparative report of the cases studied during the research. The fourth and last section identifies the deficiencies detected in the study. This is the central body of our research because it allows us, through the above analysis, to draw conclusions on the results. We will warn of the shortcomings that must be addressed in order to eliminate the ambiguity of the causal chain of logic between outputs and impacts, and also provide a measurement that contains the real value that a project generates, integrating both social and economic impacts to give a result that can be standardised, verifiable, accountable, comparable and understandable to others.

2. Materials and Methods

An ISI web of knowledge and Scopus online literature search was conducted for the years 2007–2016 for articles and conference papers on SROI analysis. Key words used included: “SROI”, “Social return on investment” in the subject area “Social Sciences”. We searched in all fields of the articles using the above-mentioned key words separately on the date of 31st October 2016. We obtained the following results: 23 results for “SROI”; 36 results for “Social Return on Investment”.

In addition, we conducted a search by entering the above terms in “Google Scholar” and the documents, reports, book chapters, articles and practical guides were also checked, and relevant documents included in the review. We decided to include non-academic material in the study material, as SROI is a methodology that is debated more frequently in professional environments than in academic ones. Where relevant to the discussion on suitable theories, articles and papers based on the methodological approach of literature review [32,33,34,35] were also added, given that the empirical studies and surveys are still limited, especially in the scientific literature.

The most relevant documents were examined and on that basis we chose the most important ones for the study. In particular, with reference to the selection process, we verified the presence of a consideration of the filter coefficients of the SROI method and its importance for the objective of the analysis. In total, twenty-three case studies from this set have been analysed to identify possible irregularities in the calculation of the filter coefficients:

• A program aimed at combating loneliness and isolation of the elderly in Scotland.
• Conservation of the natural environment surrounding the Greenlink area in Scotland.
• A project providing access to water to families who do not have access to other water sources in Stockholm.
• Promotion of healthy lifestyle habits for overweight people in Bristol.
• The Solvatten project provides household water treatment in developing countries (Kenya).
• Benefits of interaction with the rural environment for people with problems of socialisation or behaviour in England.
• Benefits of Tai Chi, in the forest, in patients of the Hospital Firth Valley in Scotland.
• A rehabilitation program based on an intensive program of activities for people with bone marrow injury in Australia.
• A programme for the development of adolescent women in areas where unemployment, poverty and lack of education are rife in Northern Ireland.
• A program that informs local women about health issues in an informal way in Northern Ireland.
• A project to provide a minimum income for rural families living in semi-arid areas in Ghana.
• A project to support, through job creation, people with mental health problems in Glasgow.
• Helping parents who care for children with disabilities develop their skills, endurance, and confidence in the UK.
• Make organically grown food accessible to the citizens of Sydney.
• Use art as a catalyst to bring about positive and lasting change in people’s lives in North Ayrshire.
• A family intervention project that works with families with children and in situations of high vulnerability in Northamptonshire (England).
• Evaluation of the social impact of a “social initiative protected employment centre”.
• A project to create a cafeteria for 30 elderly or disabled persons who are provided with transportation to the cafeteria in England.
• The Oxford Castle Renovation Project.
• Respond to complaints about antisocial behaviour in the area caused by the increasing volume of unsupervised and unauthorised bicycles in Scotland.
• Restoration of gardens, with the participation of volunteers from the municipality and students in Scotland.
• Use of a natural space for outdoor activities with the mentally ill in Scotland.
• Evaluation of the social return of investment in the creation of the Ecoclub school in Scotland.

The cases belong to different sectors and geographical areas to avoid the inaccuracies that the sample selection bias could generate in the conclusions. They are briefly described in Appendix A. Through a comparative study of the procedures followed to calculate the coefficients in each one of the projects, we have identified the shortcomings of SROI methodology in this area. The study ends with a proposal for improving the construction of these filter coefficients that make up for the shortcomings identified and respond to the needs for accuracy in the measurement.

3. Results

Before the SROI calculation can be finalised a decision has to be made as to how long the changes produced by the analysed project will last. In an SROI analysis the length of time changes endure is considered so that their future value can be assessed. The question to be answered is “if the activity stopped tomorrow, how much of the value would still be there?” This is dealt with by assuming that the value will reduce or drop off a percentage each year. As well as considering how long changes will last, it is necessary to take account of other factors that may also have impacts. The recorded change might have happened regardless of the activity, something else may have made a contribution to it or the activity may have displaced changes taking place elsewhere. In considering the influence these factors exert, a realistic approach should be adopted. The aim is to be pragmatic about the benefits actually provided by the project and to recognise that the value it creates is influenced by other factors. The SROI methodology does this by reducing the value calculated for each outcome on a percentage basis, applying several filter coefficients: deadweight, attribution and displacement.

A reduction for deadweight reflects the fact that a proportion of an outcome might have happened without any intervention. Attribution recognises that external factors, or the contribution of others, may have a part to play in the changes that are identified and displacement applies when one outcome is achieved but at the expense of another outcome, or another stakeholder is adversely affected.

This section presents an analysis that compares each of the filter coefficients of the social projects that, when analysed, produced the most significant results. There are four tables in total: deadweight (Table 1), displacement (Table 2), attribution (Table 3) and drop-off (Table 4). Our objective is to make the study of the various ways of calculating these coefficients more visual, and to allow for a more exhaustive comparison.

Table 1. Deadweight.

Case	Calculation of the deadweight
1	The method for choosing the appropriate percentage was carried out through interviews. The project analysed the percentages of each person and then they made an average to obtain a more robust figure. The family members group did not recognise the existence of deadweight; however, the evaluators felt the need to incorporate 5% to reflect a small sample size.
2	The method used in the calculation was based on a survey of households in Scotland. The results show that the volunteering rate in general in North Lanarkshire is 13.7%. Therefore, it is estimated that 13.7% of the Greenlink volunteers would be volunteers whether the Greenlink was there or not. The rest of the data is obtained through statistical data.
3	Deadweight is divided into three areas of results, but the estimates made are not specified.
4	A minimum deadweight percentage for all the results is established, which ranges from 5% to 10%.
5	Deadweight is not taken into account due to the fact that all the results are caused solely by the effect of the product.
6	The lack of absolute certainty meant that a deadweight figure of 5% was assigned to all results.
7	The percentages are obtained through estimates for each result, but there are no details on the procedure.
8	Percentages are structured on a scale that goes from lowest to highest in terms of the influence of the project on the result obtained (0% where the result would not have been achieved had the activity not been carried out; 25% where the result would have been achieved but to a limited degree; 50% where the result would have occurred, in part, in any case; 75% where the result would have occurred, for the most part, in any case; and 100% where the result would have occurred anyway).
9	For the young women group, 5% was set as the value of the deadweight. In general, it was quantified at 5%, because of a previous example in which the hospitalisation of one person was prevented. For the volunteers/employees group the deadweight figure was fixed as non-existent (0%), since they would not have developed certain skills if they had not worked on the project. For the funding entities, deadweight was set at 0%.
10	For the group formed by the elderly a deadweight value of 5% was established. In the case of the Champion, i.e., the volunteer in the Day Centre, 5% was set. For the CHIW and Ardoyne Healthy Living, deadweight was estimated as non-existent. It was understood that without the Champions Programme, the results would not have been produced. Finally, for the families of the elderly group, deadweight was set at 5%.
11	In most cases, the deadweight values were calculated on the basis of data collected in the comparison group. The percentages to be applied ranged from 0% to 10%.
12	The deadweight percentages applied in this case ranged from 0% (used, for example, in the case of the result “increase in the efficiency of the therapy” for the interest group formed by support workers and occupational therapists) to 50% (set for the result “increased workload” of the interest group “support workers”).
13	There are no details on the manner in which they have come to those estimates.
14	Calculation is standardised (0% = The result would not have been achieved without the intervention; 25% = The result would have been achieved but to a very limited degree; 50% = The result would have occurred, partially, in any case; 75% = The result would have occurred, for the most part, in any case; 100% = The result would have occurred anyway).
15	Of the eight results analysed, in five cases statistical data is used to evaluate deadweight. In two cases, information taken from surveys of participants is used. In one case, the estimate is not justified.
16	Although the consultant’s opinion is that without this intervention the results evaluated would not have occurred, the evaluation attributes 15% in one of the results (the most emotional, the most difficult to quantify) and 0% in the other two results.
17	The evaluators quantify it mostly as 0% because without this project, the employability of the severely disabled is practically non-existent. Only in the case of disabled people without special problems of employability do they consider deadweight to be the percentage of people with a job in the general population (statistic).
18	The volunteers identified that the luncheon club involved more physical exercise than they might have otherwise sought. The average was around 45% more. As such, if the benchmark is 100%, since all of them would have done some other exercise anyway, the increase is therefore 145%. The estimate of deadweight is 100%/145% or 70%. For the outcome of ‘residents having nutritious meals’, the local government organisation could have obtained another provider to deliver it, so the deadweight is 100%.
19	They calculate the deadweight as the percentage of the result that would have occurred anyway, but there is no indication of how they calculated it.
20	It is calculated by reasoning and logical estimates, without any systematics.
21	For most of the results, statistical data is used. In other cases, it can be concluded logically that there is no deadweight. In certain other cases, a value is estimated in the absence of statistics.
22	It is calculated by means of estimates.
23	It is considered non-existent.

Table 2. Displacement.

Case	Calculation of displacement
1	It is not considered.
2	The only result that always involves some displacement is getting a job. The job density in Scotland is 0.82, meaning that for each applicant there are 0.82 jobs, which is equivalent to 18% displacement. That is to say, the possibility of getting a job is now 18% lower.
3	No displacement has been identified in this study.
4	The effects of displacement are considered to be minimal and range from 0% to 10%, depending on the nature of the result.
5	It is not considered that there is displacement in this case.
6	Displacement was not considered by the vast majority of the project’s stakeholders. However, one of the volunteers suggested that they could have opted for helping elsewhere if they had not decided to spend their time on the project, and therefore a displacement rate of 20% has been assigned to the project.
7	A displacement rate of 75% is applied to the Tai Chi intervention.
8	Standardisation is as follows: 0% if no displacement; 25% if it displaces another result in a limited way; 50% if the result displaces another result partially; 75% if the result displaces another result significantly; and, finally, 100% displacement if the result displaces another result completely.
9	The same values are applied as for deadweight: young women 5%, parents and caregivers 5%, volunteers/employees 0% and funding entities 0%.
10	In this programme only 5% was applied, in the case of the Champions group.
11	In this case, due to the various contradictions in the information collected for the different groups, a decision was taken to add the speculative effect as 50% of displacement.
12	In the case of this Plan, no displacements of other services or activities were identified.
13	There are no details on the manner in which they have come to those estimates.
14	The calculation is standardised 0% = The result has not displaced any other result; 25% = The result has displaced another result, in a limited way; 50% = The result has displaced another result partially; 75% = The result has displaced another result to a significant degree; 100% = The result has displaced another result completely.
15	It is not considered.
16	It is considered non-existent, justifiably.
17	It is considered non-existent, justifiably.
18	Displacement has not been considered.
19	It is not considered, without being justified.
20	Two possible effects are considered; one is discarded after an interview with the stakeholder, and the second is seen as a negative effect on the results (it is counted).
21	The percentages of displacement reflect the time that some stakeholders stopped devoting to other possible activities because they were carrying out this initiative. The value is estimated in one case. In another, it is based on surveys.
22	The extra work caused to third parties is considered to be displacement. The figure is estimated.
23	The evaluators consider it non-existent, but the reason provided is doubtful. It is argued that since this is an analysis of a project that is taking place in the present, there is no displacement.

Table 3. Attribution.

Case	Calculation of attribution
1	The attribution rates assigned were: 19% for participants in the programme; 5% for family members; 15% for Housing Associations, and 18% for the NHS. The calculations are not specified.
2	Based on their experience, Greenlink establishes a 33% attribution rate for volunteers, since 33% of the volunteers had arrived through references from other agencies.
3	A 100% attribution coefficient is used.
4	It is calculated by reasoning and logical estimates, but the procedure is not specified.
5	It is obtained through statistics. An attribution rate of 30% is considered, since, as noted in the interviews, only 60% of patients listen to the doctor’s recommendations, and 50% of these do not follow the recommendations in their homes.
6	It is calculated by reasoning and logical estimates, but the procedure is not specified.
7	The participants were asked to assign a percentage of attribution to any change in physical or mental health that could have occurred during the course of the intervention. The average percentage of these results was 62%.
8	The calculation is standardised (0% attribution rate if no external input; if there are other organisms that had some minor role in the results 25% would be established; if these organisms had a more important role in getting the results 50% would be assigned; in the event that the role of the organisms external to the project was important it would be a 75% attribution rate; and if there was full external input 100% attribution rate would be used).
9	We applied the same values as for the deadweight: young women 5%, parents and caregivers 5%, volunteers/employees 0%. Finally, for the interest group formed by the entities that participated in funding, a figure of 10% was established for attribution calculations in the case of theatre activities, since these could have been carried out by another company.
10	In the case of the Champion group, i.e., the volunteer in the Day Centre, 5% was set. 5% was also set for the group of families of the elderly.
11	Based on the responses obtained in forums of local groups, and interviews with those who were key to the project, the overall attribution rates were set between 0% and 20%. In the individual case of a participating community that already owned an area of forest reserve, it was understood that it was already a leader in nature conservation. An attribution rate of 12% was applied, a result of rounding off the maximum and minimum values considered.
12	On many occasions the changes are not due to an activity; they are the result of more than one service or set of people working together. The attribution contemplates this overvaluation, and it was fixed as a percentage (varying between 0% and 30%), which was subsequently deducted from the total impact. In the case of the customers it was 30%.
13	The report analyses the four filter coefficients for each of the results. However, it does not specify the way in which it has estimated these percentages.
14	The calculation is standardised (0% = The result is wholly the result of the intervention and no other activity, programme or organisation has contributed; 25% = Other people or organisations have played a minor role in generating the results; 50% = Other people or organisations have played an important role in generating the results; 75% = Other people or organisations have played a significant role in generating the results; 100% = The result is entirely the result of other people or organisations).
15	In the calculation they attribute all the results to themselves, and obtain a SROI greater than 8. Then they make the assumption considering themselves responsible for the results in 75% of cases, and the SROI is still greater than 6. Then they make the same assumption, but with 50%, and the SROI is still greater than 3. Conclusion: it is still a high SROI.
16	Attribution is calculated by asking everyone involved in the project whether they believe that other organisations have also contributed to the result. The values range from 0% to 25%, although the consultant’s comments emphasise that, despite the fact that there were other organisations involved, no changes were occurring until this intervention began.
17	In general, they consider a 100% attribution rate of the results, because if there were no project, the chances of achieving the result are practically non-existent.
18	Based on the results in the questionnaire it was possible to estimate that 35% of the outcome was the result of the contributions of others.
19	The attribution rate for each of the results is different, but they do not explain how they arrive at those figures.
20	It is calculated by reasoning and logical estimates, without any systematics.
21	In most cases, attribution has been calculated by direct observation of the progress of the project, by logical causes and also by interviews/questionnaires.
22	In a more or less arbitrary way. In some cases, attribution is based on questionnaires prior to the activity, but in others, it is only based on estimates.
23	Based on estimates and questionnaires to stakeholders.

Table 4. Drop-off.

Case	Calculation of drop-off
1	A percentage drop of 33% is assumed for this SROI analysis, based on the experience of the Arts Impact team in the delivery of the programme for older people.
2	In terms of the Greenlink, we have no historical data on the extent to which the result reduces with time, which is why for most of the results (which last more than a year), a 15% drop-off rate has been used as the standard percentage.
3	The percentage is calculated by dividing the value created by the increased school attendance by the annual value of 2.5 of all the results during the first two and a half years.
4	SROI limits the duration of all results to a maximum of three years, and estimates a fall of up to 50% for many results.
5	The beneficiaries experience results that will continue during the lifespan of the product (which coincides with the period of analysis), and for this reason drop-off is considered non-existent.
6	Drop-off is estimated at 0% except for the results for young people and adults trained as users of the service, for which a drop-off rate of 20% is estimated.
7	It was considered that the impact of the intervention would not last more than six months after the pilot project’s date of completion. After this time, without continuous learning sessions, it would probably decrease, and the future benefits could be attributed to other factors. A drop-off rate of 50% and 70% is assigned.
8	The calculation is standardised: the result lasts for the entire period of time it is allocated (0%); the result falls by 25% yearly from the second year (25%); the result falls by 50% yearly from the second year (50%); the result falls by 75% yearly from the second year (75%); and the result falls completely at the end of the time period (100%).
9	Drop-off was not applied in this case. SROI was only measured for the year in which the Programme was carried out. The report stressed the difficulty of measuring the long-term effects. An adjustment was made of some of the results obtained for the calculation of the SROI, with the aim of bringing it closer to reality. To do this, a sensitivity analysis was performed.
10	For the group formed by the elderly, a variable drop-off rate was established, with a generic value of 10%, 30% for the care homes, since there may be a decrease in the impact of the Programme after the first year, and 50% for a smaller number of home-care visits. For the families of the elderly group, a drop-off rate of 15% was determined, to consider the possibility that the activity could decrease, due to its ceasing to be a novelty. For the rest 0% was valued.
11	This percentage of deduction is applied when the changes last longer than the reference period and when it determines the percentage of annual loss in the benefits generated by the project. The drop-off values applied, which were generally low, were 0%, 10% or 20%.
12	A percentage drop of a 33% was considered for outcomes with a physical component (skills to cope with situations, life skills) and 66% for results with a predominantly mental component (confidence, self-esteem, personal skills and relationships).
13	Not specified.
14	The calculation is standardised (0% = the result lasts for the entire period of time it is allocated; 25% = The result falls by 25% from the second year; 50% = The result falls by 50% from the second year; 75% = The result falls by 75% from the second year; 100% = The result disappears completely from the second year).
15	In the majority of the results, drop-off is calculated by extrapolations based on some data resulting from the observation of the project. In two cases, drop-off is considered to be 0%, because during the programme there was no reduction in the obtaining of the result (a dubious interpretation of the parameter). In the end, in the sensitivity analysis they calculate that the drop-off value would have to be above 90% in order for the SROI to approach the value of 1, which is very unlikely.
16	They calculate drop-off based on the opinion of those involved as to whether, once the intervention is finished, the results obtained will be maintained, or whether the situation will worsen.
17	Drop-off is not calculated. This is a type of project that, if the intervention stops, the drop-off rate would be 100%.
18	10% is an estimation of the likelihood that residents will use the knowledge they gain less as time goes on, as they forget the sessions.
19	Neither displacement nor drop-off is considered, without explaining the reasons.
20	It is considered that the infrastructure created will remain in a good condition for about five years. An estimate is made (with logical reasoning, without systematics) of how much each of the results will decrease each year, in the next five years, if the associated activities cease to be carried out.
21	The skills developed by the project have a duration of two or three years (estimate) and the annual drop-off depends on the assumption of whether they will be used again or not (estimates/assumptions). It is considered that the infrastructure will be maintained for five years with an annual decline of 30% (estimate).
22	The skills developed by the project have a duration of five years (estimate) and the annual drop-off depends on the assumption of whether they will be used again or not (estimates/assumptions).
23	The knowledge and behaviour developed by the project have a duration of five years, and evenly the evaluators distribute the decline in the five years until its expiry.

4. Discussion

The analysis allows us to observe that the main shortcomings of the filter coefficients are related to the heterogeneity in the estimation. Even though all the cases studied share the same conception of these filters, the way in which the percentage is obtained in each case is very different, as summarised in Table 5.

Table 5. Filters coefficients comparison (I: Collection of information C: Calculation of filters).

	Deadweight	Displacement	Attribution	Detriment
1.	I: Stakeholder feedback C: Argument and justification.	Not considered	I: Stakeholder feedback C: Percentages based on stakeholder’s judgements.	No research data available. Percentage based on the past experience of the programme team in similar cases
2.	I + C: Benchmark indicators (public statistics)+argument and justification	I: Stakeholder feedback C: Argument and justification	I: Stakeholder feedback+SROI team experience C: Argument and justification	No historical data. Used 15% as a standard percentage
3.	I: Arbitrary estimation C: Neither argument nor justification	Not considered	I: Stakeholder feedback	Benchmark indicator Neither argument nor justification
4.	I: Research evidence+stakeholder feedback C: Neither argument nor justification	Neither argument nor justification	I: Stakeholder feedback C: Argument not justification	I: Research evidence+stakeholder feedback C: Argument not justification
5.	I+C: Control group	I+C: Control group	I+C: Control group	No drop-off. No argument
6.	I: Stakeholder feedback. No methodologies specified C: Neither argument nor justification	I: Stakeholder feedback. No methodologies specified C: Neither argument nor justification	I: Stakeholder feedback. No methodologies specified C: Neither argument nor justification	I: NA C: NA
7.	I: Arbitrary estimation C: Neither argument nor justification	I: NA C: Argument not justification	I: Stakeholder feedback C: Argument and justification	I: Arbitrary estimation C: Neither argument nor justification
8.	I: Stakeholder feedback C: Structured on a five standardised intervals scale Argument+justification	I: Stakeholder feedback C: Structured on a five standardised intervals scale Argument+justification	I: Stakeholder feedback C: Structured on a five standardised intervals scale Argument+justification	I: Stakeholder feedback C: Structured on a five standardised intervals scale Argument+justification
9.	I: Research evidence+stakeholder feedback C: Neither argument nor justification	I: Research evidence+stakeholder feedback C: Neither argument nor justification	I: Research evidence+stakeholder feedback C: Neither argument nor justification	No drop-off applied.
10.	I: Research evidence+stakeholder feedback C: Neither argument nor justification	I: Research evidence+stakeholder feedback C: Neither argument nor justification	I: Research evidence+stakeholder feedback C: Neither argument nor justification	I: Research evidence+stakeholder feedback C: Neither argument nor justification
11.	I+C: Control group	I: Stakeholders and key informant feedback C: Neither argument nor justification	I: Stakeholders and key informant feedback C: Neither argument nor justification	I: Stakeholders and key informant feedback+research information
12.	I: Stakeholder feedback. Not representative sample. C: Neither argument nor justification	Not considered	I: Stakeholder feedback. Not representative sample. C: Neither argument nor justification	I: Arbitrary estimation C: Neither argument nor justification
13.	I: Stakeholder feedback. C: Neither argument nor justification	I: Stakeholder feedback. C: Neither argument nor justification	I: Stakeholder feedback. C: Neither argument nor justification	I: Stakeholder feedback. C: Neither argument nor justification One percentage for every outcome.
14.	I: Stakeholder feedback C: Structured on a five standardised intervals scale. Argument+justification	Not considered	Not considered	No drop off applied
15.	I+C: Stakeholder feedback+benchmark indicators (public statistics+research evidence)	Not considered	No attribution	I: Research evidence C: Argument not justification
16.	I: Stakeholder feedback+research evidence. C: Argument not justification	Not considered	I: Stakeholder feedback+research evidence. C: Argument not justification	I: Stakeholder feedback C: Argument not justification
17.	I: Stakeholder feedback+research evidence+benchmark indicators (public statistics) C: Neither argument nor justification	Not considered	I: Arbitrary estimation C: Neither argument nor justification	Not considered
18.	I+C: Neither argument nor justification	I+C: Neither argument nor justification	I+C: Neither argument nor justification	I+C: Neither argument nor justification
19.	I+C: Neither argument nor justification	Not considered	I+C: Neither argument nor justification	Not considered
20.	I: Stakeholder feedback C: Argument+justification	I: Stakeholder feedback C: Neither argument nor justification	I: Stakeholder feedback C: Argument+justification	I: Stakeholder feedback C: Argument not justification
21.	I: Stakeholder feedback+research evidence+benchmark indicators (public statistics) C: Argument+justification	I: Stakeholder feedback C: Argument+justification	I: Stakeholder feedback+research evidence C: Argument+justification	I: Stakeholder feedback C: Argument not justification
22.	I: Stakeholder feedback C: Argument not justification	I: Stakeholder feedback C: Argument not justification	I: Stakeholder feedback C: Argument not justification	I: Research evidence C: Argument not justification
23.	I: Stakeholder feedback C: Neither argument nor justification	I: Stakeholder feedback C: Neither argument nor justification	I: Stakeholder feedback C: Neither argument nor justification	I: Arbitrary estimation C: Neither argument nor justification

One of the obstacles we encountered is that the influence of the subjectivity of the people responsible for the identification and subsequent quantification of information is very high. The estimates are approximations, which calls into question the accuracy of the evaluation of impact. Accuracy in measurement is one of the aims of the social impact evaluations. However, the estimates of the filter coefficients involve a wide margin of error and can produce erroneous interpretations and uses of the ratios.

In most of the cases considered in this analysis, the information that is used to make these estimates comes from interviews with the project’s interest groups. In these interviews, which normally use questionnaires, the participants report on an individual basis, in accordance with their personal assessment. Therefore, the information that is incorporated into the measurement is fundamentally based on value judgements and not on objective data.

The calculation of the coefficients also supports various possibilities. Some cases associate, to each result for each interest group, a percentage of deadweight; other cases associate a percentage for all the results of each group, and others make a general estimate of the whole project. However, the majority of the cases arrive at these percentages through logical deduction or reasoning, which generates an added difficulty, since this reasoning does not always follow the same pattern.

If the quantity of the filter coefficients is high, it is assumed that the project is not of great benefit on its own, which is why, in the majority of cases, analysts set small percentages. Several of the cases studied allocate a deadweight figure (Table 1 and Table 5) of under 10%; one of them (case 5) even considers that there is no deadweight for any of its results. This means that the project recognises that without its activity, none of the results that participants have experienced would have been achieved. On the other hand, given the absence of certainty, some projects try to avoid this problem and establish a representative percentage. Cases 6 and 11, for example, use this methodology. Some projects consider that the changes observed would not have been possible if the project had not taken place, but assign a 5% deadweight figure to avoid an overestimation of the impact.

Other cases follow a different methodology to perform these calculations and establish references to comparison groups or benchmarks. For example, case 2 bases its estimates on official surveys. In this case, the calculations offer greater objectivity, since the same method could be applied for similar programs. This would make such projects directly comparable with each other.

In the analysis of deadweight there is another estimation system that consists of calculating an average of the information obtained in individual interviews with the interest groups (case 15 as example). That is to say, the individual results are added up and then divided by the number of participants. This system is based on the assumption that the groups that participate in the same programme tend to have similar personality profiles, which is why, to avoid errors of subjectivity, an average of all the information obtained is taken.

Another important element in the calculation of impact is displacement (Table 2 and Table 5). Again, the main shortcoming that this coefficient presents is that it is based on estimates. Most of the projects do not take into account the displacement that their activity can generate. This claim is based on the idea that from among the twenty-three cases analysed, only twelve of them consider a certain percentage of this indicator, while the remaining eleven do not identify any type of displacement.

The fact that this coefficient does not exist in most of the projects is a limitation to homogenizing the calculation of impact. Of the cases that contemplate the existence of displacement, a small portion of them consider its effects to be minimal, and use a percentage of below 20%. Cases 2 and 4 go over a 50% displacement rate in some of their results; however, their estimate is based on the discussions held with participating agents. Considering whether a project displaces its results to other activities is not very complex; the problem lies in what percentage it is best to assign to this displacement.

Thirdly, we analysed attribution (Table 3 and Table 5). All the cases studied allocated a percentage for each of the results, and estimates in the twenty-three cases were based on the declarations of the interest groups.

The last correction factor is drop-off (Table 4 and Table 5), although some projects do not regard it as such. The drop-off indicator is used to record whether the result obtained is maintained, or not, over time, and is only calculated for results that last for more than a year. In the study we observe a number of different procedures to calculate it. One of them is to deduct a fixed percentage each year after the completion of the project. Another procedure used is to arrange an interview with the interest groups some time after the completion of the project. However, this system can imply a self-selection bias because the people who will be willing to be interviewed are likely to be the people who are the most satisfied with the results of the project.

Within the cases analysed, we would like to highlight, for their originality, cases 8 and 14. Both cases unify their assessment criterion of the filter coefficients, dividing each result into five categories. In this way they get the interest groups to identify more clearly which percentage of deadweight, displacement, attribution or drop-off is more in line with the results that they perceive.

5. Conclusions

In our analysis we have highlighted some of the difficulties in standardising calculations of filter coefficients, since the social benefits generated by projects are primarily intangible, and difficult to estimate. The main objective of this study has been to reflect the differences in unifying criteria, the shortcomings of the methodology and the difficulty in achieving more consistent estimates. If we could obtain more homogeneous filter coefficients, it would be possible to compare the return on investment of several social projects and know which of them generates greater value.

Along these lines, we perceive clearly that an effort is required to homogenise the calculation of filter coefficients in order to: (1) Minimise, as far as possible, the degree of subjectivity that they support; (2) reinforce their quantitative character against other qualitative interpretations; (3) provide them with a universal character; by standardising them they can be used in a general way and similarly in other projects and (4) incorporate the time value in their estimation and analysis to check their development, make comparisons over time and corroborate their diagnosis. In this way they would become even more useful tools for making financial and social decisions.

In view of the results obtained, we propose some ways forward or recommendations that, as well as improving measurement and evaluation, constitute new avenues of research, to continue refining and improving these coefficients and thus obtain more valuable measurements:

(1)

Create evaluation bands or quantification stages, forcing the filters to take discrete values (in bands or stages). A good example is case 8, which, as we have seen, structures its percentages on a scale that goes from lowest to highest in terms of the influence of the project on the result obtained. By establishing standard bands, we can reduce variation and subjectivity and get more easily comparable values. Intermediate values can be included within the bands, establishing the characteristic of the band, and adjusting once within it.

(2)

Stipulate some standard or average values for certain common or standard situations. This would help spread uniform and unambiguous criteria to be used in the various analyses. For example, adjust deadweight to a certain percentage when dealing with individual exceptions (case 9 is a good example, where individual recoveries, not attributable to the project, were given a deadweight figure of 5%).

(3)

Identify comparison populations, like in case 11, where a comparison group or control group was introduced. The data collected for these groups represent the deadweight for the SROI. It was a set of stakeholders outside the project’s area of influence, since they did not participate in the project’s activities, but had equivalent characteristics.

(4)

Homogenise, as far as possible, the timeline for the project. Projects with a duration of under six months make it very difficult to quantify the coefficients appropriately. Durations of over three years are equally difficult to quantify, given the high probability of exogenous events and spurious variations beyond the control of the analysis. Setting durations or timelines for analysis of 6, 12, 24 and 36 months (allowing intermediate periods of 6 months) seems a sensible approach, with time periods that allow results to be compared and analysed better. In this way we will get filters with more accurate and comparable results, and in particular, the calculation of the drop-off will have much greater value and significance.

(5)

Identify reference estimates, benchmarks, of universal validity as the basis for calculating the filters. This happened in case 2, where to filter the result of the project for the volunteers, they used the rate of volunteers in North Lanarkshire as a deadweight percentage according to a survey of households in Scotland.

(6)

Define standard rules in the valuation of the coefficients of certain interest groups. For example, groups that provide funding. Establishing certain rules in the quantification of the filters and their importance in these groups, not being the main stakeholders, can facilitate their homogeneous, common quantification.

Applying these recommendations to one of the analysed cases, the Craft Café case, results that allow a proper comparation with to other SROI cases can be obtained. The recommendations guarantee a unified assessment criterion of the filter coefficients, and the percentages of deadweight, displacement, attribution or drop-off are more in line with other analysis, as detailed in Table 6. In any case, this re-evaluation has only illustrative purposes, as the case does not provide all the information from the interviews and the whole process should be re-done to guarantee the accuracy of the re-evaluation. In our analysis we have taken an in-depth look at filter coefficients in the calculation of SROI through an exhaustive review of twenty-three relevant projects in the social sphere.

Table 6. Craft Café case. Re-evaluation.

Stakeholder	Attribution	Attribution Re-Evaluated	Deadweight	Deadweight Re-Evaluated
Older people that participate	19%	25%	17%	25%
Family members of participants	5%	0%	5%	0%
Partner Housing associations	15%	25%	13%	25%
The National Health Service	18%	25%	15%	25%

For the drop-off, the case recognised that they do not have research data available to establish the drop-off rate. Therefore, they have assumed a drop off percentage of 33% for this SROI analysis. The drop-off should be re-evaluated as 50%. Regarding the displacement, there is no evidence for this that justifies an inclusion of displacement in the impact results. As conclusion. the level of standardisation in the evaluation has been increased.

By virtue of its analysis, this article is a contribution to research in this field, and provides different contributions for both scholars and practitioners. Specifically, we have managed to: (1) make known the cases themselves, presenting a panoply of examples of successful social projects that can serve as models of good practice and inspiration for starting similar projects promoting the social economy; (2) study these projects in a systematic way, considering the methodology, results and study of the filter coefficients, in a concise and accessible way; (3) systematise the review of the calculation and comparison of the filter coefficients, allowing a comprehensive view, through the summary and analysis of these results; (4) observe some relevant facts and significant trends in the form and results of the calculation of these filter coefficients, identifying gaps and problems in their evaluation, and, finally, (5) suggest, in view of the situation observed, some recommendations and improvements in the road towards homogenising the method of calculation of the filters, as a necessary step for their implementation and success, which in turn open up new avenues of research.

The quantitative assessment of social impact, despite the conceptual and methodological difficulties that we have seen, is of great value for society. Its fixing and dissemination are important tasks, but there is still a long way to go. This work hopes to make a small contribution to this field by taking an in-depth look at the calculation and improvement of filters, which are, ultimately, coefficients defined to make the calculation of social value more accurate, precise, credible and usable.