Assessing the Sustainability of Decentralized Renewable Energy Systems: A Comprehensive Framework with Analytical Methods

Abstract

The number of models of Decentralized Renewable Energy (DRE) systems, particularly for rural electrification, is growing globally. Most approaches to assess the sustainability of these solutions beyond simple techno-economic considerations are comparative in nature, and only allow us to evaluate performance within a set of other interventions. This leaves a gap in our understanding of the conditions for a specific model to be sustainable and whether its replication is likely to succeed. The approach suggested develops a framework to evaluate the sustainability of specific models for energy access individually and proposes analytical methods to illustrate its use. It combines the multi-dimensional analysis over five sustainability dimensions and the Multi-Tier Framework (MTF) to assess technical sustainability, extending MTF’s rigorous scoring methodology to the other dimensions. The scores are based on qualitative and quantitative data collected from key stakeholders, taking into account different perspectives and aims. The framework and analytical methods are exemplified using a subset of data collected in over 40 off-grid DRE systems utilizing a common community ownership and hybrid financial structure. The proposed methodology can be used to understand the sustainability conditions of a given approach to energy access and can therefore be used by practitioners and policy makers to develop strategies and guide policies to roll out effective solutions.

1. Introduction

For the first time in 2016, the number of people with no access to electricity around the world declined to below 1.1 billion [1]. Though central grid extension continues to be the driving force accelerating access to electricity, renewable-based decentralized solutions have gained ground, accounting for 6% of new electricity connections since 2000 [1]. Driven by falling costs and innovative business models, Decentralized Renewable Energy (DRE) systems are shaping the face of energy access, especially in rural areas of the developing world. However, the failure rate of these interventions remains high [2,3,4,5,6]. Although many approaches have been developed to evaluate the sustainability of DRE systems, they are largely designed to compare the performances of different business models and technologies [2,7,8]. To the best of our knowledge, approaches that analyze the effectiveness of different types of interventions singularly and the conditions under which each can sustain and be scaled are lacking. If we are to achieve the global sustainability target of universal electrification by 2030, it is crucial to develop tools that enable the identification of scalable models of energy access and reveal the conditions for their impactful and lasting replication.

Goldemberg et al. [9] define ‘sustainable energy services’ as those that are “environmentally sound, safe, affordable, convenient, reliable and equitable,” thus defining an impactful energy service in terms of technical, socioeconomic, and environmental considerations. Recent developments [10,11,12] also emphasize the need to assess the institutional dimensions, considering the governance of energy infrastructure and participation of beneficiaries, particularly in the case of decentralized solutions. Despite the progress towards a comprehensive approach to assessing sustainability of energy access, technological and economic considerations continue to be prioritized [13,14]. Techno-economic approaches may be effective when testing novel technologies and business models, but they risk overlooking the social and institutional dimensions and could fail to provide an objective and comprehensive analysis [4]. Similarly, when social dimensions are given the priority, considerations around the viability of an intervention from a technical and economic point of view may be overlooked, or a deeper analysis of the local institutional arrangements neglected [7], thus resulting in early failures of interventions [3]. The success of new conceptualizations of energy projects lies between the acceptability, usability, and efficacy of the technology within the particular socioeconomic and cultural context of the project [15,16]. As Miller et al. [17] argue, there is a tendency to “focus almost exclusively on energy supply, leaving aside questions about the design of socio-technical arrangements that transform energy supplies into energy services that deliver social value.” Particularly, Murali, Malhotra, Palit and Sasmal [4] and Ulsrud et al. [18] emphasize the need to look at energy solutions as ‘socio-technical systems’; a heterogeneous configuration of technical and social elements that mutually interact and shape each other. Sustainability scholars [2,7,8,19,20,21] have proposed comprehensive multi-dimensional approaches that look at technical, economic, institutional, social, and environmental aspects when evaluating the sustainability of energy interventions. Building on the work done to date, this paper reveals some of the gaps in the multi-dimensional frameworks and proposes a novel approach to analyze specific models of intervention singularly.

First, most multi-dimensional approaches to assess sustainability present limited capabilities for empirical application and analysis [3,21]. For example, Ilskog’s [7] approach is based on 39 indicators across the five sustainability dimensions and each indicator is ranked on a scale of 1 to 7 through comparison between the various models assessed, where 7 indicates the best possible performance and 1 is the poorest among those considered. The overall sustainability of a model is then represented by a score that is the average of all 39 indicators. This score is inherently comparative and does not provide information to objectively evaluate the overall effectiveness of individual solutions, such as models that are entrepreneur-run of community-owned, thus leaving the framework largely ‘model-neutral.’ This poses difficulties when assessing the intrinsic value of a given model and the conditions under which it is likely to succeed. In addition, it is unclear if and how the qualitative data gathered through SWOT analysis is integrated for sustainability conclusions. Bhattacharyya [19] proposes a similar approach to score each of the five sustainability dimensions and ranks three types of energy access interventions for electricity and cooking services. While it is important to keep the evaluation as general as possible, this methodology does not offer a way to capture data pertaining to a specific type of service, and its high-level and comparative nature leaves limited empirical applicability.

Second, Ilskog and Kjellström [20] define technological sustainability as the ability of an energy system to provide a continuous service during the entire lifespan of the investment. This definition strongly links technology and economic considerations, as it focuses on how the energy service can improve its performance and lower its costs, ensuring a certain outcome during the project lifecycle. While performing a comprehensive literature review of photovoltaic energy systems for developing countries, Feron [2] includes technical aspects such as stability of service and complying with standards under the institutional dimension, leaving out many other aspects of technical sustainability, including power reliability or duration of supply. In contrast, Bhattacharyya [19] suggests that technical sustainability is achieved when a system is able to meet present and future energy needs reliably, efficiently, and using clean and renewable sources. This latter approach is more in line with the Sustainable Development Goal 7, which focuses on energy usability and the need to provide ‘affordable and reliable’ energy services. However, it falls short of providing specific quantifiable measures and analysis methods to assess the overall energy usability of the solutions considered. Recently, the World Bank and Energy Sector Management Assistance Program (ESMAP) [22] have developed a comprehensive approach to evaluate technical sustainability, advocating the use of scores and benchmarks to assess technical performance of energy access interventions against set targets. Their Multi-Tier Framework (MTF) takes into account factors such as the system capacity, durability, quality, reliability, affordability, legality, and safety of the service for domestic and commercial uses and in public spaces, retrieving measurements from system logs and data that can be independently audited. However, the MTF focuses only on the technical sustainability, leaving out other dimensions.

Third, recent studies highlight the need to understand the influence of different stakeholder perspectives and sociocultural backgrounds within which technology adoption happens [23]. When considering energy access interventions in particular, there are a multitude of stakeholders involved, from funders to community members, local NGOs, and implementing agencies, each having different aims and expectations for sustainability and impact. The underlying alignment of intent and drivers across stakeholders at all levels could play a critical role in affecting the long-term success of the proposed intervention. Additionally, Bhattacharyya [19] uses brainstorming among energy specialists to rank different types of energy technologies for lighting and cooking purposes. This approach is relevant when comparing the effectiveness of various interventions from a high-level perspective using expert opinion but it is largely detached from the views of the local communities and users. Models that integrate data collected across multiple stakeholders, including the end users, to generate a comprehensive understanding, are lacking.

This research aims to address the specific gaps identified above, and proposes an enhanced framework that builds on the work done to date. The approach is novel as it enables sustainability assessments of a given mode of intervention independently, rather than on a comparative scale. By collecting data from key stakeholders, the model also accounts for different opinions and perspectives to be considered, thus allowing for a comprehensive evaluation across dimensions and adding completeness to its conclusions. In addition, it strengthens the analyses by adding a variety of analytical methods, including visual inspection of cross-tabulation, correlation, multiple regression analysis and hypothesis testing, generating key insights into the conditions for a given type of intervention to sustain.

This article focuses on community-owned Solar Micro-Grid (SMG) solutions for energy access in remote villages. Community managed solutions have become very common in India [3,24] and elsewhere [2,25,26]. These are especially attractive in remote rural regions [3,24,25,26], which exhibit energy poverty and are generally characterized by weak technical, economic and institutional infrastructures. Studies have highlighted how, in such remote contexts, active community involvement in managing the energy intervention and crafting local forms of governance can lead to more inclusive institutional arrangements, capable to promptly adapt to local realities and respond to upcoming issues [27,28,29]. Despite these benefits, mixed project outcomes and high levels of failures call for a systematic investigation into the nature and extent of community involvement that ensures user satisfaction and overall sustainability of the solutions. The framework, therefore, proposes specific measurements for community participation and governance for remote settings, taking into consideration singular village installations as the basic unit of analysis.

The article is structured as follows. Section 2 presents the overall structure and rationale of the framework, as well as the approach and methods for data collection and aggregation of data to assign sustainability scores. In Section 3, we demonstrate the types of analyses that can be performed when applying this framework, including the development and testing of specific propositions for sustainability of community-owned DRE solutions for remote communities. The concluding section reviews the knowledge generated by the framework and provides recommendations for its adaptation to different technologies and models and for greater application.

2. Methodological Approach

2.1. Building Blocks: Dimensions, Measures, Indicators

Following a widely accepted approach in the development sector, the methodology uses dimensions, measures, and indicators as the basic structure of analysis [30,31]. It looks at five dimensions of sustainability, namely technical, economic, institutional, social, and environmental, that were first defined by Ilskog [7] and later on modified by Bhattacharyya [19]. Each dimension is further characterized through a set of variables (Measures), which, in turn are defined through a series of clearly articulated and measurable core components (Indicators).

The Technical dimension is assessed using the MTF customizing some of the indicators for SMGs installations. Given the MTF’s robust and clear approach to aggregate raw data into scores for indicators and measures, a similar methodology has also been developed for the remaining four dimensions where scores are assigned on a scale of 1 to 5 based on clearly defined and model specific benchmarks for sustainability.

A uniform tier-based approach across all sustainability dimensions is recommended as it enables cross-comparison of measures and facilitates generation of insights across dimensions. For example, this approach can be used to understand the influence of energy quality and reliability on the economic sustainability, particularly livelihood generation. Likewise, it allows us to consider how the duration and reliability of supply affect social dimensions, particularly education and household well-being, or whether perceived affordability hinders the overall economic and institutional sustainability of an installation.

Data for the indicators across all five dimensions are mixed, qualitative and quantitative, and represents multiple stakeholder voices, corresponding to multiple levels of information, i.e., household, organizational, and village levels. Household-level data are collected through surveys with individual households, whereas structured interviews with local operators and discussions with members of local village committees are used to gather data at the village institutional level. Additionally, the data at an organizational level include perspectives from solution providers, partnering NGOs, and funders, and are collected through semi-structured interviews and group discussions. Where possible, data are also collected from meter readings in the control room, or pictures of bank statements to determine the cash accumulated from the payment of energy service, thus reducing the subjective bias (

Table 1. Illustrative multi-level data and mixed methods for data collection.

Level	Description	Data & Collection Method
Village/System	Installation and village-specific data providing village profile and key system characteristics	Site visits and inspection, data from meter readings, system characteristics, record of project finances
Village/Institutions	Village-level organization in charge of effective operations and management of the system	In-person survey with close and open-ended questions with plant operators, semi-structured interviews and guided group discussions with governance committee members
Organizational	Perspectives from solution provider, local NGOs, and financiers	Semi-structured interviews and guided group discussions
Household	Information on a variety of indicators for singular house/family	In-person survey with close and open-ended questions with household members

).

For example, the indicator satisfaction with system is quantitative and collected at the household level, whereas ownership transfer is qualitative and collected at multiple levels, taking into account stakeholders from village institutions at the organizational and household level. An illustration of how data at multiple levels are used to consider different perspectives and determine sustainability scores is provided later in this section. Data collected in this way provide opportunities to compare different perspectives, goals, and expectations towards the energy system and the effect these have on the long-term success of the intervention.

Additionally, instead of providing five overall scores corresponding to each dimension [7], every intervention is defined through scores assigned at a more granular measure level, and necessary indicator-level data can also be used to question, clarify, and triangulate information. This allows for a more detailed yet compact evaluation of each of the intervention.

Similar to other development sector studies, measures and indicators are developed using an iterative process involving discussions with key stakeholders from the solution provider, local community members, financiers and local NGOs among others. This process helps highlight areas of focus, tweaking and adjusting indicators according to specific goals and expectations for impact at various levels. Field testing allows for further refinement and selection of indicators based on consistency, ease of data collection, and to lower subjective bias and margins for error. To make this process more rigorous, each indicator is accounted through a sheet, with a detailed definition of its attributes such as name, description, data type, data characteristics, level of data collection, its source, calculation methodology, and unit of measure (

Table 2. Indicator sheet—an example.

Indicator Sheet
Dimension	Economic Sustainability
Measure	Livelihood Generation
Indicator Name	Number of Commercial Activities
Description	This indicator tracks the use of electricity for income generating activities
Type	Quantitative
Source	Household Questionnaire—Question 16
Level	Household level
Data verification	Site inspection
Calculation & unit of measure	Percentage of respondents for a unit of sample (in this case a village-level project) using electricity for commercial purposes (supported by description of activities)

). Though lengthy and tedious, this mapping exercise is of extreme importance for defining and selecting appropriate indicators, developing coherent data collection methodologies, and avoiding replication and redundancies in the framework. There are limitations to assessing sustainability using indicators, particularly as they may lack the subtlety to capture some of the ground-level socio-institutional realities. However, it is contended that, if well thought out, indicators are fit to represent the overall system functionality, highlighting linkages and trade-offs between measures and dimensions that may not be visible using basic statistical data or a purely qualitative approach.

2.2. Definitions of Dimensions and Measures

This section defines each sustainability dimension and its constituent measures. When developing the framework, it is important to define the main features of the model so as to provide a meaningful definition for sustainability that matches the model’s intrinsic characteristics. This allows for the development of indicators, measures, and appropriate benchmarks.

This research, in particular, looks at community-owned SMGs in remote villages. The installations provide electricity for domestic and commercial use and in public spaces. The financial model is hybrid, with capital costs provided with no expectations for a return on assets, for example through corporate social responsibility grants. Recurring Operations & Maintenance (O&M) costs are addressed through regular billing from metered electricity consumption from each house. In this model, the community takes on the technical and financial responsibility for the management of the plant on an ongoing basis, including basic O&M work and timely collection and deposit of energy tariffs from each household.

In this context, technical sustainability relates to the ability of the system to meet present and future energy needs of domestic users, commercial purposes, and public spaces. To capture this, usable supply is defined as one that is adequate in terms of quantity, reliability, high quality, affordability, and safety. Technical tiers are assigned by adopting the MTF, looking at three core measures of domestic supply, public lighting, and domestic energy consumption.

Economic sustainability is concerned with the cost-effectiveness of the solution, the capital cost for installation and its recovery over time, as well as the system’s contribution to income-generating activities [19]. Two primary measures reflect these concerns: the long-term financial viability of the model and the system’s ability to contribute to livelihood-related activities. The first captures specific aspects of the economic model, capital finance, and cost recovery of over time. For hybrid financing models and community-owned solutions such as the one considered in this study, long-term financial viability is achieved when the community is able to raise sufficient capital through energy tariffs, take care of recurring O&M costs, and plan for battery replacement at the end of life. For livelihood activities, it is contended that economic activities generated by the intervention should increase a project’s revenue and contribute to economic development at the village level.

Institutional sustainability requires that the energy interventions are effectively managed locally. DRE systems in off-grid remote locations require localized and decentralized governance structures to ensure an operative system and efficient issue management [12]. For community-ownership structures, institutional sustainability is central and is reflected through three core measures: effectiveness of local governance, community participation in governance, and overall satisfaction of the users. Effective local governance looks at the capacity of local governance committees to manage technical, financial, and other matters with little to no external intervention or support. It requires that communities are able to take care of the system on a daily basis, solve minor issues internally, and promptly reach out to the right players when external help is needed. It also requires that governance processes, especially the appointment of committee members, is democratic and transparent, achieved through formal election and/or unanimous appointment of its members. Tasks should be clearly defined, and a balance of responsibility and accountability should be achieved. Secondly, community participation in the governance processes must be inclusive, seeking involvement from all members of the community at all stages of the intervention. Additionally, the perception of community members of their ability to raise issues and influence change in the governance process is a key component of this measure. Finally, user satisfaction is measured across all levels, looking at the perceived satisfaction of the household with the service, for both domestic supply and in public spaces, and satisfaction with the institutional arrangements and tariffs. Satisfaction of the village committees and local operator is also considered.

Social sustainability is about improving the lives of all members of the community equitably, reducing human drudgery and providing positive effects for women and children [19]. Access to energy has the potential to benefit child education, the health of the family, and safety at home and in public places, thus enhancing overall household well-being [32]. In addition, women’s participation in energy-related systems of governance can lead to an increased sense of independence in the home and a general feeling of empowerment [32]. These considerations are captured in the household wellbeing measure. Additionally, an increased sense of connection, both within the community, arising from inclusive local governance, and with the outside world, due to communication technologies enabled by energy access, is captured in the community connectedness measure.

Environmental sustainability is measured on a local and global scale as improvements in indoor air quality and reduction in carbon emissions due to reduced kerosene usage, respectively.

The definitions above result in five dimensions, 12 measures, and 31 indicators. The framework is visually depicted in Figure 1a,b, where the last row represents the legend for the images. Surveys to capture the raw data at multiple levels are available upon request.

2.3. Definition of Benchmarks and Scores

Benchmarks for the three measures in the technical dimension, namely domestic supply, public lighting, and domestic consumption, are largely drawn from the MTF, thus allowing for a comparison of results with standardized, nationally-set targets and across technologies. These are presented in

Table 3. Tiers and benchmarks for domestic energy supply across its indicators.

Indicator	Tier 1	Tier 2	Tier 3	Tier 4	Tier 5
Capacity (W/HH)	Min 3	Min 50	Min 200	Min 800	Min 2000
Duration	>4 h >1 h @ night	>4 h >2 h @ night	>8 h >3 h @ night	>16 h >4 h @ night	>23 h >4 h @ night
Reliability (monsoon months)	Frequent outages >5 days/month		2–5 days/month	1–2 days/month	No unscheduled outages
Quality	Frequent issues with V and f affecting use of appliances		Few issues with V and f	No issues with ability to use appliances when needed
Affordability	House unable to pay at time of collection and still in debt		House unable to pay at time of collection and facing difficulties in paying on following month	House unable to pay at time of collection but easily paid back the following month	No difficulties with regular payments
Legality	Illegal connections and irregularities with payments		No illegal connections and bills paid to authorized representative
Safety	Unsafe connection and installation		Absence of past accidents and perception of risk in the future

,

Table 4. Tiers and benchmarks for public lighting across its indicators.

Indicator	Tier 1	Tier 2	Tier 3	Tier 4	Tier 5
Capacity (village coverage)	1 street light	>25%	>50%	>75%	>95%
Duration (Night hours)	>2 h/day	>4 h/day	>50%	>75%	>95% s
Reliability (monsoon)	Frequent outages >5 days/month		2–5 days/month	1–2 days/month	No outages
Quality	No functioning lights	Failures, brightness flickering issues	No early failures, no issues with brightness, flickering, etc.
Safety	Unsafe connection and installation		No perceived risk of electrocution due to poor installation or maintenance

and

Table 5. Tiers and benchmarks for energy consumption.

Indicator	Tier 1	Tier 2	Tier 3	Tier 4	Tier 5
Daily Consumption (Wh/HH)	≤12–200>	≤200–1000>	≤1000–3425>	≤3425–8219>	≥8219

. To assign a measure level score, a Tier (1–5) is assigned to each indicator based on the system characteristics, site inspection, readings from control room, and data collected at various levels. When aggregating household-level responses to a village-level score, the modal value capturing the majority voice in the village is used. Likewise, when benchmark values span across scores (tiers), the higher end of the score is used, as also proposed in the MTF. The final score for the measure corresponds to the lowest score across all indicators, thus highlighting the limiting factor for technical sustainability. Although this is a conservative approach, it provides useful information for practitioners and policymakers, highlighting bottlenecks and priority areas to improve the service.

The benchmarks pertaining to measures representing economic, institutional, social, and environmental dimensions are presented in

Table 6. Tiers and benchmarks for energy consumption.

Score	Description
1	System not operational. Information from community finances shows few or no collections happening and no sign of re-establishing the mechanism for tariff collections.
2	System is operational/partly operational. Information from community finances shows a big difference between expected and actual balances at the time of inspection and energy tariffs are not being collected regularly by the local operator.
3	System is operational. Information from community finances shows a few sparse deposits. The difference between expected and actual balance at the time of the visit may still be substantial, but money collection happens regularly.
4	System is operational. Information from community finances shows regular and timely deposit. Difference between expected and actual balance at the time of the visit is small.
5	System is operational. Information from community finances show regular deposits and consistency between expected and actual balance.

,

Table 7. Tiers and benchmarks for livelihood across its indicators.

Score	Description
1	No economic activities in the village linked to energy use. Users are not reporting an increase in productivity that can be linked to electricity being available at home.
2	Limited livelihood generation activities in the village and limited increase in productivity registered
3	Few households are starting to use electricity and are purchasing appliances to help with existing businesses. A growing number of people in the village are also reporting increased productivity.
4	Few households have invested money to actively start a new business and purchase electrical equipment. The majority of people are also noticing an increase in productivity thanks to electricity at home.
5	Engagement in income activities is extended to few houses in the village, and many examples of houses actively investing money to start new businesses.

,

Table 8. Tiers and benchmarks for effectiveness of local governance across its indicators.

Score	Description
1/2	Very ineffective. Major external intervention needed to keep the project going. Local operator is not able to collect tariffs from households nor take care of small technical issues. Institutional meetings for energy-related issues are infrequent and ineffective.
3	Medium effectiveness. Local operator and members of local institutions are able to solve small technical and financial issues autonomously. Energy-related meetings are happening, though not very frequently. External intervention from higher organizational level is still largely needed to initiate meetings and discuss issues. System for accountability and enforcement of rules is in place but not solid.
4	Effective. Local operator and members of local institutions have demonstrated ability to take care of the majority of technical and financial issues autonomously in many occasions. External intervention may still be needed to take care of more serious technical issues, or to solve major internal disputes. Despite this, representative of local governance are able to seek external help when issues arise without compromising the system’s functionality.
5	Very effective. Local institutions have demonstrated ability take care of technical and financial issues autonomously over a long period of time, timely seeking help when serious issues arise and limiting the down time of the system due to O&M to the lowest possible level. Meetings happen regularly, and local institutions have been able to craft and modify rules around the use and management of the system to accommodate local necessities.

,

Table 9. Tiers and benchmarks for community involvement in governance across its indicators.

Score	Description
1–2	Low involvement. Few people participate in meetings and there is limited sense of ownership being transferred to the community.
3	Medium involvement. About half of the village attends meetings that happen at regular intervals. Sense of ownership and trust on institutions is also good.
4–5	High level of involvement. Meetings happen regularly and the majority of people participate and interact. Confidence on local institutions and governance is also high.

,

Table 10. Tiers and benchmarks for user satisfaction across its indicators.

Score	Description
1–2	Low satisfaction. Users at various levels report high level of dissatisfaction over many indicators.
3	Medium satisfaction. Users at various levels are fairly satisfied with the system. However, there may be some emerging factors limiting to higher score.
4	Medium–high. Users at various levels report high level of satisfaction across many indicators. There are, however, still one or two limiting factors.
5	High. Very high satisfaction at all levels across all indicators.

,

Table 11. Tiers and benchmarks for household wellbeing and community connectedness across their respective indicators.

Score	Description
1–2	Low. Limited to no improvements across many indicators
3	Medium. Some improvements reported but limitations noticed across 1–2 indicators
4–5	High. Improvements noticed across all indicators and high degree of confidence

and

Table 12. Tiers and benchmarks for local and global environmental sustainability across their respective indicators.

Score	Description
1–2	- Local <30% of HHs noticing improvements in indoor air quality - Global: <30% reduction in monthly kerosene use at household level
3	- Local ≤30–65%> of HHs noticing improvements in indoor air quality - Global ≤30–65%> reduction in monthly kerosene use at household level
4–5	- Local ≥65% of HHs noticing improvements in indoor air quality - Global ≥65% reduction in monthly kerosene use at household level

.

Recognizing that the benchmarks for all measures other than those for technical dimension require manual judgment and a combination of qualitative and quantitative data, an iterative Delphi-based method is proposed to reduce researcher bias. Through this approach each researcher develops an initial score independently, then the values are cross-checked, any discrepancies discussed, differences resolved, and a final measure-level score assigned. An illustration of how multi-level data have been used in the aggregation process is provided below.

To assess the measure of effectiveness of local governance, the indicator of ownership transfer is considered, which includes data gathered at several levels. To start with, data provided by the system operator help with understanding the number, nature, and frequency of technical, financial, and other issues encountered. This information is then linked with qualitative data from discussions with local institutions to understand if and how the issues are handled by the operator and discussed during meetings, and whether they are promptly acted upon by the community. Institutional-level data are then contrasted with opinion at the household level, looking for agreements or inconsistencies on technical issues encountered by households, the ability of the operator to fix issues on time, the regularity of energy-related meetings, and the household’s ability to discuss issues openly. At a higher, organizational level, the perspective of the local partner on the ground, acting as the first point of contact when issues escalate, is also taken into consideration to determine a final score.

A sample tabulation of data across measures and dimensions is presented in

Table 13. Scores across 12 measures for four sample sites.

Village	Darewadi	Viral	Jhaliyabandh	Sukalipada
Geography	Junnar	Karnataka	Jarkhand	Jawhar
Village Code	1	2	19	24
Domestic Supply	3	3	2	3
Public Lighting	4	1	2	4
Domestic Consumption	2	1	1	1
Model Sustenance	4	4	2	3
Livelihood	3	1	2	1
Efficiency of Governance	5	4	2	2
Community Participation	5	5	2	2
User Satisfaction	5	5	3	2
Household Wellbeing	5	3	5	2
Community Connectedness	5	3	3	2
Local Environmental benefits	2	5	5	3
Global Environmental benefits	5	5	5	3

.

3. Discussion

This section demonstrates the use of the framework and different types of analyses that can be performed using scores assigned to each village installation, as well as the indicator-level data. In particular, three types of investigations are illustrated: (a) cross-tabulation and visual inspection; (b) correlation analyses; and (c) multiple regression analysis at the individual household level.

To provide the reader with an example of the analyses, results from a set of community-owned SMGs implemented by Gram Oorja, a social enterprise delivering energy access solutions through SMGs, solar water pumping systems for drinking and irrigation purposes, and biogas cooking grids in rural India, are presented. The data are part of a wider study looking at sustainability conditions for community-owned energy access solutions. Data were collected between June 2017 and January 2018 by trained field staff.

First, a visual plot of cross-tabulation to inspect relationship between two measures is presented. For example, that between model sustenance and project duration or model sustenance and effectiveness of governance. The plots also include variables such as village size and geographical location to highlight variations and possible effects of these variables. In the plots below, time dependency appears along the X axis on a logarithmic scale and measured scores are presented on the Y axis. Different colors refer to different geographical areas and the diameter of the circles is proportional to the number of households served by the installation in each village. This visual analysis of data can help us understand if concepts based on common wisdom and prior research hold good, or if unexpected outcomes and patterns seem to emerge from the data.

For example, one might expect that provision of adequate energy (i.e., high scores for domestic supply across its core indicators of capacity, reliability, quality, etc.) will cause greater amenities to be purchased or new businesses to be initiated, and hence correspond to increasing consumption over time. To examine this phenomenon, tiers for domestic supply for various villages across their respective indicators are considered to understand any issue with power usability and identify the indicators limiting performance to higher scores (

Table 14. Tiers for domestic supply across indicators for eight community-owned SMGs and internal target from solution provider.

Village	Capacity	Duration	Reliability	Quality	Affordability	Legality	Safety	Domestic Supply
Solution Provider target	3	4	4	4	4	5	5	3
Darewadi	3	4	4	5	5	5	5	3
Viral	3	5	5	5	4	5	5	3
Hedolipada	3	5	3	3	4	5	5	3
Vanvasipada	3	5	2	2	4	5	5	2
Bhinjpur	3	5	3	3	4	5	5	3
Khokmar	2	5	4	5	4	5	5	2
Raksha	3	5	3	5	4	5	5	3
Hodong	2	5	2	2	5	5	5	2

). These can also be compared with the expectations from solution providers as to the level of service delivered to identify discrepancies in expectations.

After verifying the level of service across indicators, a look at Figure 2 shows that the overall supply tier for all implementations is between 2 and 3, consistent with the expectation from the solution provider, as represented by the dotted black line in the graph. Subsequently, the average daily household consumption over time can be examined (Figure 3) to see if projects that have been in existence longer show greater energy consumption. However, as revealed in Figure 3, the household daily consumption does not appear to increase for implementations that have been around for longer, thus elevating the need for further inquiry and a formulation of hypotheses on the possible reasons.

Likewise, Figure 4 and Figure 5 suggest that the performance of the intervention is consistently high when it comes to effectiveness of governance, and the model’s economic sustenance. The consistency is observed for more recent as well as older implementations, and appears to be independent of the size of the village and geographical location. A potential explanation is that effective governance of the energy system is able to enforce timely invoicing and collections of tariffs from all households, thus resulting in adequate funds collected and hence high model financial sustenance. This preliminary visual analysis can be used to record counterintuitive or confirmatory observations. Although such observations can be thought of as lacking statistical power, they can point us in directions worthy of further inquiry.

Second, subject to adequate sample size, bivariate correlation analysis across all measures can be performed to explore those that are statistically significant and whether they confirm or refute common knowledge. For example,

Table 15. Example of bivariate correlations for 24 community-owned SMG sites.

		Cons	Model Sust.	Livelihood	Eff. Gov.	Comm. Part.	User Satisf.	HH Well Being	Comm. Conn.
Cons	P. Corr.	1	0.152	0.431 *	0.190	0.009	0.169	−0.108	0.119
	Sig		0.479	0.035	0.374	0.968	0.430	0.614	0.580
Model Sust.	P. Corr.		1	0.433 *	0.815 **	0.757 **	0.691 **	0.280	0.431 *
	Sig.			0.035	0.000	0.000	0.000	0.185	0.035
Livelihood	P. Corr.			1	0.433 *	0.499 *	0.448 *	0.383	0.641 **
	Sig.				0.034	0.013	0.028	0.065	0.001
Eff Gov.	P. Corr.				1	0.897 **	0.728 **	0.483 *	0.578 **
	Sig.					0.000	0.000	0.017	0.003
Comm. Part.	P. Corr.					1	0.821 **	0.467 *	0.559 **
	Sig.						0.000	0.021	0.005
User Satisf.	P. Corr						1	0.302	0.382
	Sig.							0.152	0.065
HH Well Being	P. Corr							1	0.590 **
	Sig.								0.002
Comm. Conn.	P. Corr.								1
	Sig.

* Correlation is significant at the 0.05 level (two-tailed); ** Correlation is significant at the 0.01 level (two-tailed).

shows that almost all significant correlations support prior understanding, such as that between consumption and livelihood, or model sustenance and effective governance. This information can be used to direct further investigations into potential multicollinearity among the measures that show correlations.

Third, to exemplify more robust statistical analysis, one single hypothesis regarding long-term sustainability for the specific community-owned model is developed and tested using stepwise multiple regression. Other situationally relevant methods could be used to draw inferences related to sustainability as deemed appropriate.

For the model under consideration, it is contended that the technological proficiency of the system is key to the long-term success of the intervention [22] and strongly predicts perceived user satisfaction. Technical proficiency is achieved through the provision of adequate, reliable, and affordable energy services to domestic users. In particular, studies that applied the MTF in grid-connected and off-grid communities in India highlight how household satisfaction responds strongly to the number of hours of supply per day [33], with an effect comparable to that of electrified and non-electrified households. The reliability and quality of the service in terms of blackout days and issues with power quality were also found to have an effect, particularly relating to driving wider socioeconomic and environmental benefits [34]. If the energy charges are perceived to be fair, users are likely to actively participate in the upkeep of the system through the local institutional apparatus [4], and witness benefits in terms of household wellbeing, therefore increasing the satisfaction with the system. As a result, it is proposed that:

Hypothesis 1.

User satisfaction is predicted by the supply tier, particularly power capacity, quality and reliability, perception of charges being fair, user participation in governance, and wellbeing of the households.

The above hypothesis can be tested using stepwise multiple regression method. A test with a sample of 162 households connected to SMGs was performed where user satisfaction was the dependent variable and all other measures in the framework were entered stepwise as independent variables. The results (

Table 16. Example of stepwise multiple regression to test Hypothesis 1.

Model Summary
Model		R		R Square		Adjusted R Square			Std. Error of the Estimate
1		0.416 a		0.173		0.168			0.532
2		0.527 b		0.278		0.269			0.499
3		0.574 c		0.330		0.317			0.482
Coefficients d
Model			Non Std. Coefficients		Std. Coeff.	t	Sig.	95.0% Confidence Interval for B
			B	Std. Error	Beta			Lower Bound		Upper Bound
3	(Constant)		−0.593	0.267		−2.224	0.028	−1.119		−0.066
	HH Well Being		0.119	0.020	0.389	5.861	0.000	0.079		0.159
	Supply Quality		0.161	0.034	0.306	4.672	0.000	0.093		0.229
	Fairness of Charge		0.262	0.075	0.232	3.490	0.001	0.114		0.410

^a Predictors: (Constant), HH Well-Being; ^b Predictors: (Constant), HH Well Being, Supply Quality; ^c Predictors: (Constant), HH Well-Being, Supply Quality, Fairness of Charge; ^d Dependent Variable: User Satisfaction–HH Level.

) indicate that household well-being, quality of supply, and perceived fairness of charges predict user satisfaction, whereas reliability of supply is not a significant predictor. A potential explanation might be that the reliability tier is consistently high in all cases and might be taken for granted by the users.

The above illustrations highlight that the framework is well suited to a variety of analyses that can help us draw significant conclusions regarding the multi-dimensional sustainability of a given form of intervention and the opportunity for its successful replication on a wider scale.

4. Conclusions

The framework including measurements and analytical methods provides a tool to comprehensively evaluate the long-term effectiveness and sustainability of specific models of energy access interventions, across several sustainability dimensions. This approach is useful to understand the effectiveness of the model analyzed and the conditions under which it can be successfully replicated and therefore scaled up.

The combination of multi-dimensional analysis with the MTF for technical sustainability and the use of the MTF’s rigorous approach to calculate scores for the other four dimensions allows for an objective analysis of the state of each installation. The scoring methodology is based on clearly defined benchmarks around sustainability levels that are model-specific and reflect desirable outcomes for sustainability and impact. This allows for an objective and direct interpretation of the results, providing insightful perspectives on connections between measures across different areas of sustainability. Collecting data from key stakeholders across multiple levels of analysis is a valuable means to corroborate and validate information, taking into account the voices of all stakeholders involved in the implementation. The framework is designed to enable progressively complex empirical analysis, beginning with visual inspection and continuing on to formulating and testing hypotheses. Taken together, these analyses can guide a comprehensive assessment of sustainability and meaningful conclusions. In particular, it is suggested that we analyze installations using similar modes of implementation but developed by different organizations. This could provide useful information on the effects of different approaches on the long-term sustainability of the model and help us to draw meaningful conclusions to guide strategies for effective replication.

As in most cases, the framework has limitations. First, acquiring data, particularly at the installation level, can be a challenge as the sample size must be adequate to ensure that statistical tests can be performed. While this is a limitation, it is also a strength in that it prevents us from prematurely drawing conclusions about the replicability and scalability of an intervention.

Second, from a fieldwork perspective, several obstacles must be addressed during the planning stages. Firstly, the remoteness of the sites can cause delays in data collection as many remote communities benefitting from DRE systems become inaccessible at certain times of the year due to severe weather conditions. To address language and cultural barriers, hiring local field workers, people familiar with the area and able to speak the local language, is of the utmost importance in order to connect with the communities and establish a more egalitarian relationship between the field worker and respondents. This allows for more open conversations and obtaining sincere answers that reflect the feelings and attitudes of the communities towards the system. Additionally, using independent field staff, not linked with local organizations or the solution provider, is key to collecting more realistic and less biased data.

Third, though very general in its definition, the proposed scores and benchmarks are specifically designed for community-owned installations in remote villages, where sustainability is shaped by the remoteness of the location and where there is a general lack of institutional environment to provide technical support or facilitate local capacity building. For the community-owned model presented in this paper, one that is based on strong social engagement, decentralized local governance structures present a greater likelihood for long-term success and become central to the long-term endurance of the energy interventions. In addition, the financial model analyzed is hybrid. With no expectation of a return on assets, the financial burden on the users is reduced and the measures for economic sustainability have been uniquely designed to reflect this scenario. Caution must be exercised when applying this framework to other scenarios, for example where there is an expectation of financial return on assets or when the model is entrepreneur-run rather than community-owned. In such instances, one should look back at the measures used to describe economic and institutional dimensions, adapting indicators and benchmarks to be in line with the model’s objectives. Additionally, it is possible to apply this framework to different types of energy access technologies for lighting as well as cooking purposes. In this case, one should go back to the MTF tiers and adapt those to be in line with the technology analyzed.