Diagnosing Structural Change in Digital Interventions: A Configurational Evaluation Framework
Abstract
1. Introduction
2. Methods
- Articulate research question(s).
- Construct a purposeful sample of cases and collect data regarding outcome and key case attributes expected to be causally linked to outcome.
- Transform the dataset into crisp or fuzzy sets by defining sets and calibrating case membership in sets.
- Construct a Truth Table.
- Analyse the Truth Table to identify set relations between attributes and outcome(s).
- Evaluate, interpret, and represent findings as well as their robustness.
2.1. Research Question
2.2. Purposive Sampling of Cases with Outcomes and Key Attributes
- Description: A concise overview of the digital intervention, outlining its purpose, institutional ownership (public, private, or donor-backed), technical architecture (e.g., open source), geographic scope, and primary functionalities. This section typically answers the following questions: What is the intervention? Who created it? What is its intended purpose?
- Removal of Systemic and Structural Barriers: An assessment of how effectively the intervention addresses deep-rooted inefficiencies, exclusions, or frictions in existing systems. These barriers might include fragmented data systems, a lack of transparency or accountability, geographical, linguistic, economic, or technological access constraints, and poor information flows. This section also critically evaluates whether the right barrier was identified and targeted.
- Sustained Well-being for All: A measure of whether the intervention leads to long-term, inclusive, and equitable improvements in well-being. This includes evidence of large-scale and enduring benefits, inclusivity across gender, geography, class, and ability, as well as integration into public systems for continuity. It differentiates between the scale of use and the depth or durability of impact.
- Value Added: The distinct contributions made by the intervention that would not have existed otherwise. This can encompass improvements in service delivery efficiency or reach; the generation of actionable data or insights; enhanced decision-making or cost savings; and tangible outcomes in the target domains (e.g., health, education, governance).
- Ability to Shift the Nash Equilibrium: The intervention’s potential to create a self-sustaining, system-wide behavioural change where all actors (e.g., users, providers, regulators) have rational incentives to adopt and continue using it. A shifted Nash implies that legacy systems are retired or replaced; behavioural norms and institutional practices have permanently altered; and there is clear local ownership and operational continuity. An intervention that merely coexists with existing practices or is dependent on external funding fails this test.
- Conclusion: A summative evaluation that synthesises findings from all prior sections and assesses the intervention’s long-term viability; ownership, financing, and policy alignment; whether it meets the necessary conditions to shift the Nash; and whether the intervention adds systemic value and is likely to endure once external support is withdrawn.
- Open-Source Code (Yes = 1; No = 0): Whether the digital intervention’s codebase is publicly accessible and can be freely used, modified, and distributed. This attribute indicates whether the technology is open source (Yes = 1) or proprietary (No = 0), based on statements regarding licensing, public access, or open infrastructure.
- Regulatory Enablement (Yes = 1; No = 0): Whether a supportive policy or regulatory framework enables the success or operation of the intervention. This includes mandates that enforce adoption, legal provisions that create use-cases (e.g., e-signatures, health records), and government endorsement or integration. A value of 1 implies clear regulatory backing, while a value of 0 indicates the absence of such support.
- Revenue Model in Place (Yes = 1; No = 0): Whether the intervention has an identifiable and sustainable financial model. This includes public sector budgetary allocations, private sector monetisation strategies, and donor or philanthropic support with plans for long-term funding. A value of 1 indicates that a credible model is described; 0 indicates that funding is ad hoc or uncertain.
- Substantial Scale Achieved (Yes = 1; No = 0): Whether the intervention has reached a large number of users or geographic spread in a sustained way. Scale may be indicated by nationwide rollout, millions of users, or integration into routine government operations. A value of 1 indicates that the scale has been clearly demonstrated; 0 implies pilots, small rollouts, or stagnant reach.
- Identifiable Systemic Barrier it Seeks to Eliminate (Yes = 1; No = 0): Whether the intervention targets a clearly articulated and systemic structural barrier, such as data fragmentation, lack of portability, absence of real-time information, or institutional inefficiency. A value of 1 implies that the intervention is specifically designed to eliminate this barrier.
- Presence of Pre-requisites (Yes = 1; No = 0): Whether the successful implementation of the intervention depends on external pre-conditions already being in place. These could include smartphone penetration, the availability of a trained workforce, digital literacy, and internet connectivity. A value of 1 indicates that such enabling factors are present; a value of 0 indicates that critical dependencies are absent.
- Sufficient Time for Implementation (Yes = 1; No = 0): Whether the intervention has been in operation long enough to allow for full deployment, feedback integration, and measurable impact. A value of 1 implies multiple years of implementation (typically five or more years); a value of 0 means the intervention is too new or still evolving.
- Well-being refers to improvements in individual or collective outcomes (e.g., health, income, education). These are the goals of development, but they may arise from short-term programs or temporary boosts in service access.
- Structural change refers to the reconfiguration of formal and informal rules, roles, or institutional capacities that shape how services are delivered or regulated. Structural changes may be imposed top-down or may emerge gradually and may not always persist.
- Shifting the Nash equilibrium refers to a specific subset of structural change—a self-reinforcing reconfiguration of actor incentives. In game-theoretic terms, a new equilibrium exists when no actor (e.g., user, provider, regulator) has an incentive to revert to prior behaviours, given what others are doing. This implies local ownership, institutional embedding, and ongoing functionality without external enforcement.
3. Results
3.1. Classify the Data into Crisp Sets
3.2. Construct a Truth Table
4. Discussion
4.1. Analyse the Truth Table
4.2. Interpret the Findings from the Truth Table
5. Limitations
- First, the selection of digital interventions was purposive and not representative; cases were chosen based on their visibility and data availability, which may have biased the results toward better-documented or more prominent initiatives. Additionally, the study examines only 13 cases, which limits the generalisability of the findings. While QCA is well-suited for small-N analysis, broader claims would benefit from replication in other sectors or geographical contexts.
- Second, while crisp-set Qualitative Comparative Analysis (csQCA) enables the identification of necessary and sufficient conditions, it does not offer probabilistic inference or generalizability beyond the sample.
- Third, outcome coding relies on interpretive assessments of whether a “Nash shift” occurred. Although triangulated with expert opinion and documentation, these judgments are inherently subjective.
- Fourth, the dataset reflects a survivorship bias: most included interventions are still operational. Failed or abandoned interventions, which might offer valuable counterfactual insights, are underrepresented.
- Lastly, while this study focuses on structural success, it does not assess unintended consequences or distributional harms, which remain critical in evaluating real-world impact.
6. Use of AI
7. Conclusions and Policy Implications
- If regulatory backing and a sustainable revenue model are absent, interventions are unlikely to persist, regardless of technical design.
- If scale is achieved without incentives aligned across actors, interventions may reach large numbers temporarily, but will not shift the equilibrium.
- If a systemic barrier is not clearly identified and targeted, interventions risk coexisting with legacy systems rather than replacing them.
- If key prerequisites for the specific intervention (infrastructure, capacity) are missing, early adoption is unlikely to translate into durable transformation.
- If sufficient time is not allowed, evaluations risk premature conclusions before incentives have stabilised.
Supplementary Materials
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Nash, J.F. Equilibrium points in n-person games. Proc. Natl. Acad. Sci. USA 1950, 36, 48–49. [Google Scholar] [CrossRef] [PubMed]
- Smith, J.M.; Price, G.R. The Logic of Animal Conflict. Nature 1973, 246, 15–18. [Google Scholar] [CrossRef]
- Taylor, P.D.; Jonker, L.B. Evolutionary stable strategies and game dynamics. Math. Biosci. 1978, 40, 145–156. [Google Scholar] [CrossRef]
- Li, J.; Xu, C.; Huang, L. Evolutionary Game Analysis of the Social Co-governance of E-Commerce Intellectual Property Protection. Front. Psychol. 2022, 13, 832743. [Google Scholar] [CrossRef]
- Chen, D.; Chen, B. Evolutionary game analysis on decision-making behaviors of participants in mega projects. Humanit. Soc. Sci. Commun. 2023, 10, 921. [Google Scholar] [CrossRef]
- Yue, X.; Durrani, S.K.; Li, R.; Liu, W.; Manzoor, S.; Anser, M.K. Evolutionary game model for the behavior of private sectors in elderly healthcare public–private partnership under the condition of information asymmetry. BMC Health Serv. Res. 2025, 25, 181. [Google Scholar] [CrossRef]
- Yao, B.; Shanoyan, A.; Schwab, B.; Amanor-Boadu, V. The role of mobile money in household resilience: Evidence from Kenya. World Dev. 2023, 165, 106198. [Google Scholar] [CrossRef]
- Avgerou, C. Discourses on ICT and development. Inf. Technol. Int. Dev. 2010, 6, 1–18. [Google Scholar]
- Madon, S. e-Governance for Development: A Focus on Rural India; Palgrave Macmillan: London, UK, 2009. [Google Scholar] [CrossRef]
- Toyama, K. Technology as amplifier in international development. In Proceedings of the 2011 IConference, Seattle, WA, USA, 8–11 February 2011; Association for Computing Machinery: New York, NY, USA, 2011; pp. 75–82. [Google Scholar] [CrossRef]
- Walsham, G. ICT4D research: Reflections on history and future agenda. Inf. Technol. Dev. 2017, 23, 18–41. [Google Scholar] [CrossRef]
- Masiero, S.; Arvidsson, V. Degenerative outcomes of digital identity platforms for development. Inf. Syst. J. 2021, 31, 903–928. [Google Scholar] [CrossRef]
- Masiero, S.; Bailur, S. Digital identity for development: The quest for justice and a research agenda. Inf. Technol. Dev. 2021, 27, 1–12. [Google Scholar] [CrossRef]
- Veeraraghavan, R. Cat and Mouse Game: Patching Bureaucratic Work Relations by Patching Technologies. Proc. ACM Hum. Comput. Interact. 2021, 5, 21. [Google Scholar] [CrossRef]
- Samuel, M.; Doctor, G.; Christian, P.; Baradi, M. Drivers and barriers to e-government adoption in Indian cities. J. Urban. Manag. 2020, 9, 408–417. [Google Scholar] [CrossRef]
- Allmann, K.; Radu, R. Digital footprints as barriers to accessing e-government services. Glob. Policy 2023, 14, 84–94. [Google Scholar] [CrossRef]
- Djatmiko, G.H.; Sinaga, O.; Pawirosumarto, S. Digital Transformation and Social Inclusion in Public Services: A Qualitative Analysis of E-Government Adoption for Marginalized Communities in Sustainable Governance. Sustainability 2025, 17, 2908. [Google Scholar] [CrossRef]
- Zelenkov, Y.; Lashkevich, E. Does information and communication technology really affect human development? An empirical analysis. Inf. Technol. Dev. 2023, 29, 329–347. [Google Scholar] [CrossRef]
- Thomas, J.; O’Mara-Eves, A.; Brunton, G. Using qualitative comparative analysis (QCA) in systematic reviews of complex interventions: A worked example. Syst. Rev. 2014, 3, 67. [Google Scholar] [CrossRef]
- Ragin, C.C. The Comparative Method; University of California Press: Oakland, CA, USA, 1987. [Google Scholar]
- Ragin, C.C. What Is Qualitative Comparative Analysis (QCA)? National Centre for Research Methods (NCRM): Tucson, AZ, USA, 2006. [Google Scholar]
- Greckhamer, T. Qualitative Comparative Analysis in Survey Research. In The SAGE Handbook of Survey Development and Application; Ford, L., Terri, A., Eds.; SAGE Publications: Thousand Oaks, CA, USA, 2023; pp. 368–379. [Google Scholar] [CrossRef]
- Ragin, C.C. Redesigning Social Inquiry Fuzzy Sets and Beyond; University of Chicago Press: Chicago, IL, USA, 2008. [Google Scholar]
- Klarin, T. The Concept of Sustainable Development: From its Beginning to the Contemporary Issues. Zagreb Int. Rev. Econ. Bus. 2018, 21, 67–94. [Google Scholar] [CrossRef]
- Manioudis, M.; Meramveliotakis, G. Broad strokes towards a grand theory in the analysis of sustainable development: A return to the classical political economy. New Polit. Econ. 2022, 27, 866–878. [Google Scholar] [CrossRef]
- Hanckel, B.; Petticrew, M.; Thomas, J.; Green, J. The use of Qualitative Comparative Analysis (QCA) to address causality in complex systems: A systematic review of research on public health interventions. BMC Public Health 2021, 21, 877. [Google Scholar] [CrossRef]
- Greckhamer, T.; Furnari, S.; Fiss, P.C.; Aguilera, R.V. Studying configurations with qualitative comparative analysis: Best practices in strategy and organization research. Strategy Organ. 2018, 16, 482–495. [Google Scholar] [CrossRef]
- DPGA Digital Public Goods Alliance. Digital Public Goods Alliance. 2023. Available online: https://digitalpublicgoods.net/ (accessed on 19 March 2023).
- Bostan, S.; Johnson, O.A.; Jaspersen, L.J.; Randell, R. Contextual Barriers to Implementing Open-Source Electronic Health Record Systems for Low- and Lower-Middle-Income Countries: Scoping Review. J. Med. Internet Res. JMIR 2024, 26, e45242. [Google Scholar] [CrossRef] [PubMed]
- Archer, N.; Lokker, C.; Ghasemaghaei, M.; DiLiberto, D. eHealth Implementation Issues in Low-Resource Countries: Model, Survey, and Analysis of User Experience. J. Med. Internet Res. JMIR 2021, 23, e23715. [Google Scholar] [CrossRef] [PubMed]
- Syzdykova, A.; Malta, A.; Zolfo, M.; Diro, E.; Oliveira, J.L. Open-Source Electronic Health Record Systems for Low-Resource Settings: Systematic Review. JMIR Med. Inform. 2017, 5, e44. [Google Scholar] [CrossRef] [PubMed]
- Jayatissa, P.; Hewapathirana, R. Enhancing Interoperability Among Health Information Systems in Low and Middle-Income Countries: A Review of Challenges and Strategies. Eur. Mod. Stud. J. 2023, 7, 334–340. [Google Scholar] [CrossRef]
- Yılmaz, E. The Importance and Economic Advantages of Using National Open Source Software in Public Institutions. Int. Sci. Vocat. Stud. J. 2024, 8, 202–210. [Google Scholar] [CrossRef]
- Silva, D.G.; Coutinho, C.; Costa, C.J. Factors influencing free and open-source software adoption in developing countries—An empirical study. J. Open Innov. Technol. Mark. Complex. 2023, 9, 100002. [Google Scholar] [CrossRef]
- Dunbar, E.; Elizabeth Olsen, H.; Salomon, E.; Bhatt, S.; Mutuku, R.; Wasunna, B.; Edwards, J.; Kolko, B.; Holeman, I. Towards Responsible Data Practices in Digital Health: A Case Study of an Open Source Community’s Journey. In Proceedings of the Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, Yokohama, Japan, 8–13 May 2021; Association for Computing Machinery: New York, NY, USA, 2021. [Google Scholar] [CrossRef]
- Yeng, P.K.; Diekuu, J.-B.; Abomhara, M.; Elhadj, B.; Yakubu, M.A.; Oppong, I.N.; Odebade, A.; Fauzi, M.A.; Yang, B.; El-Gassar, R. HEALER2: A Framework for Secure Data Lake Towards Healthcare Digital Transformation Efforts in Low and Middle-Income Countries. In Proceedings of the 2023 International Conference on Emerging Trends in Networks and Computer Communications (ETNCC), Windhoek, Namibia, 16–18 August 2023; pp. 1–9. [Google Scholar] [CrossRef]
- Gathecha, G.; Ombiro, O.; Shelden, K.; Stake, A.; Murugami, M.; Mungai, E.; Odhiambo, G.; Maree, E.; Muthusamy, R.; Marimuthu, M.; et al. Integrating digital solutions into national health data systems through public–private collaboration: An early experience of the SPICE platform in Kenya. Digit. Health 2023, 9, 20552076231203936. [Google Scholar] [CrossRef] [PubMed]
- Kozlakidis, Z.; Kealy, J.; Henderson, M.K. Digitization of Healthcare in LMICs: Challenges and Opportunities in Data Governance and Data Infrastructure. In Digitalization of Medicine in Low- and Middle-Income Countries: Paradigm Changes in Healthcare and Biomedical Research; Kozlakidis, Z., Muradyan, A., Sargsyan, K., Eds.; Springer International Publishing: Cham, Switzerland, 2024; pp. 83–90. [Google Scholar] [CrossRef]
# | Acronym | Full Form | Brief Description |
---|---|---|---|
1 | AADHAAR | Aadhaar Digital Identity Platform | A biometric-based digital identity system that assigns a unique 12-digit ID to every Indian resident. It serves as a foundational identity layer and is used for authentication in banking, welfare, and other services. |
2 | CHALO | Chalo Smart Bus Platform | A technology platform aimed at digitising and improving the efficiency of bus transport systems in Indian cities. It provides real-time bus tracking and digital ticketing. |
3 | DHIS2 | District Health Information Software 2 | An open-source health management information system used globally to collect, manage, and analyse health data, often integrated into national health reporting systems. |
4 | DIGIT | Digital Infrastructure for Governance, Impact & Transformation | An open-source platform developed by eGov Foundation to support digital service delivery for urban governance in India, offering modules for property tax, water, and sanitation. |
5 | DIKSHA | Digital Infrastructure for Knowledge Sharing | An open-source platform developed by India’s Ministry of Education to support school education through digital content, teacher training, and assessments. |
6 | Ei ASSET | Educational Initiatives’ Adaptive Learning Platform | An adaptive learning tool that personalises learning paths for students based on their performance, aiming to improve foundational learning outcomes. |
7 | ICDS-CAS | Integrated Child Development Services-Common Application Software | A digital tool designed to support frontline Anganwadi workers by digitising record-keeping, monitoring, and service delivery for child nutrition and early development. |
8 | NCDEX | National Commodity & Derivatives Exchange | A digital agricultural commodities exchange enabling price discovery and risk management for farmers and traders. |
9 | ONDC | Open Network for Digital Commerce | An initiative by the Government of India to create an open, interoperable digital commerce network to break platform monopolies and empower small retailers. |
10 | SORMAS | Surveillance Outbreak Response Management and Analysis System | An open-source digital tool developed in Germany for real-time epidemic and outbreak management, adopted by several LMICs. |
11 | Swiggy | Swiggy | A private-sector food delivery platform that has evolved into a hyperlocal logistics provider for urban consumers. |
12 | UPI | Unified Payments Interface | A real-time payment system developed by the National Payments Corporation of India, enabling instant bank-to-bank transactions via mobile devices. |
13 | X-Road | X-Road | An open-source data exchange layer developed in Estonia that enables secure and standardised communication between public and private information systems. |
# | Digital Interventions (Cases) | Open-Source Code | Regulatory Enablement | Revenue Model in Place | Substantial Scale Achieved | Identifiable Systemic Barrier It Seeks to Eliminate? | Presence of Prerequisites | Sufficient Time for Implementation | Shift in NE (Outcome) |
---|---|---|---|---|---|---|---|---|---|
1 | AADHAAR | 0 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
2 | CHALO | 0 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
3 | DHIS2 | 1 | 1 | 0 | 1 | 1 | 0 | 1 | 0 |
4 | DIGIT | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
5 | DIKSHA | 1 | 1 | 1 | 0 | 0 | 0 | 1 | 0 |
6 | Ei ASSET | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 0 |
7 | ICDS-CAS | 1 | 0 | 0 | 1 | 0 | 1 | 0 | 0 |
8 | NCDEX | 0 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
9 | ONDC | 1 | 1 | 0 | 0 | 0 | 0 | 0 | 0 |
10 | SORMAS | 1 | 1 | 0 | 1 | 1 | 0 | 1 | 0 |
11 | Swiggy | 0 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
12 | UPI | 0 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
13 | X-ROAD | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
# | Open-Source Code | Regulatory Enablement | Revenue Model in Place | Substantial Scale Achieved | Identifiable Systemic Barrier It Seeks to Eliminate? | Presence of Prerequisites | Sufficient Time for Implementation | Number of Cases | Outcome |
---|---|---|---|---|---|---|---|---|---|
1 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 1 | 0 |
2 | 0 | 1 | 1 | 1 | 1 | 1 | 1 | 5 | 1 |
3 | 1 | 0 | 0 | 1 | 0 | 1 | 0 | 1 | 0 |
4 | 1 | 1 | 0 | 0 | 0 | 0 | 0 | 1 | 0 |
5 | 1 | 1 | 0 | 1 | 1 | 0 | 1 | 2 | 0 |
6 | 1 | 1 | 1 | 0 | 0 | 0 | 1 | 1 | 0 |
7 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 2 | 1 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2025 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Mor, N.; Ramasuri, R.; Saraf, D. Diagnosing Structural Change in Digital Interventions: A Configurational Evaluation Framework. Information 2025, 16, 714. https://doi.org/10.3390/info16090714
Mor N, Ramasuri R, Saraf D. Diagnosing Structural Change in Digital Interventions: A Configurational Evaluation Framework. Information. 2025; 16(9):714. https://doi.org/10.3390/info16090714
Chicago/Turabian StyleMor, Nachiket, Ritika Ramasuri, and Divya Saraf. 2025. "Diagnosing Structural Change in Digital Interventions: A Configurational Evaluation Framework" Information 16, no. 9: 714. https://doi.org/10.3390/info16090714
APA StyleMor, N., Ramasuri, R., & Saraf, D. (2025). Diagnosing Structural Change in Digital Interventions: A Configurational Evaluation Framework. Information, 16(9), 714. https://doi.org/10.3390/info16090714