Implementation of Systems Thinking in Public Policy: A Systematic Review

Abstract

The value of systems thinking in public policy is increasingly recognised. However, the extent of its use in decision- and policy-making within the government and civil service remains unclear. This review aims to investigate the state of systems thinking application in public policy. We conducted a systematic search to identify papers published up to June 2022 from various scientific databases. We also searched the grey literature. Publications eligible for inclusion were those that used or discussed systems thinking concepts and tools relating to policy-making. We included 73 papers falling into five categories: case studies, commentary pieces, user perspectives, reviews, and methodological frameworks. Our analysis highlighted the benefits and values of systems thinking that were observed and reflected by researchers and perceived by users in policy-making. It also revealed several challenges of systems thinking implementation in public policy and mapped out recommendations to address each of these challenges. Decision-makers in public policy are yet to utilise the full benefit of systems thinking. Advancing its implementation and sustaining its use in practice requires (i) exploring how to shift decision-makers’ mental models and modify the organisational cultures under which decisions are made and (ii) developing methodological and practical guidance for application and evaluation specific to policy-making.

1. Introduction

Governments increasingly have to deal with complex and uncertain challenges, which are core characteristics of most public policy issues. All the components of such a complex situation are interconnected in multiple, complicated, and hard-to-define ways. Consequently, it is essential to consider it holistically before intending to alter it [1,2]. Traditional analytical tools and problem-solving methods, characterised by linear procedures and the notion of isolated interventions, no longer work, leading to the failure of public policies to achieve the set objectives and deliver the expected and desired outcomes [3,4]. Therefore, the current scale and nature of public policy issues require new approaches to problem-solving.

Systems thinking (ST) offers governments a way of handling this increasing complexity in public policy problems, as it holistically traverses disciplinary boundaries. Multiple approaches to ST exist, each with a different emphasis on the system structure, mental models, and underlying cognition [5,6]. Overall, ST is concerned with the approach of thinking in terms of processes, relationships, and interrelationships in trying to understand the world’s complexity [7,8]. It deals with the unpredictability of the future as the world has become increasingly interconnected across traditional boundaries, generating great uncertainty [6]. Taking an ST approach encourages decision-makers to change the nature and quality of their thought processes regarding complex situations, widen their mental boundaries, and consider issues interconnectedly and holistically [4,9,10,11]. This approach has been shown to help map the dynamics of a system, make the relationships between system components explicit to the people involved, explore how these relationships affect the system’s functioning, and estimate which interventions and policies may lead to better outcomes [12,13]. In addition, public services need to be available continuously and, therefore, cannot be ceased for redesign and restarted. ST has been able to facilitate this difficult transitioning process by navigating the roll-out of new practices while core services are still running [14,15]. ST can also be very useful for managing the complexity and uncertainty of public systems by balancing the focus on the intended outcome and the capability of tackling multiple elements within a system simultaneously.

A plethora of publications advocates the usefulness and applicability of ST to a variety of public policy problems, such as public health [16,17,18,19], social care [20,21,22], and environmental policies [23,24,25,26]. However, the extent of ST application in reality and the factors that drive policy actors’ interest, motivation, and buy-in to an ST approach remain unclear. Research is needed to develop a more systematic understanding of the current use of ST in policy-making, the barriers and challenges in its implementation, and how they may be overcome. In this study, we report on findings from a systematic review aiming to establish (i) how ST has been utilised in policy-making, (ii) what values and benefits of using ST in policy have been reported, (iii) identify barriers and enablers for adopting ST by civil servants and policy-makers, and (iv) recommendations for advancing its implementation in policy-making. It thus provides a systematic understanding that will be helpful in moving from a high interest in systems thinking in policy-making to broader implementation.

2. Materials and Methods

2.1. Information Sources and Search Strategy

We searched Web of Science, Scopus, ProQuest, and Google Scholar from the date of inception until 8 June 2022. We also performed an ad-hoc search on the internet (i.e., the first ten pages of the search results) to discover reports. Results were restricted to peer-reviewed publications and government/technical reports that were written in English, which reduced results by about five per cent. We combined search terms for systems thinking with search terms for public policy as follows:

‘Systems thinking’ OR ‘System thinking’ OR ‘System Dynamics’ OR ‘Soft Operations’ OR ‘Systems Map’ OR ‘Systems Mapping’ OR ‘CLD’ OR ‘Causal Loop Diagram’ OR ‘Causal Map’ OR ‘Causal Diagram’

AND

‘Public sector’ OR ‘Government’ OR ‘Policy-making’ OR ‘Policy-Making’ OR ‘Civil Servants’ OR ‘Public Policy’.

We searched all databases identically. Appendix A details the search strategy for each database. We also searched for relevant citations in the reference lists of the following papers: [13,27,28,29,30,31,32,33,34,35].

2.2. Eligibility Criteria

We included papers that satisfied at least one of the following criteria (i) a primary focus on discussing how ST has been/would be used in policy/public sectors, what values/benefits it offers, or what enablers and barriers there are for its use in policy/public sectors and (ii) a case study/case studies that describe(s) the application of ST in specific policy contexts.

2.3. Exclusion Criteria

We excluded papers that (i) were not related to ST’s use among civil servants and policy-making or (ii) focused on discussing systems theories, described the technical aspect of specific ST tools, or discussed the development of new ST tools or (iii) focused on quantitative results for a policy analysis without describing the participatory process, or (iv) did not discuss the aspects of interest, or (v) only contained abstracts.

2.4. Data Extraction and Analysis

We extracted and analysed data in the included papers using an inductive qualitative content analysis approach to address the objectives of this review. We also looked for the following information to set the scene of a study that contributed to addressing the first objective of this review: publication type, application context, country of research, and adopted ST tools.

2.5. Study Selection

Figure 1 shows the process of identification, screening, and selection using the PRISMA flowcharts [36]. We identified 9453 records from 4 electronic databases and 157 records from the internet. After removing duplicates and screening the publication type, title, and abstract of the remainder, we retrieved 611 records to assess their eligibility. We excluded 544 records as they did not fulfil the inclusion criteria. We also identified an additional six papers via referencing screening of the aforementioned papers. Overall, 73 papers were included and reviewed in detail [12,13,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87].

3. Results

We report first on the publication categories, then the ST tools employed in policy-making, the benefits of ST, challenges, and recommendations for how to address the challenges.

3.1. Publication Categories

The literature identified belonged to five categories (i) research which described a case study (33 papers) undertaking specific ST methods to investigate specific complex systems and problems in policy-making, (ii) commentaries (13 papers) which discussed specific aspects of ST in policy (either broadly or with regard to specific methods or areas), such as specific benefits and values, implications and potential in policy-making, exemplar misuses, and specific challenges for adoption, (iii) research which carried out experiments, interviews, and surveys with policy-makers and stakeholders (9 papers) to explore their perspectives on ST and its applications in policy, (iv) reviews (7 papers) which synthesised the insights from applications of ST (mostly with regard to specific methods and areas), (v) research which proposed a theoretical framework (4 papers) that advanced the application of specific methods in specific policy areas. The other seven papers sit across categories (i) and (ii)—these are commentaries with examples of ST applications.

3.1.1. Case Study

Most literature (i.e., 33 papers) described the use of particular ST methods to study a specific problem in a specific policy context. The primary focus of these studies was on providing insights into the problem and generating solutions using ST methods. They provided little information or justification on the reasons for selecting particular methods or tools. Discussions on the enablers and barriers of implementing ST in policy, users’ opinions and reflections on ST use, and recommendations for best practices were also limited. There were 30 case studies that presented the uses of ST methods in health and 19 that were sustainability focused. There were substantially fewer case studies reported in other policy areas (i.e., 1–3 examples), including agriculture, taxation, social policies, workforce planning, public administration, transportation, tourism, and public-private partnership. Additionally, most case studies considered policy in high-income countries, mainly from the UK, Australia, European countries, and the US, with a limited number of examples from low- and middle-income countries.

3.1.2. Commentary

The papers falling into the commentary category (i.e., 13 papers) extolled the benefits of ST in policy and discussed the challenges of adopting the approach in practice but contained little detail about what tangible, concrete actions could be taken to advance its implementation. The discussions in several papers of this type were bounded by the context of public health or the use of a specific method [12,22,28,30,31,32,44,50,54]. Other papers considered specific aspects of ST, such as the political rationality for providing an environment for democratic practice [38], research and education of ST in public sector practice [27], and contradictions between this approach and political leadership [60].

3.1.3. User Perspective

Nine studies captured the view of policy-makers and stakeholders in the public sector on ST use and explored the opportunity and feasibility of implementing ST in practice and policy. In these studies, researchers adopted semi-structured interviews, surveys, a simulated case approach, and concept mapping. Bensberg, Joyce, and Wilson [40], Morais, Kuhlberg, Ballard, Indvik, Rocha, Denise Marques, Letícia de Oliveira, Gouveia, Amélia Augusta de Lima, and Waleska Teixeira [19], Zucca, Long, Hilton, and McCann [86], El-Jardali, Adam, Ataya, Jamal, and Jaafar [51], and Trochim, Cabrera, Milstein, Gallagher, and Leischow [78] offered insights into participants’ understanding of ST, its potential uses, and constraints on applications within the scope of health policy. Bérard, Cloutier, and Cassivi [41] and Scott, Cavana, and Cameron [71] focused on the perceived usefulness of particular ST methods, namely a system dynamics (SD) based decision support system and SD in group model building, respectively. Lee [62] used a self-evaluated survey to examine the effects of an ST graduate course on public officials’ way of thinking and daily administration behaviour to inform future ST course design. The UK Government Office for Science’s systems thinking case study bank, which is based on interviews carried out by the Government Office for Science with ST practitioners from across the government of their experiences of using ST in a specific project, provided a broader picture of how different ST tools had benefited various public policies and services and what challenges the practitioners had encountered [87].

3.1.4. Review

Seven literature reviews discussed ST applications in specific fields, including the eGovernment field [37], health policy [33,34,77], environmental policy [13], energy systems [66], and tourism [84]. Four reviews included only the studies using SD, and one only covered causal loop diagrams (CLDs). The included reviews focused on discussing the insights into the complexity and dynamics of specific systems and the impacts of policy interventions and put less weight on describing how ST has been applied in policy and what lessons have been learned to improve practice.

3.1.5. Framework

Four papers under the framework category focused on addressing the methodological aspects of using ST in policy. They offered a framework and guidance about how to select appropriate ST methods and apply these methods for specific policy situations. Kelly, Jakeman, Barreteau, Borsuk, ElSawah, Hamilton, Henriksen, Kuikka, Maier, Rizzoli, van Delden, and Voinov [58] et al. (2013) discussed how five common methods, namely, system dynamics (SD), Bayesian networks, coupled component models, agent-based models, and knowledge-based models, could be integrated into the decision- and policy-making processes of environmental assessment and management. Shikhzadeh and Mirzaei [74] presented a systemic SD modelling approach for solving complex and multi-disciplinary problems in public policy. Voinov, Jenni, Gray, Kolagani, Glynn, Bommel, Prell, Zellner, Paolisso, Jordan, Sterling, Schmitt Olabisi, Giabbanelli, Sun, Le Page, Elsawah, BenDor, Hubacek, Laursen, Jetter, Basco-Carrera, Singer, Young, Brunacini, and Smajgl [80] et al. (2018) collected and analysed data on the opinions of experts and modellers to provide a systematic overview, assessment, and categorisation of ST methods in participatory modelling that can assist stakeholders and modellers with their choices. In addition to offering guidance for using ST, Hamilton, Fu, Guillaume, Badham, Elsawah, Gober, Hunt, Iwanaga, Jakeman, Ames, Curtis, Hill, Pierce, and Zare [35] et al. (2019) proposed a framework for measuring the effectiveness of environmental modelling from multiple perspectives.

3.2. ST Tools

Most ST tools reported in the literature were tools for understanding a system and co-designing solutions. There was a shortfall of tools for dialogue and collaboration, as well as tools for assessing progress (i.e., implementing, monitoring, and evaluating). There were 19 publications that discussed the general ST approach without specifying any tools, whereas 54 papers reported on one or several tools. The tools reported are described below.

• Tools for understanding the system

  o

  Tools for seeing things: CLD (21), systems mapping (6), concept mapping (4), systems archetypes (3), rich pictures (2), Bayesian belief network (1), social network analysis (1), influence diagram (1), Sankey diagram (1);

  o

  Tools for thinking strategically: systems archetypes (3), Soft Systems Methodology (SSM) (4), scenario and visioning (2), social network analysis (1), PQR (what, how, why) statement (1);

• Tools for dialogue and collaboration: stakeholder analysis (1), multi-stakeholder dialogue (1), interactive management (1);
• Tools for co-designing solutions: (participatory) SD modelling (17), group model building (14), stock and flow diagram (5).

3.3. Claimed Benefits and Values of ST in Policy

The included papers discussed a number of benefits and values of ST in policy-making, most of which were based on the authors’ observations and reflections. Studies that provided evidence on the benefits and values of ST perceived by users in policy-making were scant.

3.3.1. Understand the Complexity of Systems and Wicked Problems

Most studies (i.e., 44 papers) justified the use of ST in policy based on its capability to provide deep understanding and new insights into systems complexity and wicked problems which characterise the systems within which policy-makers and civil servants work. The ST approach provided methods and tools for more effectively dealing with these types of problems that seemed to frustrate traditional policy approaches [13,33,45,53,56,57,69,78]. The approach helped recognise such complexity and take it into account when designing solutions rather than trying to simplify it and providing instant or definite solutions [27,41,44,46,47,59,63,68,73]. It also helped manage complexity by striking a balance between tackling multiple factors within a system simultaneously and focusing on the intended outcome [25,33,43,59,67]. The act of ST allowed a deeper level of analysis by capturing knowledge systematically and interrogating that knowledge to generate and communicate new insights [12,21,47,52]. Furthermore, six papers specifically claimed that ST tools were useful in visualising and conveying the complexity of a system and the linkages of its components which facilitated system exploration and knowledge integration and consolidation [21,23,26,50,63,87].

3.3.2. Capture Multiple Perspectives and Provide a Holistic View

The second most common value indicated in 31 papers was that ST helped widen the breadth of stakeholder participation and, thus, captured diverse perspectives, interests, objectives, roles, and responsibilities to provide a holistic view of the system, which was otherwise fragmented and scattered amongst stakeholders. An experiment with 40 policy-makers assessing the impact of using SD on their boundaries in a complex situation suggested that using SD had widened the number of fields and disciplines these policy-makers took into account [41]. ST was helpful in practice for a better understanding of where responsibility lies and how an impact pathway made up of a whole system of connections and relationships occurs [27,78]. It also allowed for consideration of other programs and policies across public sectors aiming at designing joined-up interventions and leveraging opportunities for reinforcing feedback from other activities [28,40,87]. ST tools that helped collate multiple actors’ perspectives to produce a whole system picture have been considered more useful than quantitative models of narrower scope in policy areas that span different disciplines [26,47,75].

3.3.3. Facilitate Stakeholder Participation and Collaboration

The value of ST for strengthening stakeholder participation and collaboration was described in 24 papers. Participatory ST approaches facilitated a collaborative mindset as they allowed participants to provide and receive feedback and build on each other’s ideas which other methods, such as surveys and interviews, could not offer [83]. Collaborative processes could also enhance relationships among adversarial groups as they clarify their views and positions and negotiate goals and actions [48]. Furthermore, ST helped advance ‘controversial, contested and value-laden debates’ among stakeholders, thus increasing their constructive participation [34]. The benefit of collaboration and dialogue facilitation among stakeholders built resilience for a system as this allowed effective adaptive co-management by collectively identifying potential leverage points to inform decision- and policy-making [23]. Busy policy-makers in preventive health in Australia explained that they chose to participate in a Systems-Focused Collaboration due to ‘its facilitation of cross-sector connectivity and collaborative work processes with highly respected researchers and counterparts in other policy jurisdictions’ [29]. Embedding ST across the UK Net Zero program facilitated more systemic and effective collaboration [87]. Stave [76] (2002) found that adopting ST to engage public participants in environmental decisions was quicker than similar stakeholder advisory processes as it provided an explicit structure for problem scoping and solution identification.

3.3.4. Shift Thinking and Mental Models

Shifting thinking and mental models as a value of ST was discussed in 20 studies. Adopting ST encouraged a rethinking of organisations and system issues with an emphasis on understanding the whole system rather than on individual components [33,74]. This approach helped shift from a single person or institution making decisions to a more holistic approach encompassing diverse views and interests [40,43,48]. The traditional view considered decision- and policy-making as a linear and mechanistic process concerning better approaches to command and control. ST enabled them to appreciate government as a complex adaptive system and understand that policy-making is a process of learning and institutions need to be learning systems [13]. Stave [76] (2002) agreed with Forrester [88] (1987) that the best value for an SD model is the capability to change people’s mental models, making them perceive things differently rather than simply confirming what they already knew. In-depth interviews with policy-makers in preventive health in Australia revealed that ST was the key contributor to creating a conceptual shift among these policy-makers, affecting multiple aspects of their work [29]. They found that their systems perspectives had evolved and, thereby, how they think and talk about health problems and contexts, policy objectives and practices, and approaches to design solutions had changed. Several studies, which explored the use of group model building in a New Zealand public service context, showed that this approach led to mental model change and that this change was enduring [45,70,71]. The lead of a project aiming to improve the performance of a UK government directory claimed that ST is great for getting people into a holistic mindset to start thinking about complex problems and how to tackle them [87].

3.3.5. Act as a Learning and Decision-Making Support Tool

Twenty studies explained that they used ST as a learning- and decision-support tool for examining the impacts of potential policy alternatives and what-if scenarios that would not have been possible to experiment with within real public systems without service disruption. As public services must be available at all times and cannot be intermittently turned off for redesigning and testing, experimenting with policies on real systems in the public sector is extremely difficult [67]. Four of these papers suggested that embedding ST in policy allowed for continuous improvement of the knowledge base and, therefore, enhanced the adaptive management capability under uncertain and variable conditions [12,24,47,58,76]. Such incremental learning increased the possibility that decision-makers would design a policy that reached their intended goals and priorities [75]. Egan and McGill [12] and the Government Office for Science [87] suggested that ST was also useful for understanding the long-term impact of a public policy and offered an adaptive approach to impacting the system over the long term as new challenges emerge. Many public policies, such as upstream preventive health policies, are often expensive, take years to implement, impact large populations, and are impossible to undo. ST provided a quick and safe approach to measuring the potential long-term impacts of such policies. Additionally, wicked problems, such as population health problems, were widely recognised to persist despite repeated efforts to resolve them. Although specific interventions and policies might influence the system for a certain period, the system would continue to adapt, resulting in further issues. For example, the importance of the timeline of the net zero policy was emphasised as science, technology, and economics that affect the whole system would significantly change in the next 30 years [87].

3.3.6. Achieve a Shared, Collective Understanding and Enhance Problem-Ownership, Consensus, and Commitment

Less commonly discussed, 11 studies indicated that ST helped a diverse set of stakeholders across organisational and disciplinary boundaries to align their mental models, thereby arriving at a shared understanding of a problem, its underlying cause, and the challenges faced [23,34,50,61,64,65,70,71,79,80,87]. By creating shared knowledge and promoting more inclusive decision- and policy-making processes, the approach helped achieve ownership of problems, consensus on a course of action, and commitment to the expected outcomes [34,65,71,73,76]. Such consensus could improve the usefulness and value of proposed solutions to all involved, which helps reduce problems that may occur after policy implementation, thereby facilitating wider policy adoption and successful implementation and avoiding wasting time and resources in the short and long run. Employees in the public sector in New Zealand ranked consensus as a significant outcome of using ST in decision-making in a questionnaire [71]. They considered coming to an agreement a success in many cases. They also ranked commitment to supporting and implementing conclusions as the most important outcome. They explained that an agreed conclusion reached by voting had previously led to low commitment by those who preferred the unchosen solutions. Several case studies in the UK Government ST case study banks also claimed that the systems mapping workshops were highly effective in creating a shared understanding of the system by challenging participants’ perspectives [87].

3.3.7. Impact Policy and Practice

Evidence on the impacts of ST on policy and practice was fragmented, ranging from affecting policy decisions to just reporting its use as described in five papers [16,19,22,29,40]. As an example of the former, an SD model that shed light on several complex and challenging problems in health and social care in the UK led to a rethinking of the proposed legislation by national agencies and the upper house of parliament [22]. These agencies decided to invest funding for social services to address capacity issues and postpone the ‘fines’ policy for a year as a result of this work. Some other studies merely reported the uptake and implementation of ST in practice. In the study of Morais, Kuhlberg, Ballard, Indvik, Rocha, Denise Marques, Letícia de Oliveira, Gouveia, Amélia Augusta de Lima, and Waleska Teixeira [19], participants in SD workshops stated that they integrated some concepts around feedback loops within their work. Policy-makers participating in a research collaboration claimed that using ST in policy work contributed to developing practical methodologies for policy and intervention design, scaling up, implementation, and evaluation [29]. Staff within a national public health agency in England reported using systems mapping in their work with localities on co-occurring mental health and substance misuse conditions, adverse childhood experiences, and a new strategic approach to reducing health inequalities [77].

3.3.8. Useful in Dealing with a Lack of Data

Five studies claimed that ST was particularly useful in dealing with the lack of empirical data, particularly when the focus was on understanding a system structure and its behaviour [20,23,26,46,87]. The complexity of systems that public policies need to handle and the diversity of stakeholders involved made it hard to generate and obtain sufficient empirical data that capture all of the dynamics of the system. This challenge led to the need in policy-making to rely on a combination of quantitative and qualitative data from different sources. ST contributed to addressing this issue by creating opportunities to collate data across multiple government levels and stakeholder groups and handled this data mixture quite well [20,23,87]. For example, participatory systems mapping was successfully used in several policy domains in the UK government [26]. The method allowed stakeholders to collaboratively construct an illustrative causal map of a specific system with interconnected components within a workshop setting. Participants could rapidly produce maps covering a broad scope of the system and including factors and relationships that were considered important to them without requiring empirical data.

3.4. Challenges of Implementing ST in Practice

The implementation of ST poses several challenges to people used to working in a policy-making context, which are described below and summarised in the first column of

Table 1. Challenges for applying ST in policy and recommendations to address those challenges.

Challenges	Recommendations
Conceptualisation, language, and communication
Intrinsic reluctance to new ways of thinking Unfamiliarity with and misconception about ST concepts, tools, and values Confusing concepts and terminology Cross-discipline miscommunication Lack of recognition of the relevance of ST and policy-making	Clarification on ST concepts and values in policy contexts Clarification on misleading impressions about ST Investing in communication—having boundary people on both sides of the academic/policy divide to facilitate communication Developing shared ST language in policy contexts to facilitate interdisciplinary communication
Competence, methodology, and practice
Lack of competence in applying ST Lack of technical and practical support and guidance Uncertainty about the suitability and effectiveness of certain ST tools Difficulty in generalising lessons from case studies Lack of confidence in the intrinsic validity of ST	Capacity building Create opportunities for professional development in using ST Access to practical guidance, technical training, and support Assess organisational readiness to apply ST Promoting integrative and interactive learning of ST for individual competence development Integrate ST into other disciplines’ education Provide skills and familiarity in working in interdisciplinary teams Adopt a non-prescriptive approach to ST training and learning Draw lessons from experiences and real-world case studies
Stakeholder engagement
Varied quality in participation among diverse stakeholders and over time Fear of disruption to work routines Difficulty in managing stakeholder equality in participation and power dynamics Under-representation of specific stakeholder groups Lack of trust between stakeholders	Identification of the right stakeholders at the start Obtaining supportive ST champions for initial stakeholder engagement Clarification on links between ST outcomes and policy-making Adopting a learning-by-doing approach to maintain engagement Supporting dynamic and diverse networks Addressing ethical dimensions of managing stakeholder participation Modification of the participatory process
Time and resources
Lack of time and resources for engaging stakeholders and applying ST Lack of time and budget for evaluating the effectiveness of ST	Revisiting funding categories and expanding cross-discipline funding Providing incentives such as additional staff salaries or core organisational funding Allocating funding to cover costs for capacity building and evaluation
Political, structural, and operational aspects
Challenging political contexts Organisational inertia and rigidity Departmentalism Lack of organisational incentive to implement ST	Political endorsement and governmental reform Establishing a central entity at the government level to promote ST use Emphasis on ST adoption at the policy leadership level Assist policy-makers to work collaboratively with researchers and other stakeholders Exploring the impacts of characteristics of decision-makers and organisational culture on ST uptake
Evaluation and evidence of impact
Lack of guidance and expertise for evaluation Lack of funding for evaluation Lack of concrete evidence for policy impact Reluctance to scrutinise the work Subjectivity in evaluation	Identifying evaluation criteria for ST use in policy Set up evaluation strategies at the start Develop both summative and formative evaluation strategies

.

3.4.1. Conceptualisation, Language, and Communication Challenges

Challenges around conceptualisation, language, and communication were raised in 28 papers. First, people tended to hold onto established mental models and refused to embrace new ways of thinking, leading to challenges in using and even the rejection of ST [24,29,30,76,87]. Second, policy-makers’ unfamiliarity with and misconception about ST concepts, methods, and values could hamper their interest and acceptance of ST [34,35,40,51,62,65,73,81,87]. The concepts and terminology of ST can be confusing, ambiguous, and even alienating to individuals who contributed to and benefited from its use in tackling complex systems [54,86,87]. The lack of familiarity of stakeholders with ST posed significant challenges for planning ST participatory activities in the short allotted time periods and affected their engagement [65,81]. Atkinson, Wells, Page, Dominello, Haines, and Wilson [34] found that health policy-makers were unfamiliar with ST concepts and what ST methods can offer, thereby causing a lack of confidence in these methods and poor uptake. This study also revealed that some practitioners found ST concepts confusing and poorly justified; thus, they continued to make decisions based on their past experiences instead of employing this approach. Additionally, the different languages that people use to talk about the same system across the government added to the confusion about ST concepts [87]. Potential misunderstanding might also result from difficulties in finding a common language to communicate across disciplines and between the expert and layperson [48]. Third, the lack of recognition of the relevance and value of the approach among policy-makers was one of the most commonly perceived constraints of applying it in policy and practice [51,52]. Policy-makers were not motivated to adopt ST as they often expected policy solutions with a simple and more straightforward appearance than what ST generated [30]. Policy-makers also viewed ST as explanatory but not action-oriented; thus, it did not fit the purpose of policy-making [30].

3.4.2. Lack of Competence and Methodological and Practical Challenges

Lack of competence and methodological and practical challenges reported in 20 papers were likely to cause poor uptake in the public sector and government. Practitioners claimed that they did not know how to apply ST as there was little guidance for systems practice and a lack of practitioners delivering programs [29,34,40,51,71]. Participants in ST workshops could get hung up on the methods and tools and thus were distracted from the purpose of the workshops [87]. They could also interpret the ST outputs (e.g., causal loop maps) differently, dissipating the efforts of creating them. Furthermore, the variety of ST tools also created challenges for practitioners as they encountered uncertainty about the suitability and effectiveness of certain tools for their projects [80,82]. Existing studies provided little justification or discussion on the reasons for using particular tools. It seemed that personal experience, skills, preferences, and institutional pressures influenced their judgments. Furthermore, generalising the experience of ST practice learned from case studies would be difficult, as such experience was arguably bounded by social and geopolitical contexts [49,55,70,85]. Finally, policy-makers were concerned about the intrinsic validity of the approach and how they could be confident with the insights and results produced by applying ST. They perceived a lack of well-functioning data systems and the consequent paucity of good quality data would hinder the use of ST in designing and examining alternative policy interventions and validating the results of such examination [18,45,48,51].

3.4.3. Stakeholder-Related Issues

Difficulty in engaging, coordinating, and managing stakeholders in the application of ST received equal attention as the challenges around competence and methodology (i.e., 20 papers). Stakeholders were unlikely to engage actively during the entire process [23,51]. Stakeholders who denied that a problem existed or could not immediately see why the problem was relevant to them also created a barrier to involving them in the solution [44,87]. The voluntary participation of some stakeholder groups could hinder the level of commitment devoted, especially for long-term projects [48,76]. The diversity of stakeholders’ backgrounds, knowledge, experiences, interests, and priorities was also likely to influence the quality of their participation in the process [48,76,81]. The fear of dramatic disruption to work routines and the reluctance to leave urgent operational problems behind to address future challenges were also barriers to stakeholder engagement.

Another challenge was to address participation inequality and difficulty in achieving open and meaningful debate caused by hierarchical differences between stakeholders from different levels of decision-making [23,28,38,39,48,50,51,56,81]. Although ST could make significant progress towards including diverse views and information from different stakeholder groups, some specific groups and their views might be left out or under-presented [23,25,39,48,50,81]. For example, several key social science disciplines remained underrepresented in most participatory modelling practices in health policies [50]. Lack of trust or the presence of some mistrust among stakeholders could undermine the efforts of implementing ST and its impacts [48]. Bérard, Cloutier, and Cassivi [41] also found that using ST had no impact on strengthening the depth of stakeholder participation despite its effect on diversifying the involvement of stakeholders.

3.4.4. Time and Resource Constraints

Time- and resource-related constraints for engaging stakeholders and carrying out collaborative, participatory, and reflective activities using ST were reported in 17 papers [17,23,25,28,29,32,35,37,48,51,61,63,65,73,80,81,87]. The nature of a project using ST requires stakeholders to dedicate time to engage. Trying to get everybody in the same room at the same time for meetings and workshops is a classic problem [87]. A survey that was carried out with a purposive sample of 62 stakeholders in health systems across ten countries in the Eastern Mediterranean Region indicated that the costliness of adopting ST and lack of essential funding was the most commonly perceived constraint for its application in policy and practice [51].

3.4.5. Political, Structural, and Operational Barriers within the Government and Public Sector

Political, structural, and operational barriers to ST adoption within the government and public sector received less attention than the above challenges and were discussed in 11 papers [28,32,39,51,55,60,62,71,72,76,87]. A challenging political context, which included high turnover and lack of accountability at the government level, competing political interests, and political instability, was considered an important factor that might hinder efforts to build ST capacity and awareness among policy-makers [51]. Political and public pressure for a quick solution forced the ST work to be expedient [76]. ST that existed in political decision-making was filtered before it was turned into tangible policies, thereby explaining why better systems knowledge did not automatically transform into better policies and practices [55,60].

Public officials believed that organisational inertia and departmentalism in the government and public sector led to the failure of applying ST in practice to improve understanding of a system and its complex problems [62]. There was a lack of organisational incentive to implement ST. Departmentalism was an obstacle to adopting ST as it was one of the main causes of conflict among different parts of the government that challenged cooperation and collaboration. Some departments with rigid and prescriptive approaches to decision-making could be a barrier to implementing ST [55]. It also posed a hurdle to managing the impact of transformation and changes in one part of the system on other parts [87].

3.4.6. Lack of Evaluation and Evidence of Impact

Ten papers explained that evaluating the impacts of ST in a project was challenging due to a lack of standard procedures and guidance in conducting and interpreting evaluations, the limited availability of expertise in evaluation, and a lack of funding [12,19,29,30,31,33,34,35,40,44,80]. There was a gap in systematically relating the attributes of ST application, context, and practice to success and policy impact [35]. This led to a paucity of evidence for the impacts of ST on policy and practice [33]. Policy-makers interviewed by Haynes, Garvey, Davidson, and Milat [29] also emphasised this frustration that evidence for the benefit of ST on providing rich and detailed illustrations of problems outweighed hard data on impact. One interviewee highlighted: ’Telling treasury and finance and ministers how complex things are is actually not that useful’. Moreover, evaluating the effectiveness of ST applications was rarely budgeted as projects were generally funded up to the point of delivering their outputs, such as final CLDs and models, and sometimes training for using these models [35].

Reluctance to evaluate and bias were other challenges [19,35,80]. Where ST applications were not well received, there was reluctance to scrutinise the work or document the failures that, indeed, can be important and useful in advancing ST knowledge and best practice [35,80]. Subjective perceptions of the outcomes of ST applications among stakeholder groups could also affect evaluation processes, as what may be a success for one group can become a failure for another [80]. Self-reported responses to interview questions regarding the continued adoption of ST tools in participants’ current work were potentially biased indicators of actual use [19].

3.5. Recommendations for Enhancing and Sustaining ST Uses in Practice

The literature not only discussed the challenges of applying ST concepts and tools in policy, but it also formulated recommendations. A mapping of challenges, together with recommendations relevant to each challenge, can be seen in Table 1. In the subsequent sections, we will go through each recommendation.

3.5.1. Clarify ST Concepts and Values in Policy Contexts and Invest in Communication

Although the challenges around ST language were most commonly reported, a small number of papers (i.e., 14 papers) provided recommendations at a high level [12,23,26,27,29,30,31,34,48,51,66,69,78,87]. Clear messages about ST concepts in policy contexts and investment in communication were the essential first steps for promoting ST as a robust tool to inform decisions and policies [23,34]. People who had used ST in the past could not understand that they were already using its systems [87]. It is crucial for people to understand that applying ST is not something that they would do once and then return to their ‘normal’ way of thinking and doing things, but they need a shift in their perspective. The misleading impression that all ST could do was produce a tangled-looking map and confirm the complexity of problems in policy also needs to be corrected [12]. Having people on both sides of the so-called academic/policy divide could facilitate clear communication between academics and decision-/policy-makers [27,48]. The language used to present ST to people was also important so as not to put them off [87]. Developing widely accepted definitions of complexity and ST concepts in policy would also facilitate interdisciplinary communications among the people involved [66,69].

3.5.2. Build Capacity

Building capacity across the public sector—that is, people with the capability to consider systemic effects of actions and policies—with knowledge of and expertise in ST was crucial for advancing its implementation in policy and supporting the uptake of ST by civil servants and policy-makers [29,30,34,39,59,67]. Ideally, capacity-building efforts discussed in 29 papers should be collaboratively designed and delivered by both researchers and policy leaders rather than being research-led to foster learning across diverse groups. First, civil servants should have more opportunities to access tailored professional development to establish procedures and routines for ST practice within their work [24,40,51,55]. Second, they need more practical guidance and access to technical training and support [18,33]. In addition, technical support could be a partnership with ST experts, using them as facilitators for structuring group deliberations, using ST tools, fostering reflection and learning, and helping develop policy action agendas [20,24,33,65,76,87]. Finally, assessing the organisational readiness to apply ST concepts and tools would provide indicators of areas for capacity-building [55].

As part of capacity-building efforts, the literature provided more specific recommendations on promoting integrative and interactive learning of ST for individual competence development. Integrating ST into other disciplines’ training and education curricula would help familiarise civil servants and policy-makers with its concepts and values, gain their acceptance and buy-in and enhance their capability and expertise to implement ST tools in their work [28,50,59,68]. The content of such ST training and education should be customised to the disciplines into which it is integrated [59,86]. It should focus on complex problem classification and well-established ST tools with a proven track record in these disciplines. Interactive learning with opportunities for critical thinking exercises, practising ST use, and reflecting on their experience and observation of ST use would be engaging and valuable [37,68]. In addition to teaching systems theories and methods, equipping learners with skills and familiarity in working comfortably in interdisciplinary teams would enhance the practicality of adopting ST in their work [50].

A non-prescriptive approach to ST training and learning creates a flexible and adaptable style of applying ST to suit individual learning preferences and organisational contexts where they work [55]. This approach would encourage learners to adopt ST innovatively in their work instead of following rigid plans and guidance. It might, in turn, create more opportunities for collaboration as they could adapt to meet the needs and working culture of partner organisations.

Learning from real-world case studies could motivate people to adopt ST in their work [62]. Real-world case studies were useful in showcasing the values and benefits of policy utility of ST, how it had been used, what had worked well and what had not worked so well, and how it had impacted outcomes [29,78]. Future work should synthesise lessons drawn from relevant experiences and case studies to develop guidance specific for civil servants and policy-makers undertaking system change [43,67].

3.5.3. Engage Stakeholders and Maintain their Engagement

A third of the included papers suggested that stakeholder engagement was vital to ensure the correct identification of important organisational concerns that would mobilise and sustain efforts to address these concerns. Identifying and bringing in stakeholders right at the start of a project could ensure that ST activities were being carried out within the policy environment and, thus, the relevance of the insights it generated to policy-making [21,87]. Obtaining support from ST champions and highly respected people in the field was useful to corral stakeholder attendance for initial engagement [76,87]. For example, the Defence Science and Technology Laboratory of the UK government reported that their customers’ initial stakeholder engagement in systems thinking led to greater stakeholder buy-in and contextual understanding [87]. In addition, making the link between the outcomes of using ST and policy-making explicit would help ensure that stakeholders perceive the legitimacy of such outcomes [39]. The learning-by-doing approach was also suggested to maintain participants’ engagement [18].

Supporting dynamic and diverse networks would contribute to sustaining ST use. Facilitating individual talents and team dynamics was essential for optimising ST application in various contexts [20]. Clarifying the link of ST to decision- and policy-makers’ daily practice would encourage collaboration between researchers and policy actors [78]. Including social research expertise could help address the ethical dimensions of managing stakeholder participation, such as power relations, gender, and equity issues [50]. Modifying the participatory process, such as shorter workshop sessions, smaller breakout groups, anonymous contributions, and supported virtual participation, would help overcome the barriers to participation that a virtual environment could create [26,38,48,83].

3.5.4. Expand Funding

Although lack of resources was reported as one of the most common barriers to ST implementation across government directorates, only three papers discussed strategies to address this issue [51,55,78]. Revisiting funding categories and expanding cross-discipline funding were suggested to support a system-based view of financing and supporting collaborations and partnerships that span various disciplines and perspectives with a system’s focus [51,78]. How funding could be used to support ST adoption was discussed in a relatively broad sense: (i) providing incentives such as additional staff salaries or core organisational funding to address political and social barriers such as lack of an enabling environment or political support, and encourage ST adoption in policy-making [55,78] and (ii) cover costs for capacity building and evaluation which are often required as part of funding criteria for projects but not funded [51].

3.5.5. Consider Political Endorsement and Governmental Reform and Enhance Policy Leaders’ Buy-In and Support

In nine papers covering this aspect, there was an emphasis on the importance of political endorsement and adoption of ST at the policy leadership level to enhance the implementation of this approach in practice across the civil service [28,40,41,49,51,54,55,59,76]. This would help build a safe environment for critical reflection and respectful dialogue among various stakeholder groups which are part of implementing ST [59]. The emphasis on the central role of government in advancing the implementation of ST concepts and tools would require a push for governmental reform and establishing a central entity at the government level to promote ST use and coordinate related activities [51]. In addition, the willingness of policy leaders to take a risk with a novel approach such as ST and their support would offer the opportunity for adopting ST and prompt civil servants to participate in the process [30,40,51,59,76]. Engaging civil servants at an operational level without buy-in from high-level policy-makers may be a missed opportunity for enacting policy changes [28]. To gain policy leaders’ buy-in and support, it was important to know how to assist them in working collaboratively with researchers and other stakeholders toward understanding problems and co-producing policy solutions [54]. The different characteristics of decision-makers, along with organisational factors and how such characteristics affect ST uptake, should also be explored [41,60].

3.5.6. Develop Evaluation Strategies

Identifying evaluation criteria for the use of ST in policy and developing clear evaluation strategies would help highlight its values and impacts as well as reveal areas for improvement. However, this was rarely discussed in the existing literature, and therefore, little insight was gained [31,40]. Bensberg, Joyce, and Wilson [40] suggested setting up evaluation plans and tools at the start of a project. Decision-makers might also require both summative and formative evaluations to determine whether the use of ST helped achieve the project objectives and what they learned from the process [31].

Thus, overall, clear communication of ST terminology together with capacity-building, appropriate resources, clear leadership support, stakeholder support, evaluation, and a focus on a change in thinking is recommended to advance the implementation of systems thinking in policy-making.

4. Discussion

Our systematic review is the first one, to our knowledge, that broadly captures various aspects of ST implementation in public policy. Previous reviews have focused on a specific policy area and ST method and the results of the effectiveness of specific policy interventions [13,33,34,37,66,77,84]. Our review’s findings suggest that public policy is yet to take full advantage and realise the potential of ST and that there are several challenges to advancing its uptake and implementation despite a growing interest in its use for tackling complex problems in policy-making. Less than half of the included papers describe the application of ST to real-world problems. Most cases report using CLDs and participatory SD to explore health systems and environmental policies in high-income countries, such as the UK, Australia, European countries, and the US. There is a shortfall in the knowledge, discussion, and evidence around sustaining quality use of ST in public policy, user experiences, expectations, and methodological and practical improvement. The literature suggests that reasons for poor uptake in policy-making and the public sector include unfamiliarity among decision- and policy-makers with ST concepts and methods and what values the approach can offer in policy contexts, lack of locally available capacity to apply ST, lack of confidence in its rigour, lack of acceptance and buy-in among civil servants and policy-makers, time and resource constraints, political, structural, and operational barriers within the government and public sector, and lack of evidence on its usefulness and contribution. We focus on discussing areas that can be progressed on the basis of our review findings.

The reported benefits and successful use of ST in public policy appear to be consistent with the advocated values of ST (e.g., [61]). However, there is a lack of rigorous investigation of the perspectives of civil servants and policy-makers on the usefulness of ST concepts and tools for policy analysis and stakeholder engagement. Also absent from the literature is an assessment of their learning and development related to adopting ST and their commitment to findings. In addition, unsuccessful cases of adopting ST in policy are likely to be underreported, and people who do not buy into the approach are likely not to participate in relevant ST activities and interviews. The case studies are all described by researchers and policy analysts who are responsible for carrying out ST-related activities and likely to have an interest in advocating the use of ST. These limitations highlight the potential bias in drawing conclusions from the successful applications of ST for policy-making.

Our review’s findings suggest that there is a need to clarify ST concepts and tools, their relevance, and their values in policy contexts and develop practical guidance and training for civil servants. This will help them become more familiar with the approach and enhance their motivation and capability to apply it. Although ST can serve a variety of objectives, civil servants may prioritise these objectives differently in ST application. In addition, they may approach ST from different perspectives and levels of prior understanding. Hence, guidance and training need to be customised to meet the needs of civil servants. The Government Office for Science—part of the UK Government—has recently published a beta version of systems thinking for civil servants, emphasising the need to develop and test accessible and easy-to-implement guidance in policy-making [89].

There are shortcomings in the methodological aspects of implementing ST in policy that are worth further research. Understanding how civil servants can select an appropriate set of ST tools that fit the purpose of their project and how to combine ST tools with other systems approaches for complementation is lacking. There is a lack of sufficient justification for using a particular ST tool and comparisons with other candidate tools in most case studies. CLDs and SD are the tools reported most frequently in the literature, though, relatively, they are not easy tools for civil servants to begin with. As there are a variety of ST tools and each ST tool serves a different purpose, users need to deploy them with intentionality and discernment as part of best practice. This is a great challenge for civil servants without comprehensive guidance, especially for those who are at the beginning of their ST journey. In addition, those with more experience with ST may wish to explore how to combine this approach with other systems approaches and the potential of such mixed methods. Combining methods may help tackle complex problems that would be difficult to address with a single tool. Furthermore, approaches to building confidence among policy-makers in ST and the insights it produces are missing from the discussion of existing literature.

Understanding the characteristics of decision-makers and organisational factors that encourage and inhibit interest and uptake of ST requires further exploration. It is undeniably crucial to explore how decision-makers with different attributes acquire and process information, the organisational culture in which decisions occur, and how ST could help structure and guide their cognitive processes in decision-making [41,60]. Gettinger et al. [90] argued that the characteristics of users, such as their decision-making styles, determine their satisfaction with a decision-support system. Understanding this would help develop effective, targeted interventions and strategies to promote ST adoption.

An area emerging in the review that is worth further investigation is the evaluation and benchmarking of good practices of ST application in policy-making. Research and implementation must go beyond assessing whether civil servants and policy-makers use ST concepts and methods in practice and consider a multi-dimensional evaluation of the quality and efficacy of ST implementation. This requires the development of a guiding evaluation framework that captures different perspectives of a successful ST application, including but not limited to (i) innovative and fit-for-purpose use of ST methods and methodological and scientific rigour, (ii) beneficial outcomes on a variety of criteria such as sustainability and equity, and (iii) confidence building, learning from the application process, conceptual shifts, policy and systems changes, and (iv) commitment to implementing ST findings agreed by stakeholders. The insight gained from evaluation activities will, in return, help inform the best practice of applying ST in policy-making and advance the development of ST concepts. It will also help make the values of ST use explicit to potential users in the civil service and policy-makers.

5. Conclusions

This paper reported on our systematic review of ST literature in public policy and revealed the benefits, challenges, and lessons of implementing it in practice, and highlighted gaps for further exploration and improvement. It showed that there is a growing interest in how ST concepts and tools can aid decision- and policy-making in the government and public sector. However, public policy has yet to exploit the full potential of ST and its full range of methods. In addition, there are several challenges in implementing ST in practice and sustaining its use. Areas that can be progressed include a conceptual shift among civil servants and policy-makers, impacts of decision-making styles and organisational cultures on ST uptake and use, development of methodological and practical guidance specific to policy-making, and the development of an evaluation framework and benchmark of best practice.

Limitations to our study are related to the keywords used in the literature search. CLDs and SD were among the most-used tools, but these, together with ‘soft operations’, were also specific keywords that we searched for. While it conforms to the tools most frequently used in the Government Office for Science case study bank [87], a more explicit keyword inclusion of the different soft operations research, problem-structuring, and critical systems approaches might have changed results somewhat. In addition, we deliberately focused on the peer-reviewed literature, which excluded useful books and book chapters such as those by Seddon [91] or Andersen, Rich, and MacDonald [92] on ST or SD in the public sector.