A Scoping Review of Empirical Evidence on (Digital) Public Services Co-Creation

: The public sector is facing signiﬁcant challenges regarding public services provision, including declination of users’ trust and limited resources. An alternative approach to traditional public service provision with the potential to address these challenges is the co-creation of public services. Co-creation promises to foster innovative solutions to provide high-quality services that respond to users’ needs. Considering this background, we aim at critically exploring public service co-creation via a scoping review, employing the PRISMA-ScR method. Our review focuses on 25 empirical studies out of 75 analyzed articles that examine the implementation of co-creation of (digital) public services and investigates how the empirical literature portrays the concept of public service co-creation. Our ﬁndings primarily suggest that co-creation can be implemented in a wide range of sectors and settings, to improve public services and to foster innovation, throughout the whole public service cycle, using a variety of digital, analog and hybrid co-creation tools and strategies. Yet, our review has also shown that there is still an implementation gap that needs to be bridged between knowing and doing in the context of public services co-creation in a digital setting.


Introduction
Currently, the public sector is facing significant challenges regarding public services provision. On the one hand, trust from users is deteriorating. On the other hand, the public sector needs to provide better services with fewer resources. Users require accessible, user-friendly, personalized, and integrated public services that match their needs and circumstances. In light of these challenges, the active engagement of public service users as co-creators promises to foster innovative solutions via joint experiences, resources, and skills (Torfing et al. 2019;Nabatchi et al. 2017). These collaborative efforts are expected to provide not only higher-quality services but also to deliver responsive services that meet users' needs (O'Brien et al. 2016). Moreover, co-creation can be a starting point for digital transformation and innovation in the public sector (Loeffler 2021) while supporting user-centric and inclusive services.
Nevertheless, research on how co-creation works and on how the implementation of public service co-creation processes can realize the promised outcomes in digital contexts remains limited (Almeida et al. 2018;Sicilia et al. 2019; see also Ansell and Torfing 2021). On the one hand, it seems that the increasing interest in public services co-creation is not translated into its implementation (Almeida et al. 2018). On the other hand, empirical research that discusses the actual implementation of co-creation initiatives remains scarce. In the same line, Sicilia et al. (2019, p. 238) observed that empirical research "is lagging behind conceptual, theoretical, and descriptive research". Considering these observations, the objective of this scoping review is to explore the implementation of public services Table 1. Dimensions of analysis.

Dimensions Main References
Public service setting and policy field (Nabatchi et al. 2017) Location of co-creation initiatives (Clifton et al. 2020) Phases of public service cycle (Nabatchi et al. 2017) ) Aims of co-creation initiatives (Voorberg et al. 2015) Co-creators actors (Nabatchi et al. 2017) Tools and strategies to co-create (Almeida et al. 2018) Benefits of public service co-creation (Nabatchi et al. 2017) (Voorberg et al. 2015) Challenges of public service co-creation (Voorberg et al. 2015) Most of the previous systematic literature reviews have provided a comprehensive overview of the different aspects of co-creation, such as co-creation definition (Almeida et al. 2018), barriers and enablers (Amorim Lopes and Alves 2020; Clifton et al. 2020;Sicilia et al. 2019;Voorberg et al. 2015), and theoretical backgrounds (Jukić et al. 2019), among others. Nevertheless, our scoping review aims to explore the specific dimensions related to the implementation of the co-creation of public services in digital settings with a focus on empirical evidence (see Table 1). By doing this, our review advances the empirical understanding of the implementation of (digital) public services co-creation. As claimed by Leino and Puumala (2021), it is crucial to examine the actual practices of co-creation and to realize its participatory potential. In that way, we aim at bridging the gap between knowing and doing, which limits the potential impact of co-creation. In other words, our review differentiates from previous studies by exploring beyond the 'what' and 'why' dimensions of public service co-creation.
The remainder of this article is structured as follows. Section 2 presents the concept of co-creation, including its phases; Section 3 describes the research strategy; Section 4 presents the results of the scoping review and Section 5 discusses our results. Finally, Section 6 concludes the article and suggests avenues for future research.

Understanding Co-Creation
Co-creation is increasingly viewed as a key tool of governance for public sector practitioners and has been receiving growing attention from scholars (Torfing et al. 2019;Nabatchi et al. 2017;Voorberg et al. 2015;EU 2019). It represents an innovative approach for public organizations (Casiano Flores et al. 2021). This approach supports the delivery of high-quality services in a context of constrained resources through the involvement of stakeholders in the design, delivery and evaluation of public services (Fugini and Teimourikia 2016). Particularly in the context of digital transformation, co-creation is identified as an appropriate strategy for transforming services based on the adoption of digital technologies (Dugstad et al. 2019).
In the academic literature, the concept of co-creation is often used as a synonym or as being closely interrelated to the concept of co-production (Voorberg et al. 2015;Brandsen et al. 2018). In general, co-production is understood to be "a relationship between a paid employee of an organization and (groups of) individual citizens that requires a direct and active contribution from these citizens to the work of the organization" Adm. Sci. 2021, 11, 130 3 of 21 (Brandsen and Honingh 2016, p. 431). Meanwhile, co-creation of public services can be defined as "a process through which two or more public and private actors solve a shared problem, challenge, or task through a constructive exchange of different kinds of knowledge, resources, competences, and ideas that enhance the production of public value [ . . . ], or services." (Torfing et al. 2019, p. 55).
Based on the above definitions, we argue that co-creation in the public sector can provide a more holistic and inclusive picture than co-production. In this line, Torfing et al. (2019) emphasize that co-production is a phenomenon that is stretched to describe the phases of the public service cycle and does not provide insights into the new and broader interactions between the public sector and society that co-creation encompasses, while co-creation understands that all stakeholders can provide value to the provision or creation of a public service. This last perspective includes a wide variety of actors. We, therefore, agree that co-creation differs from co-production in three main aspects: (1) the actors involved; (2) the interaction purpose; and (3) the focus on public value co-creation 1 instead of public service production (Ansell and Torfing 2021). Based on the previous explanation of co-production and co-creation concepts, we aligned our understanding of co-creation with the definition developed by Torfing et al. (2019). Yet, due to the close relationship of the concept with co-production, our scoping review also considered relevant co-production literature (cf. Linders 2012;Nabatchi et al. 2017;Sicilia et al. 2016;Bovaird and Loeffler 2012).
Co-creation processes can take place in all phases of the public service cycle McBride et al. 2019;Bovaird 2007;. Building on Linders' (2012) work, we focus on the three phases: co-design, co-delivery and co-evaluation. The co-design phase provides an important frame for the conception and layout of the service that is to be designed. This phase might encompass user consultations (Bovaird 2007), design labs (Bovaird and Loeffler 2012) or other approaches that engage different stakeholders in the development of the specific public service. The key elements of codesign are the consultation and ideation of service design elements. The inclusion of a variety of stakeholders, including users, in the design of a service not only provides a more user-centric experience but also creates mutual trust between the authorities and the respective stakeholders (Sicilia et al. 2016). Co-delivery typically occurs through trained stakeholders or, more specifically, peer groups, and can be defined as concurrent to the service. Co-delivery can enhance the acceptance of the services through the involvement of peers in the process. This can happen, for instance, through peer support groups, nursefamily partnerships, or even trained youth councils that support peer education (Nabatchi et al. 2017). Including stakeholders in the delivery phase can enhance the communication between service providers and users and can allow for a more integrative user experience (Sicilia et al. 2016). Finally, the co-evaluation phase assesses the service after its delivery in an effort to learn from it or to adapt to it through possible prospective elements (Nabatchi et al. 2017). This phase is also known as co-assessment or co-monitoring (cf. Rodriguez . Through evaluating and assessing the provided service, this phase can provide an important factor in empowering involved stakeholders (Sicilia et al. 2016).

Research Strategy
In this article, we conducted a scoping review following the "Preferred Reporting Items for systematic reviews and meta-analyses", also known as PRISMA (Liberati et al. 2009). Specifically, we used the PRISMA Extension for Scoping Reviews (PRISMA-ScR) (Tricco et al. 2018). To conduct the review, we first identified scientific articles through the PRISMA method. PRISMA provides predetermined protocols for the data collection process that decreases the bias collection while enhancing the transparency and reproducibility of the review (Liberati et al. 2009). As a second step, and in order to identify the relevant aspects of the public service co-creation concept, we conducted a PRISMA-ScR method (Tricco et al. 2018). This method is useful "to identify and examine characteristics or factors related to a particular concept" (Munn et al. 2018, p. 77).

Article Selection
Following the PRISMA-ScR method (Tai 2021), we searched in the electronic databases Web of Science and Scopus using a defined search string. The search string was used consistently in the two databases and combines three types of search terms: (a) terms related to co-creation 2 ; (b) terms related to e-government; and (c) terms related to the public sector. The search process was concluded on 3 March 2021 (see Table A1 for the Scoping Review Search Log).
The selection was based on pre-defined eligibility criteria. The definition of the eligibility criteria allows us to decrease bias in the selection of articles and their scoping to enhance the validity, applicability, and comprehensiveness of the review. PRISMA-ScR differentiates between study eligibility and report eligibility criteria. Report eligibility criteria focus on more formal elements, such as year, language, publication status and field of study (Liberati et al. 2009). For our review, we selected papers from international peer-reviewed academic journal articles in English. As Walker and Andrews (2015) argue, this approach ensures a suitable publication quality and greater academic rigor. We did not specify a time limit since the main body of the literature has been published over the last 20 years, which was confirmed in the search for articles. The oldest article that was included in the full-text screening was published in 2003. Considering the multi-disciplinary nature of the topic, the query included the domains of social sciences, computer sciences and library sciences for Web of Science, and the subject areas of social sciences and computer sciences for Scopus. From 1.216 articles found in the database search, 627 articles were deemed eligible when considering the report eligibility criteria: language, field, and type of publication. Next, we checked the 627 articles that were identified in the first step for duplicates and disregarded 92 automatically, leaving a total of 535 articles.
The second step is the selection of articles based on the study eligibility criteria that focus on the topics, outcomes, and study design (Liberati et al. 2009). In our review, we included empirical studies that describe and examine co-creation cases and initiatives that (a) deal with co-creation or co-production of (b) public services in the context of (c) digital transformation or ICT implementation. We excluded theoretical or conceptual studies, reviews, or empirical studies that did not explain or at least describe co-creation cases (e.g., Linders 2012). Following the eligibility criteria, we screened the title and abstracts of the 535 articles. During this process, two researchers acted as reviewers and another as an advisor in case of disagreements. Based on this analysis, the two reviewers discussed and agreed on the definition of the study eligibility inclusion or exclusion criteria and pre-selected 13 articles (4 relevant and 9 irrelevant) which were used to train the tool that supported the screening process.
For the screening process, we employed ASReview, a machine-learning tool that supported our title and abstract screening. This AI-aided and open-source tool allows for a more efficient and less error-prone screening process (see van de Schoot et al. 2021 for more information) when compared to a regular screening process. Based on a pre-selection of training articles by the reviewers, the tool suggests the titles and abstracts of articles from the literature selection that might be relevant according to the training data. As a result, the researchers see the most relevant articles first and consequently, the tool reduces decision fatigue and speeds up the screening process.
With the support of an ASReview and a common basis of analysis, the two reviewers independently followed the same screening process for the 535 articles that were included in the abstract screening process after the exclusion of duplicates. Both researchers screened 375 articles or 70.09% of the sample. This cut-off point was selected after the ASReview tool suggested 35 irrelevant articles in succession (Ros et al. 2017). The resulting list of articles was compared afterward, and discrepancies were discussed with the advisor. The result of these steps was the selection of 75 articles for the full-text screening.
The second screening involved a full-text article review based on the report and study eligibility criteria, including the abovementioned key elements of co-creation (see Table 1). As a result of this process, 49 articles were disregarded. This decision was mainly based on the study design, topic, lack of focus on public services, or unfitness to the context of this study. The final selection of relevant articles on the topic of public service co-creation resulted in 25 empirical articles, which were included in the final scoping review (see Figure 1 for the PRISMA selection process diagram). screened 375 articles or 70.09% of the sample. This cut-off point was selected after the ASReview tool suggested 35 irrelevant articles in succession (Ros and Runeson 2017). The resulting list of articles was compared afterward, and discrepancies were discussed with the advisor. The result of these steps was the selection of 75 articles for the full-text screening.
The second screening involved a full-text article review based on the report and study eligibility criteria, including the abovementioned key elements of co-creation (see Table  1). As a result of this process, 49 articles were disregarded. This decision was mainly based on the study design, topic, lack of focus on public services, or unfitness to the context of this study. The final selection of relevant articles on the topic of public service co-creation resulted in 25 empirical articles, which were included in the final scoping review (see Fig

Data Analysis
To explore how the empirical literature portrays the concept of public service co-creation, a coding process was followed based on the main dimensions of co-creation implementation (see Table 1),. Given the exploratory nature of this study, we conducted a thematic analysis process, including two phases of coding: open coding and axial coding. The coding process was undertaken with the qualitative research tool NVIVO 20 by the first author in discussion with the second reviewer and advisor to validate emerging connections during the axial coding.

Public Services Setting
When considering the setting of the public service, we categorize them into analog, digital or hybrid categories. In the review, we identified that five articles examined analog public services. From the 25 articles, 14 articles reported co-creation processes of digital

Data Analysis
To explore how the empirical literature portrays the concept of public service cocreation, a coding process was followed based on the main dimensions of co-creation implementation (see Table 1). Given the exploratory nature of this study, we conducted a thematic analysis process, including two phases of coding: open coding and axial coding. The coding process was undertaken with the qualitative research tool NVIVO 20 by the first author in discussion with the second reviewer and advisor to validate emerging connections during the axial coding.

Public Services Setting
When considering the setting of the public service, we categorize them into analog, digital or hybrid categories. In the review, we identified that five articles examined analog public services. From the 25 articles, 14 articles reported co-creation processes of digital public services provided or supported via mobile applications, websites, digital platforms, and others. Finally, seven articles reported co-creating a hybrid public service, meaning that the public services are provided both via digital and analog channels.
The selected studies focused on a wide array of public service areas, with several examining more than one case in different areas. For instance, six articles focused on e-government services and open government, four on health and social care, and three articles addressed mobility. Others addressed initiatives on education, energy, urban transformation, food safety, employment, water and sanitation, law enforcement, and non-emergency call centers.

Geographical Focus of Public Service Co-Creation Initiatives
The majority of the selected articles focused on Europe (12), including nine from single European Countries, and three presented results of cross-country comparisons. The remaining studies focused on countries around the world, with four on the USA, two on China, two on Australia, two on Indonesia, and one each on South Africa and Korea. The selected studies mostly focused on local initiatives (28), while 11 studies focused on cases around the central level, three cases at the regional level, one at the European level, and one case at the provincial level (see Table 2). Note: * Total is higher than 25 because of articles examining more than one case.

Phases of Public Service Co-Creation
To further explore and categorize the articles, we build upon the framework of Linders (2012) that proposes a three-phase co-creation process: co-design, co-delivery, and co-evaluation. Since some articles do not explicitly mention the co-creation phase of the case in question, as part of the review, we have identified the phases for all articles. This identification was based on both Linders' (2012) framework and the article's case description. From the 25 articles examined, 11 articles examined co-design cases, codelivery was studied by four articles, and co-evaluation by seven. Meanwhile, four articles were reported to analyze the three phases of the public service cycle. In addition, six articles reported an evaluation phase in the design stage (

Public Service Co-Creation Purposes
When exploring the rationale behind the co-creation processes, we aimed to identify the reasons and purposes behind its implementation. We identified four themes among the reported aims of co-creation: co-creation to improve public service provision, to innovate, to create new public services, and user-driven co-creation (see Table 3). Moreover, within the 25 articles, four studies included more than one purpose through one or more cases (cf., Concilio et al. 2014;Hepburn 2018;Deakin et al. 2011; Soares da Silva and Horlings 2019). In the following subsection, we present each of these five purposes in more detail. Table 3. Co-creation purposes.

Purposes to Co-Create References
To improve public service provision The results of our analysis on the identification of needs are presented in Table 4. Some studies highlighted the need to improve public service performance due to low quality related to service completion rate, transparency , and users' acceptance and adoption of the public services (Tsekleves et al. 2017). For instance, Putra and van der Putra and van der Knaap (2020) examined the use of co-creation to identify solutions for traffic congestion problems. In the case of public spending, several reasons were also identified. For example, Hepburn (2018) highlighted the need to increase the efficacy of public service provision while Huang and Yu (2019) focused on avoiding extra costs and time. Table 4. Needs for improvement.

Needs References
Citizens' complaints and non-responsiveness by the government The main rationale behind co-creation processes when aiming to improve public service provision is the realization of better outcomes through both citizens' insights and citizens' active participation, including, for instance, improvements in the service completion rate and complaint handling procedures Meijer 2011Meijer , 2012. In turn, public service providers can meet users' needs in a responsive manner Concilio et al. 2014). An interesting example is provided by the optimized food inspection model based on Open Government Data (McBride et al. 2019, pp. 5-6), which shows that co-creation processes can lead to improvements in the performance of public services. We also derived from our analysis that the most mentioned purpose in the selected articles is the improvement of public service provision, as was identified in 17 articles. Table 5 below presents the identified purposes and their corresponding references. Table 5. Identified purposes of public services' improvement.

Purposes of Public Service Improvement References
Enhance public service provision (i.e., efficacy, effectiveness)  Based on our analysis, we identified seven articles that aim to foster innovation through co-creation processes (Putra and van der Knaap 2020;Concilio et al. 2014;Soares da Silva and Horlings 2019;Deakin et al. 2011;Hepburn 2018;Trischler and Scott 2015). Our findings show that innovation results not only from the cooperation between the governments, private actors, and NGOs but it is also embedded in the cooperation itself. In this sense, co-creation can be a niche of incubation and multi-scale spatial networking to promote innovation (Soares da Silva and Horlings 2019). Additionally, we identified that innovation-seeking projects can include participants in the design, evaluation, and monitoring of digital public services . Moreover, co-creation is expected to lead to the development of innovative ideas using digital techniques (Putra and van der Knaap 2020) more efficiently (Hepburn 2018), including the users in the center of the process (Trischler and Scott 2015). Finally, some articles focused on the implementation of co-creation in digital transformation processes. For instance, involving users might facilitate their transition from using analog services to digital ones, which can be less costly for public service providers.  observed that co-creation can help to understand users' circumstances, while digitalizing services based on users' insights to meet their expectations, and to avoid the trap of "one size fits all". In the case of Deakin et al. (2011), the authors reported that co-creation is an interesting process for innovation-seeking projects to allow a larger number of participants to engage in the design, evaluation, and monitoring of digital public services.

To Create New Public Services
Co-creation processes are also considered for the creation of new public services. From the 25 articles, three focused on this aspect. One is the study of Jarke (2019) that examined the co-creation process of a new application for supporting elderly people. Secondly, Soares da Silva and Horlings (2019) studied a cooperative initiated by citizens and supported by other stakeholders, including the government, to provide a wind park after the failed attempt by governmental actors. Finally, the article by Paskaleva and Cooper (2018) presents various co-creation processes with the aim of creating public services with a focus on the employment of co-designing tools such as mapping services, collaborative design, and prototyping.

User-Driven Co-Creation
It is relevant to point out that some authors of our sample did not mention a specific purpose of the co-creation process directly related to public service provision. In these cases, co-creation was implemented to empower citizens Hepburn 2018;Morton and Paice 2016), enhance trust among users (Meijer 2012) and improve the perception of users about public administrations and reputation of the service providers (Meijer 2012(Meijer , 2011. Moreover, co-creation aimed to increase inclusion and support by enabling the participation of disadvantaged groups, such as elderly people Concilio et al. 2014) or the "distant others" Morton and Paice 2016) in the social, economic and civil life.

Co-Creation Actors and Roles
The articles were also analyzed based on the role of the actors. A wide variety of co-creation actors were identified. Yet, most of the articles reported the involvement of citizens and governmental actors (See Table 6), followed by the engagement of private actors. It is relevant to note that one article mentioned the involvement of users without specifying who the users were (Hepburn 2018). The co-creation actors could initiate the co-creation process or could be part of the co-creation process by sourcing resources, such as time, funding, experience, information, expertise, among others. Concerning the roles, in most of the cases, the co-creation approaches were implemented top-down, with the government being the main co-creation actor. A few cases reported bottom-up co-creation processes initiated by citizens or nonprofits. In these cases, the government played a role as sponsor, providing the frame and funding for the initiatives. Some examples are the studies of , Soares da Silva and Horlings (2019), and Meijer (2011).
When considering the described background, Linders (2012) labels these types of co-creation processes based on the roles of the co-creation actors, such as citizen sourcing (Citizen to Government) and government as a platform (Government to Citizen). The author also presents a third category called "Do It Yourself Government". Yet, in line with similar previous studies (e.g., Rodriguez Müller 2021), we do not consider citizens' "selforganization" as a case of co-creation, because we understand co-creation in the context of public service provision.
An aspect sometimes neglected in the co-creation and co-production literature are the strategies used to invite and engage stakeholders to co-create as well as to raise awareness over these processes. When looking at it in our review, only six articles explicitly described how co-creation actors were engaged or kept informed about the co-creation process through digital and analog channels (e.g., social media, newspaper articles, leaflets)

Public Service Co-Creation Strategies and Tools
Co-creation processes take place through digital tools, analog tools, and a combination of both (hybrid co-creation). While analog public services are still highly employed by users, governments have been increasingly adopting digital tools to engage users in public service provision to extend the applicability of co-creation processes, and increase efficiency and effectiveness, among others (Rodriguez Müller 2021). Therefore, it is relevant to identify and classify the co-creation tools based on the adopted approach. We identified 10 articles that reported the use of digital tools only, such as websites, forums, SMS, mobile applications, online surveys, among others. Nine other articles reported the use of analog tools, such as interviews, surveys, focus groups, meetings, among others. In addition, six articles out of the 25 selected articles examined hybrid co-creation. Some of these tools involved co-creation actors more actively (e.g., hackathons, cocreative workshops), while more traditional tools, such as online surveys, illustrated a more passive co-creation process. We also differentiated between the tools based on the phase of public service co-creation, yet only two articles McBride et al. 2019) examined co-creation in the delivery phase. In these cases, they reported the use of Open Government Data (OGD) and data and information platforms to support the co-creation process.
An interesting distinction is the use of OGD to engage users in the co-creation of digital public services . Within this approach, different strategies can be used, such as open government data portals, code exchange through "app contests", civic issue trackers (i.e., user monitoring, crowdsourcing, users as sensors), and participatory open data (i.e., data provided by both governments and users) .
Similarly, Trischler and Scott (2015) argued that the combination of co-creation and collaborative techniques can lead to a better and more in-depth understanding of users' experiences and in turn, support the design of complex public services. For instance, Concilio et al. (2014) made use of techniques such as design tables during co-creative workshops. Another illustration is the use of design fictions, a speculative design process that includes scenarios, brainstorming, rapid prototyping, and multidisciplinary techniques to explore and define new services and "what-if" scenarios (Tsekleves et al. 2017).  also reported the use of several tools (see Table 7 and argued that online tools can provide an enhanced experience while being less costly than analog engagement methods. Table 7. Co-creation tools and approaches.

Tools References
Co-design

Benefits of Public Service Co-Creation
Through an analysis of the 25 articles, we have identified several benefits and outcomes obtained through the co-creation processes and clustered them into seven themes. These benefits seem to be more comprehensive than the co-creation purposes, as they surpass the initially expected outcomes. Yet, in general, the benefits and outcomes related to co-creation processes are not exhaustively described. The first theme concerns organizational changes as well as changes in the organizational culture, processes, and capacity. Authors argue that co-creation favors and increases public service providers' accountability De Filippi et al. 2017), and it can also lead to changes in management . Moreover, co-creation processes might promote organizational changes to overcome ethical concerns (Tsekleves et al. 2017) and to further promote co-creative practices in the public sector (Paskaleva and Cooper 2018). In the same line, co-creation can lead to cultural changes towards user-centered working processes (Morton and Paice 2016).
The second theme is the enhancement of user-centricity. Particularly, it focuses on the benefits related to the users' needs, experiences, and requirements. Public service co-creation can provide additional channels for interaction with the users where users can voice their concerns Meijer 2012Meijer , 2011. In addition, it can provide users with a support structure (Meijer 2011(Meijer , 2012 for improving emotional and social development (Meijer 2011;. Co-creation can also lead to a higher level of social inclusion when involving disadvantaged users, such as elderly people or weaker groups De Filippi et al. 2017;Clark et al. 2013). This can also result in higher-quality services for these users, covering their particular needs and requirements  and leading to increased user satisfaction ). In addition, some authors mention that engaging users in the co-creation of public services might lead to a higher users' sense of ownership of both the co-creation processes and the service (Soares da Silva and Horlings 2019; Jarke 2019). Finally, Farr (2017) highlights how co-creation can lead to increased levels of privacy.
The third theme identified through the analysis of the selected articles is enhancement and personalization of public service provision. Similar to the purpose of improving public services, this category is the most often mentioned by our sample of articles (see Table 5 for an overview). Some authors reported that co-creation has led to improved public service performance and provision Meijer 2012;Hepburn 2018;Allen et al. 2020;Xu and Tang 2020;Trischler and Scott 2015) as well as to more personalized and convenient public services Tsekleves et al. 2017; thanks to the information and resources provided by different stakeholders Meijer 2012Meijer , 2011. Moreover, co-creation can lower the costs and resources needed by public service providers Clark et al. 2013)  The fourth theme includes benefits related to the improvement of government functions, such as (data-driven) decision-making processes, capabilities, capacity, and digitalization of public services. From the 25 selected articles, three have highlighted these benefits Jacobs et al. 2018;.
The fifth theme includes articles that reported, as one of the main benefits, the sustainability of both public service provision, as well as co-creation practices. Authors agree that through co-creation, the sustainability of public service provision, collaborative practices and newly co-created services can be strengthened ; Putra and van der Knaap 2020; Soares da Silva and Horlings 2019). Moreover, co-creation can promote higher levels of user acceptance and plausibility of services and technologies, leading to more sustainable public service provision (Tsekleves et al. 2017).
The sixth theme is related to public sector innovation. These articles argue that cocreation can foster innovative ideas and solutions (Soares da Silva and Horlings Jacobs et al. 2018) and partnerships ) with a variety of expertise and resources (Putra and van der Knaap 2020). Co-creation can also result in new initiatives and public services. Particularly, it can help conceptualize new public services (Trischler and Scott 2015;Tsekleves et al. 2017; Soares da Silva and Horlings 2019) that meet users' needs with the support of innovative approaches, such as OGD . Moreover,  highlights the potential of co-creation processes to develop ICTs and digitized public data and public services. Finally, the seventh theme focuses on the enhancement of social well-being and quality of life Concilio et al. 2014;Cinderby et al. 2018).

Challenges to Public Service Co-Creation
Beyond the promises and high expectations of co-creation processes, recent studies have highlighted the need to consider the potential pitfalls and the "dark-side" of cocreation Torfing et al. 2019) in order to advance both theory and practice. In line with this call, we have explored the challenges reported in the 25 empirical selected studies. Overall, the articles included in the review highlight the benefits of co-creation processes, neglecting an explicit discussion of the potential risks and challenges. Moreover, the scoping review revealed that some articles do discuss potential challenges related to different phases of co-creation but build on academic literature instead of empirical evidence.
Among the challenges identified through the review, the most notorious is related to the inclusiveness and equality of co-creation processes Deakin et al. 2011;Huang and Yu 2019;Ma and Wu 2019;Xu and Tang 2020;Tsekleves et al. 2017;. Scholars argue that challenges regarding the inclusiveness of cocreation processes are common for both analog and digital co-creation and public services. In traditional co-creation processes, the challenge mainly lies in the skills and resources needed for users to participate. Moreover, issues related to social inequality (e.g., language, class, skills) are also reported . Concerning the cases where ICTs are adopted, the main concern is in line with the argument of the "digital divide" (Xu and Tang 2020;Huang and Yu 2019;Clark et al. 2013). These differences obstruct the willingness and ability of users to engage as co-creators. This issue might lead to the reliance of a specific group of users that are active creators, potentially leading to unrepresentative results . Furthermore, co-creators with different backgrounds and skillsets lead to co-designed public services who, therefore, represent the view of this specific group. Moreover, during co-creation processes, the interaction of stakeholders with different needs and perspectives might become a challenge (Huang and Yu 2019;Deakin et al. 2011;Tsekleves et al. 2017).
Related to the previous point, another challenge with co-creation processes is the difficulties in meeting the expectations of all involved stakeholders Cinderby et al. 2018) to achieve a co-creative synergy. For instance, Cinderby et al. (2018) argue that feedback received on their co-design case study suggested that the solutions presented were "not radical enough".
The solutions and co-creating tools, either analog or digital, need to be properly communicated to enhance awareness of all users and to increase engagement, as reported by Xu and Tang (2020). In addition, the scoping review also revealed that some concerns arise with the use of citizens' and users' data and insights. Particularly, articles identified challenges related to quality, reliability, and sampling bias. Meijer 2012Meijer , 2011. In the same line,  argues that the quality of services cocreated by citizens with no technological expertise might not be up to the expectations of the general public. Concerning digital co-creation,  claims that the adoption of digital tools to engage citizens as co-creators might replace planning with probability, leading to potential challenges in the provision of the co-created public service.
Another challenge that was discussed in the reviewed articles is the issue of power dynamics Putra and van der Knaap 2020;. In co-creation processes, actors-both government and non-governmental actors-need to take up new responsibilities and roles. This might incite some resistance to change from certain stakeholders . Therefore, the allocation of responsibilities among the co-creation actors is crucial to overcome some of these challenges . Moreover, the proposal of creating "equal partnerships" between public service providers and users might exacerbate intricate power dynamics . The same is argued by Hepburn (2018), who claims that the collaboration between the public, private and third sectors brings different ways of working and expectations that need to be handled to achieve the expected outcomes and to overcome power tensions.
Professionals at the strategic level can also be reluctant to engage external stakeholders as co-creators based on the complexity of public service provision, timing, and limited resources. In this case, early adopters in the government can help in the development of a culture of co-creation that can be sustained over time (Morton and Paice 2016). In addition, in order to drive innovation and long-lasting results, there is a need to formalize governmental policy to engage non-governmental stakeholders as co-creators while avoiding a solely technocratic and political approach (Putra and van der Knaap 2020) and ensuring accountability ). Regulations and policies should also facilitate bottom-up co-creation initiatives .
Finally, three of the selected articles reported concerns about the actual tangible and sustainable impact of co-creation. They argue that co-creation might have a lower contribution towards innovations and that, in some cases, silos structures together with a top-down governmental approach structure can lead to lower impacts and innovation development (Putra and van der Knaap 2020;Concilio et al. 2014;Paskaleva and Cooper 2018;Trischler and Scott 2015). Furthermore, Putra and van der Putra and van der Knaap (2020) argued that enough time is needed to implement co-creation and to see results. However, co-creation demands extra effort, resources, time (Huang and Yu 2019;, and capacity . A lack of these, together with a lack of expertise and relevant benchmarks of success, might obstruct the achievement of the desired outcomes (Paskaleva and Cooper 2018). For instance, Ma and Wu, 2019, explained that local governments which have well-maintained portals drove more user engagement than other cities where the resources are more limited.

Discussion
In this article, we conducted a scoping review following PRISMA-ScR methodology. We identified 25 relevant empirical articles out of 75 analyzed articles that discussed the implementation of public service co-creation initiatives in a digital context. Specifically, our scoping review provides a new lens to enhance our understanding of how the empirical literature portrays the concept of public service co-creation. Our review confirms that co-creation has also become more prevalent with digitalization ). As our analysis shows, most of the selected articles reported the co-creation of only digital public services or on a hybrid variant of offline and online. Our review reveals that the initiatives examined focused on a variety of policy sectors, yet the majority of the articles focused on e-government services, followed by social health and social care and mobility. In this regard, our results confirm previous reviews on the topic, which identified that social policy and the health sector are among the predominant focus of research (e.g., Almeida et al. 2018;Clifton et al. 2020;Voorberg et al. 2015;Jukić et al. 2019). In addition, most of the articles selected (14) are single case studies or comparative cases in single countries (nine). This methodological aspect is still considered as one of the main criticisms of co-creation and co-production research (e.g., Loeffler 2021; Jukić et al. 2019). In terms of the geographical focus, the majority of the cases were focused on Europe, followed by the USA. Moreover, most of the selected articles explored local-level initiatives.
Concerning the phases of the public service cycle, we can observe that most empirical articles that analyzed co-creation focused on co-design processes. This phase has two characteristics: it allows the inclusion and consideration of users' requirements, and it is prevalent in co-creation processes since it is more feasible to include users in this stage than in, e.g., the service delivery . In contrast, co-delivery is included in a few of the examined articles. This result is in line previous studies that highlight the limited evidence on co-delivery processes (cf. Loeffler 2021). Concerning the third phase co-evaluation, we observe that it is examined in two ways: (1) co-evaluation processes in the evaluation phase of the public service cycle, and (2) as a final phase in co-design processes. While our results highlight co-creation projects in the co-design phase, some of the selected articles have also shown the possibility of adopting co-creation in several or all phases of the public service cycle.
When exploring the rationale behind the implementation of co-creation initiatives, the results of our review show that the public service co-creation initiatives of our selected articles mainly aim to improve public service provision, particularly concerning the efficacy, effectiveness, quality, and sustainability. This result corroborates with the review conducted by Voorberg et al. (2015), who reported that the main objectives reported were related to effectiveness and efficiency. These aims tend to be reactionary since co-creation might be perceived as the answer to low-quality services or a lack of public accountability Clark et al. 2013). The second theme of aims found to be predominant in the reviewed literature is the implementation of co-creation to foster innovation, followed by the creation of new services. Nevertheless, some articles have not reported a specific purpose for the implementation of co-creation and were focused on aspects related to user-centricity and the engagement of citizens. This is in line with what Voorberg et al. (2015) describe in the context of co-creation as the "implicit assumption that involvement of citizens is a virtue in itself".
Our review has also explored the actors involved in co-creation initiatives and their roles. While a wide variety of actors are expected to be involved in co-creation processes, the cases reported in our review primarily focused on citizen-government interactions. Moreover, governmental actors have also been identified in the analyzed literature as the actors involved in all observed co-creation cases and as the most regular initiators. There is also a clear tendency towards top-down initiatives. Besides the public sector itself, citizens seem to play the most relevant role as public service users and co-creators, followed by private businesses and academic or non-profit actors. This result confirms previous research, including the work by Jukić et al. (2019), which found that 39% of studies focused on citizens as external actors, while only 7% focused on businesses. On the one hand, these results might highlight the relevance of involving citizens, also known as "experts by experience" (Meriluoto 2018), in public service co-creation. This also implies that, in the reviewed literature, user-centric service development mainly focuses on citizens as the users of co-created public services. On the other hand, the results might also reflect the use of other concepts for the involvement of external stakeholders, such as collaborative governance or public-private partnerships which were not considered in this review. This aspect is related to several of the criticisms around the conceptual variety and fuzziness of the topic (cf. Jukić et al. 2019).
Furthermore, while different online and offline recruitment methods have been observed in the literature, our review revealed that there is not yet a comprehensive understanding of which strategies can successfully engage users, and how they can do so in co-creation processes, beyond the "usual suspects" ). This finding is highly relevant as previous research identifies the lack of awareness and selfselection as two of the main challenges of (digital) co-creation to engage a wide range of users as co-creators (van den Berg et al. 2020;Fledderus et al. 2015).
Another relevant dimension explored through our scoping review is the implementation of co-creation processes. We particularly explored the tools and approaches utilized. To do so, we classified the tools and strategies into three clusters: digital tools, analog tools, and hybrid approaches. Moreover, to advance the understanding of co-creation implementation, we have clustered the tools among the different phases of the public service cycle. This distinction provides a starting point for an understanding of the differences in co-creation processes throughout the public service cycle. Furthermore, our results show a balanced use of digital and analog methods (cf. Table 7). The literature shows that these analog methods include mostly personal interactions or direct conversations, as can be found in interviews and workshops (Trischler and Scott 2015). Meanwhile, digital tools allow a wider variety of methods for co-creation, such as mapping, the use of Open Government Data, and mobile applications (Trischler and Scott 2015;Allen et al. 2020). Our review also reveals that broader conversation and higher inclusiveness can be achieved through social media. In addition, the findings also show an emergence of hybrid co-creation methods. Overall, hybrid approaches combine online and offline tools, as well as low-threshold and higher-threshold co-creation tools. Moreover, they include methods that support the design of complex services and allow for a broad involvement of user groups (Tsekleves et al. 2017;Concilio et al. 2014;Trischler and Scott 2015). This result is in line with previous studies that examine channel choice between users reporting behavior, indicating that adopting a hybrid approach can overcome challenges related to users' engagement .
We also identified common themes around the benefits obtained via co-creation processes. Contrary to previous studies (e.g., Voorberg et al. 2015), our scoping review reveals that the benefits seem to be broader than the co-creation purposes, surpassing the expected outcomes. As the purposes of co-creation indicated, change, innovation, and service improvement seem to be at the core of public service co-creation. Moreover, the overarching benefits identified in the review refer to the support of change management and a redesigned work progress, the identification of users' needs with the consequence of enhanced and personalized public service provision, and the sustainable development and implementation of public services. While co-creation is mainly seen as a method for more active user involvement in the phases of the public service cycle and an answer to users' needs regarding (digital) public services, our scoping review revealed that co-creation might have far-reaching benefits at administrational, political, and societal levels. It also has the potential to be a lever to create better functioning of public service provision, to improve quality of life, to increase inclusion either through more civic engagement or addressing underrepresented groups. Additionally, it can enhance transparency and trust between citizens and public administrations.
Nevertheless, co-creation processes are not without challenges. This aspect has been recently gaining more academic attention and has been labeled as the "dark-side" of cocreation . Some challenges reported in the selected articles are related to unbalanced power dynamics, a lack of specific policies and regulations, inclusiveness and equality divides, and limited resources and sustainability. Yet, our selected studies did not report a detailed discussion on these factors and empirical evidence was limited. Finally, it is relevant to notice the limited number of empirical cases that describe and examine cocreation initiatives which might imply that either there is still a lack of empirical evidence or that the challenges related to the implementation of public service co-creation in digital settings are greater than the benefits. The latter might suggest that co-creation is still in its infancy (see also Almeida et al. 2018). As Clifton et al. (2020) suggested in their review on ICT-enabled co-production, governments have not highly promoted the implementation of co-production processes in digital settings despite the "rhetorical enthusiasm".

Concluding Remarks and Future Research Agenda
This article aimed to explore how the empirical literature portrays the concept of public service co-creation by conducting a scoping review. To do so, we explored the relevant dimensions related to the implementation of public service co-creation. This review contributes to research and practice by identifying key aspects of implemented (digital) public service co-creation, including a wide variety of co-creation processes with different purposes, actors, and broad benefits and challenges. Moreover, this scoping review provides general insights into the co-creation of public services and showed that co-creation can be relevant when implemented in the overall public service cycle. These phases can be accompanied by different online and offline tools that need to consider the involved actors. Moreover, user-centric service improvement and sustainable public service provision are at the core of co-creation processes, and consequently, co-creation can be deemed as an appropriate approach for providing and digitalizing public services.
Before presenting the main contributions and future research agenda, we should note some limitations of the methodology used in this article. While using a PRISMA-ScR method has allowed for a more systematic and transparent review of literature reinforced by the use of an AI tool, the screening process may suffer from the researchers' subjectivity. Furthermore, this scoping review focused on empirical articles; however, the possibility of generalizing and comparing our results is restricted. Although our findings should be understood in light of these limitations, our study has revealed six main avenues for future research that are critical to start bridging the gap between knowing and doing in public service co-creation research.
First, our study has reported the potential of co-creation as an alternative strategy to traditional public service provision in diverse policy sectors. Yet, it has also revealed limited research that provides critical evidence on the actual benefits and outcomes of co-creation initiatives based on the context of the public service co-created. In addition, the articles screened show a tendency towards the analysis of successful cases, neglecting the study of failures to further understand relevant challenges and obstacles of co-creation. For instance, future research could examine under which governance modes co-creation could flourish in order to achieve the expected outcomes.
Second, our scoping review explicitly included a digital aspect when screening for the articles and showed that empirical research on co-creation practices in a digital context remains in its infancy. Therefore, we would like to take this opportunity to invite researchers to conduct future studies on the implementation of (digital) co-creation of (digital) public services and its implications in terms of strategies, actors and roles, and outcomes. The findings of such studies could contribute to a better understanding of the role of digital technology in the implementation of co-creation processes and vice versa.
Third, while co-creation has been called "the new kid on the block in public governance" (Ansell and Torfing 2021), research from a governance perspective is still limited. In this regard, our review confirms the lack of co-creation studies considering a multi-level governance perspective (cf. Loeffler 2021). As observed in Table 2, most of the revised studies have focused on the local level. While this could be understood considering the relevance of local governments when providing public services, other governmental levels, such as subnational and national, also provide key services to users. Therefore, we believe that future research should also consider initiatives of co-creation of public services provided by subnational and national governments.
Fourthly, the tools, methods, and strategies employed to co-create are fundamental for implementing public service co-creation. Yet, our review reveals that studies tend to fail in describing the specific tools adopted and in explaining how co-creation can take place. Particularly, there is a lack of understanding of how to implement co-delivery processes. In addition, comparative studies on the adoption of analog tools and their digital versions (e.g., in-place and virtual workshops) to co-create are needed.
Fifthly, the interactions between a wide array of actors are part of the nature of cocreation processes. In theory, co-creation is understood by its multi-actor perspective, yet based on the selected articles, it has been neglected by empirical research in public service co-creation in digital settings. Future research should address this limitation by exploring which actors are involved in the three phases of co-creation and what their roles and responsibilities are. Further research is also needed to analyze the circumstances of the interactions between the actors considering the challenges related to power imbalances, and what strategies have been successful (or not) in overcoming those challenges. This avenue of research also entails a consideration of how decisions are made, which stakeholders are involved in the decision-making process, and the accountability mechanisms that underline this process.
Finally, our review revealed that the implementation of co-creation differs between the different phases of the public service cycle. Yet, overall, the selected articles did not discuss in detail the implications of implementing co-creation processes in different phases. Comparative studies of initiatives implemented between different phases of co-creation would therefore be needed to start narrowing down the implementation gap. In addition, empirical research could address whether and how the implementation of co-creation through the whole public service cycle leads to higher public value creation.

Note
Conflicts of Interest: The authors declare no conflict of interest.

Appendix A
To conduct the literature search on the selected databases, we selected the following general criteria (see Table A1 for the final search log): • Key words: "co-creation" (and related terms), "e-gov" (and related terms) and "public services" (and related terms). • Time period: no start date-2021 • Field: Social Sciences-Computer Sciences • Search: Title and Abstract • Sources: published and early access journal articles Table A1. Scoping Review Search Log.