Assessing the Quality of Digital Coproduction: An Interdisciplinary Model

Abstract

The digital transformation of society raises high hopes for the improvement of the design and implementation of coproduction processes, which may lead to higher levels of acceptance, satisfaction, and trust with respect to public services. Yet and despite a growing number of studies on the role of digital technology on coproduction, our knowledge on how to assess the quality of digital coproduction is still very limited. The difficulty of defining quality indicators relates to three issues: the complexity of defining quality in public services, the lack of a clear understanding of quality in a coproduction process and the tendency to evaluate coproduction using the same approach as with other digitalised services. In this article, we adopt an interdisciplinary perspective to develop a conceptual model that builds on and bridges theoretical approaches commonly used to assess the quality of digital services, while adding elements that are specific to coproduction.

1. Introduction

This article discusses how to conceptualise and measure the quality of digital coproduction. The digital transformation of society raises high hopes for the improvement of the design and implementation of coproduction processes, which may lead to higher levels of acceptance, satisfaction, and trust with respect to public services (Fledderus 2015, 2016). Yet and despite a growing number of studies on the role of digital technology on coproduction, our knowledge on how to assess the quality of digital coproduction is still very limited (Lember et al. 2019; Clifton et al. 2020; Rodriguez Müller 2021; Cagigas et al. 2021). Policy documents show optimistic notions of digital coproduction processes and their benefits, such as customisation, efficiency, better public services, and higher engagement. However, what does quality mean in this instance?

As in any concept of quality, the underlying values may compete (for instance, effectiveness vs. efficiency). Thus, quality of coproduction encompasses a range of potential combinations of focus points and even trade-offs rather than a fixed standard. This is not surprising since coproduction often takes place over a longer period of time, in complex multi-actor settings and with varying stakeholders. These stakeholders have different roles, responsibilities, and perspectives. Expectations regarding the quality of coproduction are therefore highly context-dependent and should not be conceptualised rigidly in terms of one standard. Instead, a comprehensive framework is needed that offers a blueprint to normatively and empirically investigate stakeholders’ demands regarding quality and resulting tensions.

What makes coproduction in a digital context even more complex is that perspectives on quality are in practice at least partly driven by a technological perspective, which emphasizes values that would not normally play a significant role in assessing coproduction. From a social science perspective, some of these considerations may seem counterintuitive. However, in practice they shape not only the design and implementation but also the evaluation of coproduction processes. Adopting an interdisciplinary perspective, we will thus analyse and bring together different theoretical perspectives. Our aim is to arrive at a multidimensional conceptual and analytical framework to assess the quality of digital coproduction.

This article was developed as an interdisciplinary conceptual and empirical effort, as part of the EU-funded INTERLINK project (https://interlink-project.eu, accessed on 15 January 2023). Due to the interdisciplinary nature of our research consortium, we were able to incorporate concepts derived from various strands of the literature and disciplines that are not always easily compatible. This chance to present a more encompassing and richer perspective on quality, however, also raises new questions about the overall assessment of quality and the negotiation of tensions and trade-offs.

The remainder of this article is structured as follows. Section 2 discusses the background of digital coproduction and the way digital coproduction can be conceptualised. In Section 3, we discuss the challenges of defining and assessing quality in the context of digital coproduction, while Section 4 presents our inter-disciplinary and multi-dimensional model to analyse the quality of digital coproduction. Section 5 shares our concluding remarks and reflection on the development and further use of this framework.

2. Conceptualizing the Digital Coproduction of Public Services

To assess the quality of digital coproduction, we first need to define coproduction and consider what makes it digital. The concept of coproduction has been widely discussed in the literature (see: Brandsen and Honingh 2016; Nabatchi et al. 2017; Bovaird and Loeffler 2012). In this paper, we rely on the relatively narrow definition of coproduction developed by Brandsen and Honingh (2016) as ‘a relationship between the employee of an organization and (groups of) individual citizens that requires a direct and active input from these citizens to the work of the organization.’

In this article, the focus is on digital coproduction. What is it exactly that makes a coproduction process digital? It is certainly not difficult to come up with examples showing how the relationship between citizens and public administrations can change significantly under the influence of digital technologies. Often it is stated that opportunities to coproduce have increased due to the implementation of ICT. Lember (2018) states that it is often assumed that new technologies will foster coproduction and cocreation by making these processes more effective and more efficient, for example, through lower barriers for citizens to engage in such processes. Citizens can fill out forms from home or report dirt, loose paving stones, and broken lampposts. More substantive contact can also take place online, for example, through online citizen fora and referenda.

Digital coproduction could therefore be defined as coproduction in which ICT plays a significant role. That definition is of course rather imprecise. Should ‘digital’ be used to include only those processes which take place online? This would narrow down the object of our inquiry considerably, but it would also be misleading. Suppose we only have e-mail contact with someone; is that a ‘digital’ relationship? And does that change once we happen to meet that person at a physical conference, or in an online meeting where we see the person’s face? What if the digital interaction takes place against the background of largely analogue processes—say between two organisations and most of the actual work of the collaboration occurs in person? The distinction between analogue and digital coproduction is in practice blurry, so a strict definition is unlikely to be useful. Theoretically, it is sufficient to acknowledge that digital technologies play an important role in how the coproduction process is shaped, even if that intertwines with analogue processes. This is a gradual rather than an absolute demarcation.

This fuzzy definition is in fact less problematic than the fact that the different disciplinary perspectives that can be applied to the phenomenon each bring with them different understandings of quality, some of which are hard to reconcile. In the next section, we will point out the main difficulties which also need to be taken into consideration in the formulation of a framework for the quality of coproduction.

3. Quality: What Is Specific to Digital Coproduction in Public Services?

Quality can be considered what Pollitt and Hupe (2011) have termed a ‘magic concept.’ The very high degree of abstraction and the notion’s solely positive normative connotations overcome any opposition and have meant that aiming for qualitative public services has become an important objective across domains. However, its blurriness also means that defining and implementing a framework in the attempt to ‘assess’ quality is difficult. We will here break the difficulty down into three issues of how quality must be understood in the context of (1) public services, (2) coproduction, and (3) digital processes.

3.1. The Quality of Public Services

To understand the difficulties in defining—and ultimately measuring—the quality of digital coproduced public services, it is important to understand their origins: The concept of quality management spilled over from the industrial sector to the service sector, and finally to public services (Cole 1994; Reichwein and Broekmate 2010). This means there are three main difficulties in defining and measuring quality as a performance indicator in the public sector (Donabedian 1988).

First, many public services are delivered in complex settings, as is the case in coproduction processes. This means that stakeholders with different perspectives are involved, spanning public administration at the national, regional, and local levels, as well as public, not-for profit, and private service providers. They all see the colours of the quality prism filtered through their own lenses, since quality assessments are in the end normative judgements based on the values of the respective stakeholders (Oechler 2009; Beckmann et al. 2004). Defining quality thus usually includes the coordination of different perspectives rooted in divergent values and interests. This of course also matters in a business context, but in the public sector there are more (kinds of) actors of which to take account. Complexity is further increased in the context of digital public services as new players, like IT service providers, enter the stage.

Second, standardisation of quality, though common in the industrial sector, is more complicated in the case of public service provision as unpredictable human interactions are often an integral part of these kinds of services (Oppen 1995, pp. 20–31). This leads to a number of issues when trying to identify quantifiable (and even objective) indicators of quality, e.g., the immateriality and intangibility of the ‘product’ or the nonexistence of uniformity in service provision. Other difficulties are inherent to the public sector, such as the centrality of legal requirements and the client’s relative lack of independence compared to services in the private sector.

Finally, it is often unclear who is responsible for the level of quality that is ultimately achieved. This is because various interdependent actors are involved in creating public services and the effects of their contributions are hard to separate. This is even more complex in the context of services in which the consumer of public services is a coproducer (Beckmann et al. 2007; Honingh et al. 2020). Thus, their collaboration is necessary for the production of public services and, moreover, the acts of producing and consuming coincide. Who then is responsible for which parts of quality in public services?

For these reasons, an objective approach to quality, based on a standardised understanding of quality, is difficult in the context of public services. When designing quality indicators, it is important to include different perspectives on quality. We will do this by developing a multidimensional perspective on quality.

3.2. The Quality of Coproduction

In coproduction research, there has been relatively little attention given to the issue of quality and the control thereof. This is perhaps due to the use of coproduction in practice still being so fragmented. Perhaps fostering coproduction and giving shape to such an interactive process may have already been seen as a success and therefore as an aim or mark of quality in itself. Or perhaps the difficulty of defining good indicators has discouraged scholars from engaging with the issue. Or perhaps it has been regarded as a type of process that should not be subjected to quality control at all, because it may encourage efforts towards (undesired) standardisation. We recognize the latter risk, but also the equally real risk that coproduction will in practice be evaluated anyway, regardless of whether good indicators have been developed or not. As we will argue below, it is important to influence this process.

So where to start? There is a handhold in the academic literature as it has previously described the potential benefits of coproduction. These usually come down to the following: (1) a greater role for citizens in service delivery is an impulse for democracy (i.e., of intrinsic value), (2) it leads to greater feelings of empowerment among users, (3) it leads to a better relationship between user and provider, and (4) it leads to a functionally better service (Brandsen and Pestoff 2006). Preferably, all these benefits should be considered in developing indicators for coproduction processes.

3.3. The Quality of Digital Processes

If coproduction is conducted through digital means, such as a platform, there is a tendency to evaluate it using the same approach as with other digitalised services. This has led to reliance on two types of quality approaches. One is the technology acceptance literature, which uses an information systems perspective to assess how users accept and apply a technology. The other is consumer-based quality systems such as SERVQUAL that focus on perceptions and expectations in relation to the service that is ultimately delivered.

In one way, this fits well with existing practices, since the quality discourse in the public sector is highly embedded in a trend towards the rationalisation, marketisation, and managerialisation of public services (Hoggett 1996; Speck 1999; Power 1997). However, it does tend to emphasize certain values over others. The coalition of market-inspired and engineering perspectives can lead to an overwhelming emphasis on efficiency. Moreover, quality systems based on a consumer perspective tend to measure satisfaction rather than the actual quality of the service. In, for example, a music streaming service, that distinction is less important than in education or health care. Finally, there is the risk that less tangible, less measurable effects of coproduction (e.g., effects on trust) are ignored.

Our point is not that there is one superior way of measuring the quality of digital coproduction. The realistic approach is that all these perspectives must be taken into account. In the following section, we will suggest a conceptual model that builds on existing approaches, while simultaneously building in elements specific to coproduction.

4. A Model to Measure the Quality of Digital Coproduction

Definitions of quality can differ wildly, depending on their origins. The analysis above has shown that this will be especially the case for a fuzzy concept such as digital coproduction, which can be claimed as the stomping ground of various disciplines.

To start a process of reconciling the different perspectives, we will use and merge previously developed categorisations of quality that groups them into three types (Walsh 1991; Loeffler 2002; Alzaydi et al. 2018):

• a product-based definition understands quality as the degree to which a particular service or product conforms to its specifications;
• a user-based definition understands quality as the extent to which attributes of a product meet the customer’s requirements;
• a value-based definition understands quality as the extent to which services are in line with normative expectations regarding public services.

Most definitions of quality tend to be multidimensional in one respect or another combining different factors. However, to comprehensively assess the quality of digital coproduction and to allow a multistakeholder perspective, it is necessary to combine these three categories. By doing so, it is possible to tackle many of the issues mentioned earlier, because the three approaches jointly cover each other’s ‘blind spots’.

Product-based quality is the strictest of the three interpretations, as it ties quality to concrete standards to which a service must adhere, such as compatibility and reliability. This allows precise measurement and a clear ranking of assessments, as in a school report. A justified criticism of New Public Management was that it applied such a product-based logic to public services, although these are inherently different. However, while we agree with this regarding the service element of digital coproduction, the digital element of the process does rely on a product. Thus, the product-based understanding of quality is relevant to digital public services. For instance, when several organisations are involved in delivering a service, but the incompatibility of software precludes smooth collaboration, that incompatibility constitutes a lack of quality. Within the three categories, there are of course many potential criteria, leading to more detailed subcategories. One of the most important sources of variation is whether to focus on processes and interactions or on outcomes (Parasuraman et al. 1985; Brady and Cronin 2001). In our design of quality indicators, we have included both of these perspectives, to achieve a comprehensive measurement of quality.

User-based quality incorporates those indicators that reflect perceptions, such as whether the service is easy to access and whether it meets expectations. The understanding of users in the context of digital public services is twofold. It encompasses both the recipient of a service (for example, elderly or unemployed citizens) and public officials using previously developed digital tools (for example, street-level-bureaucrats). As noted earlier, a criticism of this type of measure is that it tends to equate with satisfaction, although satisfaction itself can be the result of user-based quality assessments (for example, how flexible a service is to individual circumstances). Not purely focusing on satisfaction may also mean a more objective approach, especially in the case of highly complex public services for which the outcome and thus the eventual degree of satisfaction can be hard to observe in the short term, if at all. For instance, satisfaction with hospital care tends to be determined as much by the friendliness of staff and by the perceptions of interactions as by the quality of medical treatment. Thus, the criticism against one indicator loses its sting when applying a multidimensional framework. The weaknesses of any single criterion are balanced out by the multitude of incorporated angles. Furthermore, we cannot neglect that (the subjective) user experiences are essential elements.

Value-based quality refers to those indicators that assess quality based on higher norms. In public services, they play a crucial role, as the renewed debate about ‘public values’ shows. In contrast, a business context mainly refers to value in relation to price—the ‘bang for your buck’—even though there is a debate on higher norms, for example, in the context of corporate social responsibility. Overall, this category of quality shows a noticeably high level of tensions and trade-offs between factors—especially in the context of the digital coproduction of public services. For example, there is the issue of how to efficiently handle highly personal data while maintaining privacy standards. There is also the question of how to empower citizens, one of the fundamental values and assumptions of coproduction, while also paying attention to efficiency, privacy, flexibility, and ease of use. Furthermore, democratic values (e.g., empowerment) are intrinsic aims of coproduction. However, they must be balanced against other values such as efficiency (coproduction can be costly and time-consuming) and accountability (coproduction can muddy the division of responsibilities). It is only in considering the trade-offs that the relevance and weight of values can be truly grasped.

Figure 1 shows our overall multidimensional conceptualization of quality. Above all, this figure reveals that bringing together different perspectives and criteria does not mean to get rid of tensions when assessing quality. Instead, the framework helps to make them more visible and to investigate empirically how the stakeholder perspectives might differ regarding the assessment of quality. We are convinced that an open discussion will help to identify and overcome barriers of respective collaborative processes.

In the conceptualisation of specific indicators, we have deliberately sought not to create something entirely new and detached from what was previously developed, but have, insofar as is possible, aimed to integrate existing measurements, indices, and indicators on quality, most notably the PSQ and SERVQUAL models (Guenoun et al. 2016; Sabadie 2003). These are important reference points for observing product-based and user-based quality in public services, with constituent dimensions that take into account the specific context of the public sphere (e.g., tangibles, reliability, responsiveness, comprehension). We have also included insights from the literature on service design and satisfaction in the context of digitalised services as well as the well-known ISO frameworks on Product Quality and Quality in Use. As there is a lack of specification of the value-based understanding of quality in the literature, we could not refer to existing measurements for this category, but ourselves designed items inspired by the literature of public values and by the specific requirements of service delivery in the public sector (Weber 1922; Li and Shang 2020; Kim et al. 2021).

These indicators have three important limitations. First, public services differ regarding contextual settings. Hence, the proposition of general indicators might not always capture the specific understanding of quality in a public service under investigation. Second, as quality is a result of negotiation processes and the power to define quality is based on existing accountability relationships, its assessment is also time-sensitive. Changes in the assessment of quality thus do not stem only from changes in the quality but also from a change in the dominant perspectives in judging quality. A third and final limitation relates to the fact that some of the indicators might refer to outcomes as well as to the process of coproduction (for example, efficiency, empowerment, and democratic values). As it is hard, if not impossible, to provide a clear conceptual demarcation, the most realistic strategy to deal with potential confusion is to clearly state whether the focus of the assessment is on the process or on the outcomes.

5. Conclusions

In this article, we have aimed to develop an interdisciplinary, comprehensive framework for measuring the quality of digital coproduction. Clearly, this is only the skeleton of what should eventually become a more refined, coherent framework, but it is nevertheless a useful starting point for addressing the main issues such a framework must face. Such issues include: (1) interdisciplinary differences in the emphasis on particular values; (2) differences in the notions of users in the disciplines; (3) the need to take account of contextual differences when applying the framework; and (4) difficulties in distinguishing the process from output/outcome measures.

One of the most important questions concerns what values the term quality represents, an issue that even within disciplines is a complex one. The literature identifies various such values: effectiveness, efficiency, satisfaction, inclusion, and more. Disciplines tend to stress some values over others, which complicates interdisciplinary discussions. For example, efficiency and ease of use are dominant values in the development of technical systems, a frame which is then transplanted into public service contexts. In contrast, values of privacy/security or Weberian principles might be more important from a public administration perspective. In working towards a reconciliation of different values within an interdisciplinary team, there is a logical tendency to compromise by simply adding up different categories of values. Yet, that can be problematic: first, because it gives no clear guidelines on how to reach an overall assessment, and second, because it tends to favour those values which can be more easily measured.

A related Issue is how the role of users is perceived. When they are seen as customers, quality is interpreted as a measure of how satisfied they are and how likely they are to remain loyal. Of course, in a public service context, satisfaction matters as well. However, such an approach can be problematic, because the use of public services is often not voluntary and there is no effective competition. This has indeed been one of the major criticisms of market-oriented public sector reforms. A more fundamental issue is whether the use of a public service is the only type of involvement that should be measured. Although in a business context there are many examples of users codesigning products and services, this is usually seen as a way of raising the perceived quality of those products and services, but is kept separate from the quality measurement itself. In a public service context, the inclusion of users in the design and delivery of public services can be a democratic aim in itself, apart from whether or not that leads to better services in a more technical sense.

Public services are also part of a more political context, in which the way that services are delivered, or the fact that they are delivered at all, is seen as an expression of democratic choices. That means the assessment of a public service partly derives from a wider context, beyond the specific characteristics of the service. To some extent this is also the case in a commercial context: one’s satisfaction with a company’s services are often influenced by one’s feelings about the brand and the reputation of the company. In the public sector, such a connection is especially strong: assessments of public services have sometimes been found to be mostly unrelated to technical quality. There is known to be a strong connection between general trust in and expectations of the public sector, and the evaluation of services. If this is so, choosing quality measurements that only incorporate aspects of the service itself is a mistake.

Moreover, it is important to acknowledge that the quality of coproduced public services might refer to the process of coproduction itself as well as to the final output. Some of the variables we have presented in Figure 1 refer both to the process as well as to the output. Think, for example, about the values of empowerment or efficiency. Both of these indicators play a role over the course of the coproduction process, as they are directly related to the way coproduction gets shaped. Ultimately, the output might be considered inefficient or the citizen might even feel disempowered. This requires the assessors to be very explicit about the focus and starting point of their assessment.

All of this influences how the measurement of quality is set up, with respect to indicators, but also with respect to the process: how and when users are approached. Achieving a unified approach in an interdisciplinary context is hard work, for various reasons. The concerns of one discipline can seem marginal in the eyes of another. Specifically, the emphasis in both IT and business on efficiency can lead to a dominant place of this value in an evaluation. Furthermore, the concepts used by various approaches are overlapping and fuzzy, which means long and difficult discussions are required between scholars who might have difficulties learning each other’s scientific languages and who may regard the concerns of the other disciplines as either shallow or overly finicky. Yet, this collaboration is necessary in order to reach a comprehensive framework.

This article has attempted to bring together the most important concepts encountered when measuring the quality of digital coproduction, in the hope that this will encourage further exploration of how the strengths of different perspectives can be combined.