Next Article in Journal
Sustainable Environmental Management System Integration and Business Performance: A Balance Assessment Approach Using Fuzzy Logic
Next Article in Special Issue
The Role of Public Innovation Labs in Collaborative Governance—The Case of the Gipuzkoa Lab in the Basque Country, Spain
Previous Article in Journal
Evaluating the Ecological Services of Roof Greening Plants in Beijing Based on Functional Traits
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

A Strategy for a Sustainable Local Government: Are Participatory Governments More Efficient, Effective, and Equitable in the Budget Process?

1
Department of Public Administration, Dongguk University, Seoul 10326, Korea
2
Korea Public Finance Information Service, Seoul 04637, Korea
3
Korea Institute of Public Finance, Sejong 30147, Korea
*
Author to whom correspondence should be addressed.
Sustainability 2019, 11(19), 5312; https://doi.org/10.3390/su11195312
Submission received: 17 July 2019 / Revised: 23 September 2019 / Accepted: 25 September 2019 / Published: 26 September 2019
(This article belongs to the Special Issue Citizen Participation in Sustainable Local Decision-Making)

Abstract

:
Citizen participation in budget processes is an attractive governance strategy for creating sustainable local finance. In participatory governance, citizens are engaged in the governmental policy decision-making process for sustainable communities. Despite the importance of a participatory government, its instrumental benefits are uncertain and remain unexamined at the local level. No one has offered any extensive evidence based on large-N data to ascertain such benefits. This article fills this gap in the literature by testing the impacts of participatory budgeting on local financial outcomes. The results show that participatory governments are financially more effective and equitable without sacrificing efficiency. Advancing a more institutional perspective, this article explains the identified effects of participation mechanisms in the budget process on different local financial outcomes.

1. Introduction

Citizen participation in administrative or policy decision-making is a strategy to create better and more sustainable development and good governance, drawing wide attention from public administration scholars and practitioners [1,2,3,4,5,6,7,8,9]. Furthermore, in 2015, the United Nations announced its 2030 agenda, which includes sustainable development goals and offers citizen participation as one of the most important policy tools contributing to sustainable development through democratic governance in the decision-making process [10]. Ideally, it has both normative benefits (e.g., citizenship, accountability, and legitimacy) and instrumental benefits (e.g., organizational performance, quality of life, economic growth, and financial soundness) [6,11,12].
However, public employees are worried about instrumental benefits in justifying participation initiatives [5,13,14,15]. Participation may become symbolic, window dressing, manipulated, or politicized, and scholars have not provided any convincing evidence regarding its instrumental benefits [14]. Existing research tends to emphasize its normative benefits or is often qualitative in nature [5,6,9,16,17,18]. The few large-N studies that examine the instrumental benefits are limited in scope (e.g., restricted to a particular policy) and measurement (e.g., perceived benefits). Accordingly, many governments and managers are reluctant to initiate participation institutions, questioning whether they are worth the effort [13,14]. This article addresses this question by examining whether, by establishing participatory mechanisms, local governments can produce desirable outcomes for a sustainable community [19,20].
In particular, it focuses on governments’ institutional efforts to establish participation mechanisms in the budget process because participatory budgeting is one of the most important participatory channels that directly affect multiple financial outcomes [21,22,23,24]. Participating budgeting is a deliberative process in which the public is involved in the decision-making of budget allocation or financial management [22]. Participatory budgeting has mainly operated at the local level, but the first participatory budgeting was recently launched in the Korean central government. Despite the considerable impacts that participatory budgeting have on the budget process, little research has directly explored the financial consequences of citizen participation in the budget process. To overcome this limitation, this study captures the varieties of citizen participation in the budget process and their effects on financial outcomes. For this study, multiple outcome dimensions of participatory budgeting such as efficiency, effectiveness, and equity were examined because a government has various institutional goals in terms of competing values [25,26,27,28,29].
The results will help governments decide to what extent they should use citizen participation as a governance strategy when building a sustainable community, given the numerous competing strategies on the local government reform landscape. This study also helps scholars reconsider and enrich the models that have been prescribed to better design citizen participation institutions. Theoretically, this article develops an institutional perspective to clarify the links between participatory budgeting and local financial outcomes.

2. Literature Review

The importance of assessing the impact of citizen participation has long been recognized in terms of its complexity and difficulty [12,30]. Even though various stakeholders have different preferences, expectations, and evaluations with respect to citizen participation, two challenges exist: the varieties of participation mechanisms, and the multiplicity of participation outcomes according to domestic and international studies.

2.1. Varieties of Participation Mechanisms

The main approach to citizen participation deals with the use of various mechanisms in order to capture citizen participation as a generalized strategy or practice [31,32,33]. This is necessary because an institutional environment promoting citizen participation influences all government decision-making across such mechanisms. None of these mechanisms are perfect, and they do not exist in isolation. Using various mechanisms may offset the weakness of a particular one with the advantages of another, allowing for greater exchanges between citizens and government [32,34]. Institutional theories offer an adequate explanation of participation mechanisms [35,36,37,38]. Citizen participation mechanisms are institutions because they enact the rules of the game that govern the government citizen relationship. Mandated or not, they shape a regularized or routinely enacted behavioral pattern or structure, becoming operational institutions [38] or sets of institutional rules. In Ostrom’s [39] words, they “permit” citizens to participate. Institutions such as public hearings can be mandated and designed to permit more participation, but citizens may not actually participate unless they believe that their preferences are reflected and the issue at stake draws their attention [13,14]. Thus, for many local governmental decisions, citizens may not provide much input, despite the existence of participation institutions.
In contrast, participation institutions play a constraining role for public managers. In Ostrom’s [39] words, they “require” managers to keep participation channels open. In fact, institutions are predominantly viewed as “humanly devised constraints that shape human interaction” [39] (p. 30), which forces humans to conform to those rules [35]. For managers, participation mechanisms are institutional rules that influence their actions and choices. When these participation mechanisms are routinely available or used, they become part of the “institutional infrastructure of participation” of a community. Their effects on citizens are contingent upon whether the policy issues at stake draw citizens’ energy. Their effects on public managers, however, are more consistent. Managers know that at any time, citizens could participate, their decisions could be questioned, and low productivity could raise red flags. In other words, participation institutions create accountability expectations for managers that affect decision-making and managerial behaviors [40]. Accountability expectations are a tool that motivates managers to justify their beliefs and actions. When accountability pressure is too high, it may lead to blame avoidance and risk-averse behaviors on the part of bureaucrats [41,42]. As a result, such accountability pressure is a necessary force to regulate human behaviors to achieve institutional goals by enabling local managers to enhance local community outcomes in response to citizens’ demands.
Even though public managers adopt various participation mechanisms because of accountability pressures, various challenges to operating such mechanisms exist. The first challenge regards the level of authenticity of citizen participation mechanisms [14]. Public managers hesitate to utilize citizen participation mechanisms effectively because they produce substantial administrative burdens by delaying decision-making or increasing additional administrative costs. Therefore, prior studies have highlighted that participation mechanisms are a window-dressing, and public managers are simply attempting to demonstrate that they establish functional tools that reflect citizen voices or ideas without considering their real instrumental benefits [13,14]. A key factor is not how many participation mechanisms are adopted, but if such mechanisms are operated effectively. The second challenge concerns how much citizens can be involved in actual decision-making [32]. Various participatory channels exist according to the citizens’ authority in the decision-making processes. Citizens are passive targets in one-way communication mechanisms such as emails or newsletters, but delegatory participation mechanisms enable citizens to have more direct authority in decision-making processes. Consequently, the substantive effects of participatory mechanisms depend on how much decision-making public managers delegate to citizens [32]. The last consideration is the duration of the implementation of citizen participation mechanisms. Citizen participation studies have paid scant attention to how long governments have operated participation mechanisms, because participation in administrative processes has only been recently introduced as an administrative reform tool. Institutional maturity results in desirable outcomes. In this sense, as a government operates participation mechanisms for longer periods, participatory outcomes can increase.

2.2. The Multiplicity of Participation Outcomes

In addition to the participatory mechanism, the outcomes of citizen participation have been emphasized. Aside from the distinction between normative and instrumental benefits, participation outcomes could be further categorized such as process-oriented (e.g., increased trust and inclusion), content-oriented (e.g., public safety and environment quality), or user-oriented (e.g., stakeholder satisfaction) [34,43]. Nabatchi [6] differentiates four types of benefits: (a) intrinsic benefits; (b) instrumental benefits for citizens; (c) instrumental benefits for communities; and (d) policy and governance instrumental benefits. Empirical studies have assessed the outcomes at different levels of analysis. Individual-level benefits include lower citizen cynicism [13], public employees’ trust in citizens [30], and trust in the government [44]. However, prior studies have paid little attention to the financial outcomes of participation budgeting in a particular administrative process. Furthermore, such participatory effects could have multiple benefits, including efficiency, effectiveness, and equity because public organizations pursue competing values in response to various external stakeholders [27,45]. Perry addresses the importance of multiple performance dimensions in government:
“The field’s historical concerns about administrative performance can be summarized by four concepts, each beginning with the letter “e”: economy, efficiency, effectiveness, and equity. Little research or theory within the field has attempted to understand how tradeoffs among prominent performance criteria affect administrative behavior or how multiple performance objectives influence administrative outcomes. Most theories of organizational effectiveness do not shed light on these issues” [45] (p. 12).
Similarly, participation budgeting could produce multiple financial outcomes through public managers’ accountability that responds to citizens’ various preferences [46]. At a given level of analysis, the financial outcomes may be in tension with one another. As commonly argued in the literature, efficiency often contrasts with effectiveness and equity [47], although Neshkova and Guo [46] find that citizen inputs improve both government efficiency and effectiveness. Other studies observe that citizen participation improves effectiveness but decreases efficiency [43]. No quantitative study in public administration literature has measured the equity dimension affected by citizen participation in the budget process. To fill this gap, this study clarifies how the varieties of participatory mechanisms in the budget process affect different local financial outcomes (3Es: efficiency, effectiveness, and equity).

3. Hypotheses

Although citizen participation is a critical factor in local decision-making, whether participation mechanisms produce various local outcomes remains unclear. This study clarifies this question by examining the relationships between the varieties of participation mechanisms in budget process and multiple financial outcomes. In particular, public participation in the budget process has direct impacts on the allocation and management of financial resources [23]. Citizens can provide their ideas or voices on how to allocate financial resources, as well as how to innovate financial systems in public hearings or budgetary citizen boards if they believe that such channels are trustworthy and valuable [13]. Citizens can also be involved in either monitoring or overseeing to check not only that administrative costs are not wasted, but also keep track of debt increase due to inefficient spending [21]. Such citizen involvement efforts result in desirable consequences, such as less wasteful and more sustainable or inclusive financial management.

3.1. Efficiency

The focus of public management studies has been on technical efficiency to evaluate public services—the extent to which an organization produces an output or outcomes with the least resources [27,28,48,49]. Prior studies have proposed different arguments about how citizen participation affects efficiency. The traditional view claims that citizen participation often causes higher administrative costs because it is time-consuming and increases conflict in policy systems, complicates decision-making, and creates decision delays in addition to increased operating costs [3,14,43]. Participation may also increase decision-making costs. Poor decisions may be made because citizens lack knowledge of complex and technical issues [50]. Furthermore, the accountability pressures generated by participation institutions may lead managers to become more cautious with decisions and to stick to bureaucratic red tape, which reduces efficiency.
The contrasting view submits that citizen participation enhances efficiency because it enables citizens to suggest new or innovative ideas that decrease wasteful projects, eliminate duplicated processes, and save unnecessary administrative costs. Citizen participation may also help governments avoid costs associated with citizens bringing litigation against the government [43] and improve implementation efficiency by reducing citizens’ resistance and cynicism [13]. Accountability pressures may also stimulate managers to concentrate on their tasks and find ways to reduce waste and inefficiency. This is particularly true when citizens are concerned with cost-related performance indicators [51]. Neshkova and Guo [46] provide empirical evidence suggesting that participation of institutions in the budget process increases the efficiency of state transportation departments. In terms of financial management, citizens provide innovative ideas that contribute to the efficiency of budget programs. Citizens’ monitoring of the budget process prevents local governments from increasing administrative costs, which are less related to actual programs. Given the competing arguments, we suggest the following neutral hypothesis.
Hypothesis 1 (H1).
Participatory budgeting does not necessarily produce more efficient financial outcomes.

3.2. Effectiveness

Effectiveness refers to the degree to which goals or objectives are achieved [45] and is often measured by the outputs and outcomes of public services [27]. It is not a cost-related measure. Most of the literature agrees on the positive impact of citizen participation on government effectiveness [5,9,20,21]. The explanation is largely based on the strengths of citizens: they have local knowledge or context-specific information, so their inputs help managers recognize citizen preferences, and they also possess ideas and resources that detect risks and problems of inappropriate financial operatives [46,52].
The accountability pressures resulting from participation institutions lead public managers to be more concerned with citizens’ voices. In this process, managers are forced to pay attention to various ideas for better decision-making and financial management. In addition, accountability pressure requires managers to seek open discussion instead of relying on their own judgment in the financial decision-making process. Managers are more likely to appreciate new evidence rather than sticking with old evidence [42]. They are more likely to make robust evaluations of the available alternatives for optimal financial decision-making [53]. The accountability mechanism reduces managerial bias and increases managerial effectiveness to promote better financial outcomes. Specifically, such managerial effectiveness is salient to debt management because effectively managing debts is indispensable to financial sustainability [54]. As a result, citizen participation in the budget process leads managers to seek effective debt management by accommodating citizens’ ideas or requests.
Hypothesis 2 (H2).
Participatory budgeting produces more effective financial outcomes.

3.3. Equity

Equity is the fair distribution of resources or benefits across various groups. Despite the importance of its democratic value, public management studies have rarely examined the impact of citizen participation on equity [47,55,56,57,58,59]. An important purpose of citizen participation is to expand social groups’ access to bureaucracy, thus empowering and enriching the groups [36,44]. Whether this actually leads to more equitable public outcomes is still a controversial question. Some worry that participation may paradoxically reduce social equality because the disadvantaged are either excluded from the participatory process or lack the time, resources, knowledge, or skills necessary for meaningful engagement [33,60]. For example, Caucasians and males are more represented on citizen advisory committees, on citizen boards, and in public hearings [61,62]. Many citizen surveys do not represent the local population [63].
Despite these concerns, the availability of participation institutions means that the disadvantaged could participate and require the government to make decisions about their welfare when salient issues affecting their wellbeing arise and when they feel their interests are at stake. Their problem of poor knowledge and skills can only be addressed via learning by engaging in participation. In a community with a weaker participatory environment, it would be harder for the disadvantaged to improve their participation capability.
Moreover, the accountability expectation that the disadvantaged might participate is strong in the budget allocation process. As such, managers can allocate more financial resources on behalf of the disadvantaged. When the accountability pressure is high from disadvantaged groups in participatory governance, managers are more likely to look for solutions that contribute to social equity, given that participation budgeting is a financial tool benefiting the disadvantaged. From the neo-institutional perspective, participation mechanisms manifest the value of citizen empowerment, creating normative pressures for managers to attend to the interests of the disadvantaged [2]. As a result, local governments allocate more financial resources to social welfare programs in response to underrepresented groups’ involvement in participation mechanisms in the budget process.
Hypothesis 3 (H3).
Participatory budgeting produces more equitable financial outcomes.

4. Data and Methods

4.1. Data

To test the hypotheses, we merged numerous datasets about South Korean local governments. First, the Korea Institute of Public Finance (KIPF) conducted a survey in 2016 to measure the varieties of participatory mechanisms in local budget processes in South Korea. The office phone numbers of local managers in participatory budgeting positions were obtained through organizational charts on local government websites. Through personal phone calls, we confirmed each managers’ email address and sent them our survey’s website link, including questionnaires about the varieties of their participatory budgeting. From a sample of 243 South Korean local governments, 168 local managers responded to our online survey, constituting a 69.5% response rate. To identify whether respondents represented South Korean local governments, we compared the differences between respondents and non-respondents across some local variables through t-tests (see Table 1). We confirmed their representativeness because we found no significant differences between respondents and non-respondents. In addition to survey variables, we used numerous archival data to measure dependent and control variables. We derived local financial data from the Local Finance Integrated Open System (LOFIN). Local demographic and political characteristics originated from the Korean Statistical Information Service (KOSIS) and the National Election Commission Statistics (NEC Statistics).

4.2. Dependent Variables

For this study, we used three dimensions of local financial outcomes to test the hypotheses, as summarized in Table 2. First, we used administrative costs, including employee wages and operating costs, to measure the efficiency of local financial outcomes. Administrative costs constitute indirect spending that is not used to produce local services directly. As administrative costs decrease, local governments can produce local services by using fewer inputs. Specifically, administrative costs are calculated using the ratio of wage and operating costs to total expenditure. Effectiveness is measured by to what extent a local government is financially sustainable. Local governments aim to sustain their financial conditions by reducing the amount of government debt [54]. Fiscal sustainability is measured by calculating the ratios of debts and liabilities to total assets in order to measure both short-term and long-term sustainability.
Finally, financial equity is measured by calculating the percentage of social welfare spending to total budget. When a local government has considerable social welfare spending, it spends much of its budget on social welfare programs for disadvantaged groups. The local financial data originated from the LOFIN, which the South Korean Ministry of the Interior and Safety operates.

4.3. Explanatory Variables

We measured explanatory variables that affect dependent variables through numerous data sources. We used the varieties of participation mechanisms in budget processes as independent variables. We obtained participatory budgeting data from the 2016 KIPF survey, which examined various aspects of local participatory budgeting through several questionnaires. In particular, we measured the effectiveness of participatory mechanisms in budget processes. The concept of effective participation was emphasized in prior studies [13,14] because the unauthenticity of participation mechanisms lead to citizen cynicism, producing undesirable participatory results. Designing effective participation mechanisms plays a core role in producing intended outcomes because citizens are not willing to be involved in window-dressing participatory channels [14]. This study used the following survey question: How effective is your local participatory budgeting in the budget process? (five-point Likert scale)
To measure the level of citizen delegation in budgetary decision-making, in our survey we also asked managers about how much of the budget decision-making they delegate to citizens who participate in participatory budgeting. Moreover, through the survey, we identified the number of participatory budget process mechanisms. The respondents verified how many participatory mechanisms their local governments have in the budget process: budget notices or emails, citizen surveys, citizen committees, citizen public hearings, citizen budget programs, and citizen performance evaluations. Finally, in our survey, we examined the duration of participation by asking how long their local governments have been operating participatory budgeting.
In addition, we selected numerous variables to control for the impacts of participatory budgeting on local financial outcomes. First, we used the chief’s party affiliation as a control variable because the progressive-party-affiliated chiefs tend to provide greater social welfare services. The turnout rate is also a critical variable that influences local financial conditions. A high level of political participation leads local governments to pay greater attention to effective financial management that leads to desirable financial outcomes. We measured these political variables using an archival source such as the NEC statistics.
We used population size, local fiscal condition, and aging population as the control variables that affect local financial conditions. Generally, local governments with large populations can efficiently manage financial systems because of the scale of their economies. A high level of per capita gross regional domestic product may relate to more economic expenditure to boost local industry, leading to lower social expenditures. In particular, aging populations may increase social welfare expenditure because local governments allocate a considerable amount of their budget to social programs for the older population. Population and economic-related variables originated from the KOSIS. We also controlled for the effect of per capita lagged total expenditure on financial outcomes because past budget sizes are closely related to current financial conditions.
Lastly, we controlled for the characteristics of participation from the citizens’ and managers’ perspective, as suggested by prior studies [11,50,63]. Citizens’ commitment to deliberation and representation in the participation process are critical elements because participatory results are sounder and fairer under high levels of deliberation and representation [50,63]. Citizens’ participatory deliberation and representation were measured by two items from the KIPF survey: (1) How much do citizens actively discuss your participatory budgeting? (deliberation); (2) How much do citizens in your participatory budgeting represent the entire community? (representation). In addition to citizens’ participatory characteristics, we measured managers’ intention to operate participatory mechanisms from two items of the KIPF survey, as suggested by prior research [11]. The survey asked managers the following questions: (1) How much do you intend to adopt public hearings in the budget process? (2) How much do you intend to involve citizens in the budget process?

4.4. Analytical Procedure

Through our hypotheses, we examined the causal relationships between participation varieties in budget processes and how such participation systems influence local financial outcomes in terms of competing values and multiple local outcomes. Since this study tests causal relationships, we examined whether participation variables used as independent variables were endogenous in response to dependent variables across all models. Through the Hausman test (χ2), we tried to identify the endogeneity of our causal models by using instrumental variables [64]. The results of the endogenous tests showed that all variables were exogenous, demonstrating that endogeneity is absent, such as reverse causality or omitted variables between participating variables and local financial variables. Therefore, as the ordinary least square (OLS) regressions are available, we ran a series of OLS regressions. We also examined the regressions’ assumptions such as the normality of residuals, multicollinearity, and homoscedasticity through k-density, variance inflation factor (VIF), and Breush–Pagan tests [65]. These assumptions were upheld.

5. Results

Table 3 shows a summary of the descriptive statistics. On average, South Korean local governments’ administrative costs constituted 21% of total expenditures. The ratios of debt and liabilities to assets are respectively 0.78% and 2.46%, and social welfare costs accounted for 31% of total expenditures. The effectiveness of participatory budgeting had a mean value of 3.30, which was larger than the median value (3.00). The level of citizen delegation in budget decision-making was low, and the mean value was 2.62, which was lower than the median value. On average, South Korean local governments had 2.09 participatory budgeting channels. The average duration of participatory budgeting was 82.2 months.
Prior to the regression analysis, we preliminarily analyzed how the dependent variables vary by the different scales of the effectiveness of participatory budgeting, as displayed in Table 4 and Figure 1. Except for administrative costs, debt, liabilities, and social expenditures showed variations consistent with our hypotheses. As the effectiveness of participatory budgeting increases, debt and liabilities tend to decrease. On the other hand, social expenditures are higher at a high level of effectiveness than at a low level of effectiveness. Administrative costs do not vary with the effectiveness of participation.
For further verification of our hypotheses, we ran several multivariate regressions. The regression analyses indicate that the F-values were statistically significant across all models, and the White test satisfied the goodness of fit for all models. Overall, Hypotheses 2 and 3 were partially supported in the effectiveness of participatory budgeting. As demonstrated in Table 5, the results indicated that the varieties of participation mechanisms did not enhance the efficiency of local financial outcomes but favored their effectiveness and equity. Firstly, none of the independent variables were statistically associated with administrative costs, which measure efficient financial outcomes. These results show that participation mechanisms in budgetary processes do not produce efficient financial outcomes because of the additional administrative burdens of operating participatory channels.
Secondly, the effectiveness of participation budgeting contributed to fiscal sustainability by lowering the amount of local debt and liabilities. As local governments operate participatory budgeting effectively, they positively affect debt management. Specifically, effective participation mechanisms in budget processes significantly reduce the ratios of debt and liabilities to total assets. The participatory effects were greater for liability, a long-term measure, than for debt. The coefficients were −0.178 (debt, p-value < 0.05) and −0.258 (liabilities, p-value < 0.1), respectively. Other participation variables were not statistically related to local debt or liabilities.
In terms of equity, the results confirmed that the varieties of participation budgeting led local governments to spend greater budget amounts on social welfare programs that enhanced social equity. The effectiveness of participation mechanisms in budget processes positively affected the ratio of social welfare expenditures to total expenditures (coefficient: 1.239, p-value < 0.05), but other varieties of participation budgeting such as citizen delegation, number, and duration of participation mechanisms in budget processes did not show statistically significant results.
Some control variables were statistically significantly related to local financial outcomes. The turnout rate was associated with the effectiveness of financial outcomes by reducing the amount of local debt. Local politicians may feel highly pressured about their financial soundness because citizens are sensitive to substantial government debts, which might in turn affect their voting behaviors. This pressure can motivate local politicians to manage debt size under a high turnout level. Population was related to financial efficiency by reducing administrative costs, but larger governments suffered from greater local debt and liabilities. Generally, larger governments tended to rely on substantial amounts of debt to cover numerous demands from diverse groups. Past administrative costs were a significant predictor of current administrative costs by controlling for time effects. Per capita gross regional domestic product decreased social welfare expenditures because wealthy communities tend to focus more on spending in times of economic prosperity than inclusive communities. An aging population positively affects debt, liabilities, and social welfare spending because more financial resources are needed to support an aging population. In particular, aging has greater effects on liabilities (coefficient: 0.0659) than on debt (0.0354), showing that aging aggravates intergenerational equity by causing greater financial burdens to younger generations. In citizens’ participatory characteristics and managers’ intentions of participation mechanisms, citizen participatory representation contributes to lower administrative costs, even though such costs increase as managers intend to operate more public hearings.

6. Implications

Citizen participation in administrative processes contributes to public outcomes to create a sustainable community [19]. Despite the positive effects, few studies have examined quantitatively how a variety of citizen participation institutions affect multiple local outcomes. In this sense, the results are meaningful because participation institutions contribute to the effectiveness and equity of local financial outcomes.
The overall finding is that participatory governments in the budget process may not be more efficient, but they are more effective and equitable. Notably, the fact that the institutions for participatory budgeting are not statistically significant in the model predicting efficiency does not indicate a negative relationship. In other words, participation institutions increase effectiveness and equity without hurting efficiency. Our results indicate that there is no association between participation mechanisms in the budget process and efficiency, contrary to the findings of Neshkova and Guo [46]. Many public administration studies view citizen participation as conflicting with efficiency [33,43], but our results show that this is not necessarily true. An institutional environment for citizen participation may decrease the efficiency of some aspects while increasing the efficiency of others. From these findings, local managers can better understand that operating participatory budgeting does not necessarily waste money or increase administrative costs.
In line with the literature [5,20,21], our results support a positive association between the effectiveness of participatory budgeting and effective financial outcomes. In an institutional environment that empowers citizens, citizens are more likely to get what they desire from a sustainable community—better jobs, income, education, homes, and safety. The precondition for such desirable community outcomes is to maintain financial soundness by investing in various public programs for community well-being. In particular, our results indicate that effectively operating participatory budgeting contributes to more effective debt management by reducing the size of local debt and liability. The recent rapidly aging population in Korea leads to greater interest in public financial sustainability because the size of government debt is expanding due to the increased spending to support the older generation. In this sense, this demonstrates that effectively operating participation mechanisms in the budget process is a desirable strategy to enhance financial sustainability by reducing the size of debt and liabilities.
The positive finding on equity is significant. It is the first time that the relationship between participation and equity is supported in a large-N study in the literature. Although a primary intention of citizen participation has always been to enhance social equity and help less educated and lower-income individuals, its actual effects have been questioned, given that the privileged are more likely to be willing to participate and influence decisions [55]. The potential biases are undeniable when participation mechanisms are used for particular issues. However, as our results suggest, when effective participatory budgeting constitutes a strong institutional environment supporting citizen participation, the equality gap is more likely to be reduced through increased social welfare spending. In short, the best way to develop a more equitable society is to invest in public programs or services that are helpful to socially disadvantaged groups by creating participatory institutions that empower citizens, particularly the less privileged.
Interestingly, several control variables draw attention to further implications for superior financial outcomes. Even though the turnout rate is a political variable, the results indicate that it can create administrative benefits by lowering local debt size. Local governments that focus on fiscal sustainability must pay greater attention to citizen voices and ideas by encouraging citizens to participate in local elections. Another notable finding is that citizens’ representation contributes to lowering administrative costs, even though all variables on the varieties of participation show no significant association with administrative costs. As many scholars emphasized the representation of participation [50,55,63], ensuring citizen representation in the participation process plays a role in enhancing financial efficiency because citizens can check financial activities in a fair and balanced way. Local governments need to consider how participation mechanisms are representative of the entire community if they want a more streamlined bureaucracy.
The practical implication of this study is straightforward: local governments should establish a stronger institutional environment that empowers citizen involvement in budget processes by mandating or regularly using a wide variety of participation channels. Doing so is not only democratic but also good for business. It is only an institutional myth that citizen participation is inefficient and has minimal instrumental benefits. This does not mean that managers can ignore the design of institutional-specific participation mechanisms in budget processes, nor does it mean that a government’s responsibilities cease once these institutions are established. Our results simply suggest that a stronger participatory infrastructure in the budget process contributes to local financial effectiveness and equity for sustainable development. Better design, better mobilization, and better management of particular participation mechanisms only make this infrastructure stronger.

7. Conclusions

We start with an important question that has long haunted scholars and practitioners: Does participatory budgeting bring instrumental benefits for local financial outcomes in the budget process? We find the answer is a resounding yes. Through a large-N study with objective outcome measures, we show that local governments with stronger institutional environments supporting citizen participation are more effective and equitable without being less efficient in the budget process. At a time when managerialism dominates the administrative reform agenda of many governments, the present findings send a clear message—building a participatory institutional environment is good for business!
Although this article discovers the meaningful instrumental benefits of citizen participation, there are some limitations that point to future research questions. Although time-related variables are included, this research design is not based on a longitudinal study. Future studies need to confirm our findings through a longitudinal research design.
This study also measures participatory outcomes at the government-wide level, ignoring the individual, group, and agency levels. Future studies may expand to all levels, particularly the individual level, directly assessing how the institutional environment for citizen participation affects the behavioral choices of citizens, managers, and other stakeholders. In developing our hypotheses, we examined the potential impact of institutional expectations on managers, but we did not empirically test such impacts. In other words, this article shows the link between participation budgeting and government-wide outcomes, but the behavioral mechanisms are not revealed. Addressing this issue requires a more systematic theorizing and a more deliberate research design.
Despite these limitations, we advance a supply-side institutional perspective that complements the explanations based on individual participation mechanisms or citizens’ actual participation. Building an institutional infrastructure that empowers citizens will increase citizens’ participation. However, in a modern society, citizens cannot, would not, and probably should not participate all the time or in all decisions. Our institutional perspective avoids this problem. An institutional infrastructure “permits” citizens to participate and “requires” managers to keep the channels open, creating normative pressures and accountability expectations for the government to better work towards a sustainable community.

Author Contributions

Conceptualization, S.-h.J.; data curation, Y.O. and H.S.; formal analysis, Y.O.; Methodology, S.-h.J.; Writing-Original Draft Preparation, Y.O.; Writing-Review and Editing, S.-h.J.; Validation, S.-h.J.; data curation, H.S.

Funding

This research received no external funding.

Conflicts of Interest

The authors declare no conflicts of interest.

References

  1. Cooper, T.L.; Bryer, T.A.; Meek, J.W. Citizen-centered Collaborative Public Management. Public Adm. Rev. 2006, 66, 76–88. [Google Scholar] [CrossRef]
  2. Dahl, R.A. Democracy and Its Critics; Yale University Press: New Haven, CT, USA, 1989. [Google Scholar]
  3. Kweit, M.G.; Kweit, R.W. Implementing Citizen Participation in a Bureaucratic Society: A Contingency Approach; Praeger: New York, NY, USA, 1981. [Google Scholar]
  4. Leighninger, M.; Mann, B. Planning for Stronger Local Democracy: A Field Guide forLocal Officials; National League of Cities Center for Research and Innovation: Washington, DC, USA, 2011. [Google Scholar]
  5. Moynihan, D.P. Normative and instrumental perspectives on public participation: Citizens summits in Washington, D.C. Am. Rev. Public Adm. 2003, 33, 164–188. [Google Scholar] [CrossRef]
  6. Nabatchi, T. Democracy for Public Administration Addressing the Citizenship and Democratic Deficits: The Potential of Deliberative. Am. Rev. Public Adm. 2010, 40, 376–399. [Google Scholar] [CrossRef]
  7. Verba, S.; Schlozman, K.L.; Brady, H. Voice and Equality: Civic Volunteerism in American Politics; Harvard University: London, UK, 1995. [Google Scholar]
  8. Walters, L.; Aydelotte, J.; Miller, J. Putting More Public in Policy Analysis. Public Adm. Rev. 2000, 60, 349–359. [Google Scholar] [CrossRef]
  9. Woolum, J. Citizen Involvement in Performance Measurement and Reporting A comparative case study from local government. Public Perform. Manag. Rev. 2011, 35, 79–112. [Google Scholar] [CrossRef]
  10. Oliver, F.; Peter, S. Citizen Participation in the UN Sustainable Development Goals Consultation Process: Toward Global Democratic Governance. In Global Governance: A Review of Multilateralism and International Organizations; Brill: Leiden, The Netherlands, 2016; Volume 22, pp. 555–574. [Google Scholar]
  11. Wang, X.H. Assessing Public Participation in U.S. cities. Public Perform. Manag. Rev. 2001, 24, 322–336. [Google Scholar] [CrossRef]
  12. Yang, K.; Pandey, S. Further Dissecting the Black Box of Citizen Participation: When does Citizen Involvement Lead to Good Outcomes? Public Adm. Rev. 2011, 71, 880–892. [Google Scholar] [CrossRef]
  13. Berman, E. Dealing with cynical citizens. Public Adm. Rev. 1997, 57, 105–123. [Google Scholar] [CrossRef]
  14. King, C.S.; Feltey, K.M.; Susel, B. The Question of Participation: Toward Authentic Public Participation in Public Administration. Public Adm. Rev. 1998, 58, 317–326. [Google Scholar] [CrossRef]
  15. Yang, K. Public administrators’ Trust in Citizens: A Missing Link in Citizen Involvement Efforts. Public Adm. Rev. 2005, 65, 273–285. [Google Scholar] [CrossRef]
  16. Roberts, N. Public Deliberation in an Age of Direct Citizen Participation. Am. Rev. Public Adm. 2004, 34, 315–351. [Google Scholar] [CrossRef]
  17. Stewart, K. Write the Rules and Win: Understanding Citizen Participation Game Dynamics. Public Adm. Rev. 2007, 67, 1067–1076. [Google Scholar] [CrossRef]
  18. Thomas, J. Public Involvement in Public Management: Adapting and Testing a Borrowed Theory. Public Adm. Rev. 1990, 50, 435–445. [Google Scholar] [CrossRef]
  19. Agyeman, J. Sustainable Communities and the Challenge of Environmental Justice; NYU Press: New York, NY, USA, 2005. [Google Scholar]
  20. Hawkins, C.V.; Wang, X.H. Sustainable Development Governance: Citizen Participation and Support Networks in Local Sustainability Initiatives. Public Works Manag. Policy 2011, 17, 7–29. [Google Scholar] [CrossRef]
  21. Ebdon, C. Beyond the Public Hearing: Citizen Participation in the Local Government Budget Process. J. Public Budg. Account. Financ. Manag. 2002, 14, 273–294. [Google Scholar] [CrossRef]
  22. Ebdon, C.; Franklin, A. Searching for a Role for Citizens in the Budget Process. Public Budg. Financ. 2004, 24, 32–49. [Google Scholar] [CrossRef]
  23. Ebdon, C.; Franklin, A. Citizen Participation in Budgeting Theory. Public Adm. Rev. 2006, 66, 437–447. [Google Scholar] [CrossRef]
  24. Franklin, A.L.; Ho, A.T.; Ebdon, C. Participatory Budgeting in Midwestern States: Democratic Connection or Citizen Disconnection? Public Budg. Financ. 2009, 29, 52–72. [Google Scholar] [CrossRef]
  25. Ammar, S.; Duncombe, W.; Hou, Y.; Jump, B.; Wright, R. Using Fuzzy Rule–Based Systems to Evaluate Overall Financial Performance of Governments. Public Budg. Financ. 2001, 21, 91–110. [Google Scholar] [CrossRef]
  26. Andrews, R.; Entwistle, T. Does Cross-Sectoral Partnership Deliver? An Empirical Exploration of Public Service. Effectiveness, Efficiency, and Equity. J. Public Adm. Res. Theory 2010, 20, 679–701. [Google Scholar] [CrossRef]
  27. Boyne, G.A. Concepts and Indicators of Local Authority Performance: An Evaluation of the Statutory Frameworks in England and Wales. Public Money Manag. 2002, 22, 17–24. [Google Scholar] [CrossRef]
  28. Hatry, H.P. The Status of Productivity Measurement in the Public Sector. Public Adm. Rev. 1978, 38, 28–33. [Google Scholar] [CrossRef]
  29. Stallmann, J.I.; Deller, S. State Tax and Expenditure Limitations, Business Climate, and Economic Performance. Public Budg. Financ. 2011, 31, 109–135. [Google Scholar] [CrossRef]
  30. Yang, K.; Callahan, K. Assessing Citizen Involvement Efforts by Local governments. Public Perform. Manag. Rev. 2005, 29, 191–216. [Google Scholar]
  31. Brady, H.E.; Verba, S.; Schlozman, K.L. A Resource Model of Political Participation. Am. Political Sci. Rev. 1995, 89, 271–294. [Google Scholar] [CrossRef]
  32. Fung, A. Varieties of Participation in Complex Governance. Public Adm. Rev. 2006, 66, 66–75. [Google Scholar] [CrossRef]
  33. Kathlene, L.; Martin, J.A. Enhancing citizen participation: Panel designs, perspectives, and policy formation. J. Policy Anal. Manag. 1991, 10, 46–63. [Google Scholar] [CrossRef]
  34. Bryson, J.M.; Quick, K.S.; Slotterback, C.S.; Crosby, B.C. Designing Public Participation Processes. Public Adm. Rev. 2013, 73, 23–34. [Google Scholar] [CrossRef]
  35. DiMaggio, P.J.; Powell, W.W. The Iron Cage Revisited: Institutional Isomorphism and Collective Rationality in Organizational Fields. Am. Sociol. Rev. 1983, 48, 147–160. [Google Scholar] [CrossRef]
  36. North, D.C. Institutions, Institutional Change and Economic Performance; Cambridge University Press: Cambridge, UK, 1990. [Google Scholar]
  37. Ostrom, E. Governing the Commons: The Evolution of Institutions for Collective Action; Cambridge University Press: Cambridge, UK, 1990. [Google Scholar]
  38. Williamson, O.E. The New Institutional Economics: Taking Stock, Looking Ahead. J. Econ. Lit. 2000, 38, 595–613. [Google Scholar] [CrossRef]
  39. Ostrom, E. Understanding Institutional Diversity; Princeton University Press: Princeton, NJ, USA, 2005. [Google Scholar]
  40. Romzek, B.S.; Dubnick, M.J. Accountability in the Public Sector: Lessons from the Challenger Tragedy. Public Adm. Rev. 1987, 47, 227–238. [Google Scholar] [CrossRef]
  41. Hood, C. Accountability and Transparency: Siamese Twins, Matching Parts, Awkward Couple? West Eur. Politics 2010, 33, 989–1009. [Google Scholar] [CrossRef]
  42. Lerner, J.S.; Tetlock, P.E. Accounting for the Effects of Accountability. Psychol. Bull. 1999, 125, 255–275. [Google Scholar] [CrossRef] [PubMed]
  43. Irvin, R.A.; Stansbury, J. Citizen Participation in Decision Making: Is Worth the Effort? Public Adm. Rev. 2004, 64, 55–65. [Google Scholar] [CrossRef]
  44. Wang, X.H.; Wart, M.W. When Public Participation in AdministrationLeads to Trust: An Empirical Assessment of Managers’ Perceptions. Public Adm. Rev. 2007, 67, 265–278. [Google Scholar] [CrossRef]
  45. Perry, J. Strategies for building public administration theory. Res. Public Adm. 1991, 1, 1–18. [Google Scholar]
  46. Neshkova, M.I.; Guo, H. Public Participation and Organizational Performance: Evidence from State Agencies. J. Public Adm. Res. Theory 2012, 22, 267–288. [Google Scholar] [CrossRef]
  47. Frederickson, H.G. The Recovery of Civism in Public Administration. Public Adm. Rev. 1982, 42, 501–508. [Google Scholar] [CrossRef]
  48. Poister, T.H. Measuring Performance in Public and Nonprofit Organizations; Jossey-Bass: San Francisco, CA, USA, 2003. [Google Scholar]
  49. Wang, X.H.; Bryer, T.A. Assessing the Costs of Public Participation: A CaseStudy of Two Online Participation Mechanisms. Am. Rev. Public Adm. 2013, 43, 179–199. [Google Scholar] [CrossRef]
  50. Oh, Y.; Lim, S. Connecting a Missing Link between Participation in Administration and Political Participation: The Mediating Role of Political Efficacy. Int. Rev. Adm. Sci. 2017, 83, 694–714. [Google Scholar] [CrossRef]
  51. Ho, A.T.; Coates, P. Citizen Participation: Legitimizing Performance Measurement as a Decision Tool. Gov. Financ. Rev. 2002, 18, 8. [Google Scholar]
  52. Vigoda, E. Administrative Agents of Democracy? A Structural Equation Modeling of the Relationship between Public-Sector Performance and Citizenship Involvement. J. Public Adm. Res. Theory 2002, 12, 241–272. [Google Scholar] [CrossRef]
  53. Wang, X.H. Assessing Administrative Accountability Results from a National Survey. Am. Rev. Public Adm. 2002, 32, 350–370. [Google Scholar] [CrossRef]
  54. Edward, S. Debt Relief and Fiscal Sustainability. Rev. World Econ. 2003, 139, 38–65. [Google Scholar] [CrossRef]
  55. American Political Science Association Task Force. American Democracy in an Age of Rising Inequality. Perspect. Politics 2004, 2, 651–666. [Google Scholar] [CrossRef]
  56. Behn, R. Rethinking Democratic Accountability; Brookings Institution: Washington, DC, USA, 2001. [Google Scholar]
  57. Berger, B. Attention Deficit Democracy: The Paradox of Civic Engagement; Princeton University Press: Princeton, NJ, USA, 2011. [Google Scholar]
  58. Peterson, P.E. City Limits; University of Chicago Press: Chicago, IL, USA, 1981. [Google Scholar]
  59. Pitts, D.A. Little Less Conversation, a Little More Action: Using Empirical Research to Promote Social Equity. J. Public Adm. Res. Theory 2011, 21, 77–82. [Google Scholar] [CrossRef]
  60. Peterson, S.A. Political Behaviors; Sage: Thousand Oaks, CA, USA, 1990. [Google Scholar]
  61. Baker, W.H.; Addams, H.L.; Davis, B. Critical Factors for Enhancing Municipal Public Hearings. Public Adm. Rev. 2005, 65, 490–499. [Google Scholar] [CrossRef]
  62. Cole, R.L.; Caputo, D. The Public hearing as an effective citizen participation mechanism. Am. Political Sci. Rev. 1984, 78, 404–416. [Google Scholar] [CrossRef]
  63. Robbins, M.D.; Simonsen, B.; Feldman, B. Citizens and Resource Allocation: Improving Decision Making with Interactive Web-Based Citizen Participation. Public Adm. Rev. 2008, 68, 564–575. [Google Scholar] [CrossRef]
  64. O’Sullivan, E.; Rassel, G.R. Research Methods for Public Administrators; Routledge: New York, NY, USA, 2003. [Google Scholar]
  65. Simonson, I.; Staw, B.I. Deescalation Strategies: A Comparison of Techniques for Reducing Commitment to Losing Courses of Action. J. Appl. Psychol. 1992, 77, 419–426. [Google Scholar] [CrossRef]
Figure 1. Effectiveness of participation budgeting and financial outcomes.
Figure 1. Effectiveness of participation budgeting and financial outcomes.
Sustainability 11 05312 g001
Table 1. Balance test (t-test) between respondents and non-respondents.
Table 1. Balance test (t-test) between respondents and non-respondents.
RespondentsNon-RespondentsDifference
Number16874
Chief party affiliation (%)0.36686390.36486490.001999
Turnout rate (%)58.8259959.00393−0.17794
Population (log)11.9907612.09726−0.1065
Lagged total expenditure (per capita)5125.6215197.208−71.587
Population aged 65 and above (%)18.4810118.235950.24506
Note: Chief party affiliation (progressive party = 1, otherwise = 0).
Table 2. Measurement and data sources.
Table 2. Measurement and data sources.
VariablesMeasuresSources
Dependent Variables
3EsEfficiency: Administrative costs
Percentage of administrative costs to total budgetLOFIN
Effectiveness: Fiscal sustainability
Percentage of debt to total assetsLOFIN
Percentage of liabilities to total assets
Equity: Social welfare spending
Percentage of social expenditure to total budgetLOFIN
Independent variables
Variety of citizen participationEffectiveness of participation mechanisms in budget processKIPF Survey 2016
Level of citizen delegation in budget decision-making
Number of participation mechanisms in budget processes
Duration of participation mechanisms in budget processes
Control variables
MiscellaneousChief’s party affiliation: Progressive Party (1/0)NEC Statistics
Turnout rate in local elections (%)
Population logarithmKOSIS
Percentage of lagged administrative cost to total budgetLOFIN
Per capita gross regional domestic product (millions of won)KOSIS
Per capita lagged total budget (thousands of won)LOFIN
Percentage of population aged 65 and aboveKOSIS
Citizens’ participatory deliberationKIPF Survey 2016
Citizens’ participatory representation
Managers’ intention of public hearing
Managers’ intention of citizen involvement in budget process
Notes: LOFIN: Finance Integrated Open System; KIPF: Korea Institute of Public Finance; NEC: National Election Commission; KOSIS: Korean Statistical Information Service.
Table 3. Descriptive statistics.
Table 3. Descriptive statistics.
NMinimumMaximumAverageSD
Administrative costs (%)2435.37802535.2451320.922095.090321
Debt (%)24309.4131160.7891481.407057
Liabilities (%)2430.1824.222.4618932.411937
Social expenditures (%)2437.8402670.3421331.1220414.78739
Effectiveness of participatory mechanisms in budget processes169153.3076921.046536
Level of citizen delegation in budget decision-making169152.6213021.199611
Number of participatory mechanisms in budget processes169162.0946751.103256
Duration of participatory mechanisms in budget processes243015382.2880726.34669
Chief’s party affiliation: Progressive Party (dummy)243010.3662550.482775
Turnout rate (%)24334.820671.3636458.880175.111405
Population (log)2439.21044116.3584312.02321.242489
Lagged administrative costs (%)2434.37039731.9284517.866334.533408
Gross regional domestic product (per capita)2427.930081388.464633.9392630.71773
Lagged total budget (per capita)243958.810116548.355147.4223736.017
Population aged 65 and above (%)2436.6537.4918.406387.714224
Citizens’ participatory deliberation169153.355031.01974
Citizens’ participatory representation169153.2189351.037681
Managers’ intention of public hearing1691105.2248522.251264
Managers’ intention of citizen involvement in budget process1691105.6745562.416585
Table 4. Effectiveness of participatory budgeting and financial outcomes.
Table 4. Effectiveness of participatory budgeting and financial outcomes.
Effectiveness of ParticipationAdministrative Costs (%)Debt (%)Liabilities (%)Social Expenditures (%)
522.501370.3308791.87727339.56176
420.82780.7893972.45764733.37742
321.594850.7451542.30741929.58436
219.796241.0889082.595223.89514
119.521221.7045143.93777828.18316
Table 5. Results of ordinary least square (OLS) regressions.
Table 5. Results of ordinary least square (OLS) regressions.
EfficiencyEffectivenessEquity
Administrative Costs (%)Debt (%)Liabilities (%)Social Expenditures (%)
Effectiveness of PB−0.116−0.178 **−0.258 *1.239 *
Level of citizen delegation0.213−0.0769−0.09480.671
Number of PB0.09580.01390.0470−0.00489
Duration of PB0.005070.001970.00188−0.00254
Progressive Party−0.244−0.176−0.288−1.104
Turnout rate−0.0347−0.0287 *−0.05440.0834
Population−0.809 ***1.252 ***1.298 ***0.106
Lagged administrative costs1.009 ***−0.0441 *−0.05840.597 ***
Gross regional domestic product−0.00140.00310.0019−0.0840 ***
Lagged total budget−0.00010.0002 ***0.0001−0.0037 ***
Population aged 65 and above −0.02170.0354 **0.0659 **0.339 *
Citizens’ deliberation−0.1260.05010.246 *−0.185
Citizens’ representation−0.308 **0.0843−0.04671.085
Managers’ intention of public hearing0.224 ***0.03620.0067−0.424
Managers’ intention of citizen involvement −0.104−0.01150.02010.002
Number168168168168
F291.42 ***5.20 ***5.15 ***30.23 ***
White test (χ2)143.91147.68151.67152.91
R-squared0.9330.6180.4450.733
Adj. R-squared0.9260.5810.3900.706
Note: * p-value < 0.1; ** p-value < 0.05; *** p-value < 0.01. PB: Participation mechanisms in budget processes.

Share and Cite

MDPI and ACS Style

Oh, Y.; Jeong, S.-h.; Shin, H. A Strategy for a Sustainable Local Government: Are Participatory Governments More Efficient, Effective, and Equitable in the Budget Process? Sustainability 2019, 11, 5312. https://doi.org/10.3390/su11195312

AMA Style

Oh Y, Jeong S-h, Shin H. A Strategy for a Sustainable Local Government: Are Participatory Governments More Efficient, Effective, and Equitable in the Budget Process? Sustainability. 2019; 11(19):5312. https://doi.org/10.3390/su11195312

Chicago/Turabian Style

Oh, Youngmin, Seong-ho Jeong, and Heontae Shin. 2019. "A Strategy for a Sustainable Local Government: Are Participatory Governments More Efficient, Effective, and Equitable in the Budget Process?" Sustainability 11, no. 19: 5312. https://doi.org/10.3390/su11195312

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop