The Official Website as an Essential E-Governance Tool: A Comparative Analysis of the Romanian Cities’ Websites in 2019 and 2022

Abstract

This paper aims to measure the quality of all Romanian cities’ websites in 2019 and 2022, before and after the disruptive event of COVID-19. Since the official websites are the core instrument of e-governance, the changes in the quality of Romanian cities’ websites reflect the changes in the development of urban e-governance in Romania. The COVID-19 lockdowns and contact restrictions and the moving of most activities into the online environment had the potential to impact the performance of Romanian cities’ websites significantly and catalyze the progress of local e-governance. The quality of Romanian cities’ official websites was measured using an adapted survey instrument proposed by Holzer and Kim and Manoharan, Melitski, and Holzer. The tool covers five critical criteria of website quality or performance: personal data security, usability, content, services, and citizen participation. The research results may seem contrary to expectations: the improvements of websites have been incremental rather than transformational. Although our research provides only a case study, we may assert that the reform of traditional administration remains a step-by-step process. Our study’s main contribution consists in showing that the progress of e-governance is an incremental process even in a situation of emergency that requires the moving of most activities into the online environment.

1. Introduction

1.1. ICTs as Means of Governance

As a convergent set of goods, applications, and services used to produce, store, process, manipulate, distribute, and exchange information in a digital form [1], information and communication technologies (ICTs) increasingly transform the way people live, work, and communicate. In order to thrive or even survive within the knowledge-based economy and information society that ICTs bring about, political organizations, public institutions, businesses, NGOs, and individual citizens must acquire, process, and use the information at a high level of quantity, quality, and speed [2].

Like any human product, ICTs create winners and losers and have unforeseen consequences [3], especially as they continue to evolve swiftly. ICTs’ access and use are still unequally distributed within and between countries [4] and will continue to be so. The digital divide—that is, the gap between the information-haves and information-have nots—will probably widen instead of narrow because digital inclusion and digital empowerment require—besides access to hardware, software, and relevant content or services—training for digital literacy skills [5]. Moreover, ICTs raise some crucial issues that must be properly addressed: (1) the loss of privacy, abusive surveillance, and violation of human rights, (2) the vulnerability to the systemic network failures of the internet or power grid, (3) the loss of “human skills” and the “crowding out” of natural communities, (4) the adverse health effects of radio waves and sedentary lifestyles, (5) electronic waste and carbon emission, (6) digital exclusion, and (7) children’s exposure to multiple forms of abuse [6].

Despite their uneven development, side effects, and risks, ICTs offer the formidable advantage of putting the interaction between any “supply side” and “demand side” [7] under the signs of reliability, efficiency, transparency, flexibility, responsiveness, accountability, and sustainability [1,8,9]. Operating in a highly competitive environment, companies have good incentives to adopt e-business models and strategies. They need to constantly lower their production costs, maintain consumers’ and the public’s confidence, adapt their product offerings to the ever-changing consumers’ demand, and demonstrate a genuine commitment to environmental protection. Therefore, it is not surprising that companies have exploited the enormous potential of ICTs much more than other types of organizations have. The primary stakeholders of companies―shareholders, managers, employees, suppliers, and customers―use ICTs at the informational, operational, and transactional level [10] more frequently and systematically than those of public institutions and NGOs.

Although it operates under monopoly conditions, may cover its inefficiency through taxes or loans, and bears the burden of bureaucratic inertia, most countries have strategically and systematically used ICTs to “facilitate more accessible government services, allow greater public access to information, and make government more accountable to citizens” [1]. The development of digital governance has been stimulated by the pressure from the business community already accustomed to using ICT on a large scale, by the increased expectations of the citizens regarding the quality of information and public services, by the tendency of the state to expand its scope of action, by the government’s interest in increasing citizens’ legitimizing participation, and by the rise of globalization. To illustrate the growing trend of the state’s scope, we mention that, in the US, total government expenditure accounted for less than 2% of the national income until 1916 but reached 44% in 2020 [11]. To highlight the decrease in citizens’ participation in political life and, implicitly, the deficit of democratic legitimacy, we mention that the participation rate in the parliamentary elections in Romania decreased dramatically from 86.19% in 1990 to 31.84% in 2020 [12,13].

States have good reasons to use ICTs as a means of governance, but they also have to overcome various barriers to digital government: (a) technological barriers (infrastructure, lack of interoperability, data access), (b) organizational barriers (lack of strategy, human resources, financial resources, digital skills, capacities of managers), (c) legal and ethical barriers (legal constraints, lack of citizens’ trust), and (d) cultural barriers (risk aversion, bureaucratic culture, fear of change) [14].

As the use of ICTs in government bears the dialectical influence of good incentives and major barriers, the emergence of e-government is proving to be a winding process, far from the pattern of linear progress. Using the composite indicator “the e-government development index (EGDI)” to measure the willingness and capacity of national administrations to utilize ICTs to deliver public services, the United Nations Department of Economic and Social Affairs (UNDESA) provides snapshots with relative rankings of the e-government development in these states [15]. The United Nations E-Government Survey highlights the uneven development of digital governance. For example, some countries such as Korea and Estonia have made spectacular progress in the development of digital governance, obtaining much better EGDI scores and superior positions in the rankings. On the other hand, we can see that other countries such as the United States, Italy, and Germany have dropped many positions in the 2020 rankings due to minor progress or even stagnation in improving e-governance. This may seem surprising given the levels of economic development of those countries. It is worth noting that Romania ranked 50th in 2003 with an EGDI of 0.48 and ranked 55th in 2020 with an EGDI of 0.76.

All hypercomplex societies, such as the Euro-Atlantic countries, need good governance (not just a government) to deal with major challenges such as the 2008 economic and financial crisis, the COVID-19 pandemic, or Russia’s military invasion of Ukraine. Only good governance can create and sustain an environment in which government agencies, businesses, NGOs, and citizens work together to foster society’s robust, equitable, and sustainable development. If e-governance based on the use of ICTs proves to be a feasible form of good governance, it is important to understand its evolution so that we may identify solutions to improve it.

In this context, we aim to assess the quality of all Romanian cities’ websites in 2019 and 2022, before and after the disruptive event of COVID-19. Being “the primary interfaces between government and citizens” [16], reliable channels for controlled information-sharing with stakeholders and media [17], effective media for providing public services [18], and the “means through which public administrations inform and educate the public, provide transparency and promote economic activities” [19], official websites and especially the cities’ websites are the core of e-governance [20]. Therefore, we may assert that the changes in the quality of Romanian cities’ websites reflect the changes in the development of urban e-governance in Romania. To manage the COVID-19 pandemic, governments imposed lockdowns, contact restrictions, and the moving of many economic, social, educational, and administrative activities into the online environment. Under these circumstances, the cities’ website performances in 2022 are expected to be significantly higher than they were in 2019. If the analysis does not confirm to this expectation, we may conclude that the progress of local e-governance in Romania is not transformational but tends to remain incremental despite the disruptive events.

The tool we used to evaluate Romanian cities’ official websites is taken and adapted from Holzer and Kim [21] and Manoharan, Melitski, and Holzer [22], who also applied it to evaluate municipalities’ websites worldwide.

In the following, we will present some essential contributions to understanding (local) e-governance as an emerging, evolving process and official websites as its essential tool. Then, we will present some instruments for measuring the website’s performance, indicate the research hypotheses, and present and discuss the data obtained from evaluating all Romanian cities’ websites.

1.2. The Official Website as an E-Governance Tool: A Brief Literature Review

Before discussing the importance of the official website in developing e-governance, it is helpful to clarify the distinction between government and governance, as many authors seem to use these terms interchangeably. Government is a system of rules and procedures enforced by a hierarchic bureaucracy and used for controlling a country, city, or community at the highest level. When we refer to the government, we also mean the activities involved in controlling a country, city, or community and the people who officially carry out these activities. If government “increasingly depends on other organizations to secure its intentions, deliver its policies, and establish a pattern of rule” [23] and is a part of a “self-organizing network” [24], we refer to it by the term “political governance” or simply “governance”. As such, governance acts as a democrat-liberal agency. Nowadays, intensifying globalization, the increase in transnational economic activity, the rise of regional institutions such as the European Union (EU), and the enhanced role of non-state actors (especially multinational corporations) in the delivery of public services make the state more “interested in various strategies for creating and managing networks and partnerships” [23]. Under these circumstances, government works as governance.

In order to understand the fundamental attributes of e-governance (that is, electronic or digital governance), it is helpful to consider some relevant definitions of government/governance recorded in the literature: “the use of IT [information technology] to enable and improve the efficiency with which government services are provided to citizens, employees, businesses and agencies” [25]; “the use of information and communication technology (ICT) applications to deliver various government services” [26]; “governments’ use of Information and Communication Technologies (ICTs) combined with organizational change to improve the structures and operations of government” [27]; “the application of electronic means for the interaction between government and citizens and government and businesses, as well as in internal government operations to simplify and improve democratic, government and business aspects of governance” [28]; “the usage of information and communication technology (ICT) in a government setting” [29]; “the use of ICT tools and applications to provide better services to citizens and businesses” [30]; “the use of information technology to raise the quality of the services governments deliver to citizens and businesses” [31]; “a way for governments to use the most innovative information and communication technologies, particularly web-based Internet applications, to provide citizens and businesses with more convenient access to government information and services, to improve the quality of the services and to provide greater opportunities to participate in democratic institutions and processes” [32]; “utilizing the internet and the World Wide Web for delivering government information and services to citizens” [33]; “a unification of information and communication technology (ICT) in all the operations to augment the potential of the government to satisfy the requirements of the local public” [34]; “an interconnected system in which the government interacts with citizens and provides augmented services to them through electronic applications” [34]; “the single most important point of contact between government and citizens, and successes and failures therein have effects that can be felt across a political system” [35]; “a technology-mediated relationship between citizens and their governments” [36]; “the use of information and communication technologies (ICTs) to improve efficiency and communication within government, enhance communication between the public and the private sectors, and facilitate the availability and delivery of services from government to the public” [37]; “the systematic and strategic use of Information and Communication Technology to promote more efficient and effective government, facilitate more accessible government services, allow greater public access to information, and make government more accountable to citizens” [1]; “the technology-enabled transformation of government” [38]; “the use of information and communication technology to disseminate information and services by governments” [39]; “web-based services from agencies of local, state and federal governments” [40].

Firstly, we note that the authors of the definitions mentioned above use the terms government and governance indiscriminately, without distinguishing between systems and political agencies that independently control societies or communities and those that exercise the control function in interdependence and collaboration with other stakeholders. ICTs have enormous transformation potential, which can be used for the benefit of society by a liberal democratic government and to the detriment of society by an authoritarian and illiberal government. The term “e-government” does not implicitly mean that ICTs are used by a political agency efficiently and beneficially for society. Digital or tech-enabled authoritarianism is not a dystopian construct but a real risk. A state may become a digital authoritarian government that uses ICTs to control and shape the behavior of its citizens through surveillance, repression, manipulation, and censorship. Moreover, the e-services it provides can prove to retain and expand political control [41].

Secondly, most definitions incorrectly refer to the forms and purposes in which ICTs are used. These forms and purposes are not all of the attributes of e-governance, nor are they necessarily the most important or defining attributes. Therefore, it is recommendable not to use them in the definition of e-governance. It is much simpler and fairer to define e-governance as ICT-enabled governance. Later, one can discuss the place it occupies in the system of governance as a whole, the instruments it uses, the forms it takes, and the goals its agents can pursue.

Thirdly, enchanted by the truly formidable potential of ICTs, theorists tend to take a hyper-optimistic view of the place of e-governance in the overall system of governance. ICT-enabled governance would not only be an ordinary component of governance, dedicated to “digitally included citizens” [5] who have the skills to work with ICT tools, but a constitutive part that transforms the entire system of governance for excellence. It is said that e-governance provides better information and public services more efficiently to citizens, other government organizations, employees, and businesses [25]. Unfortunately, many e-governance projects “fail to address the legitimate but diverse interests of many stakeholders” [42] or “have fallen short of their potential” [7]. In many countries, the percentage of failed e-governance projects is higher than that of successful ones [29]. In less developed countries, e-governance’s success has been minimal. According to some research, more than 60% of e-governance projects failed to meet the desired outcomes, and only 15% were successful [27]; of the failed projects, 35% failed completely and 50% failed partially in meeting the expected outcomes [26]. However, despite these failures, it can be argued that ICT-enabled governance is not just an extension of “on-site” governance. Since ICTs support incremental upgrades and improvements in components, features, and applications, it is expected to see only incremental changes in the first stages of e-governance. If constant, these changes could transform the ways in which the government and the public interact [26] within the whole system of e-governance.

One of the most popular and financially feasible tools for developing e-governance is the official website [33]. The website is considered a vital element of any successful e-governance strategy [43] and serves the public as “heterogeneous groups of citizens with varied personal attributes, needs, and interests” [8]. It is designed for delivering governmental information and services and strengthening the interaction between governments and citizens [43]. It carries out a variety of corresponding functions, including (a) information publication, (b) public service delivery, and (c) citizens’ participation in the decision-making process [36]. In order to engender trust and ensure successful e-governance, the website must have qualities such as security, privacy, accessibility, usability, content, and citizen participation [44], and it must be governed by the public values of transparency, accountability, openness, and fairness [8].

Although it plays an essential role in the system of government, the website planning is often done by trial and error based on subjective knowledge and intuition, with little or no formal research and evaluation [16]. Many governmental institutions copy the best features of other institutions’ websites without considering their particular objectives and values [16]. Therefore, it is not surprising that there is often a mismatch between the considerable investment made in e-governance and the low utilization of public institutions’ websites [43].

Beyond their shortcomings, the official websites enforce e-governance at three levels: (a) information, (b) transaction, and (c) engagement. At the basic level of information, government agencies use websites for publishing governmental information to the citizens. At the transaction level, a public institution enables citizens to effect various online transactions, initially in direct relation with it and later with third-party organizations from the local community. Finally, government agencies provide websites at the engagement level as platforms to facilitate citizens’ involvement in decision-making or public service processes [29]. Once developed, each e-governance level brings specific public values: (a) the transparency of information (openness, publicness, dissemination), (b) the efficiency of the transaction (citizen-centricity, cost-reduction, value for money, citizens’ satisfaction, productivity, performance, service quality), and (c) the depth of engagement (involvement, consultation, collaboration) [29].

The engagement level may, in turn, be seen as comprising four sublevels: (a) information, (b) consultation, (c) concertation, and (d) active participation. More precisely, after distributing complete, objective, reliable, relevant, and accessible information, citizens and stakeholders are involved in defining the government’s rules, goals, and obligations. Afterward, they are implicated in negotiating a compromise between confronted interests, and, finally, the citizens’ will and “wisdom” are integrated into the government’s decision making [45].

As government agencies refine their websites, delivering “beneficial new digitally enabled public services” [46], e-governance is advancing to higher stages of development, making society fairer and more sustainable. The literature contains numerous contributions to the identification and description of e-governance’s development stages depending on the improvement of the official website. Zhang and Kimathi [29] present ten such contributions, and Iannaci et al. present sixteen [25]. In general, these stages range from the simple dissemination of information of public interest and continue to the provision of a self-government platform. The stages numbers vary between 2 and 6. These contributions have the merit of indicating a possible direction for developing e-governance that all stakeholders could pursue, namely, government agencies, companies, NGOs, and citizens. On the other hand, the staged modeling of e-governance highlights the risk of creating the illusion of linear progress in the evolution of e-governance. In reality, like any social phenomenon that bears the imprint of human intentionality, e-governance has a sinuous evolution everywhere, with spectacular qualitative leaps and dramatic setbacks.

For the present research, the maturity development model of e-governance proposed by Bindu, Prem Sankar, and Satheesh Kumar is especially relevant. Using cluster analysis of the citation network, the authors identified the major topics of research on the e-governance framework design. Based on them, they defined five stages of e-governance: (1) the information stage, in which personal computers, printers, e-mail, and SMS technology enable government agencies to add a digital component focused on providing public information to conventional or manual governments; (2) the interaction stage, in which governments utilize Web 2.0 as a medium for communicating effectively with a large group of citizens and obtaining their feedback; (3) the transaction stage, in which the widespread adoption of the internet and Web 2.0 technology facilitates online money transactions, and government agencies can provide e-services safely and efficiently; (4) the transformation stage, in which the e-government transforms itself into participatory e-governance and citizens become increasingly involved in the decision-making process; (5) the symbiosis stage, in which the computing techniques of artificial intelligence, machine learning, and personal identification using cameras and other connected devices tagged with radio-frequency identification (RFID) chips enable government agencies to provide more intelligent and customized responses to citizens [28].

The model proposed by Bindu, Prem Sankar, and Satheesh Kumar is helpful in the analysis of the Romanian cities’ websites because it allows us to remark on the content of the web page elements that are partly correlated and partly disparate from the various stages of e-governance. In addition, the model seems to indicate a direction toward the sustainable development of e-governance. E-governance is not an end in itself but a means to good governance. Of course, good governance involves being inclusive and interactive with the public [47], but it also requires intelligent and personalized responses from government agencies regarding the needs and desires of its citizens, especially the poor and disadvantaged [48]. Information and communication technologies specific to the symbiotic stage of e-government can contribute to building sustainable, good e-governance—obviously not as a state of static equilibrium but as a dynamic process that involves permanent changes and transformations [39].

1.3. Instruments for Evaluating Websites’ Quality: An Adapted Holzer and Kim Model

The website has a vital role in developing e-governance; therefore, it has constantly garnered the attention of researchers. They evaluate the quality or performance of the official website with various tools at different levels of complexity. In some studies, researchers followed only a few particular aspects; in other cases, they performed holistic analyses.

We mention, for example, that Paul and Das evaluated Indian e-government websites in terms of accessibility and usability [44]. Accessibility measures how distinctively capable individuals can access, understand, interact with, and navigate the site, and it is determined by several parameters such as page layout, color choice, readability, and browser independence [44]. The usability of a website is measured by the download speed, the page size of the home page, and the broken (or dead) links [44]. In terms of accessibility, the authors concluded that Indian e-government websites “have certain moderate to critical accessibility issues”; related to usability, they found that “70% of e-government websites have a long download time and 43% of these websites have pages with broken links” [44].

Related to accessibility, the leading international standards organization for the internet proposed a set of principles—The Web Content Accessibility Guidelines (WCAG)—for making web content more reachable, especially for people with disabilities: (a) Perceivable—the information and user interface components must be presentable to users in ways they can perceive; (b) Operable—the user interface components and navigation must be operable; (c) Understandable—the information and the operation of the user interface must be understandable; (d) Robust—the content must be robust enough that it can be interpreted by a wide variety of user agents, including assistive technologies [49].

Being aware of “the importance of multidimensionality in designing, operating, and evaluating government websites”, Lee, Lee-Geiller, and Lee [8] proposed a more concise and refined variant of the Democratic E-governance Website Evaluation Model (DEWEM) consisting of 25 items under 5 factors (transparency, information suitability, service quality, security, and citizen engagement). These factors have been statistically confirmed to relate to the outcome of democratic e-governance. Moreover, the authors found “a statistically significant correlation between items in the modified DEWEM and citizens’ satisfaction and intention to use” [8].

Holzer and Kim [14] built a survey instrument for evaluating city and municipal websites consisting of 98 measures grouped into five categories: (1) Privacy/Security; (2) Usability; (3) Content; (4) Services; and (5) Citizen Participation [21].

In the article Digital Governance: An Assessment of Performance and Best Practices published by Holzer, Manoharan, and Melitski, the items that allow for a comparative survey of e-governance performance in terms of privacy/security, usability, content, services, and citizen participation appeared in a more detailed form [22].

Detailed by Manoharan, Melitski, and Holzer, the Holzer and Kim survey instrument is simple and easy to apply; it has been tested several times in evaluating the performance of municipal websites around the world. We hold that this entitles us to use a slightly modified version of it to analyze Romanian cities’ websites. The adapted tool used in evaluating the web pages of 284 Romanian municipalities is found in Table S1.

2. Materials and Methods

As mentioned above, the survey instrument used to compare the performance of the Romanian cities’ websites in 2019 and 2022 measures five dimensions of the website quality: (1) Privacy/Security; (2) Usability; (3) Content; (4) Services; and (5) Citizen Participation. The study used 98 items, with a maximum gross score of 219 and a maximum weighted score of 100. Weighting was required because each of the five dimensions had a different number of questions (18 for security and 20 for all other dimensions) and different scores (25, 32, 48, 59, 55). The five dimensions received equal weight, regardless of the number of questions used for assessment. Thus, after weighing, each dimension could receive scores from 0 to 20, and the maximum possible score is 100.

For security and the protection of personal data, concepts such as public statements on personal data protection, authentication, encryption, the management of collected information, and cookies have been operationalized. The ease of understanding the website, simple design, the length of the access page, its structure, the extent to which it addresses specific audiences, and the ability to search for information on the website were the operationalized concepts for the size of usability. In terms of content, the emphasis was on access to recent information and the existence of official documents, reports, publications, and audio-visual materials. The services provided included transactions that could occur between the local administration and citizens or business owners, as well as the submission of applications for various authorizations (permits, licenses). In terms of citizen participation, the operationalization looked at the website instruments that enable the citizens to provide feedback to local authorities, online debates on local public policies, the existence of a system for measuring citizen satisfaction, and tools for measuring local government performance.

Apart from the 98 items in the evaluation grid of the websites in 2019, in 2022, we introduced two more specifics of the pandemic period: one related to information about the pandemic (local infection rate, local regulations on the pandemic, links to other institutions responsible for public health) and another related to information on vaccination (information on the types of vaccines in the city, the presence of a list of vaccination centers in the locality and the possibility of online programs at such a center). The scores for these questions could be between 0 and 3. To ensure comparability between the results obtained in 2019 and 2022, the scores of the answers to these two COVID-19 questions were neither weighted nor added to the scores of the other measures. We have added the two questions only to see to what extent explicit information about the COVID-19 pandemic and anti-COVID-19 vaccination appears on the municipal websites. Actually, if significant, the impact of COVID-19 on the performance of the municipal websites should manifest in all five dimensions: privacy/security, usability, content, services, and citizen participation.

To ensure the fidelity of the survey instrument, we made sure that at least two evaluators evaluated each website. The evaluation grid included examples for each item, with operators receiving detailed explanations about the scoring system. If the score difference was greater than five points (5% of the maximum value of the scale), the site was evaluated once again.

The evaluation grid was applied to all Romanian cities with a functional web page during the research periods, namely, in the first two weeks of June 2019 and in the last two weeks of February 2022 (see Table S2). The information-gathering process was carried out with the help of students in the Political Sciences and Communication Sciences program of the “Alexandru Ioan Cuza” University of Iași. In the summer of 2019, we conducted a study of 268 cities with functional sites. In 2022, the number of Romanian cities with functional web pages that were surveyed increased to 284.

The hypotheses that we wanted to test are as follows:

Hypothesis 1 (H1).

The performance of e-governance in Romanian cities has significantly improved in 2022 compared to 2019.

Hypothesis 2 (H2).

The more urbanized a development region is, the higher the average e-governance score at the regional level is.

Hypothesis 3 (H3).

The larger the city’s population, the higher the e-governance score.

Hypothesis 4 (H4).

Information about the COVID-19 pandemic and vaccination can be found on most Romanian cities’ websites.

3. Results

The scores obtained after evaluating the municipal websites were systematized in a database. With the help of the PHStat statistical analysis program, several insightful outputs could be generated, which we will present and discuss in the following.

Table 1. Maximum, minimum, and average scores in 2019 and 2022.

	Average Score		Maximum Score		Minimum Score		Standard Deviation
Year	2019	2022	2019	2022	2019	2022	2019	2022
General score	23.11	27.57	49.86	61.83	9.95	11.45	7.75	8.05
Privacy/security	1.07	3.64	10.53	13.55	0	0	2.05	2.86
Usability	9.82	10.95	18.12	18.75	1.88	4.38	2.74	2.74
Content	6.65	6.63	13.20	15.2	0.8	1.2	2.64	2.33
Services	3.46	4.55	12.20	16.27	0	0	2.36	2.97
Citizen participation	2.12	1.79	10.89	12.34	0	0	2.11	1.89

shows the overall scores obtained in the years researched. The overall average score increased from 23.11 in 2019 to 27.57 in 2022. The overall performance increased by 16.43% in the pandemic years. Although significant, this improvement is below expectations. During the period analyzed, new information technologies were implemented, and many activities moved online.

In these two and a half years, the minimum scores increased from 9.95 to 11.45, and the maximum ones increased from 39.66 to 49.66, resulting in a slightly higher data spread (from 7.75 to 8.05). The fact that the average score in 2022 is only 27.57 out of 100 possible points and that 56 of the web pages (19.71%) obtained scores below 20 points shows the great distance between the ideal type of e-governance and the Romanian reality.

The histogram of the average scores is shown in Figure 1, using the steps of four units. We can notice that the distribution tends to lean even further to the right, with the number of cities with scores above 28 being smaller and smaller.

Table 2. Scores for each dimension on each development region of Romania.

Region	Privacy/Security		Usability		Content		Services		Citizen Participation		Score
	2019	2022	2019	2022	2019	2022	2019	2022	2019	2022	2019	2022
Bucharest& Ilfov	2.23	4.56	10.65	11.56	7.32	7.91	4.30	6.80	1.56	2.46	26.08	33.30
Center	1.39	4.45	9.89	11.27	7.32	6.51	3.80	4.37	2.41	2.04	24.83	28.66
North East	0.97	4.21	10.04	10.04	7.74	6.12	4.69	4.61	2.35	1.70	25.8	26.70
North West	0.97	3.1	10.36	10.35	7.05	6.92	3.51	4.86	2.39	2.30	24.31	27.55
South	0.88	3.07	9.52	10.86	6.12	6.01	3.40	4.27	1.98	1.48	21.92	25.71
South East	0.93	4.10	9.84	11.85	6.73	6.92	3.24	5.02	2.46	1.37	23.23	29.28
South West	0.75	2.60	8.97	11.00	5.37	6.54	2.65	3.76	1.49	1.27	19.24	25.19
West	1.05	3.33	9.84	11.27	5.85	7.11	2.67	4.23	1.79	1.95	21.21	27.90

shows the general score and the five component dimensions for each of the eight development regions of Romania.

Table 3. Scores for each size by the size categories of the cities.

City Type	Privacy/ Security		Usability		Content		Services		Citizen Participation		Score
	2019	2022	2019	2022	2019	2022	2019	2022	2019	2022	2019	2022
Large	2.79	6.86	11.02	12.41	8.52	9.18	5.56	8.65	4.16	3.21	32.06	40.33
Medium	0.79	4.23	10.24	11.49	8.17	7.15	4.70	5.12	2.52	2.52	26.43	30.52
Small	1.19	4.31	9.44	11.61	7.38	6.97	4.08	5.12	2.32	1.97	24.43	29.99
Very small	0.68	2.86	9.66	10.46	5.72	6.07	2.59	3.67	1.55	1.44	20.22	24.51

also shows the overall score and the five component dimensions for the four size categories of cities.

Table 4. Average scores for the pandemic questions.

Question	Medium Score
Is there information about the COVID 19 pandemic?	1.22
Is there any information on vaccination?	0.90

shows the average scores for the two pandemic questions, with the maximum possible score for each of them being 3.

4. Discussion

Although the average score increased between the two evaluations—from 23.11 to 27.57—and the difference is statistically significant (the value of the t-test being 108,057 and the p-value being practically zero), not all of the five dimensions considered have increased: the protection of personal data, the services provided, and the ease of use have higher average scores than they did in 2019, the content has remained practically constant, and the participation has decreased (Figure 2).

4.1. Has the Performance of E-Governance in Romanian Cities Significantly Improved in 2022 Compared to That in 2019?

The best results appear in the case of usability—10.95 out of a possible total of 20 (Table 4)—and in the case of the content—6.63. The other dimensions obtained low scores (3.64, 4.55, and 1.79). Regarding the evolution in the two and a half years taken into account, some changes are evident in different directions of the five dimensions. In relative terms, the progress of security and personal data protection was the most important—an increase of 340.18% compared to 2019—even if, in absolute terms, the score on this dimension is minimal (3.64). The above does not necessarily show a lack of concern for information security on city hall websites but the fact that security measures are not presented to users. The security and privacy notice is often only on the page where online users can pay taxes. The observed increase is mainly determined by inserting a note on the security and protection of personal data on more and more sites. This fact was mainly due to the implementation in Romania of the European Union legislation. In the first year of evaluation, usability obtained the absolute best score (10.95 out of a possible 20) and the best maximum score (18.75). However, this increase was more modest than the one on the security and protection of personal data (11.5%). There have been no significant changes (e.g., improved search criteria and site search capabilities) but only improvements to existing resources (e.g., shorter access pages, the ease of navigation, the coloring of already visited links). Most sites have a relatively short access page, which spans a maximum of two screens. Additionally, most of the sites analyzed have navigation elements on each page. There are very few cases where the target audience is divided into groups. At best, the sites include links for locals and tourists. There are rarely links for business owners, and there are never links for the elderly, young people, or people with special needs. Only a few sites allow citizens to fill in the forms online, and none of them offer the possibility to carry out the entire administrative procedure (from submitting an application, following the route, and obtaining the answer) exclusively electronically.

The website content score practically decreased from 6.65 in 2019 to 6.63 in 2022 (a decrease of 3.01%, which is not statistically significant, the p-value being 0.924). Most of the sites provide the town hall address and some contact details, a list of decisions/resolutions of the local council, as well as some information. The sites are not always very clear on the local budget. The minutes of the local council meetings can be found on about 50% of the sites. More than half of the sites provide information in at least two languages (Romanian and English, but there is less information in English than information in Romanian). There is no natural disaster alert system. There are only a few sites with access options for the visually impaired. Most content items refer to events that have already taken place. Upcoming events are announced in a few words and are usually related to the most festive moments of the city (legal holidays, religious festivals, festivities/commemorations of historical events, or festivals). The provision of services is a size that has increased significantly, from 3.46 to 4.55 (an increase of 31.5%), but it remains low in absolute terms. The pandemic circumstances mainly caused the increase. Because of restrictions, most sites began to allow local taxes to be paid online, either directly or through a mixed procedure. Most websites provide contact details for requesting information or filing claims, but only a few have dedicated pages for online complaints.

The score in terms of digital participation has decreased, in absolute terms, from 2.12 in 2019 to 1.79 in 2022. Although a decrease of 18.43% seems large, it is not statistically significant. The p-value is 0.0531, which is slightly higher than the 0.05 significance level. Only a few websites provide forms that can be completed online to comment on the performance of local authorities. Less than 20% of the major websites have online opinion polls, and none of them offer the possibility of holding an online referendum or signing an online petition. Mainly, the big cities had discussion forums, but some of them have disappeared, and the local authorities rarely see the ones that still exist. The authorities have transferred the part of participation involving interaction with the citizens to social networks, especially Facebook.

With the overall score and the three-dimensional scores increasing significantly, taking into account that the decreases in the other two dimensions are not statistically significant, we can consider the first hypothesis as accepted.

4.2. Does the Degree of Urbanization of the Region Influence the Performance of Municipal Websites?

To test the hypothesis about the influence of the development region on the performance of the e-government, we used ANOVA for a single factor and a significance level of 0.05, and then the Tukey–Kramer test was used, which consists of successive comparisons between groups. The ANOVA summary table and the p-value obtained in 2022 are shown in Figure 3.

The p-value is slightly lower than the significance level (0.0496), so we can say that there are significant differences between the groups. In order to be able to determine in which groups there are significant differences, we have the Tukey–Kramer test.

As shown in Figure 4, at a degree of significance of 0.05, there are statistically significant differences between the Bucharest-Ilfov Region, which is practically a conurbation around the capital of Romania, and the least urbanized region, the SW. The differences between the other regions are not statistically significant. We can say that the second hypothesis is only partially accepted.

4.3. Does the Size of the City’s Population Influence the Performance of the Municipal Websites?

The third hypothesis was that a factor in the development of e-governance is the size of the city, which means that the larger the city, the higher its overall score. Although the correlation coefficient between the two variables is relatively low (r = 0.395), to test this hypothesis, the population of cities was divided into four groups: very small cities (less than 20,000 inhabitants), small cities (20,001 to 50,000 inhabitants), medium cities (50,001 to 100,000 inhabitants), and large (over 100,001 inhabitants). For very small towns, the average score was 24.51. The same statistical information was 29.99 for small cities, 30.52 for medium-sized cities, and 40.33 for large cities. Using ANOVA for a single factor and a significance level of 0.05, followed by the Tukey–Kramer test, we tested the differences between the groups. The ANOVA summary table and the p-value obtained in 2022 are shown in Figure 5.

When the p-value is close to zero, the null hypothesis is rejected: the differences between the means are significant. Where do the differences within and between groups come from? These may be due to the different needs and resources that the cities have and the interest of the officials in the e-government. Are there significant differences between the groups? To answer this question, we used the Tukey–Kramer test again. As can be seen in Figure 6, at a degree of significance of 0.05, there are statistically significant differences between very small cities (24.51) and the rest of the city groups and between large cities and the other groups. The differences between the small- and medium-sized cities are not statistically significant.

Some citizens’ needs can be more easily met through e-government: the payment of taxes and fines, requests to local authorities, petitions, obtaining permits and authorizations, and access to information of local interest. Citizens of large cities have better access to the internet and higher computer literacy, which puts additional pressure on local authorities. We believe that the differences between the groups may have their origin in the higher administrative, financial, and human capacities of the big cities compared to those of the other groups. However, this explanation is insufficient, as the administrative and budgetary differences, especially between very small and small cities, are not large enough to justify the difference. Another qualitative research study should be able to provide better explanations for this phenomenon. We can say that the third hypothesis is still accepted.

4.4. Is There Information about the COVID-19 Pandemic and Vaccination on Most Municipal Websites?

The fourth hypothesis states that, given the pandemic conditions, most city websites will contain information about the COVID 19 pandemic and the vaccination campaign.

We would expect a positive correlation between the e-government score and pandemic information. In reality, the correlation exists, but it is weak. The correlation coefficient is 0.2534 for the first question and 0.3247 for the second (the one related to vaccination). As shown in

Table 5. Scores for the pandemic questions.

Question	Medium Score	Number of Sites with a Score of 3	Number of Sites with a Score of 2	Number of Sites with a Score of 1	Number of Sites with a Score of 0
Is there information about the COVID 19 pandemic?	1.22	51	61	73	99
Is there any information on vaccination?	0.90	39	46	50	149

, 99 sites out of 284—i.e., 34.86%—do not present any information related to the pandemic. The proportion of those that do not present any information related to vaccination is even higher—52.46%. So, we can say that the fourth hypothesis can only be partially accepted.

Due to its disruptive nature, the COVID-19 pandemic could have marked a qualitative leap in municipal sites and a shift in local e-governments to a higher, transformative (if not symbiotic) level. In reality, municipal websites reflect the impact of the pandemic to a lesser extent. Thus, the information about COVID-19 infection, protection measures, and vaccination against COVID-19 is incomplete and not adapted to the local realities. Local authorities took them over as such from central governmental agencies. Helpful information about (a) the current status of COVID-19 cases within the territory, (b) how and when to get tested or vaccinated for COVID-19, (c) whether schools and childcare facilities are open, (d) welfare options such as emergency food relief, tax relief, and rent relief, (e) the current restrictions affecting businesses in different industries [50] is scarce and hard to find on the website. Citizen participation facilities on the websites (discussion forums or “you ask, we answer” sections) could have been used extensively for (a) listening to community concerns and questions, (b) promoting the understanding of risk and health experts’ advice, (c) building resilience to misinformation, and (d) engaging or empowering communities to take positive action [51]. Paradoxically, local authorities did not use them. In fact, the citizen engagement/participation score is lower in 2022 than it was in 2019.

Considering the items that led to the scores’ decrease, the impact they could have on the citizens’ trust in the authorities, and the goal of reaching the transformative and symbiotic stage of e-governance, we propose specific tools for each dimension of the webpage. These means could be linked to some main positive results and the obstacles that stand in the way of their use (see

Table 6. Official website tools for improving e-governance.

Factors	Tools	Implementation Results	Obstacles (Implementation Risks)
Privacy/Security	Information note on the protection of personal data	Compliance with relevant European legislation Increasing user confidence	Negligence or indolence of website administrators
	Information note on the possibility of disclosing the information to a third party	Increasing transparency Protection of citizens’ privacy	Ignorance of the legal and ethical grounds of information-sharing The inability of staff to anonymize personal data
	Contact address where the user can request answers regarding the protection of personal data	Increasing user confidence	Staff shortages
	SMS auto-responder and instant messaging applications regarding the protection of personal data	Efficient communication Interactivity Increasing user confidence	Qualified staff shortages Lack of equipment
	Virus check function when downloading documents	Increasing user confidence	Budget shortage
Usability	Shortening the access page (1–2 screens)	Improving the users’ orientation on the website Increasing users’ self-efficiency in accessing data on websites	The website administrators’ habit of adding information only on the access page
	Website structure adapted to the municipality’s stakeholders (residents, tourists, businessmen, etc.) and not according to the institution’s chart	Citizen-centered orientation Increasing user satisfaction	The habit of using the website mainly in the municipality’s interest
	Site map	Increased transparency of the website	Qualified staff shortages
	Information note on completion errors associated with the forms to be completed online and how these errors can be corrected	Citizen-centered orientation Increasing users’ self-efficiency in filling in the online forms	Qualified staff shortages Lack of a vision focused on the citizens’ needs
	Sophisticated search engine	Reducing search time Facilitating complex data searches	Requires the creation or purchase of a powerful search engine
	Display the date of the most recent update for all posts	Increasing users’ confidence in up-to-date data	Staff’s inability or unwillingness to update information on the website
Content	Information note on the public policy priorities of the city administration	Increasing users’ trust in authorities Strengthening democracy	Authorities’ reluctance to promote transparency Lack of medium- and long-term strategies
	Database containing all local authorities’ decisions	Increasing transparency Increasing users’ trust in local authorities	Authorities’ reluctance to promote transparency
	Database containing all local budgets approved	Increasing transparency Increasing users’ trust in local authorities	Authorities’ reluctance to reveal their dependence on particular vested interests
	Access facilities for the visually impaired	Increasing inclusive democracy	Authorities’ reluctance to spend public money on a small public
	Database containing vacancies in the city administration	Increasing the staff recruitment base Reducing cronyism	Vested interests in public administration
Services	Online tax and fines payment function	Increasing user satisfaction	Qualified staff shortages Lack of a vision focused on the citizens’ needs
	Online tracking of applications	Increasing users’ self-efficiency	Lack of a vision focused on the citizens’ needs
	Online forms for requesting information	Increasing users’ self-efficiency in accessing data on websites	No serious obstacles
	Online bidding function for public procurement	Increasing transparency Selecting the best offers	Vested interests in public administration
	Frequently asked questions	Facilitating citizens’ access to online services	Qualified staff shortages Lack of a vision focused on the citizens’ needs
Citizen participation	Online forum	Increasing inclusive democracy Fulfilling the feedback function Collecting valuable input	Authorities’ tendency to interact with citizens on social media
	Departmental online forms for collecting specific feedback	Improving the quality of public administration	Lack of a vision focused on the citizens’ needs
	Online opinion polls	Increasing citizen participation	Lack of a vision focused on the citizens’ needs
	Newsletter	Increasing transparency Increasing citizen engagement	Qualified staff shortages
	Live broadcasting of the important public events	Increasing citizen engagement	Lack of equipment and qualified staff Authorities’ sentiment of self-inefficiency

).

Since applying e-governance tools to each dimension of municipal websites evidently produces positive results, it is easy to make a general recommendation to rapidly adopt all the means mentioned above. However, it is difficult to determine who should make the necessary changes, in what order, and at what pace.

The best way to quickly implement reforms in the Romanian e-governance system would be to set up a regulatory authority to manage municipal websites. In the absence of such a regulatory institution, local public authorities operate their web pages independently of each other, copying almost randomly specific models that seem to be successful. The immediate consequences of the lack of coordination in the management of municipal websites include the waste of material and human resources, the heterogeneity and low comparability of websites, and the lack of a strategy for using the website as an e-governance tool.

Under the coordination of such a regulatory authority, it would be easier to remove the obstacles to developing the website and e-governance. The crucial step is to adopt a vision focused on the citizens’ needs and interests. By adopting a citizen-centered perspective, local public authorities could reorganize the website’s content so that it responds as effectively as possible to their demands. A second step would be the unitary training of the staff involved in managing the websites so that they could offer similar responses to similar challenges (from requesting the online payment service of taxes and fines to informing the citizens about the measures to prevent infection with COVID-19). By applying common standards and practices, local public authorities could use their websites to increase interoperability, sharing their resources for providing better information and services to citizens.

Another important recommendation is to strengthen the feedback function at all of the website dimensions: privacy/security, usability, content, services, and citizen participation. If citizens are encouraged and helped to provide honest, prompt, clear, and informative feedback, the website can be adjusted and developed to best meet their needs and interests. Subsequently, the feedback function could be coupled with mechanisms designed to involve citizens in the decision-making process regarding issues of public interest. The citizens’ feedback should also be decisive in establishing the order and the rhythm in which the website tools and functions are introduced.

Even in the absence of state regulatory authority, municipalities could follow these recommendations in managing the COVID-19 pandemic and when they have to cope with other disruptive social problems.

5. Conclusions

Some conclusions can be drawn from this research on urban e-governance in Romania. The results also suggest new directions for research. Of the four hypotheses made, one was accepted, but the answer is not so clear for the other three. In terms of overall performance, it was observed that the average score increased significantly (from 23.11 to 27.57). However, the increase is low considering that the maximum possible score is 100. Progress can be seen in three dimensions of e-governance. The most significant advances have been made in securing and protecting personal data and services (from 1.07 to 3.64). Usability developed to a lesser extent (from 9.82 to 10.85). The content remained practically constant (from 4.46 to 4.55), but the digital participation suffered a significant decrease (from 2.12 to 1.79), although it was statistically insignificant.

The most developed dimension is technical, namely, usability. The dimensions related to politics and administration, i.e., the involvement of citizens and the information and services provided, are at shallow scores. The second hypothesis, related to the influence of the degree of urbanization of the region on the performance of e-governance, was only partially confirmed. Significant differences exist between the most urbanized region (Bucharest-Ilfov) and the least urbanized (South West). For the rest of the regions, the differences are not statistically significant. Things are a bit clearer regarding the influence of city size on e-governance performance: big cities have significantly better scores than medium, small, and very small ones. There are no significant differences between small- and medium-sized cities alone, but this could be due to the artificiality of the division into groups (from 20,001 to 50,000 inhabitants—small; from 50,001 to 100,000 inhabitants—medium), a division made in an attempt to maintain a numerical balance between groups of cities. Some citizens’ needs can be more easily met through e-government in big cities, where the cost of traveling to access public services can be higher—for example, paying taxes and fines, applying to local authorities, petitions, obtaining permits and authorizations, and accessing information of local interest. Citizens of large cities also have better access to the internet and higher computer literacy, which puts additional pressure on local authorities. These cities may also have higher administrative, financial, and human capacities than smaller cities. However, another qualitative research study could better explain this phenomenon, namely, the correlation between the overall score and the size of the city being modest (0.395).

The results of our research are in line with those obtained by Epstein in his article “Two decades of e-government diffusion among local governments in the United States” [29]. Epstein confirmed three similar hypotheses: (H1) Cities with larger populations are more likely to offer more e-government services; (H2) Wealthier cities are more likely to offer more e-government services; (H3) Cities with greater internet usage are more likely to offer more e-government services.

Given the seriousness of the last two years in terms of public health, we expected to find information about the pandemic and the vaccination campaign on many Romanian cities’ websites. However, only 65.14% of the sites have information related to COVID-19, and only 47.54% have information related to vaccination. We also expected to notice a revolutionary improvement of the municipal websites towards the transformation stage of e-governance. It is known that many companies, NGOs, and educational institutions have promptly and effectively responded to the COVID-19 pandemic, moving many of their activities or transactions into the online environment. However, the research results show that the quality of municipal websites is only gradually improving and depends on factors such as the region’s level of development or the size of the city’s population. Although the COVID-19 pandemic was disruptive, changes in the quality of municipal websites and local e-governance between 2019 and 2022 are as gradual as those described by the Stoica and Ilas [52] for the period 2006–2008, when no disruptive events affected the Romanian society.

Even if this research is a case study, it can be said that, despite the theoretical promise of rapid and substantial growth generated by e-government, the process of reforming public administration due to ICT is and remains only incremental despite the disruptive events that could catalyze a revolutionary upgrade.