How Efﬁcient Is the Implementation of Structural Funds Committed to Enhancing ICT Adoption in SMEs?

: We evaluated the execution of the operational programs (OPs) committed to encouraging the adoption of information and communication technologies (ICTs) in small and medium-sized enterprises (SMEs). To achieve this goal, we employed a novel three-stage weighted Russel directional distance (WRDD) data envelopment analysis (DEA) model in conjunction with stochastic frontier analysis (SFA), which considers indicators ofﬁcially mandated by the European Union (EU) and environmental factors, to evaluate 51 OPs from 16 EU countries. All in all, we concluded that by removing the environmental factors, about 30% of the OPs (16) reached an efﬁcient procedural performance against 20% (10). The OP more frequently viewed as a benchmark regardless of the environmental factors is “Multi-regional Spain—ERDF” that remains robustly efﬁcient within 5% and 10% tolerances. Without the removal of the environmental factors, the “number of operations supported” is the indicator that requires more attention from management authorities (MAs), whereas with their removal one-third of the OPs need to further reduce the “eligible costs decided” and improve “eligible spending”. According to our ﬁndings, more developed regions and a higher rate of ICT specialists seem to be related to an underuse of ERDF funds dedicated to boosting ICT in SMEs. These ﬁndings might be related to the administrative burden and the lack of ability of SMEs to deal with the various procedures for applying for and implementing European Regional Development Fund (ERDF) projects. Overall, it is critical to provide further support that simpliﬁes administrative procedures and addresses SMEs’ speciﬁc requirements.


Introduction
In a time of digital transformation, global markets, and pandemics, businesses must increase their internal skills to deal with dynamic environments [1]. Innovative ICTs such as the Internet of Things, mobile devices, big data, and machine learning have a proven impact on organizations' operating contexts, radically altering the business practices of many firms as well as how customers engage with these businesses and other players [2][3][4]. ICT tools are usually used to gather, save, manage, disseminate, and spread information, bringing together both physical devices (such as computers, networks, and terminals) and the software or applications which allow running them [5].
According to the literature, ICT can assist enterprises in adapting to international markets by lowering border operating costs and allowing wider access to critical innovative resources [6][7][8]. In addition, it is broadly acknowledged that ICT can provide a Another factor hampering ICT adoption by SMEs refers to the lack of perception regarding the potentialities and impacts that their digital transformation might bring [28,29]. On the one hand, SMEs risk losing competitiveness, productivity, and viability if they do not undertake digital transformation [30,31]. On the other hand, managers tend to reject digital projects since they do not know how to integrate them into the firm [28,32]. In fact, the question of whether and how innovation can be reproduced and used in a broader framework is inextricably linked to an awareness of the elements and procedures that might determine the (un)successful adoption of the technology itself [33].
To encourage ICT investment in SMEs, it is critical to adopt governmental policies that reduce the digital gap, offer free broadband Internet connectivity, and promote education [28,29,32]. In this context, digital transformation has many definitions [34]. Fitzgerald et al. [35] define digital transformation as the application of new digital technologies such as the Internet, smartphones, analytics, or smart objects to enable large business changes. These in their turn allow for boosting customer satisfaction, optimizing processes, or producing innovative business practices. As a result, digital transformation goes beyond just digitizing resources to generate value and money from digital assets. Similarly, [36] differentiates between digital transformation and digitization. Whereas digitization refers to the simple conversion of analog data into digital data, digital transformation refers to a broader understanding and shift in attitude that influences politics, business, and social concerns. This digital transition, according to the "Plattform Industrie 4.0", depicts a stride forward in which people, equipment, and goods are immediately connected to their surroundings [25].
All things considered, the advancement of ICT is critical for attaining Europe's prosperity, particularly with the continuous development of a digital globalized market. Therefore, investing in ICT projects is fundamental for achieving the European Commission's goal of preparing Europe for the digital era [37].
Over 2000-2006, ICT assistance for SMEs became a major concern, with around EUR 2.3 billion in Cohesion Policy financing, corresponding to 24% of Cohesion Policy grants for digital projects [38]. This contribution reached over EUR 2.6 billion between 2007 and 2013; however, it has since been reduced to EUR 2.1 billion for the period 2014-2020, with just 10% of Cohesion Policy funds committed to digital developments [38].
In the context of Cohesion Policy funds, Member states (MSs) were expected to conduct evaluations of their efficacy, efficiency, and impacts since 2014. Notwithstanding, there are still concerns that receive insufficient research interest, particularly when evaluations take place during the programmatic time horizon. According to [39], policymakers have to face substantial challenges in designing and implementing research and innovation policies, notably during the monitoring and evaluation stages, mainly due to a lack of appropriate data. The case of EU ICT policies is no exception since researchers have also been highlighting both the lack of scholarly attention and data on the adoption of ICTs in firms at the regional level [5]. Understanding the ICT investment strategies of EU regions is not an easy procedure [40]. Besides being considered an activity sector, ICT is also an integrated element of other linked activity sectors (e.g., e-Health) and a tool to support other activities. Because the actions implemented with European Structural and Investment Funds (ESIF) sometimes have several goals, it is difficult to identify ICT-connected activities within the allotted categories when the OPs are scheduled. The financial data of the OPs are organized into Categories of Intervention, Thematic Objectives (TOs), and priority areas. It is stated in the guideline material for regions and MSs to help them write OPs that planned ICT activities should be categorized predominantly under Thematic Objective 2 (TO2). However, ICT initiatives can also obtain funding within different TOs, and they are also incorporated into several smart specialization policies.
Hence, this study aims to contribute to the literature by proposing a novel overarching methodological framework that allows policymakers (e.g., MAs) to monitor the implementation of OPs devoted to supporting ICT adoption in SMEs through the use of a nonparametric approach. We propose a novel three-stage WRDD model in combination with SFA to evaluate 51 OPs from 16 EU MSs.
The first stage involves employing the WRDD model to calculate each OP's efficiency scores. At this stage, valuable information is computed that provides further understanding of the necessary corrections that need to occur to tackle the potential discrepancies from the OPs viewed as benchmarks. In comparison to other methodologies usually used in this context (e.g., review of benchmark case studies, econometric analysis, statistical analysis, macroeconomic and microeconomic studies), the WRDD model can be notably helpful for MAs, since it allows them to recognize the benchmarks and changes that should be operated to improve the successful implementation of the OPs. The second stage involves applying SFA to the OPs deemed as inefficient to compute the adjusted input and output factors, by removing the significant environmental effects and statistical noises. At this point, it is also possible to identify the main environmental factors that can influence the efficiency of the implementation of ERDF funds in different OPs dedicated to enhancing ICT adoption in SMEs and to see the importance of management failures. Finally, in stage 3, the previously adjusted input and output factors of every inefficient DMU are used to compute new efficiency scores with the WRDD model.
Overall, the key research questions that this work aims to address are as follows: RQ1. "Which indicators prevent the efficient utilization of ERDF allocated to boost ICT adoption in EU SMEs?" RQ2: "Which OPs were most frequently referenced as a source of best practices for the programmatic period under analysis?" RQ3: "Which OPs show more efficiency resilience in the face of probable changes in the indicators used?" RQ4: "Which environmental factors have the greatest impact on the inefficiency of the execution of OPs aimed at boosting ICT in EU SMEs?" RQ5: "How does efficiency change with the removal of environmental factors?" In this perspective, the main novelties of this work are fourfold: (1) it suggests using the WRDD DEA model in conjunction with SFA, proposing a novel three-stage WRDD model approach; (2) it provides a novel approach to conduct for the first time an evaluation strictly targeting the OPs dedicated to fostering ICT in EU SMEs; (3) it helps to identify if the changes required to attain efficiency are linked to management inefficiency or environmental factors, thus offering support that enables shaping the required policies to overcome the detected inefficiencies; (4) it computes efficiency by considering adjusted factors obtained after removing the impact that environmental factors might have on the OPs' performance.
The structure of this document is as follows: Section 2 includes an overview of the literature on Cohesion Policy evaluations of ICT in SMEs. Section 3 describes the main premises behind the approach proposed to evaluate the implementation of the OPs under scrutiny. Section 4 discusses the major reasons behind the selection of the input and output factors used in the efficiency evaluation, and also some basic statistics on the data that were used to feed the WRDD and SFA models. Section 5 examines the main results. Section 6 presents the key conclusions, offers potential policy implications, highlights the main limitations, and suggests potential future work developments.

Literature Review
Over the last ten years, several efforts have been undertaken to address the role of ICT adoption in SMEs, leading to the publication of a broad range of review studies. In this context, [41] examined two theoretical models-the diffusion of innovation theory [42] and the technology, organization, and environment structure [43]-to provide a holistic conceptual background for ICT adoption by SMEs. This integrative model comprises an overall taxonomy that categorizes some of the most internally and externally important elements influencing SMEs' ICT adoption. In a similar vein, [44] reviewed the literature on the link between ICT, SMEs, and poverty alleviation. First, this study examines the role of ICT adoption in SMEs. Then, it looks into how SMEs might adopt ICTs to help them mitigate poverty. The disparities in access and use of ICTs between firms have also been addressed in a review conducted by [45] on the digital divide (DD) among firms. This review considered the geographical area, the type of firm, and the time horizon of the study and the influence and causes of DD. Other studies reviewed the main determinants, effects, and barriers of ICT adoption in SMEs [27]. In addition, [46] analyzed the possible impacts of ICT on SMEs' performance. The importance of ICT skills and capabilities for SMEs has been targeted by [28], who discussed distinct perspectives on the topic. More recently, [26] also reviewed the subject of digital competencies in the workplace. To offer a first integrated perspective on organizational culture, sustainable development, and digitalization levels in SMEs, and their linkages, [47] proposed a conceptual model to show the relationships between these three topics. They concluded that the most studied cultural aspects were strategic alignment, organization core competencies, management, and viewpoints. Other recent studies reviewed digital innovation in SMEs [48] and concluded that it is guided by past experiences, progressing through many stages of innovation, culminating in organizational and business continuous improvement effects. In the same line of work, [49] presented a literature review to help distinguish the main difficulties and prospects faced by SMEs in the framework of digitalization and ICT advancements.
So far, none of the studies previously mentioned focused on the review of ICT policies. As highlighted by [50], there has been a lack of scholarly attention on how to select the best strategies for funding distribution across distinct ICT policies. In addition, just a few studies evaluate if this sort of funding distribution is made according to the most critical requirement of each region [51,52].
Since the effective strategy and application of suitable policy plans and programs involve familiarity with the main determinants regarding the ICT adoption and use by SMEs, we have conducted a literature review devoted to this topic, by using the keywords "ICT SMEs EU". For this purpose, the Google Scholar database was employed, only retaining the Q1 papers indexed in Scopus from 2012 onwards (representing the last 10 years). The results of the search are listed in Table 1.

Authors Main Purpose Methodologies Variables
Billon et al. [57] Explore the presence of trends that integrate innovation and the usage of ICT by firms in the EU, as well as the reasons behind them Factorial and cluster analyses Dependent variables: Individuals using the Internet, individuals using a computer, individuals ordering goods and services online, employees using a computer at work, and employees using the Internet at work. Independent variables: economic variables-regional GDP per capita; employment specialization-employment in high and medium high-technology manufacturing sectors, employment in total knowledge-intensive services, share of service employment over total, economic activity rate, research and development expenditure; human capital-life-long learning and tertiary education; ICT user characteristics-population between 15 and 64 years, population density, percentage of the population living in densely populated areas; institutional factors-the quality of government and the fiscal decentralization index Giotopoulos et al. [58] Find possible determinants of ICT adoption in Greek SMEs Ordered probit models Ruiz-Rodríguez et al. [5] Categorize and quantify DD in Spanish regions and those in EU MSs using data from firms with more than 10 workers that have adopted ICTs Propose an enterprise digital development index (EDDI) for MSs in the EU and Spanish regions through factor analysis and cluster analysis ICT connectivity to Internet-enterprises connected to the Internet, enterprises that employ ICT specialists; ICT adoption-enterprises with ERP package software, enterprises with homepage, use of social networks, workers using computers, enterprises with online publicity; e-commerce-enterprises using e-invoices; e-government-enterprises sending e-invoices to the government Reggi and Gil-Garcia [50] Examine the link between local demands and investment choices aimed at reducing regional DD in the EU Ordinary least square models Dependent variables: EU funding allocations to broadband, e-government, ICT in SMEs, and e-inclusion; independent variables: households with broadband availability, digital individuals who used the Internet for interaction with public authorities, enterprises sharing electronic information on the supply chain, individuals who have never used the Internet, Quality of Government Index, GDP per capita From the literature review conducted, it can be established that there is a scarcity of regional-level research and data on ICT at the firm level [5,50]. In addition, to the best of our knowledge, there are no studies that consider a fine-grained evaluation of the funds assigned to the OPs devoted to ICT adoption by SMEs in the programming period of 2014-2020.
Studies on the evaluation of ESIF devoted to ICT usually evaluate their impact, i.e., refer to ex post assessments [51,52]. In addition, there is a study that provides an ex ante evaluation in terms of the factors influencing the distribution of funding across distinct ICT strategies [50]. However, so far, there have been no studies that compare the implementation of the OPs devoted to ICT policies (regional or national) with their peers during the programming period, nor that identify the modifications that should occur to render an inefficient OP efficient. As a consequence, the adoption of nonparametric methodologies can be highly advantageous and appropriate, particularly since the indicators available for evaluating the Cohesion Policy can be used in conjunction with these types of methodologies.
Nonparametric techniques, including DEA, can easily handle many evaluation factors. In addition, DEA can assist in the identification of the primary factors that hinder efficiency, providing policymakers with pertinent information on how to overcome them. In this framework, [59] assessed the efficiency in the deployment of ESIF devoted to Objective 1 from 2000-2006 through a combination of the SFA and DEA techniques (the output-oriented version of the Charnes-Cooper-Rhodes (CCR) [60] and Banker-Charnes-Cooper (BCC) [61] models). Additionally, [62] applied the value-based DEA technique to appraise the execution of ESIF devoted to fostering the competitiveness of SMEs across different OPs (national and regional). In addition, [63] considered the output-oriented version of the slack-based model (SBM) combined with cluster analysis for assessing 102 OPs from 22 EU countries committed to fostering a low-carbon economy in SMEs. Finally, [64] employed the non-oriented version of the network SBM model coupled with cluster analysis to appraise 53 OPs from 19 countries dedicated to promoting research and innovation in SMEs.
Notwithstanding its utility, the DEA technique has yet to be used in the evaluation of ICT-related ESIF (see also Table A1 from Appendix A). The evaluation of efficiency through the DEA approach enables MAs to identify and follow the best practices, further understanding the necessary adjustments that must be operated on the set of metrics considered in the evaluation to attain efficiency. Therefore, we will suggest a novel three-stage WRDD DEA model approach in conjunction with SFA, whose potentialities are then explored through its application in the evaluation of the implementation of OPs committed to boosting the use of ICT in the EU SMEs. With this novel methodology, further insights can be obtained that go beyond the use of traditional DEA methods, since it allows distinguishing those inefficiencies that result from management failures from those that result from potential environmental effects.

Methodology
We adopt the WRDD model which, contrarily to the CCR [60] and BCC [61] models, provides a more thorough analysis of efficiency since it is non-radial (inputs and outputs can vary non-proportionally) and can be non-oriented [65]. In addition, unlike radial models, the WRDD model provides information on the adjustments specifically required on every input and output to make each inefficient DMU efficient [65]. Finally, when compared to the SBM DEA model [66] it additionally provides the decomposition of inefficiency.

The Weighted Russell Directional Distance Model
Consider n DMUs (DMU 1 , DMU 2 , . . . , DMU n ), and let the vectors of inputs and outputs of a given DMU o be given as x o and y o , respectively. The WRDDM formulation proposed by [65] is then obtainable as: where α r o and ζ i o are the inefficiency values for every output and input, respectively, and β R o is the overall inefficiency (when β R o = 0, the DMU under evaluation is fully efficient); ζ i o g xi refers to the reduction that should occur on the i-th input of DMU o to make it efficient. Similarly, α r o g yr refers to the increase that should occur on the r-th output of DMU o to make it efficient. The weights w y and w x assign the importance related to the outputs and inputs, and w y + w x = 1. In addition, the importance of the inefficiencies associated with every output and input is defined such that ∑ r ∈ O r y = 1, ∑ i ∈ I i x = 1. In problem (1) it is considered that the directional vectors g x and g y are defined in such a way that (−g x , g y ) = (−x o , y o ). Lastly, a variable returns to scale (VRS) technology is adopted with the imposition of ∑ n j=1 λ j = 1, λ j ≥ 0 (∀ j ). Let α r * o and ζ i * o be the optimal solutions to problem (1). The following problem allows computing the reference set of each inefficient DMU o : Let (s + * r , s − * i , λ * j ) be the optimal solution to problem (2). On the one hand, the reference point of the efficient frontier for each inefficient DMU o is: On the other hand, the reference set of inefficient DMU o is: The WRDDM model's measure of inefficiency may be translated into an SBM departing from problem (5): s.t. ∑ n j=1 λ j y rj = y ro + s + r , r = 1, . . . , s, Let (s + * r , s − * i , λ * j ) be the optimal solution to problem (5). The WRDDM inefficiency measure can be decomposed from:

Robustness Assessment
To cope with uncertainty, we assume as in [67] that the changes in the values of the factors are bounded by an interval, which is determined by utilizing a single tolerance value, ω, such that In the robustness assessment, worst-case and best-case scenarios are contemplated. In the first case, the outputs are increased while the inputs are decreased for all DMUs except for the DMU under evaluation (i.e., DMU o worsens its efficiency performance while the remaining DMUs improve their efficiency performance). In the second case, the opposite situation is assumed. Problem (7) refers to the best-case scenario and allows obtaining the upper bound, Problem (8) refers to the worst-case scenario and allows obtaining the lower bound, From problems (7) and (8), it is possible to conclude that 1 − β UR o ≤ 1 − β LR o . Let J be the index set of DMUs (j = 1, . . . , n). The DMUs can then be classified into strongly efficient, i.e., E ++ = {j ∈ J: . A DMU is robustly efficient (inefficient) to data perturbations if it stays efficient (inefficient) for the tolerance employed.

Stochastic Frontier Analysis
One of the limitations of the DEA approach is that it does not account for the impact of environmental factors and random errors in efficiency assessment. Hence, the single use of DEA can lead to unreasonable outcomes. To overcome this limitation, [68] proposed a three-stage DEA model. Firstly, the efficiency scores of every DMU and the input and output slacks are computed through the use of the DEA model. Secondly, the slacks are decomposed into environmental effects, managerial inefficiencies, and statistical noise. We use stochastic frontier analysis (SFA) [69,70] to implement the decomposition and, consequently, compute the adjusted input and output factors. The slacks are the dependent variables while the external environmental factors are the independent variables. The purpose is to remove the influence of environmental factors and random errors. The functional form of the SFA model for each input slack obtained for j inefficient DMU (j = 1, . . . , p) is given by: where s ij is the slack value of the i-th input of the j-th DMU, f Z j , β i is the deterministic feasible slack frontier and β i designates the coefficients related to the environmental variables. The term v ij + u ij is the mixed error, v ij refers to the statistical noise, and u ij refers to the management inefficiency. Generally, it is presumed that If γ is near 1, it means that management factors are in a leading position; i.e., most of the change required to attain efficiency is linked to management inefficiency. If γ is near 0, the random error is the predominant factor; i.e., most of the change required to attain efficiency is related to statistical noise.
The adjusted input and output slacks are then computed by decomposing the mixed error. In line with [71], the management inefficiency is computed as follows: , and ϕ and ∅ correspond to the density function and distribution function of the standard normal distribution, respectively. Hence, the random error term can be computed as follows: We employ an adaption of the three-stage method of [68] by computing in the first stage the slacks through the WRDD DEA model, specifically through problem (5), instead of using the SBM DEA approach.
In stage 2, the input and output variables of each DMU are adjusted based on the SFA results by removing the significant environmental effects and statistical noises.
The input data are adjusted as follows [68]: Considering a similar procedure for computing the adjusted outputs, these are adjusted as follows [72]: Finally, in stage 3, we compute the efficiency scores with the adjusted input and output values.

Data and Assumptions
To account for the ICT SME support, we consider the dimensions of intervention given in Table 2, also used in the study of ICT funding allocations considered for SMEs by [50]. These two dimensions represent EUR 1.7 billion and EUR 304 million of planned investments, respectively [40]. These amounts are inscribed under multi-TO (EUR 810 million), TO2 (EUR 790 million), and TO3 (EUR 349 million) and to a smaller level under TO1 and TO8 [40]. Productive investment linked to the cooperation between large enterprises and SMEs for developing information and communication technology (ICT) products and services, e-commerce, and enhancing demand for ICT Example of activities supported [40]: commerce and expand demand for ICT; encouragement and information initiatives (events, campaigns, consultation) aimed at increasing SME ICT preparedness, as well as the launch of innovative business ICT tools and solutions (ERP, CRM, cloud, among others) for SMEs.

82
ICT services and applications for SMEs (including e-commerce, e-business, and networked business processes), living labs, web entrepreneurs, and ICT start-ups). Example of activities supported [40]: Encourage the development of new advanced ICT solutions; collaboration between ICT companies and academic institutions through incentive and communication campaigns, events, seminars, and business consulting; commercialization and global marketing of ICT goods and services (consultation, promotion, marketing, involvement in tenders and expos, software localization); aid SMEs in the development of innovative applications and services, such as smart linked devices; smart housing and energy efficiency; ICT services and applications for health, SMEs in the health sector, domotic services.
We considered cumulative figures from previous years reported in March 2022. Furthermore, we only included in our assessment the OPs with no incomplete information (i.e., the OPs with incomplete information were not evaluated). This led to the assessment of 51 OPs from 16 countries.
We involved multiple stakeholders in the prior identification of the indicators employed in the evaluation by organizing a facilitated workshop with key policymakers and MAs on the subject "evaluating co-financed intervention policies in enterprises".

Input and Output Factors
The input and output factors selected for assessing the efficiency of the implementation of the ERDF funds committed to ICT support in SMEs were picked from a set of transversal indicators officially mandated by the EU, i.e., from the list of categorization data for the ERDF OPs (available online: https://cohesiondata.ec.europa.eu/2014-202 0-Categorisation/ESIF-2014-2020-categorisation-ERDF-ESF-CF-planned-/3kkx-ekfq (accessed 30 March 2022)), and are further explained below.

The Rate of OPs' Execution
An efficient policy implementation implies an efficient financial execution of OPs, with a particular focus on the pace of execution of the OPs ( [62][63][64]73]). In this context, costs must be certified by an approved authority, which is the entity in charge of certifying that the co-funded equipment/services were supplied, that the corresponding payments were made, and that the payments conformed with the EU and national rules. The indicators used to evaluate the capacity of the OPs' absorption are "total eligible spending" and "eligible costs decided". The first relates to the qualified costs reported and verified by this authority. Consequently, this indicator is utilized as an output because the greater the amount assigned to it, the greater the financial implementation of every project. The second is regarded as an input since it refers to the financial funds devoted to the projects selected for funding, and this should be minimized. Since the assignment of the ICT funds to distinct policy objectives is influenced by the plans that policymakers agree to follow [50], this latter indicator has also been used in the second stage of the analysis to understand how it was affected by ICT strategies in SMEs.

Operations Supported
The "number of operations supported" relates to the number of projects that obtained ERDF funding. The greater the number of supported projects, the greater the prospect of increasing/enhancing the firms' use of ICT. Hence, this indicator is viewed as an output.
Data on these indicators are available in Table A2 in Appendix A. Based on Table 3, it is reasonable to infer that the overall average financial implementation ratio (i.e., the ratio of "total eligible expenditure" and "total eligible cost") is somewhat higher than 50% (56.3%). Furthermore, the number of projects financed varies greatly. A prerequisite of DEA is that inputs and outputs should hold an isotonic relationship [74]. This property indicates that raising any input whilst holding the other evaluation factors constant must not lower any output but rather raise at least some output. A positive and significant correlation between inputs and outputs depicts an isotonic relationship. We opted to compute the Spearman correlation coefficients and the related significance tests since the assumption of normality for the applicability of the tests for the significance of Pearson's correlation was not confirmed in this case-see Table 4.

Environmental Factors
Because of the absence of data on ICT adoption at the firm level from traditional data sources at the NUTS2 level [50,55,57], we ended up using, in the second stage of the analysis, indicators available from the Regional Innovation Scoreboard in 2021 [75]. The indicators were selected according to the literature review on the main determinants that influence ICT adoption in EU SMEs (see Table 1).
Concerning independent variables (see Table 5), we have employed as a proxy of economic development the regional GDP at purchasing power parity per capita (GDPPPpc) [50,55,57]. As documented by [76], ICT tends to be promoted more efficiently in prosperous regions. Additionally, since ICT goods and services are classified as normal goods [77], their demand is also supposed to vary positively with GDP. We have also considered as an environmental factor the percentage population aged 25-34 having completed tertiary education, because the literature acknowledges a positive linkage between educational achievement and ICT adoption [55,57]. Particular reasons have been stressed in the case of ICT use to explain this possible positive association. On the one hand, education offers the skills needed to use and benefit from ICT. On the other hand, employees should be expected to be more competent at learning how to utilize new technologies, specifically ICT [57].
Additionally, since R&D expenditures foster ICT dissemination for European regions [55,57,58], we have used as an explanatory variable R&D expenditure in the business sector as a percentage of GDP and SMEs introducing product innovations as a percentage share of all SMEs.
In addition, because the ICT skills of a firm are regarded as important technologyrelated elements that are influential in determining user acceptance and ICT deployment [58], we have considered individuals who have above basic overall digital skills as a percentage of total SME employment. Finally, the percentage of ICT specialists as a percentage of total SME employment has similarly been used, i.e., people whose primary occupation is ICT and who are capable of handling a wide variety of responsibilities connected to firms' computer systems [5].
Data were normalized using the min-max procedure, i.e., the minimum score for all regions across all the available years was subtracted from the respective transformed score, which was then divided by the difference between the maximum and minimum scores observed for all data (regions and years). The maximum normalized score is equal to 1 and the minimum normalized score is equal to 0-see Tables 5 and A3.

Analysis and Discussion of Results
Results were computed through an Excel Visual Basic based application that was specifically developed by the authors for solving our DEA problems and uses Excel Solver as the backend. Table 6 displays the basic descriptive statistics of the results obtained for the OPs under evaluation. Based on Table 6, it is fair to assert that, while efficient OPs have significantly higher average scores than inefficient ones (with efficiency values ranging between 1.00 and 1.31 and with at least 50% of efficient OPs with efficiency values higher than 1.07), their evaluation factors are also more variable when compared to those of inefficient OPs (these findings are corroborated by the higher standard deviation values). Furthermore, inefficient OPs have a broad range of efficiency ratings (with a standard deviation of 35.48 and at least 50% of inefficient OPs with efficiency values lower than −6.61).
The number of OPs at distinct efficiency score subintervals is illustrated in Figure 1. The number of OPs at distinct efficiency score subintervals is illustrated in Figure 1. Only about 20% of the OPs attained an efficient procedural performance, i.e., 10 out of 51 ( Figure 1).

Spending Operations Costs Decided
The four OPs that are most frequently selected as benchmarks are "Extremadura-ERDF" (30), "País Vasco-ERDF" (25), "Multi-regional Spain-ERDF" (22) and "Provence-Alpes-Côte d'Azur-ERDF/ESF/YEI" (21)-see Table 7. Curiously, all these regions are located in countries viewed as being the highest spenders on ICT support for SMEs [38]. Furthermore, the findings obtained for Spanish regional OPs (the country with the OPs most frequently selected as benchmarks) match the conclusions of [5]. In line with the authors of [5], Spanish regions at the firm level are at a moderate level of digital advancement or at a greater level of digital advancement than their EU peers, as well as displaying a reduced DD among them (i.e., a smaller degree of difference in firm digital development) in comparison to what occurs in the remaining European MSs. Curiously, Greece and Bulgaria, countries with OPs classified as efficient, were identified in [5] as countries whose firms were last ranked in terms of digital inclusion (according to 2015 data), highlighting the efforts made by these countries in the adoption of ICT by SMEs in the latest programming period. Only about 20% of the OPs attained an efficient procedural performance, i.e., 10 out of 51 ( Figure 1).
The four OPs that are most frequently selected as benchmarks are "Extremadura-ERDF" (30), "País Vasco-ERDF" (25), "Multi-regional Spain-ERDF" (22) and "Provence-Alpes-Côte d'Azur-ERDF/ESF/YEI" (21)-see Table 7. Curiously, all these regions are located in countries viewed as being the highest spenders on ICT support for SMEs [38]. Furthermore, the findings obtained for Spanish regional OPs (the country with the OPs most frequently selected as benchmarks) match the conclusions of [5]. In line with the authors of [5], Spanish regions at the firm level are at a moderate level of digital advancement or at a greater level of digital advancement than their EU peers, as well as displaying a reduced DD among them (i.e., a smaller degree of difference in firm digital development) in comparison to what occurs in the remaining European MSs. Curiously, Greece and Bulgaria, countries with OPs classified as efficient, were identified in [5] as countries whose firms were last ranked in terms of digital inclusion (according to 2015 data), highlighting the efforts made by these countries in the adoption of ICT by SMEs in the latest programming period. Figure 2 demonstrates that the average funds committed to the eligible cost of efficient OPs (EUR 75,514,840) were much higher than those dedicated to inefficient OPs (EUR 16,621,816). Similar conclusions may be drawn for the mean eligible spending of efficient OPs (EUR 46,026,233) and inefficient ones (EUR 8,504,000)-see Figure 2. Furthermore, the number of operations enabled by efficient OPs is substantially greater than that of inefficient OPs (1310 against 190).   Figure 3 shows the contribution of each factor to the inefficiency recorded in inefficient OPs. Our findings indicate that the number of operations supported is the factor that requires more attention from MAs. The OPs' financial execution is also important but in fewer cases. While nine OPs require an enhancement of "eligible spending", there is only one OP that requires the reduction in funding dedicated to it, i.e., the reduction in "eligible costs decided".   Figure 3 shows the contribution of each factor to the inefficiency recorded in inefficient OPs. Our findings indicate that the number of operations supported is the factor that requires more attention from MAs. The OPs' financial execution is also important but in fewer cases. While nine OPs require an enhancement of "eligible spending", there is only one OP that requires the reduction in funding dedicated to it, i.e., the reduction in "eligible costs decided". Figure 3 shows the contribution of each factor to the inefficiency recorded in inefficient OPs. Our findings indicate that the number of operations supported is the factor that requires more attention from MAs. The OPs' financial execution is also important but in fewer cases. While nine OPs require an enhancement of "eligible spending", there is only one OP that requires the reduction in funding dedicated to it, i.e., the reduction in "eligible costs decided".    The WRDD DEA model also provides information on the adjustments that inputs and outputs must undergo to make inefficient OPs efficient (Table 8). These results are also displayed in Figures 4 and 5 per OP, where OPs are portrayed in ascending order of efficiency from left to right. The "number of operations supported" shows the largest improvement potential of about 376% (i.e., it should increase on average from 189.54 to 902.34), while "eligible costs decided" (−8%) and "eligible spending" (17%) only require mild adjustments-see Table 8 and Figures 4 and 5.

Robustness Analysis
The robustness study was carried out using tolerances of 5% and 10% for introducing data perturbation in all factors under scrutiny. According to our findings shown in Figure 6, it can be concluded that there are only four efficient OPs that remain efficient for data perturbations with both tolerances, which are "Provence-Alpes-Côte d'Azur-ERDF/ESF/YEI", "Extremadura-ERDF", "Enterprise and Innovation for Competitiveness-CZ-ERDF", and "Multi-regional Spain-ERDF". Contrastingly, 21 OPs remain robustly inefficient for both data perturbations (i.e., 41% of the sample of OPs under evaluation). Finally, under the same conditions, 27 OPs are potentially efficient. All in all, these outcomes suggest a poor application of ERDF funds in ICT SME support. These findings appear to be consistent with the challenges identified by [38] at the heart of the EU ICT policy in terms of digital achievements. According to their findings, the EU seems to underachieve its competitors (specifically, the United States, Japan, and South Korea) in terms of innovative ICT infrastructures and ICT adoption, in particular by SMEs. In fact, [33] ascertained that to create the encouraging circumstances and provide incentives to SMEs for adopting innovative solutions, SMEs need explicit policy initiatives to guarantee that technological services can be supplied and that the required infrastructures are accessible.
The WRDD DEA model also provides information on the adjustments that inputs and outputs must undergo to make inefficient OPs efficient (Table 8). These results are also displayed in Figures 4 and 5 per OP, where OPs are portrayed in ascending order of efficiency from left to right.

Analysis of SFA Regression Results
The slacks of output factors obtained with problem (5) are used as the explained variables at this stage, resulting in two regression models (since there was only one OP with an input surplus, these factors were not adjusted). In this context, correlation coefficients and the variance inflation factors (VIFs) are two different methods for examining multicollinearity among the variables. A high correlation between two explanatory variables is an indicator of multicollinearity. Table 9 presents the correlation analysis between the environmental variables. Fostering a…

Analysis of SFA Regression Results
The slacks of output factors obtained with problem (5) are used as the explained variables at this stage, resulting in two regression models (since there was only one OP with an input surplus, these factors were not adjusted). In this context, correlation coefficients and the variance inflation factors (VIFs) are two different methods for examining multicollinearity among the variables. A high correlation between two explanatory variables is an indicator of multicollinearity. Table 9 presents the correlation analysis between the environmental variables.
It can be seen that there is a strong correlation (0.73) between GDPPPPPC and ICT specialists. Otherwise, all the other significant correlations correspond to moderate correlations since the coefficients vary between 0.32 and 0.65. Since multicollinearity is defined as a phenomenon in which two or more explanatory variables in a multiple regression model are highly correlated, the values of VIFs were also calculated (see Table 10). The higher the VIF is, the more likely the multicollinearity among the variables is. VIF starts at 1 and has no upper limit, and values between 1 and 5 indicate a moderate correlation between two explanatory variables, but, within this range, it is considered not severe enough to require attention [78,79].  The obtained results for multicollinearity indicate that we can include all variables in the initial models. However, due to the problem of misspecification, the final models do not necessarily contain all the variables previously considered.
The R program version 4.0.5 (RStudio Team, 2021) and, specifically, the sfaR package version 0.1.1 were used to run the SFA regression models [80]. Table 11 displays the results. The γ values of the two models are close to 1 and significant at the 1% level, indicating that management factors are the main factors responsible for the efficiency scores attained. Therefore, to obtain unbiased efficiency results, we removed the effects of the environmental variables and random errors through the use of SFA.
All the regression coefficients are significant at the 1% level, suggesting that the selected environmental variables have a significant impact on the slacks of each output.
From Table 11, it can be concluded that the increases in both the percentage of ICT specialists and GDPPPP pc contribute to a higher required improvement of "total eligi-ble spending", whereas the remaining factors have a negative influence on the required improvement of this factor. These findings suggest that more developed regions and a higher rate of ICT specialists do not necessarily mean a higher absorption rate of ERDF funds dedicated to boosting ICT in SMEs. The underuse of ERDF by ICT Croatian SMEs was also acknowledged by the authors of [81] for the period of 2014-2020. These authors concluded that the intricacy and the required time to apply, develop, and appraise the projects were the potential cause of these findings [81]. In addition, the authors of [82] ascertained that SMEs' investment ends up being smaller than it would be anticipated by standard economic fundaments, indicating that these firms are particularly vulnerable to funding issues. Another reason for these findings might reflect the use of other sources of financing by these SMEs [38].
Concerning the required increase in the "number of operations" supported, the results show that this variable tends to increase as the digital skills and ICT specialists increase, whereas it tends to decrease with the percentage of SMEs with product process innovations and GDPPPP pc . Then again (and for similar previous reasons), these outcomes suggest that a higher rate of ICT skills/specialists does not necessarily lead to an efficient number of operations supported. On the other hand, more developed regions and a higher percentage of SMEs more prone to process innovations do not necessarily need to run for additional ERDF-supported operations, because these are more efficient in applying for funding.

Efficiency after Input and Output Adjustments
In the first stage of the analysis, we found that the mean efficiency scores computed were quite volatile, validating the use of the multistage technique [83]. In the second stage, we used the SFA approach to adjust the outputs by removing the effects of environmental factors and random shocks. Subsequently, we obtained the inherent production possibility set by running problem (1) with these new adjusted factors.
According to Table 12, it is possible to conclude that efficient OPs reduce their variability, with a standard deviation of 0.05 (efficiency values vary within [1.00, 1.18] and with more than 50% of efficient OPs with efficiency values higher than 1.03). Moreover, inefficient OPs significantly reduce the variability of their efficiency ratings (with a standard deviation of 0.53 and with more than 50% of inefficient OPs with efficiency values lower than 0.51) and improve their performance substantially (underlining the importance of the impact of environmental factors). About 30% of the OPs reached an efficient procedural performance against the previous 20%, i.e., 16 out of 51 (Tables 6 and 12).
The difference between the efficiency scores of the OPs with and without adjusted factors is depicted in Figures 7 and 8. From there, it can be concluded that the efficiency scores obtained without an adjustment of the factors used in the analysis were underrated. When contrasted with the first stage, "Melilla-ERDF", "Île-de-France et Seine-ESF/ERDF/YEI", and "Friuli-Venezia Giulia-ERDF" showed the largest increase in efficiency, with their values rising from −21.05 to 1.02 (105%), −19.60 to 1.03 (105%), and −17.78 to 1.02 (84%), respectively-see Figure 8 and Table 12. These findings suggest that these OPs' previous inefficiencies were not solely because of their backward technological level, but rather associated with their external environment.

Enterprise and Innovation for… Multi-regional Spain -ERDF País Vasco -ERDF Competitiveness…
Innovations and…

Aquitaine -ERDF/ESF/YEI Warmińsko-Mazurskie…
Haute-Normandie -…  When contrasted with the first stage, "Melilla-ERDF", "Île-de-France et Seine-ESF/ERDF/YEI", and "Friuli-Venezia Giulia-ERDF" showed the largest increase in efficiency, with their values rising from −21.05 to 1.02 (105%), −19.60 to 1.03 (105%), and −17.78 to 1.02 (84%), respectively-see Figure 8 and Table 12. These findings suggest that these OPs' previous inefficiencies were not solely because of their backward technological level, but rather associated with their external environment. When contrasted with the first stage, "Melilla-ERDF", "Île-de-France et Seine-ESF/ERDF/YEI", and "Friuli-Venezia Giulia-ERDF" showed the largest increase in efficiency, with their values rising from −21.05 to 1.02 (105%), −19.60 to 1.03 (105%), and −17.78 to 1.02 (84%), respectively-see Figure 8 and Table 12. These findings suggest that these OPs' previous inefficiencies were not solely because of their backward technological level, but rather associated with their external environment. The OPs that are more often selected as benchmarks are "Umbria-ERDF" (26), "Multiregional Spain-ERDF" (19), "Central Macedonia-ERDF/ESF" (13), and "Berlin-ERDF" (11)-see Table 13. Curiously, "Multi-regional Spain-ERDF" remains in the top four efficient countries viewed as benchmarks. In addition, two (out of five) of the OPs that remain efficient regardless of the adjustments belong to Spanish regions (see grey cells of Table 12). In this context, it is worth mentioning that MSs in Southern (such as Italy and Spain) and Central and Eastern Europe were the biggest recipients of financing for ICT and the digital economy [38]. This is particularly true for countries with efficient OPs, such as Spain, Greece, Hungary, the Czech Republic, and the Baltic States [38]. Curiously, it is interesting to see that in some of the countries that significantly worsened their performance because of environmental factors, such as Poland (previously ranked 10th and dropped to 50th) and Bulgaria (previously ranked 7th and dropped to 43rd), more than 50% of their firms had extremely modest degrees of digitization in 2016 [38]. Finally, other OPs from countries such as Italy, Germany, and Finland also show the importance of environmental factors, revealing that with the adjusted outputs they become efficient. Note: The grey cells identify the OPs that remain efficient regardless of the adjustments. Figure 9 shows that the adjusted average funds dedicated to the eligible cost of efficient OPs were substantially reduced for efficient OPs (from EUR 75,514,840 to EUR 48,760,091), whereas they suffered a slight increase for inefficient ones (from EUR 16,621,816 to EUR 18,756,611). An analogous trend was obtained for the mean adjusted eligible spending and number of operations supported by efficient and inefficient OPs-see Figure 9. Figure 10 suggests that if the effect of environmental factors is removed from the analysis, the number of operations supported is no longer the only factor that requires more attention from MAs. There are now 17 OPs that require a reduction in "eligible costs decided", i.e., a reduction in the amount of funding allotted to ICT in SMEs. In addition, the enhancement of "eligible spending" is also an important barrier to efficiency for the same number of OPs.

Original efficiency Adjusted efficiency
The required adjustments that inputs and outputs must endure turning inefficient OPs into efficient ones were obtained with the adjusted factors (Table 14). These computations are also depicted in Figures 11 and 12 per OP, where OPs are portrayed in ascending order of efficiency from left to right. Overall, our findings suggest that, although the average adjusted factors depart from bigger values than previously (see Table 14) if we contrast the previous projections with the ones attained with the adjusted values, the average projection for the "total eligible spending" requires an enhancement of 42%, whereas the average projection for the "number of operations" supported only increases 4% and the average projection for the "total eligible cost decided" requires a further improvement of 9%.  Figure 9 shows that the adjusted average funds dedicated to the eligible cost of efficient OPs were substantially reduced for efficient OPs (from EUR 75,514,840 to EUR 48,760,091), whereas they suffered a slight increase for inefficient ones (from EUR 16,621,816 to EUR 18,756,611). An analogous trend was obtained for the mean adjusted eligible spending and number of operations supported by efficient and inefficient OPssee Figure 9.  Figure 10 suggests that if the effect of environmental factors is removed from the analysis, the number of operations supported is no longer the only factor that requires more attention from MAs. There are now 17 OPs that require a reduction in "eligible costs decided", i.e., a reduction in the amount of funding allotted to ICT in SMEs. In addition, the enhancement of "eligible spending" is also an important barrier to efficiency for the same number of OPs. The required adjustments that inputs and outputs must endure turning inefficient OPs into efficient ones were obtained with the adjusted factors (Table 14). These computations are also depicted in Figures 11 and 12 per OP, where OPs are portrayed in ascending order of efficiency from left to right. Overall, our findings suggest that, although the average adjusted factors depart from bigger values than previously (see Table 14) if we contrast the previous projections with the ones attained with the adjusted values, the average projection for the "total eligible spending" requires an enhancement          Then again, the average "number of operations supported" shows the largest improvement potential of about 141% (i.e., it should increase on average from 387.06 to 934.29), while "eligible costs decided" (−11%) and "eligible spending" (11%) only require mild adjustments-see Table 14.
From the robustness analysis, it can be concluded that there are only three efficient OPs that remain efficient for data perturbations with both tolerances of 5% and 10%, which are "Provence-Alpes-Côte d'Azur-ERDF/ESF/YEI", "Enterprise and Innovation for Competitiveness-CZ-ERDF", and "Multi-regional Spain-ERDF". These OPs coincide with three of the previously most robust OPs without considering the adjusted factors. In addition, except for "England-ERDF" and "Integrated Infrastructure-SK-ERDF/CF" that become robustly inefficient for a tolerance of 5%, all the remaining OPs are potentially efficient for both tolerances.

Conclusions and Policy Implications
Since there are no studies available that specifically address the evaluation of the funds allocated to the OPs committed to ICT adoption by SMEs, this study is aimed at evaluating the procedural efficiency of the implementation of 51 OPs devoted to boosting ICT use in SMEs from 16 EU countries. Furthermore, because most of the methodologies used to evaluate ESIF in the context of ICT refer to ex post or ex ante assessments, we suggest a novel three-stage WRDD model framework that consists of combining this DEA model with SFA, which allows performing midterm/terminal assessments.
Our major findings concerning our research questions are given below. RQ1. "Which indicators prevent the efficient utilization of ERDF allocated to boost ICT adoption in EU SMEs?" Without the adjustment of the outputs, the "number of operations supported" is the indicator that requires more attention from MAs, whereas "eligible costs decided" and "eligible spending" only require attention in fewer cases (one and nine OPs, respectively). With the removal of the environmental factors and statistical noise, the "number of operations supported" is no more the single factor that demands more concern from MAs. With the adjusted factors, one-third of the OPs need to further reduce the "eligible costs decided". Furthermore, the improvement of "eligible spending" also becomes a relevant obstacle to efficiency for an identical number of OPs.
RQ2: "Which OPs were most frequently referenced as a source of best practices for the programmatic period under analysis?" The OP that is more often selected as a benchmark regardless of the environmental factors is "Multi-regional Spain-ERDF", which is ranked among the four efficient OPs viewed as benchmarks in both scenarios. In addition, two (out of five) of the OPs that remain efficient regardless of the adjustments belong to Spanish regions. In this regard, it is worth mentioning that the use of "vouchers" in Spain [38] seems to be an efficient approach for reaching SMEs and providing them with help that is simple to administer and targeted toward their requirements.
RQ3: "Which OPs show more efficiency resilience in the face of probable changes in the indicators used?" The robustness analysis shows that the OPs that remain efficient for data perturbations with both tolerances of 5% and 10% regardless of the adjustments are "Provence-Alpes-Côte d'Azur-ERDF/ESF/YEI", "Enterprise and Innovation for Competitiveness-CZ-ERDF", and "Multi-regional Spain-ERDF". Furthermore, "England-ERDF" and "Integrated Infrastructure-SK-ERDF/CF" become robustly inefficient for a tolerance of 5% either with or without the adjusted factors. Either way, these findings suggest that the management procedures of these OPs are the main reason behind these achievements.
RQ4: "Which environmental factors have the greatest impact on the inefficiency of the execution of OPs aimed at boosting ICT in EU SMEs?" Results show that more developed regions and a higher rate of ICT specialists lead to an underuse of ERDF funds dedicated to boosting ICT in SMEs. In addition, a higher rate of ICT skills/specialists leads to the underachievement of the number of operations supported. On the other hand, more developed regions and a higher percentage of SMEs introducing product innovations seem to be more efficient in applying for funding.
These findings might be related to the acknowledged red tape procedures associated with the EU mechanisms, financing outlets, and bureaucratic regulations, namely for SMEs.
RQ5: "How does efficiency change with the removal of environmental factors? With the adjusted factors, about 30% of the OPs (16) reached an efficient procedural performance against the previous 20% (10), highlighting the impacts of environmental factors on efficiency assessment. The regions that showed the largest increase in efficiency were "Melilla-ERDF", "Île-de-France et Seine-ESF/ERDF/YEI", and "Friuli-Venezia Giulia-ERDF", with values rising from −21.05 to 1.02 (105%), −19.60 to 1.03 (105%), and −17.78 to 1.02 (84%), respectively. These outcomes indicate that the inefficiencies of these OPs were mainly influenced by their external environment.
All in all, it can be ascertained that SMEs' access to ESIF (particularly ERDF) continues to be out of reach since they lack the administrative ability to deal with the various procedures for applying for and implementing ERDF projects. When opposed to "conventional" SME initiatives, this issue becomes more significant when it comes to ICT. In effect, increased speed and adaptability are required for initiatives in a sector known for rapid changes, such as ICT. Therefore, MAs should find ways of providing further support to SMEs that simplify administrative processes and address SMEs' specific needs.
Additionally, our study highlights one of the major limitations of this study, which is the lack of indicators available to assess the performance of the implementation of ESIF funds committed to ICT support in SMEs (only two outputs and one input were used for this purpose, despite the use of other environmental factors). As a result, national and regional policymakers should use supplemental specific indicators that account for ICT performance, tag spending that falls under other thematic objectives (rather than TO2) but has an ICT element, improve the quality and comprehensiveness of data on ICT performance at the regional and SME level, and consolidate distinct existing data sources.
Finally, while this study offered new perspectives and novel approaches to the efficiency evaluation of funding committed to boosting ICT use in European SMEs, future work should specifically assess the potential impacts of these ICT policy initiatives on economic indicators, such as GDP or employment, as well as their effect on territorial disparities. Acknowledgments: INESC Coimbra and CeBER are supported by the Portuguese Foundation for Science and Technology funds through Projects UID/MULTI/00308/2020 and UIDB/05037/2020, respectively.

Conflicts of Interest:
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.