Success or Waste of Taxpayer Money? Impact Assessment of Rural Development Programs in Hungary

The effectiveness of support directed to less developed regions is a timely question halfway through the 2014-2020 programming period. This paper describes an analysis of the impact of development support on the wellbeing of Hungarian LAU1 regions between 2008 and 2013. The aim is to measure the overall impact of all of the Rural Development Funds, covering all measures within the program. Three indices of local wellbeing are used: the multi-dimensional, local-variables-based Rural Development Index which measures the overall level of regional development. Furthermore, two simple, migration-based indices are used, as proxies for perceived quality of life. Propensity score matching and difference-in-difference estimation techniques are employed to evaluate the impact of subsidies. Irrespective of the way the amount of support is calculated, or the measure of local wellbeing or methodology employed, the impact is generally not significant, or even negative. This casts doubt on the effectiveness of Rural Development Policy.


Introduction
It is difficult to overestimate the role of Regional Development Policies (RDPs) in developed economies. Seventy-five percent of the territory of OECD countries is classified as rural, and on average a quarter of the total population live in these areas (OECD, 2006). In the past decades, the global economy has experienced unprecedented growth in agricultural productivity -itself a laudable outcome -yet, despite lavish subsidies, this has led to a fall in both agricultural employment and the importance of agriculture to national economies (at least when developed economies are considered).
Whilst agricultural output amounts to roughly 2 percent of OECD nations' GDP, the vast majority of rural land is used for agricultural purposes (e.g. 96 percent in the EU25, including forests). However, in the EU25 only 13 percent of rural labour is employed in agriculture -the OECD average is 10 percent, producing a gross value-added of only 6 percent (OECD, 2006). Whilst the aims of EU Common Agricultural Policy (CAP) with respect to agricultural production were laid down in the 1958 Treaty of Rome, and, albeit with significant amendments, have been applied until now, the importance of rural and indeed regional development -i.e. that not directly connected to production -was only recognized in the 1970s. Thus the modern CAP (AGENDA 2000) shifted the support system towards a more integrated rural development policy, creating the European Agricultural Model (Renting et al., 2009) whose primary aim is to promote a viable and liveable rural environment rather than maximize agricultural output (for further discussion, see for example 'The new rural paradigm: policies and governance ', OECD 2006). It was a key revelation that, besides production, a nation's agriculture contributes to the creation or preservation of a number of important values such as landscape, traditions-customs, social structures and environmental protection. The most important pre-condition of the creation/preservation of the abovementioned values is the existence of a sufficiently large active rural population. This highlights the importance of policies designed to slow rural-to-urban migration, and reverse the continual increase in the average age of rural inhabitants.
The economic output of Hungarian rural areas is 50 percent less than the national average, and three times less than that of the predominantly urban output (for more detail about sectoral and regional differences in the EU and OECD countries, see for example, Bollman et al. (2005), Copus et al. (2006), or Terluin et al. (2011)). To sum up, besides the economic and agricultural perspective, rural areas are also very important in terms of population, preserving landscape traditions and environmental protection. In addition, New Member States (NMS) are more rural than Old Member States (OMS), and the income gap between rural and urban areas is more predominant in NMS than OMS. Within the European Union (EU), Hungary is one of the biggest beneficiaries of Rural Development Program (RDP) payments -at least when per capita transfers are considered. In the 2007 -2013 programming period, EUR 3.8 billion was spent, whilst for the 2014 -2020 period EUR 4.2 billion (of which EUR 740 million in the form of national co-funding) is earmarked for this purpose. The question that naturally arises is: do these substantial transfers make a difference? The European Commission's mandatory ex-ante, mid-term and ex-post program evaluations -based on monitoring a set of (partial) indicators and qualitative assessment -fail to provide an answer because of the need for uniformity and comparability across Member States. On the other hand, academic research with respect to the evaluation of RDP measures is rather scanty. Most papers focus on the impact of agricultural policy on labour markets or rural income distribution (e.g. Olper at al. 2014;Breustedt and Glauben, 2007;Elek et al., 2010;Esposti, 2007;Petrick and Zier, 2012;Swinnen and Van Herck, 2010) or focus on the farm level (e.g. the impact of RDP investment support, Medonos et al. 2012, or Pufahl andWeiss, 2009 on the evaluation of farm programs). One possible reason for the scarcity of relevant literature is that the policy evaluation or impact assessment of RDP is a rather complicated issue since complex notions are hard to quantify, whilst all relevant aspects of the impact should be captured in a transparent and easy-to-handle fashion. There are two key issues here: first, the problem of applying partial indicators (such as number of projects supported, area supported, change in employment, value of investments realized, and GDP change -see Michalek and Zarnekow 2012 for a critical review); and second, the existence of a counterfactual situation which excludes the possibility of making a before-and-after comparison. Moreover, GDP data is not available at a disaggregated level, thus the analyst is basically left with unemployment levels/rates or taxes. The often employed naïve approaches to the impact evaluation of RDP such as simple case studies or partial indicators do not even attempt to address the counterfactual situation (Terluin and Roza, 2010).
Generally, the most important drawback of partial measures is the lack of clear causality between partial measures and RDP (the problem of making a distinction between the impact of RDP and other exogenous factors). These issues may, however, be solved by use of a complex Rural Development Indicator, RDI, originally proposed by Michalek and Zarnekow (2012) and counterfactual analysis.
We follow this approach, and construct an 'objective' Quality of life (QoL) index, complemented by two 'subjective', internal-migration-based QoL indices, along with the idea that, regardless of objectively computed local development scores, people tend to migrate when the perceived QoL is higher. In contrast to Michalek (2012), who investigates only the impact of the SAPARD programmes in Poland and in Slovakia between 2002 and 2005, we focus on the 2008-2013 period which covers all regional development policy measures and try to answer the following simple question: have the significant amount of Rural Development funds that have been distributed had any measurable impact? The rest of the paper is organised as follows: in the next section we detail the methodology and data we use. In Part Four, we focus on empirical results. The paper is concluded in Section Five.

Data and methodology
Hungary, covering an area of 93,000 km 2 and with an approximate population of 9.8 million, has been a Central European EU member state since 2004. The country has three NUTS 2 regions, and 20 NUTS 3 ones (19 counties plus the capital city, Budapest). On a Local Administrative Unit level (LAU1, formerly NUTS4) there are 174 small regions, composed of 3,164 administratively independent settlements. We were in a position to employ a highly disaggregated dataset of yearly data with respect to these administratively independent settlements, which we believe contributes to the unique nature of this research. The T-STAR database of the Hungarian Central Statistical Office was obtained from the CERS-HAS databank (http://adatbank.krtk.mta.hu/adatbazisok___tstar ). This source of data is designed for use in spatial studies and consists of several hundred variables relating to demographics, public health, education, pollution, unemployment, social care, economic entities, infrastructure, commerce and hospitality, tourism, culture, housing stock, municipal aid, municipal budgets, agriculture and personal income tax. These variables are available for the 2007-2013 period for all 3,166 administratively independent Hungarian settlements. An internal migration database was provided by the Hungarian Central Statistical Office. Data about development funds for the period 2008-2013 was taken from the Information Systems of National Regional Development. Using total payments per locality, we created three subsidy indicators: total support, support per km 2 , and support per capita in LAU1 regions.
Our empirical strategy consisted of three steps: 1. First, we composed a local, 'objective' development indicator based on the wealth of variables available in the T-STAR database. There are several potential approaches to this. The most prominent methods are factor/principal component analysis (i.e. 'let the data choose') and the construction of an indicator based on selected variables. Factor and Principal Component Analysis have been used by Michalek and Zarnekow (2012) to evaluate the SAPARD programme in Poland and Slovakia. In the Hungarian context, the research of Lukovics and Kovács (2008) and Lukovics (2009) uses factor and PCA analysis and employs the same TSTAR dataset as used in this paper to compute regional -Group 4: Infrastructure and environment (e.g. houses connected to a centralised sewage system from total number of houses, natural gas, electricity, running water usage normalised with either population or housing stock, services provided by local government, distance in minutes to LAU1 centre, etc.).
Each individual local variable that enters one of the groups was scaled according to the following eq. 1: The weighted sum of four Groups 1 -4 resulted in an 'objective' quality of living indicator we denote as rdi.
Group indicators are the arithmetic average of composed scaled variables (variables with negative impact such as mortality, unemployment, travel time entered with a negative sign). Once the localityspecific group variables were calculated, the data was aggregated 1 to 174 small regions, corresponding to the Hungarian LAU1 (formerly NUTS4) regions. The database was then merged with the National Regional Development data to form a balanced panel of 174 regions and 6 years.
2. Second, we calculated the region-and year-specific net migration variable and relative net migration variables: as proxy variables for 'subjective' quality of living in a given region. These in turn were also merged with the National Regional Development database.
3. Once the 'objective' and 'subjective' indices were calculated, we were in a position to actually analyse the impact of RDPs on LAU1 regions. Whilst in standard policy analysis settings, the sampleaverage treatment effects cannot be calculated because only one of the two possible outcomes for each individual (or region in our case) can be observed, this issue was solved by the RDI which allowed for the creation of the counterfactual. Following the insights from impact analysis literature, we thus adopted the counterfactual framework developed by Rosenbaum and Rubin (1983). We employed propensity score matching (PSM) to predict the probability of a region being subsidised on the basis of observed covariates for both subsidised and non-subsidised regions. The method balances the observed covariates between the subsidised and non-subsidised regions based on the similarity between the predicted probability of their being selected as subsidised regions. The aim of PSM matching is to find a comparison group of subsidised regions from a sample of non-subsidised regions that is closest (in terms of observed characteristics) to the sample of subsidised regions. More specifically, sub-regions are selected into treatment and non-treatment groups that have similar potential outcomes (rdi, relative and net migration scores). We employed a matching estimation technique to identify the treatment effects. More specifically, sub-regions selected into treatment and non-treatment groups had the potential outcomes (TE scores)  4 as: Estimating the treatment effects based on Propensity Score Matching (PSM) requires two assumptions. First, the Conditional Independence Assumption (CIA), which states that for a given set of covariates participation is independent of potential outcomes. The second condition is that the Average Treatment Effect for the treated is only defined within the region of common support. This assumption ensures that treatment observations have comparison observations 'nearby' in the propensity score distribution. For a more comprehensive discussion of the econometric theory behind this methodology, we refer the reader to Imbens and Wooldridge (2009) and Guo and Fraser (2010).
The descriptive statistics for the development subsidies (years 2008-2013, total per region, per capita and per square km) presented in Table 1  This uneven distribution is also reflected in the extremely high standard deviations. The negative minimum numbers in the table are due to two regions that had to repay RDP funds. The picture is made more nuanced by the last two rows in Table 1 (per capita and per square km subsidy) in which the inequality of distribution is less prominent.  Figure A2 in the Annex depicts the box plot graphs of total, per capita and per km 2 subsidies; here we focus on the yearly averages of all subsidy variables (Table 2).

Results and discussion
Inspection of the ten highest and lowest RDI indices confirm what seems intuitive; i.e. RDI is lowest in the North-East, East, and South-West of Hungary (e.g. the Ozdi, Berettyóújfalúi, and Fehérgyarmati regions) whilst the high RDI regions are clustered around the capital, Budapest (e.g. Gödöllői, Ráckevei, Dunakeszi, and Budapest regions) and in the North-West. The RDIs display high correlation indices across the timespan, suggesting the stability of development rankings over the years. Regional objective RDI levels for the beginning and end of the period under examination are presented in Figure 1A. Regions are coded from highest ranking (darkest shade) to lowest (lightest shade). Results also confirm our intuition: LAU1 regions in the East/North-East and South-West  Figure 1B depicts the internal migration flow relative to the size of local, LAU1, and population, whilst 1C accounts only for net migration flows. The argument for using both relative and net migration is that the government, faced with the depopulation of areas, may choose to target these, irrespective of the planned objective level of development. Further, whilst relative migration is more often used, persistent outmigration from the North, North-East and South decreases the denominator in eq. 3, thus in the long run these connected yet different concepts may diverge.   Now, turning our attention to the impact of development funds, in our analysis we specifically focus on Rural Development Subsidies, 75% of which are paid by the EU, and 25% in the form of cofinancing. In the introduction to this paper we already emphasised the magnitude and importance of these payments. With respect to subsidies, Figures 3-5 depict the regional intensity of total, per capita and per square km subsidies received. The pattern of geographical distribution is less obvious compared to that illustrated on the regional development maps. It can be seen that less-developed regions benefited most from support, especially in terms of per capita or per km 2 subsidies. In line with the current literature, we analysed the impact of regional development subsidies using propensity score matching 3 . The estimated propensity score is actually the probability of participation in a program (treatment), conditioned on control variables calculated for all sub-regions. A number of matching algorithms are available for this purpose, such as nearest neighbour, radius caliper, stratification matching and kernel matching (Abadie et al. 2004, Leuven, Sianesi 2009). Whilst asymptotically all matching procedures should result in similar conclusions, small sample estimation may pose some problems. The following criteria were used to choose the appropriate matching algorithm: a) standardised bias, b) t-test, and c) common significance and pseudo R 2 .
Since all sub-regions received some development support, a (necessarily subjective) rule had to be imposed to differentiate between treated and non-treated region. For robustness, in the research described in this paper for each treatment indicator dummy (i.e. total subsidy per region, per capita, and per km²) we used two definitions: areas where support intensity was higher than two-thirds of the yearly median/average subsidy (MSub and ASub, respectively) were qualified as subsidised (for the yearly distribution of treated and non-treated regions, see Table A1). In the first step, a Probit model was estimated for all three subsidy indicators (thus only the dependent variable changes). Two of the covariates controlled for the initial status of a given sub-region: level of objective quality of living (rdi_ob08), and unemployment rate (unemp08), both measured in 2008. In addition, yearly unemployment rate (unempit), the variables used to build the objective index (group1_n to group4_n), and finally, a dummy capturing whether the given LAU1 region is the home of the county seat were used as explanatory variables in the binary choice model. Results of the Probit estimations (see Table   A2 displaying probit estimations for PSM; results of DID are available upon request) were used to 3 We used the psmatch2 STATA routine for the estimation.
calculate the probability of participation (of being treated) of a given region in development projects.
As discussed before, to create appropriate counterfactuals the PSM methodology requires careful balancing of covariates. Thus, before turning to the actual assessment of the impact of development subsidy payments upon Hungarian regions, we present balancing tests of covariates in Tables 3-5.
Results emphasise that the correct matching approach was used (i.e. where the mean values of covariates were significantly different in the unmatched sample, after matching the difference in the mean was insignificant across treated and untreated sub-regions).  An important requisite of PSM methodology is an assessment of whether common support or overlap assumptions actually hold (Caliendo, Kopeining, 2005). The test is based on a comparison of the distribution of estimated propensity scores in the treated and untreated samples. This may be done using graphical approaches (kernel density functions or histograms) or by applying parametric/nonparametric statistical tests. The result of Smirnov-Kolmogorov tests suggest that we may not reject the null hypothesis of the equal distribution of the two groups at a 1% significance level. We assessed the ATT impact of development subsidies on sub-regions (see Abadie et al. 2004 for a discussion of pros and cons) using non-parametric Kernel matching with common support. Table 6 presents our main results for the objective (rdi) and subjective (rel_migr and net_migr) local development indices.
We reached the same conclusion -quite unfortunate from a policy perspective -that, regardless of the outcome variable (objective or subjective), or the definition of the treatment dummy, the impact of support is not significant. Next, we present the results of PSM-DID which can help overcome hidden-bias, and generally may improve non-experimental program evaluation. Table 7 displays the results of DID for the six support dummy variables. A key issue here was how the baseline and end periods are defined. Since data exist for six years, for robustness three definitions were used. First, the two extreme years (2008 and 2013), followed by the first two and last two years, and finally, the first three and last three years. used to compute significances and common support was imposed. *p<.1; **p<.05; *** p<.01.
Of the 54 diff-in-diffs estimated, only seven proved to be significant -mainly when the first three years were taken as baseline, and the last three as follow-up periods. Note, significant impacts are all negative, and generally small. That suggests a negative effect of subsidies on RDI, although these impacts are very small PSM-DID estimates partly reinforced the findings of ATT; namely, that it is difficult to identify any positive effects of RDP funds upon regional levels of development. Regardless of the subsidy variable employed or definition of baseline-end period, most results were not significant. Even more surprisingly, where significant, the impact was negative.

Conclusions
The main contribution to the literature of this paper is its assessment of (almost) an entire programming period and focus on the overall effects of Rural Development payments at a highly disaggregated level using three QoL indicators, along with six definitions of subsidies. This analysis of sub-regions' subsidy data and econometric estimations leads to several conclusions. First, we find considerable variation in terms of the level of subsidies received by regions during the period under analysis. Second, and most importantly from a policy point of view, it is very difficult to identify any impact of European development subsidies, and not only because estimates are sensitive to the chosen supported variables. Only a few estimations revealed significant impacts, and these were negative instead of the expected positive. Due to the lack of relevant papers about the topic, it is difficult to assess these results against other research to evaluate the impact of European RDP. The exception is a paper by Michalek (2012) which assessed the impact of the SAPARD program in Slovakia. Using directly comparable methodology, Michalek (2012) also finds negligible impacts for the SAPARD RD program on Slovakian rural regions. We conclude that, irrespective of estimated coefficients, the impact of regional subsidies is negligible -a result that should raise important policy-related questions. As a consequence, further research is needed to explore the impacts and mechanisms of subsidies.