Next Article in Journal
Clustering-Based Binarization Methods Applied to the Crow Search Algorithm for 0/1 Combinatorial Problems
Next Article in Special Issue
Economic Policy Uncertainty and Stock Market Spillovers: Case of Selected CEE Markets
Previous Article in Journal
Numerical Scheme for Solving Time–Space Vibration String Equation of Fractional Derivative
Previous Article in Special Issue
Analyzing the Causality and Dependence between Gold Shocks and Asian Emerging Stock Markets: A Smooth Transition Copula Approach
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

A Complex MCDM Procedure for the Assessment of Economic Development of Units at Different Government Levels

by
Aleksandra Łuczak
* and
Małgorzata Just
*
Department of Finance and Accounting, Faculty of Economics and Social Sciences, Poznań University of Life Sciences, Wojska Polskiego 28, 60-637 Poznań, Poland
*
Authors to whom correspondence should be addressed.
Mathematics 2020, 8(7), 1067; https://doi.org/10.3390/math8071067
Submission received: 18 May 2020 / Revised: 10 June 2020 / Accepted: 17 June 2020 / Published: 2 July 2020
(This article belongs to the Special Issue Advanced Methods in Mathematical Finance)

Abstract

:
Studies on the economic development of government units are among the key challenges for authorities at different levels and an issue often investigated by economists. In spite of a considerable interest in the issue, there is no standard procedure for the assessment of economic development level of units at different levels of government (national, regional, sub-regional). This assessment needs a complex system of methods and techniques applicable to the various types of data. So, adequate methods must be used at each level. This paper proposes a complex procedure for a synthetic indicator. The units are assessed at different government levels. Each level (national, regional, and sub-regional) may be described with a particular type of variables. Set of data may include variables with a normal or near-normal distribution, a strong asymmetry or extreme values. The objective of this paper is to present the potential behind the application of a complex Multi-Criteria Decision Making (MCDM) procedure based on the tail selection method used in the Extreme Value Theory (EVT), i.e., Mean Excess Function (MEF) together with one of the most popular MCDM methods, namely the Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS), to assess the economic development level of units at different government levels. MEF is helpful to identify extreme values of variables and limit their impact on the ranking of local administrative units (LAUs). TOPSIS is suitable in ranking units described with multidimensional data set. The study explored the use of two types of TOPSIS (classical and positional) depending on the type of variables. These approaches were used in the assessment of economic development level of LAUs at national, regional and sub-regional levels in Poland in 2017.

1. Introduction

Regional and local development are processes that overlap and condition each other. Local development can be considered a process which, while being similar to regional development extending over the economic, political, social, and cultural aspects, but takes place on a smaller scale. The source literature provides various definitions of local and regional development. In the American literature (e.g., [1,2]), local development is related to economic development while in the European literature it is often called socio-economic development (e.g., [3]). Generally, local development means economic development. It should be noted that most definitions view development as positive changes. In contrast, the definition by Schumpeter [4] clearly differs from classical definitions of development, and considers it to be a transition between stages of an economic system which cannot be subdivided into infinitely small steps.
Economic development is a complex process affected by multiple determinants. From the perspective of local and regional policies, it is important to identify the factors of development, i.e., certain properties of an area or occurrences which take place within or outside of area and have an impact on that economic development of territorial unit. Agarwal et al. [5] claim that some factors can comprise multiple aspects. Generally, they may be divided into endogenous and exogenous factors. Endogenous factors are the resources of a territory, including natural environment resources, landscape values, mineral resources, geographic location, human capital, technical and social infrastructure, investment capital, universities, research and development institutes, knowledge and skills of local authorities and population. In turn, exogenous factors include the government policy, economic system, historical conditions, and technological progress (cf. [6]).
The assessment of economic development level of local territorial unit is an issue often considered and investigated by economists. Eraydin [7] stated that quantitative methods were commonly used around the world in the 1950s and 1960s. This is confirmed by Anselin [8] who emphasizes that in late 1950s, the development of spatial theories brought a large number of models which became widespread in regional analyses but lost their popularity in the 1980s. Renewed interest in quantitative methods came only at the beginning of the 21st century. Despite the interest in this issue, there is no unified comprehensive procedure to assessing the development levels of local territorial units. It should be noted also that, paradoxically, different Multi-Criteria Decision Making (MCDM) methods may yield different results. Therefore, choosing the right method is the fundamental problem when using them. Although there are many methods for solving multi-criteria problems, some of them are criticized for being “ad-hoc” measures used without a complete theoretical or empirical background, only because no better options exist [9,10]. Obviously, all methods have some advantages and some limitations. Certain quantitative methods are quite complicated, time and cost consuming. However, they often turn out to be more effective in areas that are suitable for quantitative analysis. Their effectiveness consists in that they deliver more specific (quantitative) “facts” and more detailed results [11]. Their usefulness depends on the proper identification of advantages and disadvantages of the existing methods. It can be asserted that, in a sense, quantitative methods enable a greater rationalization and objectification of processes compared to qualitative methods.
It should be noted also that choosing the level of analysis (national, regional, sub-regional) is an important aspect in assessing local territorial units. Different levels of analyses involve different data types. The larger the number of units covered by a study, the greater are the differences in their values, and the tougher become the problems related to their analysis. Different data types have different properties associated with them. For these reasons, a multiway comprehensive and methodological procedure needs to be adopted to assess the economic development level of local administrative units at national, regional and sub-regional levels. Therefore, the purpose of this paper is to present the potential behind the application of a complex MCDM procedure based on the tail selection method used in the Extreme Value Theory (EVT), i.e., Mean Excess Function (MEF) and Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS), to assess the economic development level of units at different government levels. This procedure was used in the assessment of economic development level of local administrative units (LAUs) at national, regional and sub-regional levels in Poland in 2017. The study was based on 2017 statistical data retrieved from the Central Statistical Office of Poland [12].
The authors’ contribution to the development of methods to assess economic development of units includes, firstly, a proposal of complex Multi-Criteria Decision Making procedure to assess LAUs at different levels: national, regional, and sub-regional. Secondly, by the tail selection method used in the Extreme Value Theory, i.e., Mean Excess Function and positional version of Technique for Order of Preference by Similarity to Ideal Solution, the authors improved quality of assessment of economic development level of territorial units and expand possibilities of research at different government levels. The results of the study are important for decision-makers in LAUs who participate in the process of creating documents (financial plans, investment plans, strategies).
This paper, except introduction, is structured as follows: Section 2 presents the methods and data used in the empirical study, Section 3 comprises the results of research with their discussion, while the last two sections sum up the study and include conclusions and recommendations.

2. Materials and Methods

Each level of analyses (national, regional, sub-regional) can be described with variables of different types (i.e., ones which follow a normal, asymmetric or strongly asymmetric distribution). Different data types have different properties associated with them. For these reasons, this paper proposes a comprehensive multiway procedure for assessing economic development level of local administrative units at national, regional and sub-regional levels. The construction of a synthetic measure is a multi-stage process which requires the researcher, at each stage, to make many decisions regarding aspects such as: method for selecting variables to be studied, normalization procedure and method of values aggregation. These are important and often difficult tasks, especially if actual datasets include extreme values which may be due to the particularities of the complex phenomenon considered.
This paper proposes to application a synthetic measure with the use of MCDM methods combined with MEF. At the beginning of the 21st century, MCDM methods have gained in importance and are developing very fast, boosted by progress in computer-aided calculations. MCDM methods are suitable for complex problems which are characterized by uncertainty and often include conflicting objectives and different data and information types in comprehensive, evolving socio-economic systems [13]. Generally, there are two groups of MCDM: Multi-Attribute Decision Making (MADM) and Multi-Objective Decision Making (MODM) [14]. The first one includes a finite number of alternatives. In the latter, the number of alternatives is infinite (cf. [15]). Usually, problems related to the selection and assessment of alternative solutions are finite, i.e., belong to the first group. In turn, as regards development-related problems, the decisive attribute may take any value from a specific interval. Therefore, the number of potential alternative solutions may be infinite [10]. Many various MCDM methods exist e.g.: the Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS), the Analytic Hierarchy Process (AHP), VIšekriterijumska optimizacija i KOmpromisno Rešenje (VIKOR), the Weighted Sum Method (WSM), ELimination Et Choix Traduisant la REalité (ELECTRE), and the Preference Ranking Organization METHod for Enrichment Evaluation (PROMETHEE) (see [16]). Currently, modifications, enhancements and new unique approaches to MCDM emerge around the world (see, e.g., [17]). With its broad range of versatile uses, TOPSIS stands apart from the MCDM methods. Behzadian et al. [18] identified following application areas: supply chain management and logistics, design, engineering and manufacturing systems, business and marketing management, health, safety and environment management, human resources management, energy management, chemical engineering, water resources management and other issues.
Different approaches with TOPSIS methods appear: fuzzy TOPSIS based on fuzzy sets, hesitant fuzzy sets or intuitionistic fuzzy sets (see also [19,20,21,22,23]), interval type-2 fuzzy TOPSIS [24,25], extended intuitionistic fuzzy TOPSIS [26], fuzzy TOPSIS with emerging interval-valued spherical fuzzy sets [27], rough cloud TOPSIS [28], TOPSIS-Sort for sorting problems [29], and fuzzy rough number-based TOPSIS [30]. There also exist many applications of TOPSIS with other MCDM methods: fuzzy DEMATEL and TOPSIS [31], AHP and fuzzy TOPSIS [32], fuzzy AHP and TOPSIS [33], fuzzy TOPSIS and fuzzy COPRAS [34]. We propose to combine TOPSIS with MEF which allows to identify extreme values of variables. The appropriate identification of extreme values is a demanding task which can be done using a wide variety of methods. This paper is a follow-up to authors’ previous research which relied on the use TOPSIS with MEF to identify the extreme values and eliminate their impact on the synthetic measure used to assess the financial situation of Polish municipalities (see [35]).
Proposed multiway procedure includes four approaches based on TOPSIS [36] and MEF (see e.g., [37]) to assess the economic development level of LAUs at different government levels. This process includes the construction of a synthetic measure which enables a synthetic assessment of a phenomenon described by multiple variables. It consists of eight key stages (Figure 1).
The first stage of constructing a synthetic measure is to determine a hierarchical structure for the multi-criteria assessment of economic development levels of LAUs. This is done by decomposing the problem into the following components: main assessment criterion, sub-criteria, variables, and units to be assessed (e.g., LAUs) (Figure 2).
Particular attention should be paid to selecting the variables. For this purpose, they need to be validated in substantive and statistical terms. Substantive validation precedes statistical ones. The substantive analysis may rely on expert opinions, the researcher’s knowledge of the issue concerned, or guidelines developed as part of theories of economic phenomena (see [38]). Once the substantive validation is complete, in statistical analysis descriptive statistics can be used, including e.g., arithmetic mean, variance, correlation coefficient. The analysis of the inverse correlation matrix between values of variables R 1 = [ r i j ] , where r i j ( 1 , + )   ( i ,   j =   1 ,   ,   K ) , facilitates the elimination of variables strongly correlated with each other. If some values are excessively correlated with other ones, the elements on the main diagonal r i i are much greater than ten [39]. Excessively correlated variables are eliminated from the initial data set. Main assessment criterion is a primary standard for judging or assessing complex phenomenon (first level). Sub-criteria are set of related factors related to main criterion (second level). Each sub-criterion is described by package of variables (third level). All variables describe each unit (fourth level).
The variables describing the economic development level of local territorial units (i.e., districts) at country or voivodeship level usually include extreme values or they follow a strongly (or extremely strongly) asymmetrical distribution. That problem can be solved by analyzing the graph of the Mean Excess Function in order to identify the extreme values; and by using the positional TOPSIS method together with the Weber spatial median to restrict the impact of the strong asymmetry (cf. [35]).
In the second stage, the graph of the Mean Excess Function is used to identify the extreme values. The analysis of the MEF graph is one of the methods for identifying the threshold level which defines the beginning of the distribution tail modeled with the Peaks Over Threshold (POT) model. In the POT method (see e.g., [40]), the distribution tail of the variable X k ( k -th variable) is modeled based on the generalized Pareto distribution:
F ^ ( x k ) = 1 N u l k N ( 1 + ξ ^ x k u l k β ^ ) 1 ^ ξ ,
where u l k is threshold value, N the number of observations (units, e.g., districts), N u l k the number of observations above the threshold value u l k . ξ ^ and β ^ are estimates of shape and scale parameters, respectively, and F ^ ( x k ) is a tail estimator of the distribution function of the variable X k .
The starting point for analyzing the MEF graph is the conditional expected value [37]:
E ( X k u l k | X k > u l k ) = β ( u l k ) 1 ξ ,   ξ < 1 .
β ( u l k )   is linearly dependent on u l k , and therefore the empirical estimator of conditional expected value must also be linearly dependent on u l k . Hence, the MEF graph:
{ ( u l k , 1 N u l k i = 1 N u l k ( x i k u l k ) ) : u l k < x k m a x }
should be linear above u l k . The method described above is used to identify the beginning of the right tail of the variable (extreme values in the right tail of the variable). In the left tail, the values of the variable are multiplied by minus one to calculate the threshold value l l k . Following the identification of thresholds, the values of the variable located in the distribution tails were replaced by the corresponding threshold values. This transformation is referred to as winsorization.
The third stage consists in determining the nature of variables, and grouping them by whether they have a stimulating or destimulating effect. Variables with a destimulating effect need to be converted into ones with a stimulating effect using the negative coefficient transformation.
In the fourth stage, the variables are normalized in order to make them mutually comparable. This consists in rescaling the variables and unifying their orders of magnitude. This study uses four different types of normalization of variables depending on data type:
  • classical standardization: for all variables that follow a normal or near-normal distribution (approach I):
    z i k = x i k x ¯ k s k ,
  • classical standardization for winsorized data: for a set of variables which includes extreme values (approach II):
    z i k = { l l k x ¯ k s k     for   x i k l l k x i k x ¯ k s k     for   l l k < x i k < u l k u l k x ¯ k s k     for   x i k u l k ,
  • classical median standardization based on Weber spatial median: for a set of variables which includes variables that follow a strongly asymmetric distribution (approach III) [41,42]:
    z i k = x i k m e ˜ d k 1.4826 m a ˜ d k ,
  • modified median standardization based on Weber spatial median: for a set of variables which includes variables that have extreme values and follow a strongly asymmetric distribution (approach IV) [35]:
    Z i = { l l k m e ˜ d k 1.4826 m a ˜ d k for   x i k l l k x i k m e ˜ d k 1.4826 m a ˜ d k for   l l k < x i k < u l k u l k m e ˜ d k 1.4826 m a ˜ d k for   x i k u l k .
In Formulas (4)–(7) x i k is value of the variable k ( k = 1, 2, …, K ) in the i -th unit (e.g., LAU) ( i   =1, 2, …, N ); x ¯ k —mean value of the variable k ; s k —standard deviation for the variable k ; z i k —normalized value of the variable k in the i-th unit; m e ˜ d k —Weber median for the variable k; m a ˜ d k = m e d i | x i k m e ˜ d k | —absolute median deviation for the variable k ; 1.4826—constant scaling coefficient which depends on the distribution of variables (see [41,42]). The calculations were performed with robustX package in the R program [43].
The fifth stage will consist in determining the positive ideal solution (PIS):
A + = ( m a x i ( z i 1 ) ,   m a x i ( z i 2 ) , , m a x i ( z i K ) ) = ( z 1 + , z 2 + , , z K + )
and the negative ideal solution (NIS):
A = ( m i n i ( z i 1 ) , m i n i ( z i 2 ) , , m i n i ( z i K ) ) = ( z 1 , z 2 , , z K ) .
Next, in the sixth stage, L1 (Manhattan) distances are calculated between each multi-attribute object and the PIS:
d i + = k = 1 K | z i k z k + |         ( i = 1 ,   2 , ,   N )
and between each multi-attribute object and the NIS:
d i = k = 1 K | z i k z k |         ( i = 1 ,   2 , ,   N ) .
In the seventh stage, the aggregation formula proposed by Hwang and Yoon [36] is used to construct the synthetic measure of development:
S i = d i d i + d i +         ( i = 1 ,   2 , ,   N ) .
The synthetic variable S i ranges from 0 to 1 in value. The higher values of the synthetic measure of development, the higher is the level of economic development.
In the eighth stage, the calculated values of the synthetic measure of development are used to linearly order the LAUs and to typological classification. The entire range of the synthetic variable may be divided into classes by defining numeric intervals for the measure S i (Table 1).
The classes of level of economic development for districts were evaluated by measure of homogeneity. Total clusters homogeneity measure H O   base on the idea of Hubert and Levin cf. [45] was proposed by Łuczak and Just [35]:
H O = 1 r c = 1 C       1 N c i P c k = 1 K | x i k * m e d i P c ( x i k * ) | m i n i ( k = 1 K | x i k * m e d i P c ( x i k * ) | ) m a x i ( k = 1 K | x i k * m e d i P c ( x i k * ) | ) m i n i ( k = 1 K | x i k * m e d i P c ( x i k * ) | ) ,
where x i k * —standardized value of the variable k (k = 1, 2, …, K) in the i-th unit ( i   = 1, 2, …, N ); N c —the number of objects in c-th class (c = 1,…, C); P c —set of subscripts of objects in the c-th class; r—the number of non-empty classes. The lower the value of the measure H O the more homogeneous the classes are.
Phenomena involved in economic development level of territorial units are described using three fundamental dimensions: units covered by the study (e.g., districts); diagnostic variables used to describe a phenomenon related to development processes; and time. The proposed multiway procedure was used to assess the level of economic development of LAUs (districts) at national ( N = 314 districts in Poland), regional ( N = 31 districts in the Wielkopolskie voivodeship) and sub-regional ( N   = 8 districts in the Pleszew sub-region—Pleszew district, and neighboring districts) levels in Poland in 2017. The study was based on 2017 statistical data obtained from the Central Statistical Office of Poland [12].
Based on substantive criteria were selected variables that represent: public finance, technical infrastructure, social infrastructure, and economy. Eight variables were selected based on a substantive and statistical analysis, as follows:
  • x1—own revenue of municipal budgets in the district (PLN per capita),
  • x2—population per public pharmacy,
  • x3—share of population served by water supply in the total population (%),
  • x4—share of population served by a sewerage network in the total population (%),
  • x5—entities of the national economy per 10,000 working-age population,
  • x6—investment outlays in enterprises per capita (in current prices) (PLN),
  • x7—average monthly gross wages and salaries (PLN),
  • x8—registered unemployment rate (%).
A five-year (2013–2017) average was calculated for the variable x1 which represents own revenue of municipal budgets in the district. The distributions of variables for the districts at national and regional levels are shown in Figure 3 and Figure 4.

3. Results

This study proposed and used a procedure for the assessment of economic development level of LAUs at national, regional and sub-regional levels in Poland in 2017. The analysis assumed that two variables (x2 and x8) have a destimulating effect while others have a stimulating effect.
At national level, most variable distributions describing districts (x1, x2, x5, x6 and x7) demonstrated strong or extremely strong positive skewness. Only one variable distribution (x3) was found to demonstrate extremely strong negative skewness. These variables also had high positive kurtosis values which mean that extreme values are highly likely to appear (Table 2). The distribution analysis of these variables, based on the graphs of the empirical density function (Figure 3) corroborated these findings. On the basis of the Jarque-Bera test, it was stated that the variables describing districts at national level were not normally distributed (at the significance level of 0.05). It should be noted that only one variable (x4) followed a near-normal distribution (there were no reason to reject the hypothesis that it is normally distributed at the significance level of 0.01). As regards variables that demonstrated strong or extremely strong positive skewness, MEF graphs were used as a basis for determining the threshold values for the right tail of the distribution (or the left tail for the variable with extremely strong negative skewness) (Table 2).
Also, the skewness coefficient indicated strong or extremely strong asymmetry in distribution for three variables (x1, x2 and x5) attributed to districts at regional level (Figure 4, Table 3). As regards these variables, kurtosis also indicated that extreme values were more likely to emerge in the right tail of their distribution than in a normal distribution. Based on the analysis of measures of volatility (range, standard deviation) and density function graphs, these variables can be concluded to be less volatile than the corresponding variables describing districts at national level. In the case of these three variables attributed to districts at the regional level, there were only a few outliers.
Considering the descriptive statistics of variables attributed to districts at regional and sub-regional levels (Table 3 and Table 4), and the relatively small number of districts at these levels, this study did not identify the extreme values based on the corresponding MEF graph.
Afterwards, winsorization was performed to replace tail values with calculated threshold values for districts at national level (Table 2). Once transformed, the variable distributions exhibited moderate skewness and smaller kurtosis than for non-winsorized variables (Table 2).
This study used and compared four approaches at national, regional, and sub-regional levels (Table 5, Table 6, Table 7 and Table 8):
  • classical TOPSIS with classical standardization of values of variables (approach I),
  • classical TOPSIS with classical standardization of values of winsorized variables (approach II),
  • positional TOPSIS with positional standardization of values of variables (approach III),
  • positional TOPSIS with positional standardization of values of winsorized variables (approach IV).
In the approach I at national level, LAUs are primarily concentrated in one class (nearly 88% of the total number of LAUs), representing a low level (Table 5). This results from extreme values and a strong asymmetry of variables. An improvement in the classification is not provided by the positional approach (III) which should restrict the impact of the strong asymmetry through the use of the positional standardization. The results obtained with winsorized variables (approach II) were much better. The reduction of impact of extreme values through data winsorization enabled a more adequate attribution of LAUs to classes. In turn, in the approach IV, while restricting the impact of extreme values through data winsorization, also reduced the impact of the strong asymmetry by using the positional standardization formula. This approach gave a similar result to approach II, i.e., the substantively best classification which reflected the differences in levels of economic development between LAU classes. As a consequence, it provided a more complete and more correct identification of economic development types. Similar results of approaches II and IV resulted from the lack of significant asymmetries in the central part of the distributions of variables describing districts.
At the regional level, the approaches (I and III) gave similar results as at national level (Table 6). In the classical approach I, LAUs are mainly concentrated in the two classes at medium-low (nearly 38.7% of LAUs) and low (51.6% of LAUs) levels. Winsorization could not be used because there was the relatively small number of districts at regional level. No extreme values were identified at sub-regional level. At that level of analysis, only I and III approaches were used (Table 7). In approach I, four classes were distinguished, and in approach III, only three classes.
Furthermore, Table 8 presents the descriptive statistics for synthetic measures of economic development level of LAUs by approaches and analysis levels.
In the final ranking, objects described by variables with atypical values may have an excessively high or low rank. Outlier-robust approaches (the II, III and IV approaches) were proposed as a way of solving these problems. As shown by empirical research, the approaches with winsorized data (II and IV approaches) proposed in this paper improve the rankings if the variables include extreme values.
Values of homogeneity measures according to approaches were shown in the Figure 5. At national level classifications, based on data set including variables with extreme values, gave more homogenous classes for robust approaches (II and IV). Whereas, positional approach (III) led to more homogenous classes at regional level, because data set includes variables with asymmetry. However, at sub-regional level classification, based on data sets including variables with a normal or near-normal distribution, provided more homogenous classes for classical approach (I). It should be noted, if more classes have been distinguished, the classes are more homogeneous.
Additionally summarizing, the study revealed also advantages and disadvantages methods used (Table 9).

4. Conclusions

The proposed multiway procedure to a multidimensional analysis of economic development levels of LAUs was implemented with the use of a complex MCDM procedure. The critical analysis of four different approaches was presented. Complex approaches (approach II and approach IV) are the original proposals, that reduce the impact of strong asymmetry and extreme values of variables attributed to objects (LAUs at national level). These approaches use the MEF to identify extreme values and set up the positive and negative ideal solutions. The occurrence of even one extreme (very large or very small) value for LAU can significantly affect its final position in ranking, resulting in it being ranked too high or too low. This is evidenced by the use of the classical and positional TOPSIS methods (the approach I and approach III). The main advantage of the proposed hybrid positional approach (IV) is that (opposite to other approaches considered) it can be apply intended for variables with extreme values and strongly asymmetric distribution. It was shown that the use of other approaches considered can lead to the wrong rankings and incorrect typologies of units. Approaches I–III are less suitable for the assessment of units described simultaneously by variables with extreme values and strongly asymmetric distribution, as they mistakenly create the ranking.
At national level the extreme values were identified, so the results obtained with the positional TOPSIS based on winsorized data has revealed at the national level in 2017 more accurate typological classification of districts in Poland in terms of the level of economic development. The reduction of impact of extreme values through winsorization of data enabled a more adequate attribution of objects to classes, and this typological classification was according with the expectations. At regional and sub-regional levels, this study did not identify the extreme values and only classical and positional methods could be used.

5. Recommendations

The analysis of economic development level of local administrative units (i.e., districts) at different (national, regional and sub-regional) levels involves different data types. If the data set includes atypical values of variables, the use of classical MCDM methods (e.g., TOPSIS) may result in excessively reducing the range of variation of the synthetic measure. As a consequence, it may become problematic to properly identify the development types of the complex phenomenon under consideration. The reason for these problems is that empty classes may appear if the types of development are identified based on an arbitrary manner.
The authors recommend using:
  • The hybrid positional approach: analyzing the MEF graphs and using the positional TOPSIS method with winsorized data, if the variables follow an asymmetrical distribution and include extreme values.
  • The hybrid approach: analyzing the MEF graphs and using the classical TOPSIS method with winsorized data, if the distribution of the variables includes extreme values.
  • The positional approach: using the positional TOPSIS method with positional standardization based on Weber spatial median, if the variables follow an asymmetric distribution without extreme values.
  • The classical approach: using the classical TOPSIS method based on classical standardization, if all variables follow a normal or near-normal distribution.
It should be noted that the choice of the most appropriate approach mainly depends on the distribution of variables.
Moreover, the proposed procedure to assessing the economic development levels of LAUs is a universal technique that may be used for other administrative units. The multiway procedure discussed in this paper may also provide a basis for development strategies and other strategic documents (e.g., financial plans, investment plans, operating plans).

Author Contributions

Conceptualization, A.Ł. and M.J.; methodology, A.Ł. and M.J.; validation, A.Ł. and M.J.; formal analysis, A.Ł. and M.J.; investigation, A.Ł. and M.J.; resources, A.Ł. and M.J.; data curation, A.Ł.; writing—original draft preparation, A.Ł. and M.J.; writing—review and editing, A.Ł. and M.J.; visualization, A.Ł. and M.J.; supervision, A.Ł. and M.J.; project administration, A.Ł. and M.J. All authors have read and agree to the published version of the manuscript.

Funding

This research received no external funding.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Reese, L.A.; Rosenfeld, R.A. Local Economic Development in the United States and Canada: Institutionalizing Policy Approaches. Am. Rev. Public Adm. 2004, 34, 277–292. [Google Scholar] [CrossRef]
  2. Morgan, J.Q. The Role of Local Government in Economic Development: Survey Findings from North Carolina; UNC School of Government: Chapel Hill, NC, USA, 2009. [Google Scholar]
  3. Stanny, M.; Rosner, A.; Kozdroń, E. Changes in the Spatial Distribution of the Level and Dynamics of Socio-Economic Development of Rural Areas in Poland. Village Agric. 2016, 4, 29–47. [Google Scholar]
  4. Schumpeter, J.A. Development. J. Econ. Lit. 2005, 43, 108–120. [Google Scholar] [CrossRef]
  5. Agarwal, S.; Rahman, S.; Errington, A. Measuring the determinants of relative economic performance of rural areas. J. Rural Stud. 2009, 25, 309–321. [Google Scholar] [CrossRef]
  6. Parysek, J.J. Foundations of Local Economy; Adam Mickiewicz University in Poznań: Poznan, Poland, 1997. (In Polish) [Google Scholar]
  7. Eraydin, A. Changing importance of quantitative methods in regional development and planning. A|Z ITU J. Fac. Archit. 2011, 8, 4–16. [Google Scholar]
  8. Anselin, L. Spatial Econometrics: Methods and Models; Kluwer Academic Publishers: Dordrecht, The Netherlands, 1988. [Google Scholar]
  9. Stewart, T.J. A critical survey on the status of multiple criteria decision making theory and practice. OMEGA Int. J. Manag. Sci. 1992, 20, 569–586. [Google Scholar]
  10. Xu, L.; Yang, J.-B. Introduction to Multi-Criteria Decision Making and the Evidential Reasoning Approach; Working Paper, No. 0106; Manchester School of Management, University of Manchester Institute of Science and Technology: Manchester, UK, 2001; Available online: https://phps.portals.mbs.ac.uk/Portals/49/docs/jyang/XuYang_MSM_WorkingPaperFinal.pdf (accessed on 5 December 2013).
  11. Timmreck, T.C. Planning, Program Development, and Evaluation: A Handbook for Health Promotion, Aging, and Health Services; Jones & Bartlett Learning: London, UK, 2003. [Google Scholar]
  12. Local Data Bank, Central Statistical Office of Poland. Available online: www.stat.gov.pl (accessed on 22 July 2019).
  13. Wang, J.-J.; Jing, Y.-Y.; Zhang, C.-F.; Zhao, J.-H. Review on multi-criteria decision analysis aid in sustainable energy decision-making. Renew. Sustain. Energy. Rev. 2009, 13, 2263–2278. [Google Scholar] [CrossRef]
  14. Pourjavad, E.; Shirouyehzad, H. A MCDM Approach for Prioritizing Production Lines: A Case Study. Int. J. Bus. Manag. 2011, 6, 221–229. [Google Scholar] [CrossRef] [Green Version]
  15. Pedrycz, W.; Ekel, P.; Parreiras, R. Fuzzy Multicriteria Decision-Making: Models. Methods and Applications; John Wiley & Sons: Chichester, UK, 2011. [Google Scholar]
  16. Velasquez, M.; Hester, P.T. An Analysis of Multi-Criteria Decision Making Methods. Int. J. Oper. Res. 2013, 10, 56–66. [Google Scholar]
  17. Mardani, A.; Jusoh, A.; Zavadskas, E.K. Fuzzy multiple criteria decision-making techniques and applications—Two decades review from 1994 to 2014. Expert Syst. Appl. 2015, 42, 4126–4148. [Google Scholar] [CrossRef]
  18. Behzadian, M.; Otaghsars, S.K.; Yazdani, M.; Ignatius, J. A state-of the-art survey of TOPSIS applications. Expert Syst. Appl. 2012, 39, 13051–13069. [Google Scholar] [CrossRef]
  19. Zyoud, S.H.; Kaufmann, L.G.; Shaheen, H.; Samhan, S.; Fuchs-Hanusch, D. A framework for water loss management in developing countries under fuzzy environment: Integration of Fuzzy AHP with Fuzzy TOPSIS. Expert Syst. Appl. 2016, 61, 86–105. [Google Scholar] [CrossRef]
  20. Han, H.; Trimi, S. A fuzzy TOPSIS method for performance evaluation of reverse logistics in social commerce platforms. Expert Syst. Appl. 2018, 103, 133–145. [Google Scholar] [CrossRef]
  21. Hatami-Marbini, A.; Kangi, F. An extension of fuzzy TOPSIS for a group decision making with an application to Tehran stock exchange. Appl. Soft Comput. 2017, 52, 1084–1097. [Google Scholar] [CrossRef]
  22. Palczewski, K.; Sałabun, W. The fuzzy TOPSIS applications in the last decade. Procedia Comput. Sci. 2019, 159, 2294–2303. [Google Scholar] [CrossRef]
  23. Salih, M.M.; Zaidan, B.B.; Zaidan, A.A.; Ahmed, M.A. Survey on fuzzy TOPSIS state-of-the-art between 2007 and 2017. Comput. Oper. Res. 2019, 104, 207–227. [Google Scholar] [CrossRef]
  24. Wu, T.; Liu, X.; Liu, F. An interval type-2 fuzzy TOPSIS model for large scale group decision making problems with social network information. Inf. Sci. 2018, 432, 392–410. [Google Scholar] [CrossRef]
  25. Yucesan, M.; Mete, S.; Serin, F.; Celik, E.; Gul, M. An Integrated Best-Worst and Interval Type-2 Fuzzy TOPSIS Methodology for Green Supplier Selection. Mathematics 2019, 7, 182. [Google Scholar] [CrossRef] [Green Version]
  26. Shen, F.; Ma, X.; Li, Z.; Xu, Z.; Cai, D. An extended intuitionistic fuzzy TOPSIS method based on a new distance measure with an application to credit risk evaluation. Inf. Sci. 2018, 428, 105–119. [Google Scholar] [CrossRef]
  27. Gündoğdu, F.K.; Kahraman, C. A novel fuzzy TOPSIS method using emerging interval-valued spherical fuzzy sets. Eng. Appl. Artif. Intell. 2019, 85, 307–323. [Google Scholar] [CrossRef]
  28. Li, J.; Fang, H.; Song, W. Sustainable supplier selection based on SSCM practices: A rough cloud TOPSIS approach. J. Clean. Prod. 2019, 222, 606–621. [Google Scholar] [CrossRef]
  29. de Lima Silva, D.F.; de Almeida Filho, A.T. Sorting with TOPSIS through boundary and characteristic profiles. Comput. Ind. Eng. 2020, 141, 106328. [Google Scholar] [CrossRef]
  30. Zhu, G.-N.; Hu, J.; Ren, H. A fuzzy rough number-based AHP-TOPSIS for design concept evaluation under uncertain environments. Appl. Soft Comput. 2020, 91, 106228. [Google Scholar] [CrossRef]
  31. Zhang, X.; Su, J. A combined fuzzy DEMATEL and TOPSIS approach for estimating participants in knowledge-intensive crowdsourcing. Comput. Ind. Eng. 2019, 137, 106085. [Google Scholar] [CrossRef]
  32. Rajak, M.; Shaw, K. Evaluation and selection of mobile health (mHealth) applications using AHP and fuzzy TOPSIS. Technol. Soc. 2019, 59, 101186. [Google Scholar] [CrossRef]
  33. Chou, T.-Y.; Chen, Y.-T. Applying Fuzzy AHP and TOPSIS Method to Identify Key Organizational Capabilities. Mathematics 2020, 8, 836. [Google Scholar] [CrossRef]
  34. Dhiman, H.S.; Deb, D. Fuzzy TOPSIS and fuzzy COPRAS based multi-criteria decision making for hybrid wind farms. Energy 2020, 202, 117755. [Google Scholar] [CrossRef]
  35. Łuczak, A.; Just, M. The positional MEF-TOPSIS method for the assessment of complex economic phenomena in territorial units. Stat. Transit. New Ser. 2020, 21, 157–172. [Google Scholar] [CrossRef]
  36. Hwang, C.L.; Yoon, K. Multiple Attribute Decision-Making: Methods and Applications; Springer: Berlin, Germany, 1981. [Google Scholar]
  37. Coles, S.G. An Introduction to Statistical Modeling of Extreme Values; Springer: London, UK, 2001. [Google Scholar]
  38. Walesiak, M. Variable selection and weighting problems in cluster analysis. Res. Pap. Acad. Econ. Wrocław 2005, 1076, 106–118. (In Polish) [Google Scholar]
  39. Malina, A.; Zeliaś, A. Taxonomic analysis of the spatial diversity of the quality of life of the population in Poland in 1994. Stat. Rev. 1997, 1, 11–127. (In Polish) [Google Scholar]
  40. McNeil, A.J. Extreme Value Theory for Risk Management; Department Mathematics ETH Zentrum: Zurich, Switzerland, 1999. [Google Scholar]
  41. Młodak, A. Taxonomic Analysis in Regional Statistics; Difin: Warsaw, Poland, 2006. (In Polish) [Google Scholar]
  42. Młodak, A. History of the Weber problem. Appl. Math. 2009, 10, 3–21. (In Polish) [Google Scholar]
  43. Stahel, W.; Maechler, M. robustX: eXperimental eXtraneous eXtraordinary. Functionality for Robust Statistics; R package, Version 1.1–3; 2012; Available online: https://cran.r-project.org/web/packages/robustX (accessed on 3 July 2012).
  44. Wysocki, F. The Methods of Taxonomy for Recognition of Economic Types in Agriculture and Rural Areas; Poznań University of Life Sciences: Poznan, Poland, 2010. (In Polish) [Google Scholar]
  45. Hubert, L.J.; Levin, J.R. A general statistical framework for assessing categorical clustering in free recall. Psychol. Bull. 1976, 83, 1072–1080. [Google Scholar] [CrossRef]
  46. Wuertz, D.; Setz, T.; Chalabi, Y. fExtremes: Rmetrics—Modelling Extreme Events in Finance; R package, Version 3042.82; 2017; Available online: https://cran.r-project.org/web/packages/fExtremes/index.html (accessed on 15 March 2018).
Figure 1. Procedure of construction of the synthetic measure.
Figure 1. Procedure of construction of the synthetic measure.
Mathematics 08 01067 g001
Figure 2. Hierarchical structure for the multi-criteria assessment of units.
Figure 2. Hierarchical structure for the multi-criteria assessment of units.
Mathematics 08 01067 g002
Figure 3. (a) Estimated density for the variable x1—own revenue of municipal budgets in the district at national level. (b) Estimated density for the variable x2—population per public pharmacy at national level. (c) Estimated density for the variable x3—share of population served by water supply at national level. (d) Estimated density for the variable x4—share of population served by a sewerage network at national level. (e) Estimated density for the variable x5—entities of the national economy per 10,000 of population in working age at national level. (f) Estimated density for the variable x6—investment outlays in enterprises per capita at national level. (g) Estimated density for the variable x7—average monthly gross wages and salaries at national level. (h) Estimated density for the variable x8—registered unemployment rate at national level.
Figure 3. (a) Estimated density for the variable x1—own revenue of municipal budgets in the district at national level. (b) Estimated density for the variable x2—population per public pharmacy at national level. (c) Estimated density for the variable x3—share of population served by water supply at national level. (d) Estimated density for the variable x4—share of population served by a sewerage network at national level. (e) Estimated density for the variable x5—entities of the national economy per 10,000 of population in working age at national level. (f) Estimated density for the variable x6—investment outlays in enterprises per capita at national level. (g) Estimated density for the variable x7—average monthly gross wages and salaries at national level. (h) Estimated density for the variable x8—registered unemployment rate at national level.
Mathematics 08 01067 g003
Figure 4. (a) Estimated density for the variable x1—own revenue of municipal budgets in the district at the regional level. (b) Estimated density for the variable x2—population per public pharmacy at regional level. (c) Estimated density for the variable x3—share of population served by water supply at regional level. (d) Estimated density for the variable x4—share of population served by a sewerage network at regional level. (e) Estimated density for the variable x5—entities of the national economy per 10,000 of population in working age at the regional level. (f) Estimated density for the variable x6—investment outlays in enterprises per capita at regional level. (g) Estimated density for the variable x7—average monthly gross wages and salaries at regional level. (h) Estimated density for the variable x8—registered unemployment rate at regional level.
Figure 4. (a) Estimated density for the variable x1—own revenue of municipal budgets in the district at the regional level. (b) Estimated density for the variable x2—population per public pharmacy at regional level. (c) Estimated density for the variable x3—share of population served by water supply at regional level. (d) Estimated density for the variable x4—share of population served by a sewerage network at regional level. (e) Estimated density for the variable x5—entities of the national economy per 10,000 of population in working age at the regional level. (f) Estimated density for the variable x6—investment outlays in enterprises per capita at regional level. (g) Estimated density for the variable x7—average monthly gross wages and salaries at regional level. (h) Estimated density for the variable x8—registered unemployment rate at regional level.
Mathematics 08 01067 g004
Figure 5. Values of homogeneity measure according to approaches and different government levels. Note: II and IV approaches were not appropriate to data at regional and sub-regional levels, so did not calculate values of synthetic measures and homogeneity measure.
Figure 5. Values of homogeneity measure according to approaches and different government levels. Note: II and IV approaches were not appropriate to data at regional and sub-regional levels, so did not calculate values of synthetic measures and homogeneity measure.
Mathematics 08 01067 g005
Table 1. Classes, levels of economic development and value intervals of synthetic measure.
Table 1. Classes, levels of economic development and value intervals of synthetic measure.
Class (c)123456
Level of economic developmentvery highhighmedium-highmedium-lowlowvery low
S i [0.8, 1.0][0.6, 0.8)[0.5, 0.6)[0.4, 0.5)[0.2, 0.4)[0.0, 0.2)
Source: Own adjustment based on Wysocki [44].
Table 2. Descriptive statistics and threshold values of the variables of LAUs at the national level.
Table 2. Descriptive statistics and threshold values of the variables of LAUs at the national level.
SpecificationVariables
x1x2x3x4x5x6x7x8
Input Data
Mean1802.73389.290.058.51426.22774.13780.99.3
Median1688.63208.093.560.41354.42020.03693.68.3
Max4621.58989.099.995.23051.723 687.07516.025.8
Min951.62176.031.111.6796.1250.02960.21.7
Range3669.86813.068.883.62255.623 437.04555.824.1
St. dev.552.2926.010.616.0354.22660.1439.24.3
Ex. kurtosis5.08.18.1−0.53.421.618.90.2
Skewness1.72.3−2.6−0.31.43.83.10.7
Var. coef.30.627.311.827.324.895.911.647.0
Winsorized Data
Mean1740.33298.291.158.51394.12432.13746.39.3
Median1688.63208.093.560.41354.42020.03693.68.3
Max2402.14622.099.995.21861.05268.04429.025.8
Min951.62176.075.311.6796.1250.02960.21.7
Range1450.52446.024.683.61064.95018.01468.824.1
St. dev.391.1645.47.016.0271.11456.1310.14.3
Ex. kurtosis−0.8−0.50.1−0.5−1.0−0.7−0.20.2
Skewness0.20.5−1.1−0.30.10.70.50.7
Var. coef.22.519.67.727.319.459.98.347.0
llk××75.3×××××
ulk2402.14622.0××1861.05268.04429.0×
Note: × cannot calculate. The calculations of thresholds were performed with fExtremes package in the R program [46].
Table 3. Descriptive statistics of the variables of LAUs at the regional level.
Table 3. Descriptive statistics of the variables of LAUs at the regional level.
SpecificationVariables
x1x2x3x4x5x6x7x8
Input Data
Mean1871.13042.395.964.01577.73429.53711.64.7
Median1856.12968.096.565.61561.53465.03713.54.3
Max2940.55542.099.382.72507.88524.04482.711.1
Min1376.22176.090.434.31200.5884.02960.21.7
Range1564.33366.08.948.41307.37640.01522.59.4
St. dev.313.7652.32.311.8247.01881.1316.52.1
Ex. kurtosis3.45.8−0.20.15.21.00.21.3
Skewness1.52.0−0.8−0.61.51.10.11.0
Var. coef.16.821.42.418.415.754.98.545.0
Table 4. Descriptive statistics of the variables of LAUs at the sub-regional level.
Table 4. Descriptive statistics of the variables of LAUs at the sub-regional level.
SpecificationVariables
x1x2x3x4x5x6x7x8
Input Data
Mean1691.33486.696.754.81480.33438.63561.45.1
Median1739.13231.097.256.21457.72917.53521.73.8
Max1932.65542.098.278.61795.38111.03928.611.1
Min1376.22482.094.834.31262.7884.03340.72.9
Range556.43060.03.444.3532.67227.0587.88.2
St. dev.175.0901.61.213.0163.22390.6191.52.8
Ex. kurtosis−0.52.8−1.40.40.3−0.2−0.11.6
Skewness−0.41.6−0.50.10.70.80.81.6
Var. coef.10.325.91.223.611.069.55.453.6
Table 5. Typological classification of districts in Poland in terms of the level of economic development at national level in 2017.
Table 5. Typological classification of districts in Poland in terms of the level of economic development at national level in 2017.
Class (c)Level of Economic DevelopmentSiApproaches
I
(CT)
II
(CT & W)
III
(PT)
IV
(PT & W)
Nc%Nc%Nc%Nc%
1very high[0.8, 1.0]00.0010.3200.0010.32
2high[0.6, 0.8)10.326420.3810.326621.02
3medium-high[0.5, 0.6)41.2710433.1251.5910132.17
4medium-low[0.4, 0.5)206.377824.8492.878226.11
5low[0.2, 0.4)27687.906621.0227587.586320.06
6very low[0.0, 0.2)134.1410.32247.6410.32
Note: Si—synthetic measure, Nc—number of districts in class c, CT—classical TOPSIS, PT—positional TOPSIS, W—winsorized data.
Table 6. Typological classification of districts in Poland in terms of the level of economic development at regional level in 2017.
Table 6. Typological classification of districts in Poland in terms of the level of economic development at regional level in 2017.
Class (c)Level of Economic DevelopmentSiApproaches
I
(CT)
II
(CT & W)
III
(PT)
IV
(PT & W)
Nc%Nc%Nc%Nc%
1very high[0.8, 1.0]00.00××00.00××
2high[0.6, 0.8)13.23××13.23××
3medium-high[0.5, 0.6)26.45××00.00××
4medium-low[0.4, 0.5) 1238.71××929.03××
5low[0.2, 0.4)1651.61××2167.74××
6very low[0.0, 0.2)00.00××00.00××
Note: Si—synthetic measure, Nc—number of districts in class c, × cannot calculate—the method is not appropriate to data, CT—classical TOPSIS, PT—positional TOPSIS, W—winsorized data.
Table 7. Typological classification of districts in Poland in terms of the level of economic development at sub-regional level in 2017.
Table 7. Typological classification of districts in Poland in terms of the level of economic development at sub-regional level in 2017.
Class (c)Level of Economic DevelopmentSiApproaches
I
(CT)
II
(CT & W)
III
(PT)
IV
(PT & W)
Nc%Nc%Nc%Nc%
1very high[0.8, 1.0]00.00××00.00××
2high[0.6, 0.8)112.50××00.00××
3medium-high[0.5, 0.6)225.00××225.00××
4medium-low[0.4, 0.5)112.50××112.5××
5low[0.2, 0.4)450.00××562.5××
6very low[0.0, 0.2)00.00××00.00××
Note: Si—synthetic measure, Nc—number of districts in class c, × cannot calculate—the method is not appropriate to data, CT—classical TOPSIS, PT—positional TOPSIS, W—winsorized data.
Table 8. Descriptive statistics of the synthetic measures of economic development of districts in Poland in 2017 according to approaches.
Table 8. Descriptive statistics of the synthetic measures of economic development of districts in Poland in 2017 according to approaches.
SpecificationApproaches
National LevelRegional LevelSub-Regional Level
IIIIIIIVIIIIIIIVIIIIIIIV
Max0.600.830.640.840.63×0.61×0.66×0.57×
Min0.150.150.130.150.28×0.27×0.21×0.27×
Range0.450.690.510.690.35×0.35×0.45×0.30×
Median0.310.510.290.510.39×0.37×0.41×0.38×
Mean0.310.500.290.500.39×0.38×0.42×0.40×
Skewness0.63−0.160.97−0.160.89×0.99×0.33×0.40×
Ex. kurtosis2.05−0.403.49−0.371.23×1.76×−0.57×−1.39×
Note: × cannot calculate—the method is not appropriate to data, I—classical TOPSIS, II—classical TOPSIS with winsorized data, III—positional TOPSIS, IV—positional TOPSIS with winsorized data.
Table 9. Advantages and disadvantages of different variants of TOPSIS.
Table 9. Advantages and disadvantages of different variants of TOPSIS.
MethodsAdvantagesDisadvantages
classical TOPSIS
  • − the most popular decision-making method,
  • − simple procedure,
  • − easy to apply,
  • − intended for all variables with a normal or near-normal distribution,
  • − do not intended for variables with strongly asymmetric distribution and extreme values,
classical TOPSIS with winsorized data
  • − intended for variables with extreme values,
  • − limits impact of extreme values on the ranking of objects,
  • − partly restrict the impact of the strong asymmetry,
  • − do not intended for variables with very strongly asymmetric distribution,
  • − partial loss of information by winsorization of data,
  • − more complicated procedure than classical TOPSIS,
positional TOPSIS
  • − intended for variables with strongly asymmetric distribution
  • − restricts the impact of the strong asymmetry,
  • − do not intended for variables with extreme values,
  • − more complicated procedure than classical TOPSIS,
positional TOPSIS with winsorized data
  • − intended for variables with extreme values and strongly asymmetric distribution,
  • − significantly limits impact of extreme values on the ranking of objects,
  • − restrict the impact of the strong asymmetry.
  • − partial loss of information by winsorization of data,
  • − much more complicated procedure than classical TOPSIS.

Share and Cite

MDPI and ACS Style

Łuczak, A.; Just, M. A Complex MCDM Procedure for the Assessment of Economic Development of Units at Different Government Levels. Mathematics 2020, 8, 1067. https://doi.org/10.3390/math8071067

AMA Style

Łuczak A, Just M. A Complex MCDM Procedure for the Assessment of Economic Development of Units at Different Government Levels. Mathematics. 2020; 8(7):1067. https://doi.org/10.3390/math8071067

Chicago/Turabian Style

Łuczak, Aleksandra, and Małgorzata Just. 2020. "A Complex MCDM Procedure for the Assessment of Economic Development of Units at Different Government Levels" Mathematics 8, no. 7: 1067. https://doi.org/10.3390/math8071067

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop