An Integrated Indicator System and Evaluation Model for Regional Sustainable Development

Regional sustainable development has become a worldwide issue in recent years, but there is no single and universally agreed method of choosing indicators for sustainable development assessment. The subjective selection of indicators will affect the results of assessment. Each evaluation method has its own advantages and disadvantages, and the methods used to determine indicator weight also differ. Regional sustainable development is a complex system, which is difficult to evaluate objectively and scientifically using a single method. Therefore, a new integrated indicator system and evaluation model is constructed here to more accurately reflect regional sustainable development level. The indicator system and evaluation model were constructed using a case study of 17 cities in Shandong Province, China. The indicator system includes 4 subsystems, i.e., economy, society, resource, and environment. These indicators were selected through correlation analysis and discrimination analysis. A back propagation neural network was applied to evaluate the respective scores of the 4 subsystems. The comprehensive score for regional sustainable development was evaluated using the analytic hierarchy process with entropy correction. The results show that sustainable development levels in these 17 cities show a gradually decreasing trend from east to west and from coast to inland. Cities with an underdeveloped economy usually display poor levels of social development and serious environmental pollution. Through the improvement of indicator screening, evaluation model, and result correction, the error caused by a single evaluation method can be reduced significantly. This new methodology for indicator selection and comprehensive evaluation provides a new perspective for the assessment of regional sustainable development.


Introduction
Excessive resource consumption, high population levels, severe environmental pollution and rapid climate change represent significant challenges for achieving sustainable development around the world.Breaking the globe into regions provides an approach to tackling worldwide sustainable development.A quantified indicator system and evaluation model can be used to reflect the current level of regional sustainable development; this baseline can then inform the development of corresponding policies and plans.
There has been a good deal of research in the field of regional sustainable development.Sala et al. [1] present a novel methodological framework for sustainability evaluation models and indicators by addressing critical decision-making elements to recognize the ontological, epistemological, and methodological foundations of sustainability.Arushanyan et al. [2] present a new sustainability assessment framework to assess the environmental and social risks and opportunities in future scenarios.Their qualitative assessment of environmental and social aspects uses a consumption perspective.Popovic et al. [3] developed a clear definition of quantitative indicators that can be used to perform social sustainability assessment.Research has also explored the application of multi-methodology strategies to define a set of indicators for quantitative assessment of social sustainability in supply chains based on uncertainty analysis of different sustainability assessment methods.Reznichenko et al. [4] propose that modern institutional conversions confirm the relevance of the research on problems of sustainable development of socio-economic systems and the identification of effective management tools at the level of regions.Some of the most important issues in ensuring sustainable regional development are the state of the social and production infrastructure, the provision of the region with food, fuel, and energy resources.For example, high dependence on outdated and inefficient manufacturing processes is likely to impact on resources and environmental aspects of the region.Similarly, the choice of renewable or non-renewable energy sources is likely to impact on environmental quality.Mally [5] presents the findings of a study that used a selection of 32 economic, social, and environmental indicators to evaluate the extent of achieving these objectives in Slovenian statistical regions from 2010 to 2014.
China has been undergoing rapid economic development over the past few decades, but this progress has also brought serious environmental pollution.In order to achieve sustainable development, the Chinese government has issued a series of policies and plans [6][7][8].However, a key issue in measuring sustainable development is how to establish an indicator system and evaluation method.Liu [9] presented an effective framework of general sustainability indicators for renewable energy systems.Naganathan and Chong [10] addressed issues of quantification and policy intent through the proposed State Sustainable Transportation Performances matrices, and ranked sustainable transportation performance based on the indicators of the USA.Wu et al. [11] proposed a new hybrid evaluation method based on an analytical hierarchy process (AHP)-entropy weight and the cloud model to evaluate community sustainability.This method makes use of the superiority of the cloud model to transform qualitative remarks into quantitative representations to reflect fuzziness and randomness.Davor et al. [12] presented an empirical study to assess the sustainability performance of European countries using differential multi-criteria analysis.Their Preference Ranking Organization Method for Enrichment Evaluations was applied on 38 headline and operational sustainable development indicators defined under the EU Sustainable Development Strategy.
Sustainable development has been a basic principle of national development in many countries, at both the regional level and city level, but existing problems and the level of sustainable development vary across different regions [13,14].A quantitative evaluation of the sustainable development level of different regions is useful in understanding constraints and proposing corresponding counter measures to enhance sustainability.Phillis et al. [15] selected 46 indicators across 7 aspects to evaluate the sustainable development level of 106 cities in different countries using the fuzzy evaluation method.The results can guide decision makers in allocating their available resources to one or more indicators to obtain the largest improvement of sustainability.Elgadi et al. [16] constructed an indicator system with three levels and four categories to evaluate the sustainable development of Tripoli through primary index screening.Using an indicator system of three fields, Widomski et al. [17] compared the indicator values of Lublin, Poland with other EU countries, and evaluated its sustainable development level.Tsai [18] selected a set of indicators divided into six dimensions to analyze the sustainable development trend of Taiwan over several years using the Pressure-State-Response (PSR) model.Davor et al. [19] investigated the feasibility of modeling municipal waste generation for countries at different levels of development using artificial neural networks (ANN) and selected generic indicators of sustainability.Based on a comparison of actual municipal waste generation data with predictions given by the model, it showed that ANN can be applied to modeling and forecasting municipal waste generation on a national scale.Ugwu and Haupt [20] proposed a comprehensive model to assess the sustainability of infrastructure projects from the aspects of economy, environment, society, resource utilization, health and safety, and project management.Marynych [21] used a methodology based on the vector autoregressive and Johansen VEC approach procedures.The results of the estimation of the aggregate index of sustainable development showed a significant impact and variability in the economic component, which had further influence on social and environmental indicators.These studies demonstrate that the international evaluation of sustainable development started years ago, but there is no unified determination method for indicator systems, and the conclusions using different systems are often quite different.
Existing indicator systems for assessing sustainable development can be divided into two kinds: single and comprehensive evaluation indicator systems.The former adopts a hybrid indicator system to summarize progress on regional sustainable development.Examples are genuine saving [22], energy analysis [23,24], ecological footprint [25,26], the human development index [27,28], and the social progress index [29].These indicator systems work well in respect of specific criteria, but less well as a general measure of sustainability.
The latter, comprehensive indicator systems, analyse regional sustainable development levels with multi-hierarchy indicators, such as the PSR model, three component model [30], or an indicator system based on AHP.They can reflect sustainable development levels widely and clearly, but data acquisition and results calculation become difficult once the indicator system is complex.With respect to the evaluation methods of sustainable development level, AHP [31], principal component analysis (PCA) [32], data envelopment analysis (DEA) [33], technique for order preference by similarity to ideal solution (TOPSIS) [34], and ANN [35] are often applied.The AHP method is used most frequently, and is usually combined with the entropy method and fuzzy comprehensive evaluation because of its strong subjectivity.The weight determination of PCA is objective, but the correlation among indicators should be significant.The DEA model is also objective, but it can only evaluate relative efficiency.TOPSIS has simple principles and fast calculations, but can only evaluate the relative strengths and weaknesses of objects, and the weight component may suffer from subjectivity.ANN can realize the fitting of arbitrary nonlinear functions, which is consistent with the characteristics of the complex nonlinear system of regional sustainable development.At present, the ANN algorithm is complex.If the sample cannot guarantee accuracy, the findings may be incorrect.
In summary, two existing problems need to be improved upon in the current study.Firstly, there is no single and universally agreed upon method for choosing indicators.The subjective selection of indicators will affect the results.Secondly, each evaluation method has its own advantages and disadvantages, and the methods used to determine indicator weight also differ.It should be ensured that the evaluation system can evaluate the regional sustainable development level accurately and comprehensively.
In this research, a comprehensive selection of indicators was the first step.These indicators were then screened by means of discrimination analysis and correlation analysis to reduce the subjectivity of indicator selection.Spurious relationships among sustainable development indicators were found and improved using partial correlation analysis.This method ensures the accuracy of the selected indicators.ANN and AHP with entropy correction were applied to evaluate and rank the levels of regional sustainable development.This improvement of indicator screening, evaluation model, and result correction can reduce the error caused by single evaluation method significantly.This new indicator system and evaluation model can reflect the regional level of sustainable development accurately, which provides theoretical guidance for increasing the capability of sustainable development in different regions.

Area of Study
Shandong Province is one of the more economically developed provinces in China, and covers an area of 157.1 thousand km 2 .Its GDP amounted to 7646.97 billion RMB in 2018, which accounts for 8.5% of national GDP [36].The population of Shandong Province is 0.1 billion, ranking it second place in Chinese provinces.The social and economic situation of Shandong Province is similar to China as a whole, with more economically developed regions on the eastern coastline and underdeveloped regions in the western inland areas.The level of development of economy, society, resources, and environment in the 17 cities of Shandong Province is variable.Thus, Shandong Province is a typical representative of the whole country.Figure 1 shows the location of Shandong Province in China.Figure 2 shows the geographical locations of the 17 cities in Shandong Province.In order to construct a more objective and scientific indicator system and evaluation model for regional sustainable development, discrimination analysis, Pearson correlation analysis, partial correlation analysis, back propagation (BP) ANN, and AHP with entropy correction were applied.

Construction of Indicator System
To construct the indicator system for regional sustainable development, it is necessary to follow principles of hierarchy, simplicity, comprehensiveness, and operability [37].Referring to the viewpoints of the China Sustainable Development Strategy Report 2016 [38], Widomski et al. [17] and Phillis et al. [15], and the framework of the Drive-State-Response (DSR) model proposed by UNCSD [39], the indicator system is divided into four layers, i.e., object layer, system layer, feature layer, and indicator layer.The sustainable development system is divided into four subsystems, i.e., the economy, society, resources, and environment.Figure 3 shows the layout of the indicator system for regional sustainable development.The indicators were screened using discrimination analysis and correlation analysis after the initial comprehensive selection of indicators.

Discrimination Analysis
The discrimination of an evaluation indicator refers to its ability to distinguish the feature differences of the objects evaluated.If an indicator has similar values for all evaluated regions, it means the discrimination of this indicator is too weak to recognize the difference of sustainable development level for these regions.In practice, the coefficient of variation is usually used to describe the discrimination of the indicators [40], see Equation (1). where X i is the average value, and Si = is the standard deviation.The indicator discrimination was analyzed using SPSS 19.0 software to determine the descriptive value, and assess the discrimination of indicators.The smaller the coefficient of variation V i , the weaker the indicator discrimination will be.By comparing V i and the critical value, smaller V i values were deleted to ensure the simplicity and comprehensiveness of the selected indicators.

Pearson Correlation Analysis
There were some correlations among the indicators in the system, and the repeated information reflected by them would impact the results of the regional sustainable development evaluation.The correlation coefficient between two indicators was calculated by Pearson correlation analysis [41].One of the indicators was deleted if the correlation coefficient between two indicators was high, so as to reduce the impact of repeated information reflected by the indicators on the evaluation results.This was done in three steps [42]: Firstly, the standardization of the original value was calculated with Equation (2): where Z i is the standardized value, X i is original value of an indicator, and S i is the standard deviation.Secondly, the correlation coefficient R ij between two indicators was calculated with Equation (3): Thirdly, a critical value A was set.If R ij > A, the indicator or whichever has less significance was deleted.Otherwise, both indicators were retained.

Partial Correlation Analysis
Partial correlation analysis is a method where the impact of other elements is regarded as constant when researching the correlation between two factors in a multi-element system [43].In this way, the partial correlation between two evaluation indicators can be identified, which can enhance the accuracy and reliability of the evaluation system.The partial correlation coefficient is calculated by Equation (4).
Due to the complicated relations among various factors in the regional sustainable development system, it was difficult to select control variables.Some policy-or management-related factors cannot be quantified, which leads to partial correlation analysis failure due to the lack of data availability.Therefore, an expert judgment method was adopted and the correlation between two indicators was classified on a four-point differential as one of: no correlation, weak, strong, or very strong.One of the two indicators in pairs with a strong or very strong correlation was deleted.

Back Propagation Artificial Neural Network
BP ANN is a multi-layer feed-forward neural network; it can approximate complex nonlinear functions with any precision [44].It is widely used due to its simple structure, rich algorithm, and strong nonlinear mapping capability.The learning algorithm of BP ANN is the error back propagation algorithm.Therefore, as long as the parameters are fully adjusted and the samples are accurate and reliable, errors generated by the neural network can be significantly reduced.The indicator system and evaluation model can accurately reflect the sustainable development level of each region.MATLAB software was applied to establish the BP ANN model of regional sustainable development evaluation and calculate the scores of the four subsystems for sustainable development in different sample cities.
(1) Data normalization In Equation ( 5), X i is the indicator value of one evaluated object.X i is the normalized value.X min is the minimum indicator value.X max is the maximum indicator value.For the negative indicators, the final value is 1 minus the normalized value.
(2) Building training samples The calculation of BP ANN requires predicted output, which is unknown for evaluation.Based on the linear interpolation of sample data between min and max, influence levels are set to generate training samples [45][46][47][48][49].In this research, 500 training samples were established.Referring to Kennedy et al. [45] and Sun et al. [46], and combining with the expert judgment method, sustainable development capacity was classified into 5 levels: (0, 1] means very low level, (1,2] means low level, (2,3] means moderate level, (3,4] means high level, (4,5] means very high level. (3) The establishment of the back propagation artificial neural network a.The quantification of hidden layers The three-layer neural network applied in this research can approach continuous function precisely [49].Increasing the number of hidden layers can reduce errors, but it will also increase the training time.
b.The quantification of hidden layer nodes Fewer hidden layer nodes will decrease the ability to find the trend of samples in a network and result in too many minimum points.By contrast, more hidden layer nodes can increase the training time.The general way to determine the optimal number of hidden layer nodes is cut and trial method.Varying the number using a cut and trial method can be useful to optimize the number of hidden layer nodes [50].The estimated value was used as the initial value of the cut and trial method, which was obtained by Equations ( 6)-( 9) respectively.The number of hidden layer nodes was increased gradually.After comparing the prediction performance of the network, the number of nodes with the best performance was selected as the number of hidden layer nodes.
where S is the number of hidden layer nodes, m is the number of input nodes, n is the number of output nodes, and a is a constant between 1 and 10. c.The selection of transfer function For nonlinear mappings, the sigmoid transfer function [51] and linear transfer function are usually applied for the hidden layer and output layer.The output values of the entire network can be arbitrary.In this research, logsig or tansig was selected as the hidden layer transfer function, purelin was selected as the output layer transfer function, and traingdx or trainlm was selected as the training function.
d.The determination of learning parameters The range of learning rates was set in the range from 0.01 to 0.8.A higher rate may result in network instability, while lower rates may increase training time.Generally, a lower rate is used in order to make the network converge easily. e

. The selection of training function
The improved training algorithm is usually adopted for BP ANN.Furthermore, the additional momentum factor may cause the network to slip through the local minimum point, which can speed up the convergence rate.Adaptive learning efficiency can adjust learning efficiency automatically and enhance the network's stability.The Levenberg-Marquardt (LM) algorithm has a quick convergence rate, which is suitable for small and medium sized networks.Based on these analyses, the gradient descent method, traingdx with adaptive learning efficiency and additional momentum, and trainlm of LM algorithm were adopted in this research.

Analytic Hierarchy Process with Entropy Correction
AHP with entropy correction was selected to calculate overall scores of regional sustainable development levels.The main steps included establishing a multi-hierarchy model, structuring a judgment matrix, level ranking, and consistency checking [52].The weight coefficients were modified by entropy correction in order to ensure the consistency of the judgment matrix.The main steps in entropy correction were as follows: (1) Calculating the output entropy e j of indicator f j .
The judgment matrix (a ij ) m×n was normalized by formula a ij / n j=1 a ij in order to get the standard matrix A = (a ij ) n×m .The output entropy e j of indicator f j was calculated with Equation ( 10).
(2) Calculating the deflection degree d j of indicator f j with Equation (11).
(3) Calculating the information weight µ j of indicator f j with Equation ( 12).
Figure 4 is the technical roadmap, which shows the concept and methods of constructing the indicator system and evaluation model for regional sustainable development.

Confirmation of the Indicator System
Due to data unavailability, d10, d16, and d58 were deleted.Indicators d14 and d15 were the moderate indicators, which had different optimum populations for different regions.In order to ensure the accuracy of the results, d14 and d15 were deleted [54].The 53 remaining indicators were then tested using discrimination analysis, Pearson correlation analysis, and partial correlation analysis.

Discrimination Analysis
Table 1 shows the coefficient of variation of each indicator calculated by SPSS.According to the data distribution and the importance of each indicator in sustainable development of Shandong Province, 0.12 was selected as the critical value of the coefficient of variation, referring to Arushanyan et al. [2], Chen et al. [13], and Kılkış [23], The indicators with variation coefficients less than 0.12 were d4, d6, d21, d23, d26, d27, d28, d37, d38, d53, d54, d55, and d57.Indicator d6 was retained because it was one of the most appropriate indicators to reflect economic development.Forty one indicators were retained after discrimination analysis.Bold means value less than 0.12.

Pearson Correlation Analysis
An absolute value of correlation coefficient less than 0.3 indicates no linear correlation, 0.3-0.5 indicates low correlation, 0.5-0.8indicates moderate correlation, and above 0.8 indicates significant correlation [55,56].If the correlation coefficient ≥0.8, it means that the correlation is high for the two indicators and one of these two indicators should be deleted.In this way, d7 and d11 were deleted in the Economic subsystem.All indicators were retained in the Resources subsystem.In the Social subsystem, d24, d25, and d35 had high correlation, but these indicators were retained because they were the crucial indicators in relation to the total emissions control target in The 13th Five-Year Plan for National Eco-Environmental Conservation [7].

Partial Correlation Analysis
Analyzing the partial correlation of d1 and d3 as examples, d12 was regarded as a control variable.When the control variables were not controlled, the correlation coefficient between d1 and d3 was 0.961, and the significance level was 0.000, which is highly correlated and significant.The correlation coefficient dropped down to 0.587 when considering d12 as a control variable; as a result, the significance level was 0.017.See Table 2.It can be seen that d12 was significantly correlated with d1 and d3 at the same time.According to the correlation logic, d1 and d3 should also be significantly correlated, but partial correlation analysis showed that when d12 was regarded as a control variable, the correlation coefficient between d1 and d3 decreased significantly.Indicators d1 and d3 were moderately correlated.Therefore, d1 and d3 can be considered to reflect the characteristics of different aspects within the economic subsystem.Both d1 and d3 were retained in the final indicator system.
According to partial correlation analysis, there may be a spurious relationship between two indicators with a high correlation coefficient.After correlation analysis, expert judgement was applied to delete indicators with high correlation.Indicators d7, d11, d24, d25, and d35 were deleted in this step.
Following indicator selection, the final indicator system with 36 indicators is shown in Table A2 in the Appendix A.

Subsystem Assessment with Back Propagation Artificial Neural Network
The network training parameters for the subsystems are listed in Table 3.According to Equations ( 5)-( 9), the training results are described in Figure 5.The simulink model is used to simulate the trained network.The network output for different subsystems is listed in Table 4, which represents the scores of different subsystems in each city.The Economic, Social, Resources or Environmental level of each city can be compared separately.A higher score represents a higher level of development in this subsystem.

Comprehensive Evaluation for Regional Sustainable Development by Analytic Hierarchy Process with Entropy Correction
The weight of different subsystems was calculated by AHP with entropy correction through Equations ( 10)- (13), see Table 5.The overall score of regional sustainable development level for the 17 different cities can be obtained through their overall score and weight for the four subsystems, see Table 6 and Figure 6.

Construction of Indicator System
To construct the indicator system for regional sustainable development, it was necessary to follow principles of hierarchy, simplicity, comprehensiveness, and operability.But it is difficult to ensure both the comprehensiveness and simplicity of the indicator system at the same time.Among the indicators with variation coefficients less than 0.12, only d6 was retained, due to its characteristic to reflect economic development.In each subsystem, the retained indicators were more representative than those deleted in discrimination analysis.
One of two related indicators will be deleted by correlation analysis if the correlation coefficient ≥0.8, but the remaining one may be deleted in the discrimination analysis, which leads to incomplete information being reflected by the indicators.Therefore, discrimination analysis should be given priority.Compared with the indicator systems such as the United Nations Sustainability Development Goals and China Sustainable Development Strategy Report, this indicator system applied multiple methods, such as correlation analysis, discrimination analysis, back propagation neural network, and AHP with entropy correction, and can reflect regional sustainable development more comprehensively and concisely.

Back Propagation Artificial Neural Network and Analytic Hierarchy Process with Entropy Correction
For nonlinear mappings, the sigmoid transfer function is usually applied for hidden layer and output layer.The output values of the entire network may be arbitrary.Three error curves in Figure 5 represent the training sample, validation sample, and test sample, respectively.The neural network automatically divides training samples into three types according to the default values of 75%, 15%, and 15%.The error of all three samples was very low, which means the neural network simulation was good.It also shows that ANN can realize the fitting of arbitrary nonlinear functions, which is consistent with the characteristics of the complex nonlinear system of regional sustainable development.
In order to ensure the consistency of the judgment matrix, the weights of different subsystems were calculated by AHP with entropy correction.The Economic, Social, Resources, and Environmental subsystems were weighted as 0.1028, 0.2681, 0.3146, and 0.3146, respectively.The economic development of Shandong Province has been rapid, but this also has also led to serious resource consumption and environmental pollution.The larger weight for the Resources subsystem and the Environmental subsystem shows these two subsystems are more important than the other two subsystems.

Analysis of Sustainable Development Levels
The overall score of regional sustainable development levels shows that Weihai and Qingdao have a very high level of regional sustainable development.Yantai, Dongying, Jinan, Linyi, Rizhao, and Weifang have a high level.Tai'an, Zaozhuang, Dezhou, Zibo, Jining, Binzhou, and Liaocheng have a general level.Heze and Laiwu have a low level.
Table 4 shows that coastal cities have comparatively high scores for different subsystems.The development of subsystems is coordinated, and economic and social development will not cause serious negative impacts on the Resources and Environment subsystems.As an inland city, the geographical advantage of Jinan is not obvious, but its Economic and Social development levels are better because it is the capital city of Shandong Province.The Resources subsystem is the main limiting factor affecting sustainable development for Qingdao, Jinan, and Dongying, since it is lower than the other three subsystems in these cities.For other inland cities, the four subsystem levels of Binzhou and Liaocheng are generally low, and the levels of Laiwu and Heze are even lower than these latter two cities.The constraints of Laiwu are shortage of resources and low environmental quality.For Heze, the levels of economic and social development are low, and the level of environmental quality also needs to be improved.
The sustainable development levels of different cities in Shandong Province present the following characteristics.Firstly, regional sustainable development shows a gradually decreasing trend from east to west and from coast to inland (see Figure 6).Secondly, coastal cities have higher regional sustainable development capacity, due to their geographical advantages.Their foreign trade capabilities are strong, and their economic development does not rely on heavy industries.Their proportions of tertiary industry are large and environmental quality is superior.Thirdly, cities such as Jining, Liaocheng, Dezhou, and Heze, which suffer from poor economic development, usually lag behind in social development.This is reflected in aspects such as residential living standards, educational attainment levels, urban infrastructure construction, levels of medical care, science and technology, and social security.In order to improve the level of social development, these cities need to show improvement in education, technology, and urban construction.At the same time, accelerating economic development and providing financial support for social development are further issues.Fourthly, environmental pollution is serious in economically backward cities.In these cities, the industrial structure is dominated by secondary industry.More efforts should be made to enhance science and technology development, optimize industrial structure, and reduce pollutant emissions.
This research provides a new integrated indicator system and evaluation model for regional sustainable development, which can provide a useful reference for developing policies to increase the capability of regional sustainable development in Shandong Province.Compared with the 234 indicators in the United Nations Sustainability Development Goals, this indicator system with 36 indicators and the evaluation model with correlation analysis, discrimination analysis, back propagation neural network, and AHP with entropy correction can reflect regional sustainable development more comprehensively and concisely.This methodology is applicable for different cities or different provinces in a country, but it is not suitable for making comparison between countries, because some indicators may represent different factors in different countries.Additionally, the data for some indicators are not available in some countries.In the future, a common index system that is suitable for evaluating national sustainable development needs to be explored.

Conclusions
An indicator system for regional sustainable development was established using the processes of initial indicator selection, discrimination analysis, and partial correlation analysis, which includes 3 layers, 4 subsystems, and 36 indicators.BP ANN was used to calculate the scores of the four subsystems.The overall score of the regional sustainable development of the 17 cities was evaluated by AHP with entropy correction.The results show that the cities with a very high level of sustainable development are Weihai and Qingdao, and those with a low level are Heze and Laiwu.The sustainable development levels of these 17 cities in Shandong Province show a gradually decreasing trend from east to west and from coast to inland.Cities with an underdeveloped economy usually have backward social development and serious environmental pollution.Through the indicator system and the scores of subsystems, the restrictive factors of the sustainable development level of each city were identified.These findings provide a useful reference for developing policies to increase the capability of regional sustainable development.

Figure 1 .
Figure 1.Geographical map of Shandong Province in China.

Figure 2 .
Figure 2. Detailed map of the 17 cities in Shandong Province.

Figure 3 .
Figure 3. Layout of indicator system for regional sustainable development.

Figure 4 .
Figure 4. Technical roadmap to construct the indicator system and evaluation model for regional sustainable development.
The input data of the 17 sample cities in Shandong Province were retrieved from the Shandong Provincial Statistical Yearbook 2016 [53], and the statistical yearbooks and statistical bulletins of the 17 cities, respectively.Referring to the views of Phillis et al. [15], Popovic et al. [3], and Tran [39] there were 58 indicators in the indicator system, namely: (1) 13 indicators in 5 categories for the Economic subsystem, (2) 26 indicators in 5 categories for the Social subsystem, (3) 7 indicators in 3 categories for the Resource subsystem, and (4) 12 indicators in 3 categories for the Environmental subsystem, see Table A1 in the Appendix A.

Table 1 .
Coefficient of variation of each indicator after initial selection.

Table 2 .
Correlation coefficient between d1 and d3 with or without control variable d12.

Table 3 .
The network training parameters of subsystems.

Table 4 .
Network output results of different subsystems.

Table 5 .
The weights of different subsystems.

Table 6 .
The overall score of regional sustainable development levels of different cities.
Figure 6.Graphical distribution of cities based on regional sustainable development levels.

Table A2 .
The indicator system for evaluation of regional sustainable development.