Clustering Analysis of Classified Performance Evaluation of Higher Education in Shanghai Based on Topsis Model

Round 1

Reviewer 1 Report

Dear atuthors, 

I appreciated your work, although some important issues must be fixed before publication. Indeed, it can be reconsidered after major revision.  

I suggest proofreading the paper, and concentrating especially on these 3 points:

- You declare in the Introduction (and even in the abstract) that "diversification is a fundamental attribute of higher education". The point is interesting, but unfortunately in the rest of the article there is no trace of this issue. Please, you should develop this issue, above all into section 5. Discussion and Conclusions even implement new references.

- The core of your work is the method of clustering analysis of classified performance evaluation of universities and colleges. Please, you should add also a (quick?) critical discussion about the methods you take into account, especially in sections 2 and 3. And the same also for the Topsis model. Obviously, you should implement even a wider reference quotation.

- Your Conclusions should be improved, and the results should be discussed more extensively. So, even the no-specialist reader can understand the usefulness of the proposed method.

Author Response

Point 1: You declare in the Introduction (and even in the abstract) that "diversification is a fundamental attribute of higher education". The point is interesting, but unfortunately in the rest of the article there is no trace of this issue. Please, you should develop this issue, above all into section 5. Discussion and Conclusions even implement new references.

 

Response 1: Thanks very much to point out the problem. I add and adjust following content.

1. In section 2, I adjust the content of introduction of diversification of higher education and the classification of universities.

“It produces some university rankings. However the current rankings in higher education usually evaluates universities and colleges with evaluation indicator system of a harmonized standard. To a certain extent, the evaluation indicator system with harmonized standards guides and promotes the development of higher education [16]. However, it is a little homogenized to universities and colleges of different types, which ignores the differentiation development in them. Some universities and colleges develop themselves according to the homogenized evaluation indicator system in order to get good rankings, which results to homogenization in higher education development [17]. For example, they all pursue developing comprehensively and in large scale, and set up similar talent cultivation mode, cultivation objectives, curriculum system etc [18]. The homogenization development is adverse for universities and colleges to form their own characteristics. In the long run, it will be difficult to meet the needs of diversified development in the society. In 1976, UNESCO formulated the International Standard Classification of Education (ISCED). Universities and colleges are classified into three types, academic research type, professional application type and vocational skill type, which cultivate different types of students [19]. Carnegie Foundation for the Advancement of Teaching proposes the Carnegie classification of institutions of higher education, which classifies universities and colleges in US into some categories, Large Research Universities, Smaller Doctorate-Granting Universities, Comprehensive Institutions, Baccalaureate Institutions, Associate Degree Institutions, Specialized Institutions, Vocational/Technical Institutions [20]. U-Map, the tool for European classification of higher education Institution, classifies European higher education institutions through dimensions of teaching and learning, research, knowledge exchange, international orientation, regional engagement [21]. In some countries, universities and colleges are classified into different types according to their own characteristics. Universities and colleges are classified into 3 types in Germany, University, University of Applied Sciences, University of Arts and Music [22]. Universities and colleges are classified into 4 types in England, Ancient University, Red Brick University, Plate Glass University, Post-1992 University [23]. Universities and colleges are classified into 3 types in France, University, Grandeecole Universite, Institut Universitaire Detechnologie [24]. However, there is no detailed evaluation indicator system for different types of universities and colleges in above classifications.”

2. In section 3, I add following content to introduce how to do diversification in China.

“The clustering algorithm of classified performance evaluation of universities and colleges based on Topsis model aims to promote the development characteristics and running goals of universities and colleges. To a large extent, it solves the homogenization problems in current evaluation, which helps to better diversified construction and sustainable development in higher education in China.”

3. In section 3, I describe the diversification of the classified performance evaluation indicator system

Part1: “In accordance with the development characteristics and running goals, the classified performance evaluation indicator system contains two sets of evaluation indicators with the principles of dominance, measurability, systematicity, scientificity and practicality, one for universities focusing on scientific research, the other for colleges focusing on technical skills. From the perspective of diversification, non-uniform evaluation standards are adopted for different types of universities and colleges according to their special history and current situations. The evaluation standards not only reflect the general character and the inherent features of higher education, but also reflect the individual characteristics of universities and colleges. Different evaluation standards correspond to different development goal, construction task, policy support, talent cultivation type and education quality. From the perspective of sustainability, it is necessary to pay attention not only to current status, but also to the development and improvement of universities and colleges. The indicators about development increment, such as the growth ratio of faculty team, teaching resources, research projects, etc., in the evaluation period, reflects the notion of sustainability.

The design principles of the classified performance evaluation indicator system follows four points. Firstly, quantitative indicators are combined with qualitative indicators. Secondly, indicators about scale are combined with indicators about quality. Thirdly, indicators about external performance are combined with internal improvement. Fourthly, indicators about total amount are combined with indicators about per capita. Fifthly, indicators about accumulation are combined with indicators about development increment.”

Part2: “The diversification of the classified performance evaluation indicator system is mainly listed in the following aspects.

Firstly, each set of the evaluation indicators has some different tertiary indicators since development characteristics and running goals are different for different types of universities and colleges. For example, the tertiary indicator, Ratio of part-time teachers in enterprises is only available in the type of colleges focusing on technical skills. Comparing to the universities focusing on scientific research, the colleges pay more attention on practical operation.

Secondly, the weight of the same indicator may be different in each set. For example, the secondary indicator, Research Projects, is available in both sets of classified performance evaluation indicator of universities and colleges. The value of the weight is a little larger in the indicator for universities focusing on scientific research than that for the colleges focusing on technical skills according to the calculation result of Topsis model. Relatively speaking, the colleges focusing on technical skills pay more attention on talent cultivation.

Thirdly, each set of the evaluation indicators has some similar primary, secondary and even tertiary indicators, while the real meaning of similar indicators for each set is different. It reflects the characteristics of different types of universities and colleges. These indicators may be inconsistent both in names and content, or the same name with inconsistent content. For example, the tertiary indicator, Number of Outstanding Talents, is available in both sets of classified performance evaluation indicator of universities and colleges. It includes the number of academicians of the Chinese Academy of Sciences (CAS), the Chinese Academy of Engineering (CAE), members of the National Academy of Sciences (NAS), members of the Academia Europaea, etc, as to the content of the indicator for universities focusing on scientific research. However, it includes the number of masters of technical skills, craftsmen of the Nation, directors and members of teaching guidance committee of China Ministry of Education (MOE), etc, as to the content of indicator for colleges focusing on technical skills.

Fourthly, the evaluation criteria of the same indicator may be different in each set. For example, some tertiary indicators in Platform are available in both sets of classified performance evaluation indicator of universities and colleges. Due to the different evaluation criteria, the evaluation value is different even if the data of the indicator is identical.

Fifthly, the classified performance evaluation indicator system shall take into account completeness and simplicity. The evaluation indicators need cover to reflect all aspects of performance of universities and colleges, which leads to complexity. At the same time, too complicated indicator system makes performance evaluation more difficult. Focusing on too much detail may not reflect the performance of universities and colleges scientifically. It is necessary to balance the contradiction of completeness and simplicity appropriately. 

The classified performance evaluation indicator system is dynamic. Universities and colleges, government, experts, etc, are consulted to for their opinion every year. Minor revision and improvement are made accordingly. Major revision may be made every five years.”

 

Point 2: The core of your work is the method of clustering analysis of classified performance evaluation of universities and colleges. Please, you should add also a (quick?) critical discussion about the methods you take into account, especially in sections 2 and 3. And the same also for the Topsis model. Obviously, you should implement even a wider reference quotation.

 

Response 2: Thanks very much to point out the problem. I add following content.

1. In section 2, I add following content to introduce and and analyse performance evaluation in higher education.

“The performance of higher education mainly refers to the effective allocation and use of resources in higher education. It affects the level of scientific research, the quality of talents, and even reflects the potential of technology innovation in a country. The performance evaluation of higher education can promote continuous improvement of internal management in order to improve the efficiency of running universities and colleges [11]. However, the management of universities and colleges is a complicated system. Diversification is a fundamental attribute of higher education. Running level, developing orientation, developing stage, discipline focus, geography location, and social needs are diversified in various universities and colleges. The performance evaluation of higher education is a multi-attribute comprehensive work [12]. The United States and the United Kingdom have conducted performance evaluation of higher education earlier, and have established specialized institutions to carry out research in performance evaluation [13].”

2. In section 2, I add following content to introduce and analyse Topsis model.

“Topsis model is a common method which is commonly used for multi-objective decision making [51]. It calculates the relative adjacency between evaluated objects and the optimal and inferior solutions of all objects [52]. According to the principle of Topsis model, the ideal evaluation object is the one in which all data of evaluation indicators are closest to the optimal ones and farthest from the inferior ones. it can be used in the field of performance evaluation in higher education advantageously comparing with other evaluation method. Firstly, there is no strict restriction in data of evaluated objects, including the type of data, the sample size, etc. Secondly, the data of evaluated objects with different meaning have various units and valid ranges, which is called dimensions. The Topsis method standardizes dimensions in advance so as to eliminate the impact of various dimensions. Thirdly it is better to do comparative analysis on different evaluated objects. It introduces evaluation indicators of various dimensions for comprehensive evaluation.”

3. In section 2, I add following content to introduce and analyse to clustering algorithm.

“The systematic clustering algorithm classifies evaluated objects on their own characteristics. The evaluated objects with higher similarity are grouped into one category, which is called cluster, while those with lower similarity are grouped into different cluster [53]. The number of clusters is determined based on similarity of evaluated objects instead of being set in advance. The number of evaluated objects in each cluster is calculated by the systematic clustering algorithm. The systematic clustering algorithm effectively solves the classification problem of multiple indicators, avoiding subjectivity and arbitrariness. ”

 

Point 3: Your Conclusions should be improved, and the results should be discussed more extensively. So, even the no-specialist reader can understand the usefulness of the proposed method.

 

Response 3: Thanks very much to point out the problem.

1. In section 4.4, I add more content about evaluation result analysis.

“Take 23 universities focusing on scientific research as an example. Through clustering algorithm of classified performance evaluation, when the threshold value is set to 0.010, there are 6 universities in level 1 with high performance, account for about 26.09% of the total universities. It includes 4 comprehensive universities (66.67%), 1 university of science and engineering (16.67%) and 1 normal university (16.67%). There are 6 universities in level 2 with medium performance, account for about 26.09% of the total universities. It includes 1 university of science and engineering (16.67%), 2 medical universities (33.33%), 2 universities of liberal arts, economic and management (33.33%), 1 university of arts and sports (16.67%). There are 11 universities in level 3 with relatively low performance, account for about 47.82% of the total universities. It includes 3 comprehensive universities (27.27%), 4 universities of science and engineering (45.45%), 2 universities of liberal arts, economic and management (18.18%), 2 universities of arts and sports (18.18%). Among all the universities, 57.14% of comprehensive universities are in level 1 with high performance, 42.86% of them are in level 3 with relatively low performance. 16.67% of universities of science and engineering are in level 1 with high performance, 16.67% of them are in level 2 with medium performance, and 66.67% of them are in level 3 with relatively low performance. 50.00% of universities of liberal arts, economic and management are in level 2 with medium performance, 50% of them are in level 3 with relatively low performance. 33.33% of universities of arts and sports are in level 2 with medium performance, 66.67% of them are in level 3 with relatively low performance.

Based on the above data, the performance of universities under the Ministry of Education is usually stable with relatively higher performance. There are about half of the universities with relatively low performance, which are expected to improve the performance effectively.

By analyzing the evaluation indicators of universities, it shows as follows.

Firstly, the number of outstanding disciplines in top-ranked universities is significantly higher than other universities. The outstanding disciplines are the top 50 disciplines in QS, or top 1‰ disciplines in ESI. They act as the pioneers in the process of disciplines development in universities. They drive the common development of other relevant disciplines while enhancing themselves. There are breakthroughs in one or two fields to be famous worldwide, and then the overall quality of universities improves slowly. Most outstanding disciplines develop gradually in the development of universities, which are the foundation of universities. Therefore, universities can focus on discipline development and strengthen the advantages.

Secondly, some top-ranked universities and medium-ranked universities collaborate with each other closely, such as interdisciplinary collaborations, scientific research platform collaborations, etc. There are some advantageous disciplines in the top-ranked and medium-ranked universities. The advantageous disciplines are the top 500 disciplines in QS, or top 1% disciplines in ESI. The outstanding and advantageous disciplines collaborate closely to make the university improve effectively.. The collaboration of these disciplines is beneficial to the development of universities.

Thirdly, the top-ranked universities have an obvious advantage in the number of outstanding talents, funds, science and research achievements and government support, etc, comparing to the other universities. These aspects are important for the development of universities focusing on scientific research.

Fourthly, some indicators about development increment, such as the growth ratio of faculty team, teaching resources, research projects, etc, have a slight impact on the performance. Two universities enter different clusters due to a little difference of these increment indicators while most other indicators values are similar. From the perspective of sustainability, these evaluation indicators can reflect the potential of university development.”

2. In section5, I add more content.

“Based on the result of classified performance evaluation, specific policies shall be implemented to universities and colleges in Shanghai effectively as follows.

Firstly, establish funding classification support and management model to allocate funds efficiently. Universities and colleges can serve the country and society more effectively, and enhance social recognition and international influence.

Secondly, establish short-term, medium-term, and long-term classified evaluation system based on achievements of discipline construction, quality of talent cultivation, international influence, national serving strategies, etc. Concentrate resources on constructing outstanding and advantageous disciplines to drive the development of universities and colleges.

Thirdly, create University and College Community to form strong alliance and collaboration among universities and colleges, and to avoid homogenization.

Fourthly, encourage to construct interdisciplinary Integration in universities and colleges. Promote the mutual development of disciplines.

The clustering analysis of classified performance evaluation of higher education is in initial stage. The relevant work is under exploration. It lacks effective reference and experience, and needs more knowledge to support. As far as the algorithm is concerned, latest data of continuous five years are collected from 2017 to 2021. It evaluates the performance of universities within one certain period of time. In the future, a rolling five-year period is considered to evaluate the performance of universities and colleges more dynamically and sustainably. In addition, we will take further research on outlier in clustering algorithm to make the clustering of universities and colleges more scientifically. The complexity of the algorithm is considered be reduced in the future.”

 

Reviewer 2 Report

Paper could benefit from use of other quality models and indicators for learning organizations like EFQM. A thorough desk research to identify the relevant indicators could support or change the choices made in terms of the set adopted. Paper could benefit from the application of the method to schools from other contexts besides Shangai. It is not clear how the indicators were obtained from each of the 62 institutions and that could be helpful to add value to other researchers. The use of decision theory algorithms may be useful if parameters are significant. The classification in Universities focusing on Scientific Research and in Colleges focusing on Technical Skills is not an objective one and many institutions are comprehensive.

Author Response

Response to Reviewer 2 Comments

 

Point 1: It is not clear how the indicators were obtained from each of the 62 institutions and that could be helpful to add value to other researchers.

 

Response 1: Thanks very much to point out the problem. On one hand, I add content about the design principles of the indicator system, and describe aspects about diversification of theindicator system in section 3.1. On the other hand, I add content about how the indicator data are obtained from each 62 institutions in section 4.1.  

1. In section 3.1, I add content about the design principles of the indicator system, and describe aspects about diversification of theindicator system.

“In accordance with the development characteristics and running goals, the classified performance evaluation indicator system contains two sets of evaluation indicators with the principles of dominance, measurability, systematicity, scientificity and practicality, one for universities focusing on scientific research, the other for colleges focusing on technical skills. From the perspective of diversification, non-uniform evaluation standards are adopted for different types of universities and colleges according to their special history and current situations. The evaluation standards not only reflect the general character and the inherent features of higher education, but also reflect the individual characteristics of universities and colleges. Different evaluation standards correspond to different development goal, construction task, policy support, talent cultivation type and education quality. From the perspective of sustainability, it is necessary to pay attention not only to current status, but also to the development and improvement of universities and colleges. The indicators about development increment, such as the growth ratio of faculty team, teaching resources, research projects, etc., in the evaluation period, reflects the notion of sustainability.

The design principles of the classified performance evaluation indicator system follows four points. Firstly, quantitative indicators are combined with qualitative indicators. Secondly, indicators about scale are combined with indicators about quality. Thirdly, indicators about external performance are combined with internal improvement. Fourthly, indicators about total amount are combined with indicators about per capita. Fifthly, indicators about accumulation are combined with indicators about development increment.

The diversification of the classified performance evaluation indicator system is mainly listed in the following aspects.

Firstly, each set of the evaluation indicators has some different tertiary indicators since development characteristics and running goals are different for different types of universities and colleges. For example, the tertiary indicator, Ratio of part-time teachers in enterprises is only available in the type of colleges focusing on technical skills. Comparing to the universities focusing on scientific research, the colleges pay more attention on practical operation.

Secondly, the weight of the same indicator may be different in each set. For example, the secondary indicator, Research Projects, is available in both sets of classified performance evaluation indicator of universities and colleges. The value of the weight is a little larger in the indicator for universities focusing on scientific research than that for the colleges focusing on technical skills according to the calculation result of Topsis model. Relatively speaking, the colleges focusing on technical skills pay more attention on talent cultivation.

Thirdly, each set of the evaluation indicators has some similar primary, secondary and even tertiary indicators, while the real meaning of similar indicators for each set is different. It reflects the characteristics of different types of universities and colleges. These indicators may be inconsistent both in names and content, or the same name with inconsistent content. For example, the tertiary indicator, Number of Outstanding Talents, is available in both sets of classified performance evaluation indicator of universities and colleges. It includes the number of academicians of the Chinese Academy of Sciences (CAS), the Chinese Academy of Engineering (CAE), members of the National Academy of Sciences (NAS), members of the Academia Europaea, etc, as to the content of the indicator for universities focusing on scientific research. However, it includes the number of masters of technical skills, craftsmen of the Nation, directors and members of teaching guidance committee of China Ministry of Education (MOE), etc, as to the content of indicator for colleges focusing on technical skills.

Fourthly, the evaluation criteria of the same indicator may be different in each set. For example, some tertiary indicators in Platform are available in both sets of classified performance evaluation indicator of universities and colleges. Due to the different evaluation criteria, the evaluation value is different even if the data of the indicator is identical.

Fifthly, the classified performance evaluation indicator system shall take into account completeness and simplicity. The evaluation indicators need cover to reflect all aspects of performance of universities and colleges, which leads to complexity. At the same time, too complicated indicator system makes performance evaluation more difficult. Focusing on too much detail may not reflect the performance of universities and colleges scientifically. It is necessary to balance the contradiction of completeness and simplicity appropriately. 

The classified performance evaluation indicator system is dynamic. Universities and colleges, government, experts, etc, are consulted to for their opinion every year. Minor revision and improvement are made accordingly. Major revision may be made every five years.”

2. In section 4.1, I add content about how the indicator data are obtained from each 62 institutions.

“1. Official statistic data from Ministry of Education of China and Shanghai Municipal Education Commission, including Educational Statistics Yearbook of China, Compilation of Basic Statistics of Universities and Colleges directly under the Ministry of Education of China, Analysis Report on Education and Teaching Data of Universities and Colleges directly under the Ministry of Education of China, Basic Level Statistic Report of Higher Education, Annual Statistic Report of Science and Technology of Universities and Colleges, Platform of Data Collection and Management of Talent Cultivation in Higher Education, etc.

2. Officially released information for public use in official website, including National Data Monitor Platform for Higher Education Quality, Platform of Data Collection and Management of Talent Cultivation in Higher Vocational Colleges, etc.
3. Materials submitted by the universities and colleges including Report of Innovation Information Collection of each university and college, etc.
4. Information from third party organization, including Shanghai Soft Science Education Information Consulting Company, Elsevier B.V. Company, Clarivate Company, etc.

All the data are focusing on tertiary indicators. They are subject to vigorous verification.”

 

Point 2: The use of decision theory algorithms may be useful if parameters are significant. The classification in Universities focusing on Scientific Research and in Colleges focusing on Technical Skills is not an objective one and many institutions are comprehensive.

 

Response 2: Thanks very much to point out the problem. Reviewer is correct. More classification of institutions will be designed in the future.

Reviewer 3 Report

The topic of the article is relevant from a practical point of view. The authors suggest applying mathematical methods to university performance evaluation. The authors of the article proposed a systematic clustering algorithm to analyze and evaluate the performance of universities and colleges in the same type. Universities focusing on scientific research and colleges focusing on technical skills were chosen. In the theoretical part of the article, a detailed analysis of higher education quality assessment tools and their mathematical justification is performed.

In the theoretical part, a more detailed analysis of the quality assessment criteria of higher education should be performed. It is recommended to review studies on the performance assessment of higher education, to describe performance assessment criteria.

It is recommended that the authors of the article pay attention to the harmony between the theoretical part and the research. We are concerned about the use of the construct sustainability. It is mentioned in the theoretical part, the bibliography contains several sources. However, this construct is forgotten in the research, summarizing the results of the study. We would recommend that this construct be either dropped or kept in mind in the results section, discussion, and conclusions.

The results of the cluster analysis should be described in more detail in the Classified Performance Evaluation Result Analysis section. It should be explained what outstanding disciplines and advantageous disciplines are. It should be justified how the results of this study complement the theory and practice of performance evaluation in higher education. Limitations of the study should be mentioned.

 

Author Response

Response to Reviewer 1 Comments

 

Point 1: In the theoretical part, a more detailed analysis of the quality assessment criteria of higher education should be performed. It is recommended to review studies on the performance assessment of higher education, to describe performance assessment criteria.

 

Response 1: Thanks very much to point out the problem. I add and adjust following content.

1. In section 2, I add following content to introduce and and analyse performance evaluation in higher education.

“The performance of higher education mainly refers to the effective allocation and use of resources in higher education. It affects the level of scientific research, the quality of talents, and even reflects the potential of technology innovation in a country. The performance evaluation of higher education can promote continuous improvement of internal management in order to improve the efficiency of running universities and colleges [11]. However, the management of universities and colleges is a complicated system. Diversification is a fundamental attribute of higher education. Running level, developing orientation, developing stage, discipline focus, geography location, and social needs are diversified in various universities and colleges. The performance evaluation of higher education is a multi-attribute comprehensive work [12]. The United States and the United Kingdom have conducted performance evaluation of higher education earlier, and have established specialized institutions to carry out research in performance evaluation [13].”

2. In section 2, I add following content to introduce and analyse Topsis model and clustering algorithm

“Topsis model is a common method which is commonly used for multi-objective decision making [51]. It calculates the relative adjacency between evaluated objects and the optimal and inferior solutions of all objects [52]. According to the principle of Topsis model, the ideal evaluation object is the one in which all data of evaluation indicators are closest to the optimal ones and farthest from the inferior ones. it can be used in the field of performance evaluation in higher education advantageously comparing with other evaluation method. Firstly, there is no strict restriction in data of evaluated objects, including the type of data, the sample size, etc. Secondly, the data of evaluated objects with different meaning have various units and valid ranges, which is called dimensions. The Topsis method standardizes dimensions in advance so as to eliminate the impact of various dimensions. Thirdly it is better to do comparative analysis on different evaluated objects. It introduces evaluation indicators of various dimensions for comprehensive evaluation.”

“The systematic clustering algorithm classifies evaluated objects on their own characteristics. The evaluated objects with higher similarity are grouped into one category, which is called cluster, while those with lower similarity are grouped into different cluster [53]. The number of clusters is determined based on similarity of evaluated objects instead of being set in advance. The number of evaluated objects in each cluster is calculated by the systematic clustering algorithm. The systematic clustering algorithm effectively solves the classification problem of multiple indicators, avoiding subjectivity and arbitrariness. ”

 

Point 2: It is recommended that the authors of the article pay attention to the harmony between the theoretical part and the research. We are concerned about the use of the construct sustainability. It is mentioned in the theoretical part, the bibliography contains several sources. However, this construct is forgotten in the research, summarizing the results of the study. We would recommend that this construct be either dropped or kept in mind in the results section, discussion, and conclusions.

 

Response 2: Thanks very much to point out the problem. I add following content.

1. In section 3, I describe the sustainability of the classified performance evaluation indicator system

“In accordance with the development characteristics and running goals, the classified performance evaluation indicator system contains two sets of evaluation indicators with the principles of dominance, measurability, systematicity, scientificity and practicality, one for universities focusing on scientific research, the other for colleges focusing on technical skills. From the perspective of diversification, non-uniform evaluation standards are adopted for different types of universities and colleges according to their special history and current situations. The evaluation standards not only reflect the general character and the inherent features of higher education, but also reflect the individual characteristics of universities and colleges. Different evaluation standards correspond to different development goal, construction task, policy support, talent cultivation type and education quality. From the perspective of sustainability, it is necessary to pay attention not only to current status, but also to the development and improvement of universities and colleges. The indicators about development increment, such as the growth ratio of faculty team, teaching resources, research projects, etc., in the evaluation period, reflects the notion of sustainability.

The design principles of the classified performance evaluation indicator system follows four points. Firstly, quantitative indicators are combined with qualitative indicators. Secondly, indicators about scale are combined with indicators about quality. Thirdly, indicators about external performance are combined with internal improvement. Fourthly, indicators about total amount are combined with indicators about per capita. Fifthly, indicators about accumulation are combined with indicators about development increment.”

 

Point 3: The results of the cluster analysis should be described in more detail in the Classified Performance Evaluation Result Analysis section. It should be explained what outstanding disciplines and advantageous disciplines are. It should be justified how the results of this study complement the theory and practice of performance evaluation in higher education. Limitations of the study should be mentioned.

 

Response 3: Thanks very much to point out the problem.

1. In section 4.4, I add more content about evaluation result analysis.

“Take 23 universities focusing on scientific research as an example. Through clustering algorithm of classified performance evaluation, when the threshold value is set to 0.010, there are 6 universities in level 1 with high performance, account for about 26.09% of the total universities. It includes 4 comprehensive universities (66.67%), 1 university of science and engineering (16.67%) and 1 normal university (16.67%). There are 6 universities in level 2 with medium performance, account for about 26.09% of the total universities. It includes 1 university of science and engineering (16.67%), 2 medical universities (33.33%), 2 universities of liberal arts, economic and management (33.33%), 1 university of arts and sports (16.67%). There are 11 universities in level 3 with relatively low performance, account for about 47.82% of the total universities. It includes 3 comprehensive universities (27.27%), 4 universities of science and engineering (45.45%), 2 universities of liberal arts, economic and management (18.18%), 2 universities of arts and sports (18.18%). Among all the universities, 57.14% of comprehensive universities are in level 1 with high performance, 42.86% of them are in level 3 with relatively low performance. 16.67% of universities of science and engineering are in level 1 with high performance, 16.67% of them are in level 2 with medium performance, and 66.67% of them are in level 3 with relatively low performance. 50.00% of universities of liberal arts, economic and management are in level 2 with medium performance, 50% of them are in level 3 with relatively low performance. 33.33% of universities of arts and sports are in level 2 with medium performance, 66.67% of them are in level 3 with relatively low performance.

Based on the above data, the performance of universities under the Ministry of Education is usually stable with relatively higher performance. There are about half of the universities with relatively low performance, which are expected to improve the performance effectively.

By analyzing the evaluation indicators of universities, it shows as follows.

Firstly, the number of outstanding disciplines in top-ranked universities is significantly higher than other universities. The outstanding disciplines are the top 50 disciplines in QS, or top 1‰ disciplines in ESI. They act as the pioneers in the process of disciplines development in universities. They drive the common development of other relevant disciplines while enhancing themselves. There are breakthroughs in one or two fields to be famous worldwide, and then the overall quality of universities improves slowly. Most outstanding disciplines develop gradually in the development of universities, which are the foundation of universities. Therefore, universities can focus on discipline development and strengthen the advantages.

Secondly, some top-ranked universities and medium-ranked universities collaborate with each other closely, such as interdisciplinary collaborations, scientific research platform collaborations, etc. There are some advantageous disciplines in the top-ranked and medium-ranked universities. The advantageous disciplines are the top 500 disciplines in QS, or top 1% disciplines in ESI. The outstanding and advantageous disciplines collaborate closely to make the university improve effectively.. The collaboration of these disciplines is beneficial to the development of universities.

Thirdly, the top-ranked universities have an obvious advantage in the number of outstanding talents, funds, science and research achievements and government support, etc, comparing to the other universities. These aspects are important for the development of universities focusing on scientific research.

Fourthly, some indicators about development increment, such as the growth ratio of faculty team, teaching resources, research projects, etc, have a slight impact on the performance. Two universities enter different clusters due to a little difference of these increment indicators while most other indicators values are similar. From the perspective of sustainability, these evaluation indicators can reflect the potential of university development.”

2. In section 4.4 I explain what are outstanding disciplines and advantageous disciplines.

“The outstanding disciplines are the top 50 disciplines in QS, or top 1‰ disciplines in ESI. They act as the pioneers in the process of disciplines development in universities. They drive the common development of other relevant disciplines while enhancing themselves.”

“The advantageous disciplines are the top 500 disciplines in QS, or top 1% disciplines in ESI.”

3. In section 5, I add more content about theory and practise, and about the limitations.

“Based on the result of classified performance evaluation, specific policies shall be implemented to universities and colleges in Shanghai effectively as follows.

Firstly, establish funding classification support and management model to allocate funds efficiently. Universities and colleges can serve the country and society more effectively, and enhance social recognition and international influence.

Secondly, establish short-term, medium-term, and long-term classified evaluation system based on achievements of discipline construction, quality of talent cultivation, international influence, national serving strategies, etc. Concentrate resources on constructing outstanding and advantageous disciplines to drive the development of universities and colleges.

Thirdly, create University and College Community to form strong alliance and collaboration among universities and colleges, and to avoid homogenization.

Fourthly, encourage to construct interdisciplinary Integration in universities and colleges. Promote the mutual development of disciplines.

The clustering analysis of classified performance evaluation of higher education is in initial stage. The relevant work is under exploration. It lacks effective reference and experience, and needs more knowledge to support. As far as the algorithm is concerned, latest data of continuous five years are collected from 2017 to 2021. It evaluates the performance of universities within one certain period of time. In the future, a rolling five-year period is considered to evaluate the performance of universities and colleges more dynamically and sustainably. In addition, we will take further research on outlier in clustering algorithm to make the clustering of universities and colleges more scientifically. The complexity of the algorithm is considered be reduced in the future.”

 

 

 

Round 2

Reviewer 1 Report

Thank you for revising your article. The authors did a good effort to improve their work. In present form the paper has acquired more strength and effectiveness. So, it can be published.