Exploring University Performance through Multiple Factor Analysis: A Case Study

Round 1

Reviewer 1 Report

The authors have done an excellent job in developing a university evaluation framework but to make it appealing for the journal's diverse readership, I suggest that they undertake the following revisions.

It is necessary for the authors to start with an argument why we need a university performance evaluation framework. I suppose that universities in contexts other than Columbia need it, too. It may be also the case that there are other frameworks available. Then the authors need to relate the Columbian context to other contexts and present an argument why we need a new framework or why the framework developed in the Columbian context is an important knowledge contribution to higher education.  I believe that the literature review needs to include an overview of what other frameworks available and what problems they have to justify the need to develop this new framework. And how does this framework relate to the journal's title? Any reference to sustainability will be quite important so that the manuscript can fit this journal.  Again, I would like to see a robust argument about the framework and the study's contributions to higher education performance assessment research. It is not that important for readers to know which universities in Columbia have better performance than others. It is important to know that the newly developed framework works better or yield better insights into university performance. It is also important to argue that the newly developed framework is a much more sustainable one. I hope. 

I think that the authors need to attend these issues through revsion before the manuscript can be accepted for publication. 

Author Response

Answer to Referee

 

First of all, we want to thank the reviewer for their useful comments and suggestions. We think all of them have noticeably improved the original draft of the paper. In the following we explain point by point how these suggestions have been incorporated in the revised version of the paper.

 

------------------------------------------------------------------------

Referee #1 (Comments for the Author):

 

Referee 1): It is necessary for the authors to start with an argument why we need a university performance evaluation framework. I suppose that universities in contexts other than Columbia need it, too. It may be also the case that there are other frameworks available. Then the authors need to relate the Columbian context to other contexts and present an argument why we need a new framework or why the framework developed in the Columbian context is an important knowledge contribution to higher education.  I believe that the literature review needs to include an overview of what other frameworks available and what problems they have to justify the need to develop this new framework. And how does this framework relate to the journal's title? Any reference to sustainability will be quite important so that the manuscript can fit this journal.  Again, I would like to see a robust argument about the framework and the study's contributions to higher education performance assessment research. It is not that important for readers to know which universities in Columbia have better performance than others. It is important to know that the newly developed framework works better or yield better insights into university performance. It is also important to argue that the newly developed framework is a much more sustainable one. I hope. 

 

Authors: We agree with the referee suggestion. Following the referee suggestion, we have we have updated the text as follow in the introduction and in de discussion:

 

“The Belgrade Charter (United Nations Educational, Scientific and Cultural Organization-United Nations Environment Programme) [1], is one of the first documents in which the term Environmental Education (ED) began to take concrete form. Later, the Tbilisi Declaration [2], built on the Belgrade Charter, suggests that:

 

The basic aim of environmental education is to help individuals and communities understand the complex nature of the natural and the built environments resulting from the interaction of their biological, physical, social, economic, and cultural aspects, and acquire the knowledge, values, attitudes, and practical skills to participate in a responsible and effective way in anticipating and solving environmental problems, and in the management of the quality of the Environment”. (p. 92).

In 2002, the United Nations (UN) declared de period 2005-2014, the decade for Education for Sustainable Development (SD) [3].  In addition, the World Commission on Environment and Development (WCED), who aims to address the nations of the world towards the goal of sustainable development, published in 2014 the Brundtland Report. The concept of sustainability, developed in this report became an important concept in the vocabulary of politicians, practitioners, and planners [1, 4] and highlights the intersections of the three critical aspects, economic, social and environmental of corporate actions. Nowadays, the principles of Sustainable Development (SD) are becoming increasingly important since the development of Agenda 2030 for Sustainable Development, which includes 17 Sustainable Development Goals (SDGs), which organizations must progressively incorporate. In this study, we will highlight the importance of Sustainable Development Goal 4 (SDG 4), which is the educational goal and aims to "ensure inclusive and equitable quality education and promote lifelong learning opportunities for all", and governments must face the challenge of establishment the strategies and actions plans to develop a high quality Environmental Education for Sustainability (EEfS). The UN declared education for sustainability (EFS) the key in solving many educational, environmental and social issues affecting all nations [1].  Education for sustainability can help ensure the conservation of nature and the future existence of humanity [5-8].

Universities are aware of the relevance of the concept of sustainable development and given the growing global interest on the university’s role towards promoting sustainability, an increasing number of universities are committing themselves to sustainability and are acting as agents in promoting these principles within society.

In recent years, several definitions of sustainable higher education institutions have emerged [9]. In addition, different studies has been aimed at assessing students', professors’ and rectors' perceptions of the factors that contribute to sustainable development in higher education [10,11]. Furthermore, several studies related to the implementation of sustainability in the higher education context, as Shriberg and Balas-Ferrer [12,13] stated that the main factors influencing the implementation of sustainability practices at universities are the following: public or private institutions, size, university leadership on sustainability and political orientation.

Universities could play their role effectively if they could identify the main characteristics or indicators to improve. In this context, the establishment of strategies and plans for the improvement of any system should address as a first instance the knowledge of the current state of the same, which is achieved through the formulation, study and analysis of performance indicators of the dimensions considered important for the achievement of objectives. This is also totally true in the Colombian State University System (SUE). The public Colombian universities are very different in size and financial resources. Traditionally the government has assigned different budget regarding the location of the University. In this context, improving the performance of Higher Education Institutions (HEIs) is one of the challenges facing the education system in Colombia.

This paper seeks to create a characterisation of the Colombian Universities by studying the outcome indicators established in the Progress Index of Higher Education (IPES) developed by the Ministry of National Education of Colombia through Multiple Factor Analysis (MFA) as tool and proposes the creation of a Synthetic Performance index based on Multiple Factorial Analysis (MFA)”.

Reviewer 2 Report

The idea presented in this manuscript is interesting. It shows how multiple factor analysis (MFA) is used to compare universities’ performance in regards to their strengths/weaknesses. The comparison can contribute to better decision-making in the administration of the university system.

However, the connection between this manuscript with the scope of this special issue is weak. It is difficult to find the sustainability aspects this special issue is bringing within the manuscript. The authors are suggested to submit this manuscript to the other journal whose aims and scopes are more suitable, such as Education Sciences (https://www.mdpi.com/journal/education).

Nevertheless, the manuscript is worth for publication within this special issue, if the authors could improve and revise some of the following;

(1)

The authors are suggested to highlight the sustainability issues as shown in the special issue information. Please read the information carefully, and try to make a strong connection between the information with the manuscript. For example, by connecting the goal number 4 of the sustainable development goals “Ensure inclusive and equitable education and promote lifelong learning opportunities for all” with the manuscript. The issues are recommended to be highlighted in the introduction section, and then followed by the discussion section.

(2)

The authors are also suggested to include in the Introduction section some references on previous studies of how a university performance has been analyzed so far, what the knowledge/research gap(s) obtained from the previous studies, and what this current research is offering in order to close the gap(s).

(3)

Given the broad readers’ background of this journal, the authors are suggested to provide more portion of references written in English in the list of references. It is difficult for international readers to follow the ideas presented within the manuscript without digging further into referenced papers. In addition, the authors are also suggested to provide the digital object identifier (DOI) of each reference in order to facilitate the readers to read the cited references.

(4)

Regarding the citation on the references, the authors are suggested to utilize reference manager software in order to improve the accuracy of the citation style as well as the list of references. For example; in the line 168, the citation style is not common. Instead, this style [11-22] is more common in a journal paper. Hence, the use of reference manager software is suggested. In the line 260-261, 306-308, and 412 the style is also not common. Please check the author’s guidelines in the preparation of manuscript.

(5)

The authors need to check and recheck the manuscript regarding technical aspects of writing. The comma in expressing thousands and period in expressing decimal should be consistent, for example in line 49 and 52.

(6)

The authors need to check and recheck the manuscript regarding the typo within the manuscript, for example; in the line 328 and the caption of Table 5.

(7)

The authors need to carefully check and recheck the manuscript regarding technical aspect of writing and presentation aspect of the manuscript. Any tables and figures produced by the authors are not necessarily written as the “Source: Author’s own”. Please check the author’s guidelines in the preparation of manuscript.

(8)

Although not a crucial part, the authors need to check and recheck in the Table 1 why the Resources RE is not consistent with IR1, IR2, and IR3, while the other dimensions such as Quality IC is consistent with IC1, IC2, IC3, and so on. In addition, in the Table 9, resources dimension is abbreviated by R1, R2, and R3 instead of IR1, IR2, and IR3.

(9)

The authors need to check and recheck the manuscript regarding the abbreviation presented in the manuscript. An abbreviation should be mentioned with their full or expanded meaning prior to its mentioning afterwards. For example; TCE in the Table 1 and Dim in the Table 4 are mentioned without prior explanation. At least, mention what the abbreviations stand for.

(10)

The authors need to check and recheck the manuscript regarding the presence of non-English terms within this English manuscript, especially in the figures.

(11)

The authors need to check and recheck the manuscript regarding the caption of figures and tables. For example; both Figure 7 and Figure 8 have the same caption but different figures. In addition, Table 12 is missing although it has been mentioned in the line 619.

Author Response

Answer to Referee

 

First of all, we want to thank the reviewer for their useful comments and suggestions. We think all of them have noticeably improved the original draft of the paper. In the following we explain point by point how these suggestions have been incorporated in the revised version of the paper.

---------------------------------------------------------------

Referee #2 (Comments for the Author):

The connection between this manuscript with the scope of this special issue is weak. It is difficult to find the sustainability aspects this special issue is bringing within the manuscript. Nevertheless, the manuscript is worth for publication within this special issue, if the authors could improve and revise some of the following;

Referee 1): The authors are suggested to highlight the sustainability issues as shown in the special issue information. Please read the information carefully, and try to make a strong connection between the information with the manuscript. For example, by connecting the goal number 4 of the sustainable development goals “Ensure inclusive and equitable education and promote lifelong learning opportunities for all” with the manuscript. The issues are recommended to be highlighted in the introduction section, and then followed by the discussion section.

Authors: We also agree with the suggestion made by the referee 2. Following the referees suggestion, we have we have updated the text of Introduction as follow:

 

“The Belgrade Charter (United Nations Educational, Scientific and Cultural Organization-United Nations Environment Programme) [1], is one of the first documents in which the term Environmental Education (ED) began to take concrete form. Later, the Tbilisi Declaration [2], built on the Belgrade Charter, suggests that:

 

The basic aim of environmental education is to help individuals and communities understand the complex nature of the natural and the built environments resulting from the interaction of their biological, physical, social, economic, and cultural aspects, and acquire the knowledge, values, attitudes, and practical skills to participate in a responsible and effective way in anticipating and solving environmental problems, and in the management of the quality of the Environment”. (p. 92).

In 2002, the United Nations (UN) declared de period 2005-2014, the decade for Education for Sustainable Development (SD) [3].  In addition, the World Commission on Environment and Development (WCED), who aims to address the nations of the world towards the goal of sustainable development, published in 2014 the Brundtland Report. The concept of sustainability, developed in this report became an important concept in the vocabulary of politicians, practitioners, and planners [1, 4] and highlights the intersections of the three critical aspects, economic, social and environmental of corporate actions. Nowadays, the principles of Sustainable Development (SD) are becoming increasingly important since the development of Agenda 2030 for Sustainable Development, which includes 17 Sustainable Development Goals (SDGs), which organizations must progressively incorporate. In this study, we will highlight the importance of Sustainable Development Goal 4 (SDG 4), which is the educational goal and aims to "ensure inclusive and equitable quality education and promote lifelong learning opportunities for all", and governments must face the challenge of establishment the strategies and actions plans to develop a high quality Environmental Education for Sustainability (EEfS). The UN declared education for sustainability (EFS) the key in solving many educational, environmental and social issues affecting all nations [1].  Education for sustainability can help ensure the conservation of nature and the future existence of humanity [5-8].

Universities are aware of the relevance of the concept of sustainable development and given the growing global interest on the university’s role towards promoting sustainability, an increasing number of universities are committing themselves to sustainability and are acting as agents in promoting these principles within society.

In recent years, several definitions of sustainable higher education institutions have emerged [9]. In addition, different studies has been aimed at assessing students', professors’ and rectors' perceptions of the factors that contribute to sustainable development in higher education [10,11]. Furthermore, several studies related to the implementation of sustainability in the higher education context, as Shriberg and Balas-Ferrer [12,13] stated that the main factors influencing the implementation of sustainability practices at universities are the following: public or private institutions, size, university leadership on sustainability and political orientation.

Universities could play their role effectively if they could identify the main characteristics or indicators to improve. In this context, the establishment of strategies and plans for the improvement of any system should address as a first instance the knowledge of the current state of the same, which is achieved through the formulation, study and analysis of performance indicators of the dimensions considered important for the achievement of objectives. This is also totally true in the Colombian State University System (SUE). The public Colombian universities are very different in size and financial resources. Traditionally the government has assigned different budget regarding the location of the University. In this context, improving the performance of Higher Education Institutions (HEIs) is one of the challenges facing the education system in Colombia.

This paper seeks to create a characterisation of the Colombian Universities by studying the outcome indicators established in the Progress Index of Higher Education (IPES) developed by the Ministry of National Education of Colombia through Multiple Factor Analysis (MFA) as tool and proposes the creation of a Synthetic Performance index based on Multiple Factorial Analysis (MFA)”.

 

Referee 2): The authors are also suggested to include in the Introduction section some references on previous studies of how a university performance has been analyzed so far, what the knowledge/research gap(s) obtained from the previous studies, and what this current research is offering in order to close the gap(s).

Referee 3): Given the broad readers’ background of this journal, the authors are suggested to provide more portion of references written in English in the list of references. It is difficult for international readers to follow the ideas presented within the manuscript without digging further into referenced papers. In addition, the authors are also suggested to provide the digital object identifier (DOI) of each reference in order to facilitate the readers to read the cited references.

 

Authors: Following the referee suggestions 2) and 3) we have included thirteen references:

 

Barry, J. The Belgrade Charter: A global framework for environmental education. UNESCO-UNEP Environmental Education Newsletter. 1976, 1, 1-9. UNESCO-UNEP. The Tbilisi Declaration: Final report intergovernmental conference on environmental education. Organized by UNESCO in cooperation with UNEP, Tbilisi, USSR, 14-26 October 1977, Paris, France: UNESCO ED/MD/49. Venkataraman, B. Education for sustainable development. Environment: Science and Policy for Sustainable Development. 2009, 51(2), 8-10. DOI: https://doi.org/10.3200/ENVT.51.2.08-10 Burton, I. Our common future: The world commission on environment and development. Environment: Science and Policy for Sustainable Development. 1987, 29(5), 25–29. Didham, R. J. and Ofei-Manu, P. Advancing policy to achieve quality education for sustainable development. In Leicht, A., Heiss, J., Byun and W.J. (Eds.).. Issues and trends in Education for Sustainable Development. Paris: United Nations Educational, Scientific and Cultural Organization. 2018. Jickling, B. & Wals, A. E. J. Globalization and environmental education: looking beyond sustainable development. Journal of Curriculum Studies. 2008, 40, 1-21. DOI: https://doi.org/10.1080/00220270701684667 Loughland, T., Reid, A., Walker, K. and Petocz, O. Factors influencing young people conceptions of environment. Environmental Education Research. 2003, 9, 3-20. DOI: https://doi.org/10.1080/13504620303471 Stevenson, R. B. Tensions and transitions in policy discourse: recontextualizing a decontextualized EE/ESD debate. Environmental Educational Research. 2006, 12, 277-290. DOI: https://doi.org/10.1080/13504620600799026 Madeira, A.C., Carravilla, M.A., Oliveira, J.F. and Costa, C.A.V. A methodology for sustainability evaluation and reporting in higher education institutions. Higher Education Policy. 2011, 24 (4), 459-479. Nejati M. & Nejati M. Assessment of sustainable university factors from the perspective of university students. Journal of Cleaner Production. 2013, 48, 101–107 Yuan, X. and Zuo, J. A critical assessment of the Higher Education for Sustainable Development from students' perspectives-a Chinese study. Journal of Cleaner Production. 2013, 48, 108-115. Shriberg, M.P. Sustainability in   US   higher   education:   Organizational   factors influencing   campus   environmental   performance   and   PhD dissertation, University of Michigan. Ann Arbor, Michigan, 2002. Ferrer-Balas, D., Lozano, R., Huisingh, D., Buckland, H., Ysern, P. and Zilahy, G. Going beyond the rhetoric: system-wide changes in universities for sustainable societies. Journal of Cleaner Production. 2010, 18(7), 607–610. DOI: http://dx.doi.org/10.1016/j.jclepro.2009.12.009.

 

Referee 4): Regarding the citation on the references, the authors are suggested to utilize reference manager software in order to improve the accuracy of the citation style as well as the list of references. For example; in the line 168, the citation style is not common. Instead, this style [11-22] is more common in a journal paper. Hence, the use of reference manager software is suggested. In the line 260-261, 306-308, and 412 the style is also not common. Please check the author’s guidelines in the preparation of manuscript.

Authors: Authors: Following the referee suggestions, we make the suggested modifications and review and adjust the all text.

 

Corrections

Line 168: correction, Line 203

Line 260-261: correction, Line 272, 273

Line 306-308: correction, Line 318

Line 412: correction, Line 419

 

Referee 5): The authors need to check and recheck the manuscript regarding technical aspects of writing. The comma in expressing thousands and period in expressing decimal should be consistent, for example in line 49 and 52.

Authors: Following the referee suggestion, we check the manuscript regarding technical aspects of writing, and we corrections

Referee 6): The authors need to check and recheck the manuscript regarding the typo within the manuscript, for example; in the line 328 and the caption of Table 5.

Authors: Following the referee suggestion, we make corrections the typos within the manuscript

Referee 7): The authors need to carefully check and recheck the manuscript regarding technical aspect of writing and presentation aspect of the manuscript. Any tables and figures produced by the authors are not necessarily written as the “Source: Author’s own”. Please check the author’s guidelines in the preparation of manuscript.

Authors: Following the referee suggestion, it was deleted on the tables and figures “Source: Author’s own”.

Referee 8):  Although not a crucial part, the authors need to check and recheck in the Table 1 why the Resources RE is not consistent with IR1, IR2, and IR3, while the other dimensions such as Quality IC is consistent with IC1, IC2, IC3, and so on. In addition, in the Table 9, resources dimension is abbreviated by R1, R2, and R3 instead of IR1, IR2, and IR3.

Authors: Following the referee suggestion, we have included the modifications in table 1 and table 9, and the text was adjusted.

 

Referee 9): The authors need to check and recheck the manuscript regarding the abbreviation presented in the manuscript. An abbreviation should be mentioned with their full or expanded meaning prior to its mentioning afterwards. For example; TCE in the Table 1 and Dim in the Table 4 are mentioned without prior explanation. At least, mention what the abbreviations stand for.

Authors: Following the referee suggestion, the corresponding abbreviations were added and the names in tables 1, 2, 3 and 4 were modified and the text was adjusted. (Pages 8, 9 and 10)

Referee 10): The authors need to check and recheck the manuscript regarding the presence of non-English terms within this English manuscript, especially in the figures.

Authors: Following the referee suggestion. The model MFA was rerun by writing the names of the variables in English.

 

Referee 11): The authors need to check and recheck the manuscript regarding the caption of figures and tables. For example; both Figure 7 and Figure 8 have the same caption but different figures. In addition, Table 12 is missing although it has been mentioned in the line 619.

Authors: Following the referee suggestion, we modifications the name of figure 8 (page 15). The table 12 is in the page 20.

 

Round 2

Reviewer 2 Report

The authors have improved of the manuscript following the comments the reviewers had given. Therefore, I agree to accept this manuscript in the present form for publication in the journal.