Next Article in Journal
The Causal Relationship between Urbanization, Economic Growth and Water Use Change in Provincial China
Previous Article in Journal
How Firms Can Get Ideas from Users for Sustainable Business Innovation
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Challenging the Sustainability of an Education System of Evaluation and Labor Market Outcomes

College of Economics, Sungkyunkwan University, 25-2, Sungkyunkwan-ro, Jongno-gu, Seoul 03063, Korea
*
Author to whom correspondence should be addressed.
Sustainability 2015, 7(12), 16060-16075; https://doi.org/10.3390/su71215808
Submission received: 1 November 2015 / Revised: 26 November 2015 / Accepted: 30 November 2015 / Published: 3 December 2015

Abstract

:
Students’ inattention to the importance of teaching evaluations may undermine the sustainability of the education evaluation system. This study analyzed the effects of the personality variable reflected by monotonic response patterns, which is a typical example of student indifference, on the employability of graduates using Career-SET (student evaluations of teaching) matched data of college graduates from 2008–2012. The results from various estimation models consistently indicated that graduates with a higher ratio of insincere responses in student evaluations of teaching are less likely to be employed, or are hired for lower prestige jobs than other comparison groups. This means that unlike the current practice in which firms rely simply on specifications to hire employees, applicants’ invisible characteristics, such as personality, can also be screened by job interviewers.

1. Introduction

The very purpose of student evaluations of teaching (SET) is to provide colleges with basic information to improve the quality of education [1] as well as for the instructor to improve in their teaching discipline. Hence, it can be argued that students’ indifference to teaching evaluations results in a lack of fair assessments of the instructors’ productivity, which perhaps undermines the sustainability of the educational evaluation system (When a semester is completed, every student has the opportunity to evaluate whether the original syllabus in all classes was correctly completed, if the instructor was suitably prepared for classes, or if any improvements could be made. This evaluation can be reported as part of the instructor’s achievement evaluations, and could be useful for generating information for registration in the next semester). Students’ thinking patterns, behavior, and lifestyle can be negatively affected if the students adopt a careless or indifferent attitude toward the evaluation. For example, monotonic response (MR) patterns, which are typical examples of student indifference to teaching evaluation, are shown to be the result of an uncooperative attitude (noncooperation), surpassing the cost-benefit level of the individual, for the purpose of presenting an objective evaluation to other students who will take the class or the instructor; therefore, this incurs costs for organizations while the students reduce their time costs. This behavior by students may be penalized in the labor market by lower employability. If students recognize this penalty in advance, then their incentive to be indifferent is weakened, and hence the soundness of the SET system may be maintained.
There has always been a degree of asymmetric information between employers and potential employees that is lost in job marketing signaling because certain hidden characteristics of college graduates are not included in their “specs” (In South Korea, “specs” is an abbreviation for “specifications”, and is used as a word that sums up education, including GPA, language scores, and qualifications, among those seeking employment) [2]. Although specs could be representative of the potential job ability of employees, it has many limitations as a proxy variable for the important factors that influence actual job achievements when performing at work, such as personal characteristics of personality, interests, and human nature, which cannot be converted into values and records. As a result, firms have been modifying job interviews to screen for their applicants’ hidden characteristics through the utilization of various personality assessment models (e.g., the Big Five personality traits) for managing employees (The Big Five personality traits refer to five broad dimensions used to describe human personality. The five factors have been labelled: extraversion, neuroticism, conscientiousness, agreeableness, and openness to experience, which is a summary of the common features shown by the most socially successful people in specific working environments. The big five personality traits incorporate results from various groups, including different cultures, and are explained by various tools and theoretical frames [3,4,5,6]). However, there has been criticism regarding whether the screening process is properly performed [7].
Previous studies have analyzed the effects of curriculum and non-curriculum activities on employment. Through analysis of the effects on salary after university graduates gain employment in the USA, Arcidiacono [8] explained that SAT scores and GPA are the main factors that determine future achievements in the labor market, while Lassibille et al. [9] reported that students in college who attended irregular academic programs for employment had reduced periods of unemployment. However, limited research has been conducted on how the use of employment prestige is affected by students’ unseen personality traits, revealed through campus life habits, such as behavioral patterns in teaching evaluations, because of the difficulty of collecting data on those personality factors.
Even with a great deal of sources, there are few variables that account for personal behaviors, personalities, and psychological characteristics. Accordingly, studies dealing with personal tendencies and psychology are mostly dependent on surveys that measure potential factors of personal traits; however, behavior in real environments can provide representative information about personality [10]. Mischel [11] mentioned that personal traits are manifested completely under specific circumstances, and that these traits can be identified through personal behaviors in specific situations during college, such as in responses on student evaluations of teaching (SET). The validity of the items in the SET is considered to be reasonably good, as the items have been verified by experts for several years and similar items are widely used in Korean universities. Also, a number of studies have analyzed and confirmed the validity of SET. SET constitutes measurements of teaching effectiveness [12], and illustrates the relationships between ratings of teaching effectiveness and other variables related to different dimensions or factors of students’ evaluations [13,14]. The studies on SET explore the validity and reliability of student opinions and offer suggestions aimed at improving instruction.
Thus, this study uses the students’ responses to educational assessments as a proxy variable for personality. To encourage students to participate in SET, most universities allow only those students who have completed the SET to check their grades and raise objections during grade announcements. However, some students choose the same answers to every question in the SET. This pattern of identical answers to all questions on the SET is called the monotonic response (MR). This study substantially provides the link between MR and personality traits, and how a student’s personality, which is reflected in their behavioral patterns in SET, influences employability.
This paper is organized as follows. The next section describes the theoretical framework for this analysis. Section 3 contains a literature review of MR and the five-factor model of personality and employability. Section 4 provides a summary of the data used herein, and the empirical models. The results are reported in Section 5, followed by the conclusion in Section 6.

2. Theoretical Framework

What reasons or incentives motivate students to engage in MR? First, those that select the same number for all of the questions in SET, rather than carefully reading each question before responding, can reduce the time cost required to check their grades, which is beneficial. Especially, if students have a high time cost opportunity, they could answer all items equally on the evaluation and save time in order to invest it in other valuable ventures. This behavior may correspond with what is considered a kind of cost-benefit rationality in economics. In a similar vein, Choi and Kim [15] explained that selecting the same numbers for each response does not necessarily indicate insincerity by the student.
MR could also be due to insincere responses made without understanding the purpose of individual questions, and without reading the questions carefully. This could have a negative effect when the focus is on organizations and contributions to teamwork. As a result, it can be considered a negative factor for demonstrating productivity in the labor market. Figure 1 shows the theoretical framework for the impact of students’ personality in SET and their specs on employability. The image relates the personal traits derived from MR to the big five personality traits (Big Five), which are well known as the most comprehensive and confidential analysis frame for recruits and HR management [16].
The Big Five personality model in Figure 1 shows five items for recruiting preferable candidates, and corresponds to the traditional norms of HR management [17,18,19,20]. Costa and McCrae [18] and John and Srivastava [21] presented evidence that most of the variables used to assess personality in the field of personality psychology can be mapped to one or more of the dimensions of the Big Five. In this study, we hypothesized that three personality constructs, “conscientiousness”, “agreeableness”, and “neuroticism”, would be related to MR.
Figure 1. Theoretical framework for the impact of students’ personality in SET (student evaluations of teaching) and specs on employability.
Figure 1. Theoretical framework for the impact of students’ personality in SET (student evaluations of teaching) and specs on employability.
Sustainability 07 15808 g001
The Big Five describes conscientiousness as a sense of responsibility, orderliness, dutifulness, self-discipline, and deliberation. In this respect, MR could be an indicator of a sense of responsibility to faithfully respond as students, even though responding to the student evaluation is not mandatory. Also, MR measures whether individuals themselves participate as members of an organization by providing the right information to other students or the instructor. SET aims to improve the quality of education through an objective assessment of students’ classes or instructors. Conscientiousness, which has most consistently emerged as the Big Five construct related to performance across jobs [22,23], is related to deliberation by an individual. The Big Five suggests that a sincere person who performs given tasks responsibly due to “conscientiousness” also has the property of deliberation and dutifulness.
Altruism and straightforwardness are sub-properties of “agreeableness” [18]. Agreeable persons are also cooperative (caring of others), representing the agreeableness factor in the Big Five. MR shows a tendency toward more non-cooperative and selfish behavior, which falls agreeableness in the Big Five. If MR is considered to be a disabling act to reduce the cost opportunity for time in the respect of evaluation systems to improve the education quality in college, then this indicates the opposite of altruism, meaning noncooperation with organization regulations. Also, by reflecting the student’s desire to view their grades after providing quick evaluations, it signals the insincerity property which goes against deliberation. The other factors from the Big Five, such as dutifulness or straightforwardness, can also be found indirectly through the linkage of MR.
Neuroticism in the Big Five is also related to MR. According to Costa and McCrae [18], the neuroticism dimension assesses adjustment or emotional stability versus maladjustment or neuroticism. Impatience is explained as a characteristic of neuroticism in addition to impulsiveness and anxiety [24]. The act of providing a quick evaluation by answering all items equally could be described as impatience. A few studies have found negative associations between impatience and labor market outcomes [25,26]. This relation has been most often interpreted in terms of the debilitating effects of anxiety [27], and impatient individuals are thought to experience anxiety, impairing their performance.
Thus, MR is hypothesized to be a good measure of graduates’ personalities. In the analysis, we also consider spec-related variables, which reflect the graduates’ academic abilities or human capital such as total GPA, IT-related qualifications, foreign language skills, study abroad experience, etc.
It is still under theoretical dispute whether MR can have a positive or negative effect on the employability of students in the labor market, but this study attempted to measure which influences are stronger through substantial analysis. If either the positive or negative elements have a significant influence, then it will provide important evidence that recruiters should not be dependent solely on specs like GPA [3]. Accordingly, by applying various models for the substantial analysis, such as Logit, Multinomial Logit, and Tobit, this study sought to determine whether MR frequency during the college years may influence employability and employment prestige in a specific period.

3. Literature Review

3.1. Monotonic Response (MR) Pattern in SET

The SET system has been adopted in a number of universities. The results of SET could provide a chance to review professorial pedagogy at a college by indicating the attitudes of students, who are education consumers, regarding the lectures of instructors, who are the suppliers. The results of SET are also included as part of an important criteria for institutional evaluation to make the system more effective in terms of financial programs by the government. With emphasis on the importance of this system, a number of universities push students to evaluate their courses [28,29].
Due to this encouragement to participate in SET and the implementation of a system for Internet evaluation, the response rates each semester are very high, around 90%, at Sungkyunkwan University (Seoul, Korea), which is the research target for this study. However, even with this high rate, there is a problem with the anonymous nature of the system. Under the present system, personal information from SET is not revealed to the pertinent instructors, who are provided only with the results of the comprehensive evaluation and the suggestions by the students. This system is disadvantageous because it allows students to complete the evaluation subjectively. Cho et al. [30] reported that the teaching quality may have to be discounted because of subjective behavior by students such as expectations about their course grade. Especially, in cases where completion of the evaluation is mandatory, the phenomenon of choosing the same number for all questions, rather than responding after carefully reading the evaluation items in detail, is increasingly observed. Because of this issue, studies on MR have recently gained attention in South Korea. Choi and Kim [15] analyzed the influences behind monotonic response patterns (Choi and Kim [15] defined students’ monotonic response patterns as the selection of identical numerical ratings to all of the questions on SET), targeting students who took cultural studies at private universities in South Korea in 2011. Multilevel modeling revealed that the monotonic responses of students could be explained by several properties on a student-level, including age, sex, major, GPA, and school years, rather than course feature variables. The MR rate was greater for students from the college of engineering compared to those from the college of humanities. Because almost all of the courses in majors at the college of engineering are engineering certified courses and are mandatory, a higher probability of SET also occurred in these courses (Based on a program with the purpose of developing talented and excellent engineers for industries, engineering certified courses refer to courses with education certification from ABEEK (Accreditation Board for Engineering Education of Korea)). The above study mentioned that marking the same numbered responses does not necessarily indicate insincerity by the student, but it occasionally reflected insincere or consistent response patterns [29,31,32,33]. On surveys about the SET responses of students, 34% of students responded that they did not even read the questions on the evaluation, while over 50% responded that they checked the same number consistently [34]. Therefore, an examination of the SET pattern provides an opportunity to analyze personal traits, like insincerity, during the college years.

3.2. Five Factor Model of Personality and Employability

Many studies have adopted personality systems that match the Big Five personality traits. These studies showed that “conscientiousness”, including achievement and reliability, is generally a productive factor for all types of occupations and job-related criteria [19,22]. Through these studies, responsible and confident people were also found to be sustainable and have a high motive for achievement. That is, conscientiousness people generally have higher achievement than others. In addition, extraversion is a significant predictor for opportunities that require social interaction, like management and sales, whereas the other three parameters are associated with several groups of jobs and categories, but are reported to have insignificant magnitudes [22].
Barrick and Mount [17] analyzed the relationship between personalities and working achievements as evaluated by managers, and reported that the Big Five personality traits are valid for interpreting personality. Based on the responses of 146 managers, conscientiousness and extraversion had considerable correlations to working achievement, and were significant predictors, particularly at highly self-controlled jobs. “Agreeableness” was also appropriate to jobs with high autonomy, but had a negative correlation. Also, Mueller and Plug [20] used the five-factor model to analyze whether personality influences salary by studying graduates from Wisconsin high schools in 1957. The participants were re-interviewed in 1964, 1975, and 1992 during the Wisconsin Longitudinal Study. For men, there were “substantial earning advantages” associated with antagonism, emotional stability, and openness to experience, whereas sincere and open-minded females had higher salaries. In short, agreeableness had the greatest influence on earnings.
In addition, a stable personality is an important factor in labor market outcomes [22,25,26,35] Barrick and Mount [22] claimed that “Neuroticism” was a good predictive indicator of job performance, and Nyhus and Pons [35] showed a close relationship between wages and “Neuroticism”. “Neuroticism” is associated with anxiety, impulsiveness, impatience, etc. [18,24], and Golsteyn et al. [26] documented the relationship between time preference during adolescence and long-run social and economic outcomes. The results indicated that impatient children earn less lifetime income and are more often unemployed. Cadena and Keys [25], using NLSY data to investigate the effects of impatience on human capital formation, found that impatient people show dynamically inconsistent behaviors.
As summarized above, many studies have analyzed the impact of employees’ personalities on wages and job performance after employment. Furthermore, the big five personality traits model is the most common personality assessment model related to job performance. The present study was undertaken to explain the personal traits derived from MR using the Big Five personality traits that affect HR management. While controlling for other spec variables, the effect of personality variables on employability was examined.

4. Empirical Analysis

4.1. Career-SET Matched Data

We used the combined data from student evaluations of teaching (SET) (2004 Spring–2012 Spring) and the Career data for graduates (2008–2012) to determine the effects of an individual’s personality on employability as identified by MR from individuals that complete the SET.
The information on employment status at the time of this study, performed between April and June, are included in the career data along with information about the targets, who were August graduates in the previous year and February graduates in the same year. In this paper, the corporate information disclosed by companies in the South Korean Data Analysis, Retrieval and Transfer System (DART) [36] of the Financial Supervisory Service was used to determine the employment prestige, incorporating information about sales, net profits, and average annual wages of the firms (DART is a comprehensive corporate disclosure system in South Korea that allows anyone to view all disclosed data filed by the listed companies through the Internet via the Financial Supervisory Service. The disclosed documents, such as annual reports, contain information about financial matters and the status of employees). In general, the rankings of companies are determined by a variety of criteria such as total assets or sales. The employment prestige in this study was determined by company rankings in accordance with sales, net profits, and average annual wages. Average annual wages can represent the expected wages for applicants, and were used to measure the employment prestige. Thus, this study excluded employees of public institutes, administrative institutes, and schools, and targeted 9083 graduates who had the opportunity to enter firms to analyze the employment prestige. In addition, specs such as admission, major, final grades, and language skills, or computer related certifications were included, and were used as a measure of educational achievement by using the career data.
Data regarding student evaluations of teaching (SET) included individual students who entered Sungkyunkwan University, which is one of the major universities in South Korea, after March 2004. The SET consists of nine rated questions and 1 free-answer question, with measures coded into five different levels: (1) excellent; (2) very good; (3) fair; (4) poor; and (5) very poor. The SET is normally conducted on-line after completion of the final examination. Most universities in South Korea have a computer-based evaluation system and the SET is conducted on-line in order to increase the students’ course evaluation response rates. Students can assess overall aspects such as do-ability, course relevance, clarity of course objectives, course materials, and difficulty level of the examination, encompassing the adequacy of the course after the final examination. If SET is conducted after the final grade is reported, students with good grades tend to high ratings, and the teaching quality may have to be discounted [30]. Students have to answer a set of teacher evaluation questionnaires if they want to check their grades for a course prior to receiving a formal transcript via mail. Hence, this provides an incentive to respond insincerely on the SET, for the purpose of checking their grades quickly. In this paper, the frequencies and ratios of MR among all responses to courses taken were calculated and used as a proxy variable to represent a personality variable. Table 1 is the ratio of MR in regard to the SET rating. It shows that most MR were marked 1, which means excellent. In fact, monotonic response patterns are clearly associated with higher ratings [15]. Even if students provide above average ratings for fear that instructors may recognize their evaluations, such responses also reflects negative attitudes such as insincerity. Therefore, MR reflects negative behaviors that violate communication spirit and personality, which is lack of truthfulness and sincerity (The evaluation system guarantees anonymity. The authors, affiliated with the HRD (human resource development) center in Sungkyunkwan University, received the data for SET and graduates’ employment information from Sungkyunkwan University with an agreement to provide the basic information for improving the quality of education. SET is conducted anonymously, and both instructors and students take advantage of the evaluation system).
Table 1. Ratio of monotonic response (MR) patterns regarding the SET rating.
Table 1. Ratio of monotonic response (MR) patterns regarding the SET rating.
SET Rating
12345Total
Ratio of monotonic response (MR) patterns0.275 (0.267)0.150 (0.170)0.086 (0.139)0.004 (0.013)0.004 (0.019)0.519 (0.285)
Table 2 shows the component items of SET, and an example of MR. MR refers to selecting the same rating (straight excellent, very good, fair, poor, or very poor) for all questions. Responses with the same answers for all nine questions were classified as MR, while those in which at least one question was answered differently were not counted as MR (Participation in SET may be one of the variables that can serve as a useful proxy variable to measure individual’s personality. However, we do not include the ratio of SET participation in the analysis because graduates’ average participation in SET during their college years is about 0.984 and is highly correlated with MR. Normally, response rate for students is about 91.3% on average each semester).
Table 2. Component items of SET and an example of a monotonic response (MR).
Table 2. Component items of SET and an example of a monotonic response (MR).
ExcellentVery GoodFairPoorVery Poor
1.Amount you learned in the course was:
2.Clarity of student responsibilities and requirements was:
3.Do you expect your grade in this course to be:○ A○ B● C○ D○ F
4.Course organization was:
5.Student confidence in instructor’s knowledge was:
6.Evaluative and grading techniques (tests, papers, project, etc.) were:
7.Instructor’s enthusiasm was:
8.Availability of extra help when needed was:
9.The course as a whole was:
10.Express your opinion for improvement of course
Notes: ● means a student’s selection mark in a check circle ○ in SET (student evaluations of teaching).
Table 3 shows a summary of the independent variables for the employed and unemployed groups. The independent variables included the ratio of MR, student characteristics (student’s gender and age), background (region and high school graduation), type of admission, graduation year, spec-related variables (total GPA, participation in programs abroad or student exchange, IT master and advanced foreign language certification), double major and the field of study (Sungkyunkwan University offers advanced foreign language certifications to students with scores of 950 points or more on the TOEIC, 250 or more on the 6th level of the HSK, and 950 points or more on the JPT. In addition, those who are awarded prizes in IT-related competitions or hold patents are given IT master certifications).
The ratio of monotonic response (MR) patterns was higher for unemployed graduates. The proportion of university graduates with advanced IT and foreign language certification was higher for the employed than the unemployed, and total GPAs were higher in the employed group. Thus, we could expect that graduates’ personalities reflected in their evaluation response patterns also have an effect on employability with their spec.
Table 3. Summary statistics.
Table 3. Summary statistics.
Employed (n = 4297)Unemployed (n = 4786)
Graduates’ personality
Ratio of monotonic response (MR) patterns0.5130.524
(0.282)(0.499)
Graduates’ spec
Total GPA3.5573.548
(0.340)(0.437)
Study abroad participation0.0360.017
(0.185)(0.130)
IT master certification0.5280.415
(0.499)(0.493)
Advanced foreign language certification0.1920.126
(0.394)(0.332)
Sciences and engineering0.5260.475
(0.499)(0.499)
Double major0.1740.199
(0.379)(0.399)
Graduates’ characteristics
Female student0.4050.558
(0.491)(0.497)
Age25.31724.877
(1.786)(2.084)
Graduates’ background
Specialized high school graduation0.0510.044
(0.221)(0.205)
Capital (Seoul)0.4240.449
(0.494)(0.497)
Metropolitan0.1510.128
(0.359)(0.334)
Local city0.4240.423
(0.494)(0.494)
Special admission0.2120.184
(0.409)(0.388)
Field of study
Business/Economics0.1990.114
(0.399)(0.317)
Humanities/Education0.1770.243
(0.382)(0.429)
Social science0.1050.189
(0.306)(0.391)
Science/Bioengineering0.0560.155
(0.230)(0.362)
Information and Communication Engineering0.1540.096
(0.361)(0.294)
Engineering0.3090.203
(0.462)(0.402)
Year of graduation
2008 graduation0.0140.112
(0.118)(0.315)
2009 graduation0.0350.225
(0.185)(0.418)
2010 graduation0.2270.183
(0.419)(0.387)
2011 graduation0.3850.256
(0.487)(0.437)
2012 graduation0.3390.224
(0.473)(0.417)
1. Standard deviations are in parentheses; 2. Specialized high schools include elite schools such as foreign language high school and science high schools; 3. For special admission, Sungkyunkwan University selects those with superior grades, essay writing, or language skills.

4.2. Empirical Framework

This study employed a Multinomial Logit model to analyze the employment prestige, along with a Logit model to determine the effects on employment. The dependent variable of the Logit model y = 1 indicates employed graduates. Also, to examine any influences on employment prestige, a Multinomial Logit Model was used to estimate four groups, namely non-employee, top 50 firms, middle-ranked 150 firms, and other low-ranked firms, based on the yearly sales, net profits, and annual average wages of the firms. The non-employee group was chosen as the baseline category.
log [ P ( y = j ) P ( y = J ) ] = k = 1 K β j k x k ,   j = 1 , 2 , 3
The Multinomial Logit model had four categories of dependent variables, with the baseline category J representing non-employees. The probability that the dependent variable can belong to each category was as follows.
P ( y = J ) = 1 1 + j = 1 J 1 e k = 1 K β j k x k
P ( y = j ) = e k = 1 K β j k x k 1 + j = 1 J 1 e k = 1 K β j k x k
The independent variables, x k , are the ratio of MR to the total number of responses to course evaluations that are representative of a student’s personality as well as a graduate’s spec-related, and characteristics-related variables.

5. Main Results

The results of analysis of the Logit and Multinomial Logit models, regarding the employability of the college graduates are presented in Table 4 and Table 5. Model 1 of Table 4 presents the results of Logit analysis, with the employee encoded as the dummy variable 1. Model 2 and Table 5 indicate the results of Multinomial Logit after categorizing the orders based on the sales, net profits of the year, and average annual wages of the firms. Logit analysis revealed that the ratio of MR had a negative effect on employment. In addition, Table 4 and Table 5 show that once other variables were controlled, the ratio of MR has a significantly negative effect on the probability of work in top 50, middle, or low ranked firms. The Multinomial Logit indicated that the marginal effect of the MR variable on employees in the top 50 firms was exp(−0.503) = 0.605. This means that the probability of employment in the top 50 firms was estimated to decrease by 60.5%, compared with non-employees, when the ratio of MR increased by one unit. In addition, the probability of being employed in the low-ranked firms decreased to 71.3%.
Therefore, those with higher MR on the SET for each course were less likely to be employed than those who did not. In addition, they were more likely to start work at firms whose sales, net profits, and average annual wages were lower. Barrick and Mount [17] claim that conscientiousness was significantly related to job performance, and the validity of agreeableness was also negatively correlated. In addition, some studies demonstrate that impatience is negatively related to labor market outcomes [25,26]. Consistent with our assumption about the association with MR, these previous studies support the validity of our results. Although MR does not necessarily indicate the student’s insincerity, it has a negative effect, highlighting parameters like insincerity, noncooperation, and impatience within the personality traits related to MR. Furthermore, most firms recognize that leadership, an upright personality, and patience play an important role in facilitating a good team environment and job performance. So, they tend to use personality tests for personnel selection before and after hiring people. Personality traits, such as deliberation, straightforwardness, altruism, and impatience, which are revealed through the response patterns of individuals while completing the course evaluations, can be described using the big five personality traits. The results of the analysis demonstrated that those personality traits influence employability.
Table 4. Logit and Multinomial Logit estimates of employment and employment prestige.
Table 4. Logit and Multinomial Logit estimates of employment and employment prestige.
Model 1 (Employed = 1)Model 2 (Sales of the Firms)
Top 50 FirmsMiddle RankLow Rank
Graduates’ personality
Ratio of MR−0.311 ***−0.503 **−0.194−0.338 **
(0.091)(0.159)(0.121)(0.113)
Graduates’ characteristics
Female student−0.277 ***−0.066−0.331 ***−0.310 ***
(0.057)(0.107)(0.079)(0.069)
Student’s age0.067 ***0.0190.083 ***0.071 ***
(0.014)(0.030)(0.019)(0.016)
Specialized high school0.1570.386 **0.0760.121
(0.111)(0.193)(0.155)(0.132)
Seoul0.014−0.1560.0520.051
(0.053)(0.096)(0.072)(0.065)
Metropolitan0.196 **0.0840.173*0.268 **
(0.073)(0.119)(0.097)(0.091)
Special admission0.245 ***0.271 **0.367 ***0.148 *
(0.062)(0.105)(0.082)(0.076)
Graduates’ spec
Total GPA0.162 **0.352 **0.181 **0.107
(0.065)(0.120)(0.090)(0.079)
Study abroad participation0.777 ***0.505 **0.987 ***0.659 **
(0.164)(0.246)(0.191)(0.201)
IT master certification0.290 ***0.344 ***0.370 ***0.218 ***
(0.050)(0.086)(0.067)(0.061)
Advanced foreign language certification0.454 ***0.475 ***0.698 ***0.296 ***
(0.068)(0.123)(0.088)(0.081)
Double major0.241 ***0.0200.255 **0.267 ***
(0.069)(0.167)(0.099)(0.078)
Field of study
Business/Economics0.106−0.926 ***−0.308 **0.898 ***
(0.080)(0.148)(0.101)(0.098)
Humanities/Education−0.695 ***−2.244 ***−1.284 ***0.250 **
(0.082)(0.194)(0.114)(0.101)
Social science−1.047 ***−2.626 ***−1.821 ***−0.035
(0.082)(0.226)(0.126)(0.101)
Science/Bioengineering−1.429 ***−2.297 ***−1.838 ***−0.652 ***
(0.093)(0.207)(0.137)(0.120)
Information−0.0400.611 ***−0.406 ***−0.374 **
(0.081)(0.105)(0.103)(0.125)
Year of graduationYesYes
Pseudo R20.168 ***0.149 ***
Observations9,0839,083
Source. The Career-SET matched data from Sungkyunkwan University in South Korea. Sales refers to data published in the DART using company names. Notes. 1. *** p < 0.001, ** p < 0.05, * p < 0.1; 2. The reference group for the field of study consists of engineering students; 3. The baseline category for the equation in model 2 was non-employee; 4. Standard errors are shown in parentheses; 5. Yearly sales were more than 25 trillion Korean won for the top 50 firms, 1.4–25 trillion for middle-ranked firms, and below 1.4 trillion for low-ranked firms.
Table 5. Multinomial Logit estimates of employment and employment prestige.
Table 5. Multinomial Logit estimates of employment and employment prestige.
Model 3 (Net Profits)Model 4 (Average Annual Wages)
Top 50MiddleLowTop 50MiddleLow
Ratio of MR−0.451 **−0.249−0.296 **−0.431 **−0.297 **−0.286 **
(0.156)(0.156)(0.101)(0.164)(0.128)(0.108)
Total GPA0.475 ***0.382 **0.0410.543 ***−0.0270.168 **
(0.118)(0.117)(0.072)(0.129)(0.096)(0.075)
Study abroad participation0.609 **1.002 ***0.729 ***0.493 *0.878 ***0.756 ***
(0.238)(0.224)(0.179)(0.278)(0.199)(0.183)
IT master certification0.313 ***0.317 ***0.280 ***0.399 ***0.398 ***0.219 ***
(0.085)(0.085)(0.055)(0.093)(0.071)(0.058)
Advanced language certification0.580 ***0.666 ***0.377 ***0.493 ***0.540 ***0.414 ***
(0.117)(0.109)(0.075)(0.137)(0.098)(0.075)
Business/Economics−0.817 ***−0.312 **0.500 ***−0.195−0.580 ***0.674 ***
(0.140)(0.127)(0.088)(0.141)(0.110)(0.093)
Humanities/Education−2.165 ***−1.236 ***−0.210 **−1.626 ***−1.612 ***−0.027
(0.188)(0.146)(0.091)(0.221)(0.130)(0.094)
Social science−2.575 ***−1.676 ***−0.548 ***−2.043 ***−2.061 ***−0.309 **
(0.218)(0.161)(0.091)(0.220)(0.142)(0.096)
Science/Bioengineering−2.155 ***−2.057 ***−1.014 ***−2.348 ***−2.062 ***−0.794 ***
(0.191)(0.196)(0.106)(0.264)(0.154)(0.110)
Information0.429 ***−0.434 ***−0.220 **0.388 ***−0.103−0.401 ***
(0.105)(0.131)(0.098)(0.116)(0.099)(0.117)
Year of graduationYesYes
Pseudo R20.138 ***0.178 ***
Observations90839083
Source. The Career-SET matched data from Sungkyunkwan University in South Korea. Net profits and average annual wages refer to the data published in the DART using company names. Notes. 1. *** p < 0.001, ** p < 0.05, * p < 0.1; 2. The baseline category for the equation was non-employees; 3. The reference group for the field of study consists of engineering students; 4. Standard errors are in parentheses; 5. Student’s gender (female = 1), student’s age, dummy variables of Seoul, metropolitan, and local (reference group of region is local city), high school graduation (specialized school = 1), admission type (special = 1), double major (complete = 1), and graduation year are included in the analysis, but not reported; 6. Net profits were more than 1.4 trillion Korean won for the top 50 firms, 0.2–1.4 trillion for middle rank, and below 0.2 trillion for low ranked firms. Annual average wages were more than 76 million Korean won for the top 50 firms, 47–76 million for middle-ranked firms, and below 47 million for low-ranked firms.
Those who had higher IT and language skills were more likely to work for firms with higher sales, net profits, and annual average wages, compared with the unemployed. Participation in programs or student exchange experience also showed a positive effect on employability. In general, grades, as well as foreign language and IT-related certifications, are representative of important specs for employment. In addition, globalization and an IT-based business environment have made foreign language and IT related certifications more important for employment.
Total GPA was not found to affect employment in the group of low-ranked firms based on sales and net profit, or middle-ranked firms based on the average annual wage, compared with non-employment (Table 4 and Table 5). However, total GPA was highly correlated with employment and getting a job in the top 50 firms (Model 3 of Table 5). GPA is a proxy variable for the ability to perform duties or obtain student achievements during college. However, several studies determined that when students with a high GPA had low interest in participating in additional career research activities, GPA had a reverse effect on employment [37]. Cho et al. [38] also revealed that while GPA might be a parameter affecting employment, its influence is decreasing and GPA may instead have a negative effect. However, Jung and Lee [39] reported that GPA was a very important factor for getting a job at large companies. Thus, since students with lower GPAs were employed based on their grades, it becomes a very important parameter to get jobs at large companies, or to gain good opportunities.
Students’ gender and major also significantly affect employment. Graduates with engineering majors get jobs in the top 50 and middle ranked firms, compared to students that major in business and economics, humanities and education, and science and bioengineering (Table 4). Kim and Kim [40] explained that engineering majors are more likely to have “good occupations” during the early career stages. In addition, most of the top 50 firms based on sales, net profits, and average annual wage are enterprises in the manufacturing sector, such as Samsung, Hyundai, LG, etc., which naturally means that there is high demand for graduates with engineering majors. Especially, the probability of getting a job in the top 50 firms for graduates with information and communication engineering majors is higher than for those with engineering majors. The undergraduate program for information and communication engineering at Sungkyunkwan University covers broad areas of electronics, electrical and computer engineering which integrates information technology, nano-technology and bio-technology and places a great deal of emphasis on meeting requirements from the industry as exemplified by major companies such as Samsung Electronics.
When we look at the influence of the MR on employment according to the field of study, graduates in business and economics (see Table A1), humanities and education, science and bioengineering, and engineering with a higher ratio of MR are less likely to be employed. These results show that conscientiousness in graduates reflected by dutifulness, responsibility, and sincerity, or agreeableness reflected by straightforwardness and altruism are important determinants of employment in most fields of study. Total GPA has a significantly positive effect on employment in the field of humanities and social sciences including business and economics, but has a negative effect in the field of natural sciences. This observation reflects the reality that graduates with science and engineering majors are highly in demand, even those with lower total GPA (The annual employment rate of graduates with engineering majors is more than 60%. Specifically, the employment rate of graduates majoring in humanities, social sciences, education, engineering, and natural sciences, are 48.4%, 54.4%, 49.0%, 67.5%, 52.2%, respectively [41]).
Female students experienced a negative effect on employment and the probability of employment in top 50, middle ranked, and low ranked firms compared with male students. Some studies determined that the non-economically active population more than doubled for women compared to men, and women were also more likely to be employed in the relatively weak peripheral labor market in South Korea [42,43]. Thus, the empirical results in this study reflect the phenomenon explained by previous studies.
We also used the Tobit model, because sales, net profits, and average annual wages, which are dependent variables related to employment prestige, are usually zero if there are no data disclosed by companies. The Tobit estimates were also consistent with the results in Table 4 and Table 5 (see Table A2).

6. Conclusions

This study analyzed the effects of graduates’ personalities, as indicated by monotonic responses (MR) to all the questions in a SET, on their employability by using Career-SET matched data for South Korean college graduates from 2008–2012. In this study, the response patterns in SET performed continually during the college years were examined to construct a variable for personality. The reasons for MR could be categorized according to a hypothesis of cost-benefit rationality from economics, with an aspect of increasing personal gains by selecting the same number for all questions in order to save time. MR may also be a negative factor in the labor market, as a reflection of insincerity, noncooperation, and impatience. The results of empirical analysis in this study suggest the latter case, the hypothesis that insincerity, noncooperation, impatience, etc. are more persuasive than the former.
The results from various estimation models, including Logit, Multinomial Logit, and Tobit, consistently indicated that those with a higher ratio of MR in SET are less likely to be employed or hired in lower prestige firms than the other comparison groups. More specifically, the results indicated that those who have a greater MR rate to all questions on the SET for every course are less likely to be employed than those who have a lower MR rate. In addition, the probability of gaining employment in the top 50, middle ranked or low ranked firms, as classified by yearly sales, net profits, and annual average wages, was lower. Other spec-related variables that influence employment, such as advanced IT and foreign language certification, and participation in programs abroad or student exchange, displayed positive influences on employability. In the case of total GPA, a positive impact on employment in the top 50 firms was observed.
Overall, we attempted to measure individual personality traits that are reflected by MR and explain them using the Big Five. The results indicate that graduates’ personalities, including deliberation, altruism, or impatience as well as their specs have significant effects on employability. This means that unlike the current practice in which firms rely simply on specs to hire employees, applicants’ invisible characteristics, such as personality, can also be screened by job interviewers. This suggests that the personnel management process still struggles to overcome the hurdle of insufficient information regarding students’ hidden and productive traits.
In this study, we demonstrated that MR has a negative correlation with employability using empirical results. This study provides useful information to identify students’ personality traits that are reflected by MR, and suggest that their attitude and personality are important factors in addition to specs in the labor market.

Author Contributions

Joonmo Cho designed the research; Wonyoung Baek analyzed the data. Both authors wrote the paper, read, and approved the final manuscript.

Conflicts of Interest

The authors declare no conflict of interest.

Appendix

Table A1. Logit estimates of employment by the field of study.
Table A1. Logit estimates of employment by the field of study.
Business/EconomicsHumanities/EducationSocial ScienceScience/BioengineeringInformationEngineering
Ratio of MR−0.617 **−0.698 **0.384−0.508 *−0.188−0.304 *
(0.261)(0.221)(0.238)(0.306)(0.251)(0.171)
Total GPA1.485 ***−0.0371.124 ***−0.681 **−0.516 **−0.556 ***
(0.181)(0.147)(0.165)(0.226)(0.196)(0.144)
Study abroad participation1.4201.029 **−0.5991.940 **0.3100.855 **
(1.066)(0.354)(0.615)(0.918)(0.358)(0.274)
IT master certification0.1680.508 ***0.1410.326 *0.295 **0.147
(0.136)(0.108)(0.138)(0.171)(0.142)(0.098)
Advanced language certification0.373 **0.723 ***0.0190.414 *−0.1540.537 **
(0.152)(0.135)(0.169)(0.245)(0.225)(0.179)
Observations13981927135398311202302
Source. The Career-SET matched data from Sungkyunkwan University in South Korea. Sales, net profits, and average annual wages refer to the data published in the DART using company names. Notes. 1. *** p < 0.001, ** p < 0.05, * p < 0.1; 2. Standard errors are in parentheses; 3. Student’s gender (female = 1), student’s age, dummy variables of Seoul, metropolitan, and local (reference group of region is local city), high school graduation (specialized school = 1), admission type (special = 1), double major (complete = 1), and graduation year were included in the analysis, but not reported.
Table A2. Tobit estimates of employment prestige.
Table A2. Tobit estimates of employment prestige.
Model 5Model 6Model 7
Dependent VariableSalesNet ProfitsAnnual Wage
Ratio of MR−0.377−0.498 *−0.673 **
(0.239)(0.289)(0.324)
Total GPA0.673 ***0.931 ***0.549 **
(0.181)(0.220)(0.245)
Study abroad participation1.199 ***1.315 **1.886 ***
(0.356)(0.426)(0.484)
IT master certification0.673 ***0.727 ***1.040 ***
(0.130)(0.157)(0.178)
Advanced language certification0.784 ***0.847 ***1.614 ***
(0.174)(0.210)(0.238)
Business/Economics−1.664 ***−1.769 ***−2.301 ***
(0.199)(0.239)(0.272)
Humanities/Education−4.020 ***−4.423 ***−5.466 ***
(0.228)(0.274)(0.310)
Social science−4.483 ***−5.064 ***−6.397 ***
(0.239)(0.292)(0.327)
Science/Bioengineering−4.118 ***−4.498 ***−6.151 ***
(0.258)(0.311)(0.356)
Information1.952 ***2.060 ***0.597 **
(0.193)(0.232)(0.262)
Observations9083
Source. The Career-SET matched data from Sungkyunkwan University in South Korea. Sales, net profits, and average annual wages refer to the data published in the DART using company names. Notes. 1. *** p < 0.001, ** p < 0.05, * p < 0.1; 2. The reference group for the field of study consists of engineering students; 3. Standard errors are in parentheses; 4. Student’s gender (female = 1), student’s age, dummy variables of Seoul, metropolitan, and local (reference group of region is local city), high school graduation (specialized school = 1), admission type (special = 1), double major (complete = 1), and graduation year were included in the analysis, but not reported; 5. The units of yearly sales, net profits, and annual average wages are 10 trillion, 1 trillion, and 10 million Korean won, respectively.

References

  1. Lee, J.; Cho, J. Who teaches economics courses better?: Using student–professor matched data for the principle of economics course. Appl. Econ. Lett. 2014, 21, 934–937. [Google Scholar] [CrossRef]
  2. Spence, M. Job market signaling. Q. J. Econ. 1973, 87, 355–374. [Google Scholar] [CrossRef]
  3. Digman, J.M. Five robust trait dimensions: Development, stability, and utility. J. Pers. 1989, 57, 195–214. [Google Scholar] [CrossRef] [PubMed]
  4. Goldberg, L.R. An alternative “description of personality”: The big-five factor structure. J. Pers. Soc. Psychol. 1990, 59, 1216–1229. [Google Scholar] [CrossRef] [PubMed]
  5. McCrae, R.R.; Costa, P.T. Updating norman’s “adequacy taxonomy”: Intelligence and personality dimensions in natural language and in questionnaires. J. Pers. Soc. Psychol. 1985, 49, 710–721. [Google Scholar] [CrossRef] [PubMed]
  6. Norman, W.T. Toward an adequate taxonomy of personality attributes: Replicated factor structure in peer nomination personality ratings. J. Abnorm. Soc. Psychol. 1963, 66, 574–583. [Google Scholar] [CrossRef] [PubMed]
  7. Cappelli, P. Why Good People Can’t Get Jobs: The Skills Gap and What Companies Can Do about It. Available online: http://wdp.wharton.upenn.edu/book/why-good-people-cant-get-jobs/ (accessed on 2 December 2015).
  8. Arcidiacono, P. Ability sorting and the returns to college major. J. Econom. 2004, 121, 343–375. [Google Scholar] [CrossRef]
  9. Lassibille, G.; Gómez, L.N.; Ramos, I.A.; de la O Sánchez, C. Youth transition from school to work in Spain. Econ. Educ. Rev. 2001, 20, 139–149. [Google Scholar] [CrossRef]
  10. Borghans, L.; Duckworth, A.L.; Heckman, J.J.; Ter Weel, B. The economics and psychology of personality traits. J. Hum. Resour. 2008, 43, 972–1059. [Google Scholar] [CrossRef]
  11. Mischel, W. Personality and Assessment; Psychology Press: East Sussex, UK, 2013. [Google Scholar]
  12. Marsh, H.W.; Roche, L.A. Making students’ evaluations of teaching effectiveness effective: The critical issues of validity, bias, and utility. Am. Psychol. 1997, 51, 1187–1197. [Google Scholar] [CrossRef]
  13. Arnold, I.J. Do examinations influence student evaluations? Int. J. Educ. Res. 2009, 48, 215–224. [Google Scholar] [CrossRef]
  14. Brockx, B.; Spooren, P.; Mortelmans, D. Taking the grading leniency story to the edge. The influence of student, teacher, and course characteristics on student evaluations of teaching in higher education. Educ. Assess. Eval. Account. 2011, 23, 289–306. [Google Scholar] [CrossRef]
  15. Choi, B.; Kim, J. The influence of student and course characteristics on monotonic response patterns in student evaluation of teaching in South Korea. Asia Pac. Educ. Rev. 2014, 15, 483–492. [Google Scholar] [CrossRef]
  16. Mount, M.K.; Barrick, M.R. Five reasons why the ”Big Five” article has been frequently cited. Pers. Psychol. 1998, 51, 849–857. [Google Scholar] [CrossRef]
  17. Barrick, M.R.; Mount, M.K. Autonomy as a moderator of the relationships between the big five personality dimensions and job performance. J. Appl. Psychol. 1993, 78, 111–118. [Google Scholar] [CrossRef]
  18. Costa, P.T.; McCrae, R.R. Revised NEO Personality Inventory (NEO pi-r) and NEO Five-Factor Inventory (NEO-ffi); Psychological Assessment Resources Odessa: Lutz, FL, USA, 1992. [Google Scholar]
  19. Hough, L.M.; Eaton, N.K.; Dunnette, M.D.; Kamp, J.D.; McCloy, R.A. Criterion-related validities of personality constructs and the effect of response distortion on those validities. J. Appl. Psychol. 1990, 75, 581–595. [Google Scholar] [CrossRef]
  20. Mueller, G.; Plug, E. Estimating the effect of personality on male and female earnings. Ind. Labor Relat. Rev. 2006, 60, 3–22. [Google Scholar] [CrossRef]
  21. John, O.P.; Srivastava, S. The big five trait taxonomy: History, measurement, and theoretical perspectives. In Handbook of Personality: Theory and Research; Guilford Press: New York, NY, USA, 1999; volume 2, pp. 102–138. Available online: http://www.rc.usf.edu/~jdorio/Personality/BIG%20FIVE%20TRAIT%20TAXONOMY.pdf (accessed on 2 December 2015).
  22. Barrick, M.R.; Mount, M.K. The big five personality dimensions and job performance: A meta-analysis. Pers. Psychol. 1991, 44, 1–26. [Google Scholar] [CrossRef]
  23. Salgado, J.F. The Five Factor Model of personality and job performance in the European Community. J. Appl. Psychol. 1997, 82, 30–43. [Google Scholar] [CrossRef] [PubMed]
  24. Lobel, T.E. Personality correlates of Type A coronary-prone behavior. J. Pers. Assess. 1988, 52, 434–440. [Google Scholar] [CrossRef] [PubMed]
  25. Cadena, B.C.; Keys, B.J. Human capital and the lifetime costs of impatience. 2012. Available online: http://spot.colorado.edu/~cadenab/Research_files/cadena_keys_lifetime_impatience.pdf (accessed on 2 December 2015).
  26. Golsteyn, B.H.; Grönqvist, H.; Lindahl, L. Adolescent time preferences predict lifetime outcomes. Econ. J. 2014, 124, F739–F761. [Google Scholar] [CrossRef]
  27. Chamorro-Premuzic, T.; Furnham, A. Personality And Intellectual Competence; Psychology Press: East Sussex, UK, 2014. [Google Scholar]
  28. Han, S.; Kim, H.; Lee, J. A comprehensive study of Korean students’ evaluations of university teaching. J. Educ. Adm. 2005, 23, 379–403. [Google Scholar]
  29. Park, I. A study on effects of student ratings of their learning on consistent responses in student ratings of college teaching. Korean J. Educ. Methodol. Stud. 2012, 24, 257–281. [Google Scholar]
  30. Cho, D.; Baek, W.; Cho, J. Why do good performing students highly rate their instructors? Evidence from a natural experiment. Econ. Educ. Rev. 2015, 49, 172–179. [Google Scholar] [CrossRef]
  31. Chae, S.; Lim, K. A trend study of student’ consistent responses to course evaluation. Korean J. Med. Educ. 2009, 21, 307–311. [Google Scholar] [CrossRef] [PubMed]
  32. Han, K.; Choi, S.; Park, J. Problems in mandatory course evaluations. Commun. Stat. Appl. Methods 2011, 18, 35–45. [Google Scholar] [CrossRef]
  33. Kim, H.; Kim, S.; Kwon, O.; Lee, C.; Row, K. Improving course evaluation system of engineering education. J. Eng. Educ. Res. 2007, 10, 58–77. [Google Scholar]
  34. Hong, K. An analysis of students’ response to course evaluation. J. Korean Assoc. Educ. Inf. Media 2006, 12, 97–127. [Google Scholar]
  35. Nyhus, E.K.; Pons, E. The effects of personality on earnings. J. Econ. Psychol. 2005, 26, 363–384. [Google Scholar] [CrossRef]
  36. Corporate Information Database (2008–2012). Available online: http://englishdart.fss.or.kr (accessed on 30 December 2014).
  37. Ihm, C.; Yoo, J. Determinants of college graduate employment selection in service industry. J. Educ. Adm. 2004, 22, 311–332. [Google Scholar]
  38. Cho, Y.; Kim, B.; Kim, B. Investigation into the influential factors on employment outcomes of university graduates. J. Educ. Adm. 2008, 26, 437–462. [Google Scholar]
  39. Jung, T.; Lee, K. Determinants of job finding among college graduates with emphasis on the effects of GPA. Korean Manag. Rev. 2005, 8, 159–184. [Google Scholar]
  40. Kim, C.; Kim, H. Occupational labor market activities by college majors: On the crisis of science and engineering majors. Korean J. Popul. Stud. 2006, 29, 1–27. [Google Scholar]
  41. Employment of Higher Edcuation Graduates Statistical Survey (2008–2012). Available online: http://kess.kedi.re.kr/eng/index (accessed on 17 November 2015).
  42. Cho, J.; Lee, J.; Kwon, T. Gender exclusion in social security protection: Evidence from Korea. Asian Pac. Econo. Lit. 2013, 27, 62–78. [Google Scholar] [CrossRef]
  43. Cho, J.; Lee, J. Persistence of the Gender Gap and Low Employment of Female Workers in a Stratified Labor Market: Evidence from South Korea. Sustainability 2015, 7, 12425–12451. [Google Scholar] [CrossRef]

Share and Cite

MDPI and ACS Style

Baek, W.; Cho, J. Challenging the Sustainability of an Education System of Evaluation and Labor Market Outcomes. Sustainability 2015, 7, 16060-16075. https://doi.org/10.3390/su71215808

AMA Style

Baek W, Cho J. Challenging the Sustainability of an Education System of Evaluation and Labor Market Outcomes. Sustainability. 2015; 7(12):16060-16075. https://doi.org/10.3390/su71215808

Chicago/Turabian Style

Baek, Wonyoung, and Joonmo Cho. 2015. "Challenging the Sustainability of an Education System of Evaluation and Labor Market Outcomes" Sustainability 7, no. 12: 16060-16075. https://doi.org/10.3390/su71215808

Article Metrics

Back to TopTop