Evaluation of Online Teaching Quality: An Extended Linguistic MAGDM Framework Based on Risk Preferences and Unknown Weight Information

: Evaluation of online teaching quality has become an important issue because many universities are turning to online classes due to the Corona Virus Disease 2019 (COVID-19) pandemic. In this paper, online teaching quality evaluation is considered as a linguistic multi-attribute group decision-making (MAGDM) problem. Generally, the evaluation sematic information can be symmetrically or asymmetrically distributed in linguistic term sets. Thus, an extended linguistic MAGDM framework is proposed for evaluating online teaching quality. As the main contribution, the proposed method takes into account the risk preferences of assessment experts (AEs) and unknown weight information of attributes and sub-attributes. To be speciﬁc, the Delphi method is employed to establish a multi-level evaluation indicator system (EIS) of online teaching quality. Then, by introducing the group generalized linguistic term set (GLTS) with two risk preference parameters, a two-stage optimization model is developed to calculate the weights of attributes and sub-attributes objectively. Subsequently, the linguistic MAGDM framework was divided into two stages. The ﬁrst stage maximizes the group comprehensive rating values of teachers on different attributes to obtain partial ranking results for teachers on each attribute. The latter stage maximizes the group comprehensive rating values of teachers to evaluate the overall quality. Finally, a case study is provided to illustrate how to apply the framework to evaluate online teaching quality.


Introduction
Due to the outbreak of Corona Virus Disease 2019 (COVID- 19), universities in many countries have adopted online teaching methods to prevent the spread of the epidemic on campus. According to data published by the United Nations Educational, Scientific and Cultural Organization (UNESCO), as of 28 April 2020, 168 countries worldwide had closed their campuses due to the epidemic and have adopted online teaching methods [1]. From the Chinese Ministry of Education, as of 8 May, 1454 colleges and universities nationwide were conducting online teaching, 1.03 million teachers were offering 1.07 million courses online, and a total of 17.75 million college students had participated in online learning. Currently, the global epidemic situation is still grim, and online teaching will continue to be one of the primary forms, especially in universities where students come from a wide variety of regions [2].
Large-scale online teaching has effectively solved the problem of students learning at home, but the changing of teaching environment from offline to online has also posed significant challenges to teaching quality. For instance, the changing scenarios and the lack of a sense of presence make it difficult for teachers and students to interact effectively. Meanwhile, network congestion occurs frequently when a large number of students attend the classes at the same time. Against this backdrop, how to assess and promote the quality cept of the numerical scale, and on this basis, extended the 2-tuple linguistic representation model to compute with words in an unbalanced LTS. Subsequently, Rodriguez et al. [19] introduced the concept of a hesitant fuzzy LTS, which provided a linguistic and computational basis to increase the richness of linguistic elicitation. Dong et al. [20] proposed a novel computing with words method to construct the hesitant fuzzy LTS by using an unbalanced LTS and the numerical scale. Li et al. [21] presented a personalized individual semantics model to personalize individual semantics by the 2-tuple linguistic model and an interval numerical scale. Liu et al. [22] proposed a new fuzzy envelope for the hesitant fuzzy LTS in form of type-2 fuzzy sets for representing comparative linguistic expressions in decision contexts, which overcomes the limitation of existing representations in dealing with inherent uncertainties. However, in actual online teaching evaluation, different AEs may have different expectations for semantic values of linguistic terms, which can be described by their risk preferences [23]. Therefore, it is necessary to investigate the linguistic representation model in linguistic MAGDM problems based on risk preferences.
With regard to linguistic MAGDM based on risk preferences, Zhou and Xu [24] extended the sigmoid function and presented the concept of a generalized linguistic term set (GLTS). They applied it to linguistic multi-attribute decision making involving risk preferences. Lin and Wang [23] presented the concept of GLTSs with triangular fuzzy semantic values, and developed a new method for coping with linguistic MADM problems considering decision makers' risk preferences. Lin and Wang [25] developed a new framework to solve the linguistic MAGDM problem based on risk preferences, where attribute evaluation values are represented by linguistic terms and the attribute weight information is incomplete. Guo and Sun [26] combined single-valued neutrosophic LTSs with prospect theory, and introduced a new comprehensive fuzzy decision approach to deal with linguistic MAGDM. Focusing on decision makers' risk preferences and regret theory, Ma et al. [27] initiated an approach to solve the behavioral multi-attribute decision-making problem based on probability expressions. Liao et al. [28] launched a Choquet integralbased hesitant fuzzy gained and lost dominance score method based on prospect theory to deal with MAGDM problems with experts' risk preference psychology. Obviously, the above methods and case studies are all based on a single-level EIS. This paper extends the linguistic MAGDM based on risk preferences to the context of a multi-level EIS.
Different from group decision problems with preference relations [29,30], the MAGDM problems involves the determination of attribute weights. To evaluate the quality of online teaching according to the given EIS, it is an important issue of whether to provide attribute weights. In fact, it is difficult for AEs to give attribute weight information in many cases. Thus, some scholars have conducted studies on this issue. For example, based on the standard deviation and average deviation of the interval type-2 fuzzy set, Gong [31] established an optimization model to calculate attribute weights in linguistic MAGDM, in which linguistic terms are transformed into interval type-2 fuzzy sets. Gupta et al. [32] proposed a new method for solving linguistic MAGDM problems, in which linguistic terms are converted into trapezoidal intuitionistic fuzzy numbers and Shannon's entropy theory is used to determine attribute weights. Wang and Mendel [33] initiated a new method to deal with linguistic MAGDM problems based on interval type-2 fuzzy sets, where an optimization model is established to determine completely unknown attribute weights. Touqeer et al. [34] used a combination of grey relational analysis (GRA) method and projection method to calculate completely unknown attribute weights in linguistic MAGDM problems. We can see that these literatures are all based on the researches of linguistic MAGDM under the single-level EIS. Therefore, considering multi-level EIS and risk preferences, this paper develops a two-stage optimization model to derive the attribute weights and the sub-attribute weights.
In response to the above research deficiencies, this paper designs an extended framework to assess the quality of online teaching. The novelties of the framework are as follows:

•
Taking into account the problem of online teaching quality evaluation, based on the existing traditional classroom teaching quality EISs [4,[13][14][15], the Delphi method is employed to construct a brand-new multi-level EIS of online teaching quality, including five attributes and 15 sub-attributes. • Inspired by the decision method in Lin and Wang [25], this framework extends the linguistic MAGDM method involving risk preferences from the single-level EIS to the multi-level one.

•
In teaching quality evaluation problems based on multi-level EISs, many works of literature adopted the AHP approach to derive the weights of attributes and subattributes [8][9][10][11][12][13]. Sometimes AEs cannot make pairwise comparisons on the importance of attributes or sub-attributes, that is, they cannot use the AHP method to calculate attribute weights. Thus, based on the principle of maximizing the group comprehensive evaluation value of each alternative, a two-stage attribute and sub-attribute weight solution optimization model is established in this framework.

•
The evaluation ranking results obtained by this framework are divided into the overall ranking result and partial ranking results, which are helpful for teachers and universities to understand their own situation and make targeted improvements.
The rest of this paper is structured as follows: Section 2 introduces the basic concepts related to LTSs, group GLTSs with individual risk preferences, the Euclidean distance between any two positive triangular fuzzy numbers, and the score function. Section 3 constructs a multi-level EIS for assessing online teaching quality of teachers, and then establishes a two-stage optimization model to solve the weights of attributes and sub-attributes in the EIS. Section 4 devises an extended evaluation framework and then describes the detailed evaluation process. Section 5 proposes a case study to verify the validity and feasibility of the proposed evaluation framework. Section 6 carries on the practical discussion to the evaluation ranking results including the overall and partial ranking results. Finally, Section 7 concludes this paper.

Preliminaries
In this section, we introduce some concepts about LTSs, group GLTSs, the Euclidean distance, and the score function.
be any two positive triangular fuzzy numbers, then the Euclidean distance between them is defined as

Score Function
For comparing and ranking triangular fuzzy numbers, assume that v = (v L , v M , v U ) is a positive triangular fuzzy number, then its score function could be denoted as [37]

Group Generalized Linguistic Term Set Based on Risk Preferences
To describe the influence of the evaluation group's risk preference on the semantic information of linguistic terms in an LTS, Lin and Wang [25] introduced the concept of a group GLTS.

Determination of Risk Preference Parameters
In order to determine the group GLTS defined above, it is assumed that is an expected set of triangular fuzzy semantic values offered by the AE e k ∈ E(k = 1, 2, . . . , q) according to linguistic terms in the LTS S. Then, an optimization model is constructed below: . . , −τ 2 }, θ 1 and θ 2 are decision variables, and d(., .) denotes the Euclidean distance defined by (2).
By solving Equation (8), the values of the group risk preference parameters θ 1 and θ 2 are obtained. Then by plugging them into Equations (4)-(7), the group GLTS is derived.

Selection of Indicators for Evaluating the Online Teaching Quality
The selection of appropriate indicators is the fundamental to conducting the evaluation. In order to effectively identify the critical factors affecting the quality of teaching, the literature analysis and Delphi methods were used to construct indicators for evaluating the online teaching quality.
First, analysis of the existing literature reveals that in the traditional evaluation of teaching quality, scholars have focused on indicators of content, teaching methods, and effectiveness. Similarly, these indicators also could be selected for assessing online courses.
Then, through interviews with six teachers and 14 students who participated in online teaching, it was found that all interviewees focused on the classroom atmosphere in the online education process. Besides, the online teaching environment and the teaching platform were also key factors mentioned by the interviewees. Based on literature and interview research, this paper identified five attributes of the online teaching environment, online teaching platform, online teaching content and methods, online classroom climate, and online teaching effectiveness. These attributes were then decomposed to constitute 15 sub-attributes. Based on the literature study and interview research, this paper identified five primary indicators of online teaching environment, online teaching platform, online teaching content and methods, online classroom atmosphere, and online teaching effectiveness. Then, the primary indicators were decomposed to formulate 15 secondary indicators. Finally, the five primary indicators were treated as attributes a j (j = 1, 2, . . . , 5) and the secondary indicators were treated as sub-attributes a jl (j = 1, 2, . . . , 5; l = 1, 2, 3). Sub-attributes represents a specific refinement of the attribute. In other words, the five attributes in this multi-level EIS contained 15 sub-attributes. The multi-level EIS is shown in Table 1 below.
The attributes of the online teaching environment mainly refer to the external environmental requirements that support online education, including the network, hardware equipment, and teaching environment. The network environment requires the instructor to have a smooth network to ensure video and audio transmission when conducting instruction. Hardware equipment mainly refers to the teacher having the equipment to conduct online instruction, including computers, microphones, and cameras. Additionally, the physical environment primarily refers to space where classes are conducted without outside interference. The online teaching platform requires that the online teaching system used by teachers and students be stable and easy to use. Personalized support requires the system to be able to sign in, take attendance, interact, and submit assignments. Different from the traditional teaching mode, the online platform plays the role of a "classroom" connecting students and teachers. Therefore, the stability of the online platform and personalized support functions are vital guarantees for the effective implementation of online teaching.
Regarding the content and methodological attributes of online teaching, teachers are required to teach comprehensive content and to highlight key points. The combination of theory and practice is also very crucial. Secondly, the teaching time is arranged in such a way that students can easily absorb and digest the course knowledge. It is also necessary to use a variety of teaching formats, such as a combination of live and recorded lectures, monologues and inspirational lectures. Unlike the traditional classroom teaching quality assessment, the content and methods for online teaching put forward higher requirements for universities and teachers. Specifically, teachers need to adjust teaching strategies, improve teaching methods, and optimize instructional design according to the needs of the scenario in order to compensate for the lack of presence and interaction resulting from online teaching.
The online classroom atmosphere contains the evaluation of teachers and students. Among them, the influence of teaching requires teachers to be full of energy, affable, use body language reasonably and appropriately, and teach in a natural and generous manner. The sub-attribute of student's attitude requires students to have a right attitude towards learning and to pay attention to the class. Besides, the interaction and discussion between teacher and students is also an important basis for assessing the classroom atmosphere. This can be interpreted as a lively classroom atmosphere reflects the positive interaction between the teacher and the students and also makes the teacher more enthusiastic in teaching and helps the students to listen attentively to the lesson.
The teaching effectiveness is the reflection of the quality of instruction and can be used to measure if the teaching objectives are being met. This attribute includes if students master the content of the course in a timely and effective manner and whether the lectures inspire students to think. In addition, fostering creativity and improving problem-solving skills are also incorporated into this attribute.
Based on the above-detailed description of the evaluation indicators, for the sake of more intuitively understanding the MAGDM problems of online teaching quality evaluation, a multi-level hierarchic structure of online teaching quality evaluation composed of an objective, attributes, sub-attributes, and alternatives (teachers) is shown in Figure 1.
master the content of the course in a timely and effective manner and whether the lectures inspire students to think. In addition, fostering creativity and improving problem-solving skills are also incorporated into this attribute.
Based on the above-detailed description of the evaluation indicators, for the sake of more intuitively understanding the MAGDM problems of online teaching quality evaluation, a multi-level hierarchic structure of online teaching quality evaluation composed of an objective, attributes, sub-attributes, and alternatives (teachers) is shown in Figure 1.

Determination of the Weights of Evaluation Indicators
Since this paper adopted a MAGDM approach to assess the online teaching quality of teachers, the determination of attribute weights was particularly vital. Obviously, the importance weights of different sub-attributes to the same attribute will also be different. Due to the limited knowledge of AEs and the complexity of evaluation problems, it may be difficult for them to give the weights of attributes and sub-attributes, that is, the weights of attributes and sub-attributes are completely unknown. It is supposed that

Determination of the Weights of Evaluation Indicators
Since this paper adopted a MAGDM approach to assess the online teaching quality of teachers, the determination of attribute weights was particularly vital. Obviously, the importance weights of different sub-attributes to the same attribute will also be different. Due to the limited knowledge of AEs and the complexity of evaluation problems, it may be difficult for them to give the weights of attributes and sub-attributes, that is, the weights of attributes and sub-attributes are completely unknown. It is supposed that w = (w 1 , w 2 . . . , w m ) T represents the attribute weight vector containing m attributes. w j = (w j1 , w j2 , . . . , w jp ) T means the sub-attribute weight vector, which contains p subattributes under the attribute a j . When the GLTS S of the evaluation group is determined, the AE e k gives a linguisticterm-based decision matrix D jk = (d jk s il ) n×p of the alternative x i over the sub-attribute a jl under the attribute a j where i = 1, 2, . . . , n, l = 1, 2, . . . , p, j = 1, 2, . . . , m, k = 1, 2, . . . , q.
Subsequently, each linguistic-term-based decision matrix D jk = (d jk s il ) n×p can be converted into a triangular fuzzy decision matrix expressed by where Based upon triangular fuzzy decision matrices R jk (j = 1, 2, . . . , m; k = 1, 2, . . . , q), assuming that the importance weight vector of AEs is ω = (ω 1 , ω 2 , . . . , ω q ) T , ω k ∈ [0, 1] and q ∑ k=1 ω k = 1. Then a triangular fuzzy weighted average operator and the AEs' weight vector ω = (ω 1 , ω 2 , . . . , ω q ) T are utilized to aggregate individual triangular fuzzy decision matrices related to the attribute a j into a group one as follows: where Next, combining the sub-attribute weight vector w j = (w j1 , w j2 , . . . , w jp ) T and applying a triangular fuzzy weighted average operator, the triangular fuzzy evaluation values in the elements of the ith row in R j can be aggregated into a group comprehensive evaluation value for the alternative x i (i = 1, 2, . . . , n) over the attribute a j , denoted by h j i , namely: Once h j i is obtained, according to the attribute weight vector w = (w 1 , w 2 . . . , w m ) T and the triangular fuzzy weighted average operator, the group comprehensive evaluation value h i of the alternative x i can be determined, that is Since both w = (w 1 , w 2 . . . , w m ) T and w j = (w j1 , w j2 , . . . , w jp ) T (j = 1, 2, . . . , m) are completely unknown in this paper, thus, a two-stage optimization model is constructed to determine the weights of attributes and sub-attributes, shown as follows: Stage 1: As mentioned in Equation (13), h j i represents the group comprehensive evaluation value for the alternative x i with respect to the attribute a j , the larger h j i represents the better the alternative x i with regard to the attribute a j , which leads to a better alternative x i . Therefore, based on Equation (3), we first need to find the sub-attribute weight vector w j = (w j1 , w j2 , . . . , w jp ) T (j = 1, 2, . . . , m), so that S( h As for the attribute a j (j = 1, 2, . . . , m), each alternative x i (i = 1, 2, . . . , n) is feasible and non-inferior, and the optimization model of each alternative has the same constraint conditions. Therefore, the aforementioned multi-objective optimization model can be transformed into an aggregated optimization model by setting the same sub-attribute weights for each alternative under the same attribute as follows: Symmetry 2021, 13,192 10 of 21 Based on Equations (3) and (13), the optimization model (16) can be equivalently converted into Solving the above optimization model (17) yields the optimal sub-attribute weight vector w * j = (w * j1 , w * j2 , . . . , w * jp ) T related to the attribute w j (j = 1, 2, . . . , m). The optimal solution of the model can be expressed as Since the weight vector satisfies the constraint condition of unitization, it is also necessary to normalize w * j , namely: The optimal sub-attribute weight vector w * j = (w * j1 , w * j2 , . . . , w * jp ) T after normalization is obtained. Then, substituting w * j into Equation (13), the optimal group comprehensive evaluation value h j * i of the alternative x i over the attribute w j is obtained by Stage 2: Once h j * i is determined, the larger the triangular fuzzy evaluation value h i , the better the alternative x i is. Similarly, based on Equation (3), we need to derive the attribute weight vector w = (w 1 , w 2 . . . , w m ) T so that for any i = 1, 2, . . . , n, S( h i ) is maximized. The following multi-objective optimization model of the second stage is constructed as In the same way, the above multi-objective optimization model (21) is converted into an aggregated optimization model: Symmetry 2021, 13, 192 11 of 21 Based upon the score function of triangular fuzzy numbers, we substitute h j * i into Equation (14), and the optimization model (22) can be equivalently converted into: Solving the optimization model (23) yields the optimal attribute weight vector w * = (w * 1 , w * 2 . . . , w * m ) T . Then, utilize Equation (24) to normalize w * j (j = 1, 2, . . . , m), the normalized attribute weight vector w * = (w * 1 , w * 2 . . . , w * m ) T is derived by Subsequently, substituting w * into Equation (14), and at the same time, according to Equation (20), the group comprehensive evaluation value of the alternative x i (i = 1, 2, . . . , n) is calculated as As the summary of this section, the brief flow of the two-stage optimization model used to solve the weights of attributes and sub-attributes is shown in Figure 2.
Subsequently, substituting * w into Equation (14), and at the same time, according to Equation (20) As the summary of this section, the brief flow of the two-stage optimization model used to solve the weights of attributes and sub-attributes is shown in Figure 2

A Method for Linguistic MAGDM Involving Risk Preferences and Completely Unknown Weight Information
Taking into account linguistic MAGDM problems involving decision makers' risk preferences, let X= {x 1 , x 2 , . . . , x n } be a set of alternatives, A= {a 1 , a 2 , . . . , a m } be a set of attributes, and a j = {a j1 , a j2 , . . . , a jp (j = 1, 2, . . . , m) be a set of sub-attributes related to the attribute a j . The weights of attributes and sub-attributes are completely unknown. Suppose that q AEs represented by E = e 1 , e 2 , . . . , e q assess the alternative x i ∈ X with regard to the sub-attribute a jl ∈ a j to the attribute a j ∈ A according to the LTS S defined in Definition 1. Then they offer linguistic-term-based decision matrices as D jk = (d jk s il ) n×p (j = 1, 2, . . . , m; k = 1, 2, . . . , q), where d jk s il ∈ S and the weights of AEs are known. Based on the risk preferences of AEs, a method for solving linguistic MAGDM problems with the completely unknown weights of attributes and sub-attributes is provided. The detailed evaluation process is described as follows: Preprocessing: Step 1: Based on the LTS S, each AE e k ∈ E assesses the alternative x i ∈ X with regard to the sub-attribute a jl ∈ a j relative to the attribute a j ∈ A. The evaluation process could be expressed by linguistic-term-based decision matrices D jk = (d jk s il ) n×p (j = 1, 2, . . . , m; k = 1, 2, . . . , q). Then, each AE provides his or her respective set of expected triangular fuzzy semantic values Step 2: Compute the group risk preference parameters θ 1 and θ 2 by solving the optimization model (8), and substitute them into Equations (4)-(7) to obtain the group GLTS S.
Step 8: Obtain the ranking of all the alternatives over each attribute in descending order according to the size of scores S( h j * i )(i = 1, 2, . . . , n; j = 1, 2, . . . , m), and use x i α a j x i β to indicate that x i α is superior to x i β with respect to the attribute a j . Stage 2: Step 9: Get the optimal attribute weight vector w * = (w * 1 , w * 2 . . . , w * m ) T by solving the nonlinear programming model (23). Then employ Equation (24) to derive the normalized optimal one denoted by w * = (w * 1 , w * 2 . . . , w * m ) T .
Step 10: Calculate the optimal group comprehensive evaluation value h * i (i = 1, 2, . . . , n) of each alternative in X by making use of Equation (25).
Step 12: Assess and rank the alternatives x i (i = 1, 2, . . . , n) in descending order according to the size of scores S( h * i )(i = 1, 2, . . . , n), and select the best alternative. Then, make use of Based on the detailed description of the above evaluation process, an extended framework of online teaching quality evaluation is devised in Figure 3.

A Case Study about Online Teaching Quality Evaluation
This section aims to provide a case to clarify the feasibility and effectiveness of the method and models presented in this paper.

Problem Description
During the outbreak of COVID-19, almost all universities across the countries postponed the opening of their classes. In order not to delay the progress of classroom teaching, a certain university requires teachers to teach students courses through online teaching. Because the online teaching method is relatively novel and unfamiliar to teachers, it is very crucial to control the quality of online teaching. For the sake of improving the quality of online teaching in this university, three AEs

A Case Study about Online Teaching Quality Evaluation
This section aims to provide a case to clarify the feasibility and effectiveness of the method and models presented in this paper.

Problem Description
During the outbreak of COVID-19, almost all universities across the countries postponed the opening of their classes. In order not to delay the progress of classroom teaching, a certain university requires teachers to teach students courses through online teaching. Because the online teaching method is relatively novel and unfamiliar to teachers, it is very crucial to control the quality of online teaching. For the sake of improving the quality of online teaching in this university, three AEs e k (k = 1, 2, 3) are invited to assess the online teaching quality of the four excellent teachers x i (i = 1, 2, 3, 4) selected by the university. Then the best teacher is chosen to share his or her experience to improve the overall quality of online teaching in this university.
In the same way, the ranking results of teachers' online teaching quality over different attributes are obtained, as shown in Table 2. Table 2. Ranking results of teachers' online teaching quality evaluation over different attributes.
Based on Equation ( , the overall ranking result of four teachers is x 3 x 2 x 4 x 1 , which means that the teacher with the best online teaching quality is x 3 . From the perspective of Table 2 and the overall ranking result, if the teacher with the best online teaching quality is chosen in terms of a certain attribute, then teacher x 1 is the best in aspects of online teaching platform and online teaching effectiveness. The teacher x 2 is better than other three teachers in terms of online teaching environment and atmosphere. The teacher x 4 has an absolute advantage in terms of online teaching content and methods. Compared with other teachers, although the teacher x 3 is not the best in every aspect, it almost always ranks in the second place, which is high and stable. On the contrary, the teacher x 2 ranks the worst in aspects of online teaching content and methods and online teaching effectiveness. The teacher x 4 has the worst ranking with respect to online teaching environment and platform. Except for the best ranking of the teacher x 1 in terms of the above-mentioned attributes, the first teacher ranks next to last for the other attributes, which indicates that teacher x 1 is at the lower end of the ranking and not stable in general. Although teachers x 1 , x 2 , and x 4 have the best online teaching quality among a certain attribute(s), in view of all the evaluation attributes, the teacher x 3 is the best.

Practical Discussion
The above-simulated case verifies the feasibility and effectiveness of the proposed evaluation framework. On the one hand, several inspirations to AEs are given by the evaluation process and results of the case. Initially, in the actual evaluation, AEs often have different expectations and considerations for semantic values of linguistic terms. Thus, the risk preference is used to describe and depict this phenomenon, which makes the evaluation results more accurate and persuasive. Meanwhile, we can find that the importance weights of attributes and sub-attributes in many multi-level EISs are subjectively given by AEs for previous evaluation researches. But in this paper, the importance weights of attributes and sub-attributes can be objectively derived through the proposed evaluation method. Therefore, it makes the evaluation process fairer and makes teachers more convincing. Next, from the evaluation results of the above case, we can see that there are not only the overall evaluation ranking result, but also partial ranking results of teachers over each attribute. This form of evaluation results can help universities and teachers better improve the quality of online teaching. For universities, they can not only select the best teacher in the overall ranking for teaching experience, but also choose the best one in a certain aspect to impart teaching experience in this aspect. For teachers, they can not only know their ranking in the comprehensive evaluation, but also understand their own deficiencies in some aspects to improve themselves.
On the other hand, the research in this paper also has some limitations, which need to be improved in the future. Firstly, this paper regards online teaching quality evaluation as a linguistic MAGDM problem, assuming that the AE group uses the same granular LTS for evaluation. In real life, because different AEs have different knowledge and different preferences, LTSs with different granularities may be employed by them for evaluation. How to convert LTSs with different granularities into ones with the same granularity under the consideration of individual AEs' risk preferences is the direction of future research. Secondly, the form of evaluation information given in this paper is single. When different AEs give different types of evaluation information, how to transform and aggregate them is also the direction to be studied in the future. Thirdly, the proposed evaluation framework is applicable to online teaching quality evaluation, but it is also applicable to other application cases, which is worth further research in the future.
Of course, there are some other MAGDM methods that can be used to solve attribute weights, such as TOPSIS, VIKOR and PROMETHEE II methods [38]. Which method is suitable to be used in a realistic assessment environment should be measured by one or some benchmarks. In this paper, the online teaching evaluation process takes into account the risk preferences of AEs. However, the difference in risk preferences will affect the weights of attributes and sub-attributes in the EIS [23]. Thereby, in an evaluation environment that considers risk preferences, the presented attribute and sub-attribute weight solution optimization model and the devised evaluation framework are more suitable and applicable than some other MAGDM approaches.

Conclusions
In the current way of epidemic prevention and control, online teaching has gradually transformed into a mainstream learning method in universities. In order to ensure the quality of online teaching in universities, evaluation of online teaching quality is particularly important. By introducing and expanding the linguistic MAGDM method, this paper mainly copes with the evaluation problem of online teaching quality that considers AEs' risk preferences and the completely unknown weights information in the multi-level EIS. Its contributions and advantages are manifested in four aspects. First of all, we construct a multi-level EIS of online teaching quality based upon the Delphi method, which can be well applied to evaluation of online teaching quality. Then, we extend Lin and Wang's method [25] to apply to the evaluation problem based on a multi-level EIS which commonly exists in reality. Furthermore, according to the principle of maximizing the group comprehensive evaluation value of each alternative, we put forward the two-stage optimization model that can objectively solve the weights of attributes and sub-attributes. The presented two-stage optimization model has lower requirements for the AEs than other attribute solving models, and is more practical and objective. Lastly, in the process of evaluation, the evaluation results given by our framework include both the overall evaluation result and partial evaluation results. These evaluation results can help teachers and even universities to understand the specific problems existing in some aspects of current online teaching in more detail for timely improvement.