Internet Public Opinion Risk Grading under Emergency Event Based on AHPSort II-DEMATEL

: The internet often serves as a communication platform for users to freely express opinions. However, as it supports the quick and wide spread of public opinions, it often results in great uncertainty and is very di ﬃ cult to control. As a result, evaluating the risk levels of internet public opinion becomes very challenging. The safety of internet public opinion a ﬀ ects the sustainability and stability of society. In this paper, a novel conceptual model of internet public opinion governance is proposed. The model can be used to grade risk levels by combining the Analytic Hierarchy Process Sort II (AHPSort II) and Decision-Making Trial and Evaluation Laboratory (DEMATEL) under a fuzzy environment with triangular fuzzy sets. A numerical example is provided to verify the e ﬃ ciency of the proposed method, and some managerial implications are also discussed.


Introduction
With the rapid development of the internet, online social platforms have become increasingly important channels for people to post, access and exchange information. Some examples of such social network websites and applications include Tencent QQ, WeChat, Micro-blog, Facebook, and Twitter, which connect users in different geographical locations and time zones. Research has demonstrated that public opinion propagation has taken place on these sites. More and more people tend to express their standpoints of an emergency event on the internet platform, which could easily lead to public opinions [1]. Moreover, people are increasingly relying on internet public opinions to make decisions [2]. Different from traditional media public opinions, internet public opinions have the characteristics of high liquidity, accessibility and concealment [3]. Gushes of internet public opinions may trigger unexpected incidents that significantly affect social security and stability, especially in the public area [4]. Meanwhile, compared to traditional media views, internet media users may hold more radical views due to the breeding of extreme views through communication among like-minded people [5]. To ensure public safety, it is useful to study the governance of internet public opinion and classify its risk levels.
A number of recent reviews have demonstrated the growing academic interests in internet public opinion. It has been shown that online social networks have become the main communication platform to grade internet public opinion risk level. Section 4 provides a case study which applies the AHPSort II-DEMATEL in internet public opinion grading, and, from there, managerial suggestions are provided. Finally, conclusions and future works are summarized in Section 5.

Conceptual Model of Internet Public Opinion Governance
The internet is the major platform for the public to access information under emergent events. Public opinion governance plays an important role in governmental credibility. In the following, we introduce a conceptual model to capture the formation process of internet public opinions.

Importance of Internet Public Opinions Risk Management to Sustainability of Society
The management of internet public opinions plays an important role in maintaining social stability and harmony. The benefit and the need of internet public opinion risk management can be summarized as the following.
(1). Internet public opinion risk management is beneficial for the response to emergency events.
The evolution process of internet public opinions reflects the process of emergencies. The internet has become an important channel for public emergencies, and it also makes public opinions spread widely and uncontrollably [16]. Meanwhile, internet public opinion has a great impact on the solutions of public emergency events, judicial decisions, and even government decisions. (2). Internet public opinion risk management is conducive to the sustainable development of society.
The internet is a platform on which people express their demands and views. Due to the openness, timeliness and participation of the internet platform and the distortion of information, the risk of emergencies is expanded in the internet public platform, even with rumors. It is easy to cause social mood fluctuations, even social panic, and this affects social stability. Meanwhile, positive public opinion promotes the sustainable and harmonious development of society [36]. Therefore, it is particularly necessary to intervene, control, restrain and guide internet public opinion. (3). Internet public opinion risk management effectively weakens the negative impact of emergencies.
Guidance by governments determines the development of internet public opinions. It is an effective measure to control the leader of public opinions, like a well-known blogger, who greatly affects and guides the public opinion.

Process of Internet Public Opinions Formation
The formation process of an internet public opinion is shown in Figure 1. The internet contains a collection of cognitions, attitudes, emotions and behavioral tendencies of people who spread opinions regarding emergency events. Studying these opinions' formation processes, therefore, is of great significance. II-DEMATEL in internet public opinion grading, and, from there, managerial suggestions are provided. Finally, conclusions and future works are summarized in Section 5.

Conceptual Model of Internet Public Opinion Governance
The internet is the major platform for the public to access information under emergent events. Public opinion governance plays an important role in governmental credibility. In the following, we introduce a conceptual model to capture the formation process of internet public opinions.

Importance of Internet Public Opinions Risk Management to Sustainability of Society
The management of internet public opinions plays an important role in maintaining social stability and harmony. The benefit and the need of internet public opinion risk management can be summarized as the following.
(1). Internet public opinion risk management is beneficial for the response to emergency events. The evolution process of internet public opinions reflects the process of emergencies. The internet has become an important channel for public emergencies, and it also makes public opinions spread widely and uncontrollably [16]. Meanwhile, internet public opinion has a great impact on the solutions of public emergency events, judicial decisions, and even government decisions. (2). Internet public opinion risk management is conducive to the sustainable development of society.
The internet is a platform on which people express their demands and views. Due to the openness, timeliness and participation of the internet platform and the distortion of information, the risk of emergencies is expanded in the internet public platform, even with rumors. It is easy to cause social mood fluctuations, even social panic, and this affects social stability. Meanwhile, positive public opinion promotes the sustainable and harmonious development of society [36]. Therefore, it is particularly necessary to intervene, control, restrain and guide internet public opinion. (3). Internet public opinion risk management effectively weakens the negative impact of emergencies.
Guidance by governments determines the development of internet public opinions. It is an effective measure to control the leader of public opinions, like a well-known blogger, who greatly affects and guides the public opinion.

Process of Internet Public Opinions Formation
The formation process of an internet public opinion is shown in Figure 1. The internet contains a collection of cognitions, attitudes, emotions and behavioral tendencies of people who spread opinions regarding emergency events. Studying these opinions' formation processes, therefore, is of great significance.  The internet promotes quick information transmission when emergencies occur. An emergency event promotes internet public opinions, and the internet expands public opinions. However, users cannot obtain full information about emergency events because of information asymmetry. Meanwhile, users carry different educational backgrounds, information sources, and values; thus, their attitudes to emergency events are different. Negative public opinions, neutral public opinions and positive public opinions can all be produced at the occurrence of an emergency event. These opinions are then disseminated by the internet, making global internet public opinions. Meanwhile mainstream views influence global internet public opinions and play an important role in government decision making. The attitude and decision of governments to emergencies affect their credibility. A failure of an official response, improper media response or improper media reports can bring negative effects to public ecology and social harmony. Meanwhile, the internet speeds up the spread of mainstream ideas. Therefore, it is important to classify the risk levels of emergency internet public opinions in time, a classification which is also a good way to promote the governance of internet.

Conceptual Model of Internet Public Opinion Risk Grading
The conceptual model of internet public opinion risk grading is shown in Figure 2, and this model includes two-stages and double loops. The two-stages are the preparation stage and the field work stage. The double loops contain reference point update and information update. The internet promotes quick information transmission when emergencies occur. An emergency event promotes internet public opinions, and the internet expands public opinions. However, users cannot obtain full information about emergency events because of information asymmetry. Meanwhile, users carry different educational backgrounds, information sources, and values; thus, their attitudes to emergency events are different. Negative public opinions, neutral public opinions and positive public opinions can all be produced at the occurrence of an emergency event. These opinions are then disseminated by the internet, making global internet public opinions. Meanwhile mainstream views influence global internet public opinions and play an important role in government decision making. The attitude and decision of governments to emergencies affect their credibility. A failure of an official response, improper media response or improper media reports can bring negative effects to public ecology and social harmony. Meanwhile, the internet speeds up the spread of mainstream ideas. Therefore, it is important to classify the risk levels of emergency internet public opinions in time, a classification which is also a good way to promote the governance of internet.

Conceptual Model of Internet Public Opinion Risk Grading
The conceptual model of internet public opinion risk grading is shown in Figure 2, and this model includes two-stages and double loops. The two-stages are the preparation stage and the field work stage. The double loops contain reference point update and information update.  In the preparation stage, the main target is to obtain the experience curve of every criterion. A group of experts are selected first, and then the criteria are selected by brainstorming and text analysis approaches. The weights of criteria are further determined by the experts. The public opinion about In the preparation stage, the main target is to obtain the experience curve of every criterion. A group of experts are selected first, and then the criteria are selected by brainstorming and text analysis approaches. The weights of criteria are further determined by the experts. The public opinion about an emergency event is monitored in real time, which provides evaluation information for each criterion. In the fieldwork stage, the goal is to grade the risk of internet public opinion. We can determine the risk levels of internet public opinions based on the information of the emergency event and the experience curve of each criterion. Note that the risk levels of internet public opinions toward an emergency event can be evaluated in real time.
As shown in Figure 2, the updates of reference points and emergency information are interrelated. In the reference point update loop, reference points can be selected according to text analysis, expert discussions, historical case studies, brainstorming or the types of current emergencies. The emergency internet public opinion risk level grading system can deal with uncertainty and complexity. With advances of internet technology, the reference point and profile point can be adjusted belong to the emergency event.
The grading of internet public opinion risk levels is critical in developing early warnings of internet public opinion risk. The steps are as follows.

Internet Public Opinion Intelligent Grading Based on AHPSort II-DEMATEL
The Decision-Making Trial and Evaluation Laboratory (DEMATEL) is built on the basis of graph theory and is best suited for analyzing complex structural models [37,38]. It can effectively assess total relations and construct a map between system components (criterion). AHPSort II [39] is a new multi-criteria sorting method for a high number of alternatives which extends from the Analytic Hierarchy Process (AHP) method [40]. It is a good technique to reduce the complexity of calculation. According to the conceptual model of internet public opinion governance, we propose an intelligent grading approach whose steps are as follows.

Definition of Internet Public Opinion Grading Problem
Step 1. Define the risk levels of internet public opinions.
The risk levels are defined as very high risk (L 1 ), high risk (L 2 ), low risk (L 3 ), and very low risk (L 4 ), as shown in Figure 3, and the description of the risk levels is shown in Table 1.
lp i t t = 1, 2, 3; i = 1, 2, · · · , m is the set of limiting profiles of criterion C i , which can be defined by experts based on the risk levels. If the emergency internet public opinion risk value a k is assigned to the level L i , then Let E x |x = 1, 2, · · · , p be the set of experts, and the weights of experts are given as W E = w E 1 , w E 2 , · · · , w E p . { C i |i = 1, 2, · · · , m} are the sets of criteria with respect to the internet public opinion risk grading problem.

Risk Level Description
Very Public safety and social order pose a serious threat. The public opinion which is concerned by domestic and foreign media involves negative public opinion

Risk Level Description
Very High (VH) I Public safety and social order pose a serious threat. The public opinion which is concerned by domestic and foreign media involves negative public opinion and has serious effect. The internet users of most countries know the emergency event.
High (H) II The emergency causes the public opinion which mainly refers to the harm and influence to the public safety and social order in a large scope. The domestic media and communication platform APPs involve a great deal of negative public opinion. Internet users in China and a few international countries know the emergency event.

Low (L) III
Public safety and social order are threated in a certain scope. A small number of domestic media and websites follow the emergency event. The negative public opinion refers to the emergency event effecting the public emotion. The emergencies are well known in China.

Very Low (VL) IV
This mainly refers to the general harm and influence to the public safety and social order in this county in a small scale. A few media and websites understand the emergency event, and the negative public opinion will be forgotten over time. The emergencies are known in an area.
Step 2. Determine references points r i j for each criterion. AHPSort II requires experts to determine the reference points in advance; this is a vital step of the method. Different reference points can have a great influence on drawing the judgment curve. There are two ways to define the reference points r i j (i = 1, 2, · · · , m; j = 1, 2, · · · , n) for each criterion C i (i = 1, 2, · · · , m).
(1). Based on the similarity case of the emergency event, select the reference points for criterion C i , as shown in Figure 4. The number of reference points N can be given by the top t% of similarity cases, or N = N (the number of the case).

Risk Level Description
Very High (VH) I Public safety and social order pose a serious threat. The public opinion which is concerned by domestic and foreign media involves negative public opinion and has serious effect. The internet users of most countries know the emergency event.

High (H) II
The emergency causes the public opinion which mainly refers to the harm and influence to the public safety and social order in a large scope. The domestic media and communication platform APPs involve a great deal of negative public opinion. Internet users in China and a few international countries know the emergency event.

IV
This mainly refers to the general harm and influence to the public safety and social order in this county in a small scale. A few media and websites understand the emergency event, and the negative public opinion will be forgotten over time. The emergencies are known in an area.
Step 2. Determine references points i j r for each criterion.
AHPSort II requires experts to determine the reference points in advance; this is a vital step of the method. Different reference points can have a great influence on drawing the judgment curve. There are two ways to define the reference points i   (2). Determine the maximum and minimum values of each criterion, and the number of reference points N can be defined by experts. The reference points can be determined by equal distance (see Figure 5).

Definition of Internet Public Opinion Grading Problem
Step 3. The expert pairwise compares the criteria by linguistic term and transform the linguistic terms to their corresponding triangular fuzzy sets (TFS) based on Table 2.

Definition of Internet Public Opinion Grading Problem
Step 3. The expert pairwise compares the criteria by linguistic term and transform the linguistic terms to their corresponding triangular fuzzy sets (TFS) based on Table 2. Step 4. Aggregate the experts' points by Equation (1) and obtain the comprehensive evaluation matrix Z, when i = j, z ij = ((0, 0, 0), 1, 0).
Step 5. Calculate the total relation matrix. Due to the comprehensive evaluation matrix Z, the total relation matrix can be derived by Equation (2), which contains direct and indirect relations between the factors. Step 6. Construct the DEMATEL map. Based on the total relation matrix, the sum of each row (i = 1, 2, · · · , n) and column C i (i = 1, 2, · · · , n) are calculated based on Equations (4) and (5). Then, (R i + C i ) and (R i − C i ) can be obtained by Equations (6) and (7). p i = R i + C i indicates the impact of i-th influential factor and its degree of being impacted. q i = R i − C i is defined as the importance of factors and classifies the i-th influential factor into the cause or effect category in the researching system.
Sustainability 2019, 11, 4440 8 of 16 Step 7. Obtain the weights of criteria by the value of R + C.

Judgment Curve of Internet Public Opinion Grading
Step 8. Pairwise compare the limiting profiles with the reference points for each criterion C i . In this step, we can reduce the number of pairwise comparisons using the method by Miccoli and Ishizaka [33]. Assume that the number of reference points is N and the references points are divided to k clusters. The priorities are calculated for the k clusters, as shown in Figure 6.
Step 7. Obtain the weights of criteria by the value of R + C.

Judgment Curve of Internet Public Opinion Grading
Step 8. Pairwise compare the limiting profiles with the reference points for each criterion i C . In this step, we can reduce the number of pairwise comparisons using the method by Miccoli and Ishizaka [33]. Assume that the number of reference points is N and the references points are divided to k clusters. The priorities are calculated for the k clusters, as shown in Figure 6.  The priorities are calculated for the k clusters. Local priorities of the latter clusters are linked to the previous one by multiplying them by the ratio of the scores of joining points. The clusters are shown in Figure 7.

Application of DEMATEL-AHPSort II for Internet Public Opinion Grading
The internet public opinion can be classified into political public opinion, economic public opinion, cultural public opinion, and social public opinion [41]. The social public opinion is the most common internet public opinion, and we thus present a case on social public opinion risk grading. In this section, we first present an illustrative example of the performance of the proposed DEMATEL-AHPSort II for internet public opinion grading problem and then conduct further discussion about

Application of DEMATEL-AHPSort II for Internet Public Opinion Grading
The internet public opinion can be classified into political public opinion, economic public opinion, cultural public opinion, and social public opinion [41]. The social public opinion is the most common internet public opinion, and we thus present a case on social public opinion risk grading. In this section, we first present an illustrative example of the performance of the proposed DEMATEL-AHPSort II for internet public opinion grading problem and then conduct further discussion about our proposed method.

Problem Description
China is particularly vulnerable to earthquakes and has suffered a high number of earthquake casualties [42]. From 2004 to 2018, 446 damaging earthquakes with Ms > 5.0 magnitude have occurred in China. Over the past 15 years, earthquakes have brought great economic losses. Sichuan Province of China is located in the southwestern seismic region, which is a prominent population and economic growth center. Meanwhile, Sichuan Province is an earthquake-stricken region, with one of the most dangerous cities in China. The regional distribution of earthquakes in Sichuan Province is shown in Figure 8. The red in the map of Sichuan Province represents the area where a fatal earthquake with Ms ≥ 7.0 magnitude has occurred and caused serious economic losses. The orange represents the area where earthquakes have occurred with 5 ≤ Ms < 7.0 magnitude, and the white represents the area where earthquakes have occurred with Ms < 5.0 magnitude or no earthquake.  Yibin City is located at the junction of Sichuan Province, Yunnan Province and Guizhou Province. There are many mountains around Yibin City, like the Qixing, Longtou and Guandou Mountains. At 22:55 on 17 June 2019, an Ms = 6.0 magnitude earthquake occurred in Yibin City. The internet public opinions about the Yibin earthquake arose from 17 June to 19 June. We should take more effective and efficient measures to cope with internet public opinions propagation. The hot degree of internet public opinions with time is shown in Figure 9. We can determine the risk level of internet public opinions at 2 p.m. on 18 June 2019. In this paper, the evaluation system was constructed by some previous studies [35], and the evaluation criteria were selected by expert's brainstorming, as shown in Table 3. We can determine the risk level of internet public opinions at 2 p.m. on 18 June 2019. In this paper, the evaluation system was constructed by some previous studies [35], and the evaluation criteria were selected by expert's brainstorming, as shown in Table 3. Table 3. Evaluation criteria.

Criteria Description
Scale (C 1 ) The time of the impact, the spread of the crowd and the range of areas involved.
Negative evaluation (C 2 ) The more intense the negative evaluation of the public opinion of the incident network is, the faster the negative emotion public opinion spreads; the greater the influence spreads, the greater the influence scope is and the easier the network public opinion crisis happens.

Secondary hazards (C 3 )
The secondary harm caused by the emergency network public opinion, such as the panic of the crowd, the harm to the personnel involved in the matter, and even the larger network public opinion crisis.

Moral perception (C 4 )
Users feel an emotional response in posting or forwarding emergency information, such that they are meeting ethical expectations they have for themselves or toward others.
Next, we will introduce our method step by step, according to Section 4.
Step 1. The risk levels of internet public opinions are shown in Table 1. There are four experts participating in the evaluation E = {E 1 , E 2 , E 3 , E 4 }, and the weight of experts are given as W E = {0.3, 0.25, 0.25, 0.2}.
Step 2. Select the reference points by equal distance. Determine the minimum and maximum value of each criterion and then divide it by ten reference points, in which r 0 = minimum, r 10 = maximum. Take C 1 as an example, and the reference points can be obtained as shown in Table 4.

Determination of Criteria Weights
Step 3. Establish the comparison matrices by the experts E = {E 1 , E 2 , E 3 , E 4 }, which are shown in Table 5. Then, transform the linguistic variable to triangular fuzzy number (TIFN). Table 5. Decision matrices of criteria by experts.
Step 4. Obtain the comprehensive evaluation matrix. (0, 0, 0) Step 5. Normalize the comprehensive evaluation matrix. (0, 0, 0) Step 6. Calculate the value of R, C, R + C and R − C by Equations (22) and (23), as shown in Table 6. Table 6. Value of R, C, R + C and R − C. Step 7. Transform the TIFN to an accurate value by Equations (9) and (10), as shown in Table 7. Then, the weights of criteria are obtained based on the value of R + C.

Acquirement of Judgment Curve of Internet Public Opinion Grading
Step 8. Pairwise compare the limiting profiles with reference points. Using C 1 as an example; the reference points are shown in Table 4. Cluster reference points and limiting profiles are shown in Figure 10. Three clusters of "Scale (C 1 )" are shown in Tables 8-10.

Acquirement of Judgment Curve of Internet Public Opinion Grading
Step 8. Pairwise compare the limiting profiles with reference points. Using 1 C as an example; the reference points are shown in Table 4. Cluster reference points and limiting profiles are shown in Figure 10. Three clusters of "Scale ( 1 C )" are shown in Tables 8-10.    Table 9. Second cluster of "Scale ( 1 C )".  Table 10. Third cluster of "Scale ( 1 C )". Step 9. Obtain the local priority and normalized local priorities. Taking the 1 C as an example, the result of normalized local priorities are shown in Table 11.   Table 9. Second cluster of "Scale (C 1 )".  Table 10. Third cluster of "Scale (C 1 )". Step 9. Obtain the local priority and normalized local priorities. Taking the C 1 as an example, the result of normalized local priorities are shown in Table 11. Step 10. Derive the local priority of each criterion and then obtain the global priority. The local priority curve of C 1 is shown in Figure 11.  Step 10. Derive the local priority of each criterion and then obtain the global priority. The local priority curve of 1 C is shown in Figure 11.

Risk Grading
Step 11. Collect information about the event and obtain the values of the criteria. According to Step 3 and Step 7, we can grade internet public opinion risk levels 2 p.m. 17 June 2019.

Managerial Suggestions
The results provide some suggestions to internet public opinion risk management for managers and government officials, detailed as the following: (1). Guide internet public opinion by opinion leaders. An opinion leader is the core of an internet public opinion who disseminates internet public opinions. They can be selected based on user influence and user activity. Policy makers can lead public opinion to the positive direction and reduce risk by controlling the opinion guidance of opinion leaders [43].

Risk Grading
Step 11. Collect information about the event and obtain the values of the criteria. According to Step 3 and Step 7, we can grade internet public opinion risk levels 2 p.m. 17 June 2019.

Managerial Suggestions
The results provide some suggestions to internet public opinion risk management for managers and government officials, detailed as the following: (1). Guide internet public opinion by opinion leaders. An opinion leader is the core of an internet public opinion who disseminates internet public opinions. They can be selected based on user influence and user activity. Policy makers can lead public opinion to the positive direction and reduce risk by controlling the opinion guidance of opinion leaders [43]. (2). Construct and quantify a criteria system. A standard and systematic criteria system is necessary for risk grading. The quantitative evaluation of an internet public opinion criteria system is the basic premise of efficiently carrying out mass internet public opinion information monitoring and early warning.
(3). Monitor active internet public opinions by machine learning. An active opinion is the basis for forming an internet public opinion or rumor. We should focus on the risk levels and pay attention to the threshold, and we should monitor the hot opinions in real time. Advance control and timely interventions are necessary when a hot opinion is beyond the threshold.
The suggestions are provided to internet users who are the main body of internet public opinion propagation as following: (1). Do not make a comment easily when you do not know the facts of an emergency event.
Emergencies are prone to rumors which have a negative impact on emergency management. Secondary damage will be caused by rumors. (2). Do not follow the opinion leaders and obey their opinions about emergencies. You should have your own opinion when you know the facts of emergencies, and you should express your opinion in the right way.

Conclusions
The internet is an open platform with high liquidity, accessibility and concealment for the public to express their respective views on certain events. Therefore, it is vital to form an effective response and management system to classify and grade risk levels. In this paper, the mechanism of internet public opinion formation was proposed, and it was found that the internet plays a vital role in promoting the dissemination of public opinions. The attitude and decision of the government have important influences on the development of internet public opinion. Additionally, a conceptual model of internet public opinion, two stages and double loops, was built. Next the AHPSort II-DEMATEL approach was proposed to grade the risk levels of internet public opinions. Finally, an application of the proposed method to grade the risk levels of internet public opinions was provided to demonstrate the effectiveness of the proposed method. This research of internet public opinion risk management can greatly promote stability and harmony, both of which are of great significance to ensure the sustainable development of society.
However, there are also some limitations of this research work: (1) The decision-making situation in a variety of language environments was not considered. (2) The uncertainty of expert weight was not taken into account.
For future research, we aim to first reduce the impact of the approximation-we will pay more attention to the proportional relationship between the number of reference points and data scales to guarantee highest possible precision, along with fewer comparisons in AHPSort II. Then, in order to expand methods for grading the risk level of internet public opinion, we will expand the DEMATEL-AHPSort II to different linguistic environments. These aspect will all be considered in future research.