Internet Public Opinion Risk Grading under Emergency Event Based on AHPSort II-DEMATEL

Abstract

The internet often serves as a communication platform for users to freely express opinions. However, as it supports the quick and wide spread of public opinions, it often results in great uncertainty and is very difficult to control. As a result, evaluating the risk levels of internet public opinion becomes very challenging. The safety of internet public opinion affects the sustainability and stability of society. In this paper, a novel conceptual model of internet public opinion governance is proposed. The model can be used to grade risk levels by combining the Analytic Hierarchy Process Sort II (AHPSort II) and Decision-Making Trial and Evaluation Laboratory (DEMATEL) under a fuzzy environment with triangular fuzzy sets. A numerical example is provided to verify the efficiency of the proposed method, and some managerial implications are also discussed.

1. Introduction

With the rapid development of the internet, online social platforms have become increasingly important channels for people to post, access and exchange information. Some examples of such social network websites and applications include Tencent QQ, WeChat, Micro-blog, Facebook, and Twitter, which connect users in different geographical locations and time zones. Research has demonstrated that public opinion propagation has taken place on these sites. More and more people tend to express their standpoints of an emergency event on the internet platform, which could easily lead to public opinions [1]. Moreover, people are increasingly relying on internet public opinions to make decisions [2]. Different from traditional media public opinions, internet public opinions have the characteristics of high liquidity, accessibility and concealment [3]. Gushes of internet public opinions may trigger unexpected incidents that significantly affect social security and stability, especially in the public area [4]. Meanwhile, compared to traditional media views, internet media users may hold more radical views due to the breeding of extreme views through communication among like-minded people [5]. To ensure public safety, it is useful to study the governance of internet public opinion and classify its risk levels.

A number of recent reviews have demonstrated the growing academic interests in internet public opinion. It has been shown that online social networks have become the main communication platform for public emergencies, which has also made public opinion’s influence spread more widely [6]. Soffer and Gordoni [1] showed that people are willing to express opinions in the public platform. Kim et al. [7] applied social internet analysis (SNA) to examine the mechanism of opinion diffusion. Based on the super-internet theory, Ma and Liu [8] established a super-work model which included social, environmental and psychological-viewpoint sub-internets. From the perspective of public opinion propagation, Huang et al. [6] used SNA and emotion mining analysis to study the interaction and evolution mechanism between the subject and the environment. Social media has been shown to be more likely to facilitate the emergence of internet public opinions [9], and internet usage has turned out to be the most effective means of civic engagement for emergency events [10]. Hampton [11] explored the relationship between the use of social media, attitudinal strength, perceived opinion agreement with social ties, and willingness to discuss a political issue in different online and offline contexts. Cao and Fu [12] proposed three types of internet public opinion problems based on uncertainty theory and the propagation process, and Huo and Ma [13] found that there exists a golden time for government intervention.

Some studies have focused on using super-internets for prediction. Liu et al. [14] proposed a super-internet model for public opinion prediction and governance. Another prediction model was built based on the grey system theory and the fuzzy neural internet [15]. Fidino et al. [16] assessed online public opinions toward wildlife by collecting comments made on YouTube videos. Chen et al. [17] proposed a model that could accurately predict public opinion duration and identified its influencing factors. Other examples of internet public opinions can be found regarding autonomous vehicles [18], automated driving [19], and sex education [20].

The government controls internet public opinions by enhancing its claims to legitimacy, circumscribing dissent, identifying malfeasance in its agents and using internet public opinions to adapt policies and direct propaganda efforts [21]. Lian et al. [22] described the topology of internet public opinions, and their description of the degree distribution of the internet public opinion was better than the traditional degree. Xu et al. [23,24,25,26] conducted a systematic study on rumors, and Zhang et al. [27] and Zhang and Xu [28] proposed a rumor-spreading model in the fuzzy set context. Liu et al. [29] presented a novel rumor propagation SEIR (Susceptible Exposed Infected Removed) model and identified the rumor spreading threshold.

Crisis management [30] and the SARF (social amplification of risk framework) theory [31] are two research streams about risk management. There are a great deal of uncertainties of emergencies, like geographical and spatiotemporal uncertainty [30], which increase the difficulty of crisis management. Gutteling and De Vries [32] paid attention to risk information seeking of emergencies, and they introduced a risk information seeking model. Kasperson et al. [33] proposed the phenomenon that the interpretation and response to information flow form the social amplification of risk. In other words, internet public opinions will be amplified with rumors and even cover the truth of the emergencies. This paper aims to study internet public opinion risk intelligent grading based on the crisis management and the SARF theory.

The aforementioned existing research on internet public opinion has mainly been aimed at identifying the influencing factors of public opinion, the prediction of public opinion development and the research of internet public opinion structure. However, research is scarce on public opinion governance and public opinion risk classification, and few have applied a multiple criteria decision making method [34,35]. This research aims to analyze the formation mechanism of internet public opinions and construct an intelligent grading model of public opinion risk levels, a model which can correctly estimate the impact and consequences of rumors. This research provides a new approach to grade the risk level of internet public opinion, and it gives some suggestions on how to control and take measures for internet public opinion risk.

The remainder of this research is structured as follows. The conceptual model of internet public opinion governance is proposed in Section 2. The Analytic Hierarchy Process Sort II (AHPSort II)-Decision-Making Trial and Evaluation Laboratory (DEMATEL) approach is presented in Section 3 to grade internet public opinion risk level. Section 4 provides a case study which applies the AHPSort II-DEMATEL in internet public opinion grading, and, from there, managerial suggestions are provided. Finally, conclusions and future works are summarized in Section 5.

2. Conceptual Model of Internet Public Opinion Governance

The internet is the major platform for the public to access information under emergent events. Public opinion governance plays an important role in governmental credibility. In the following, we introduce a conceptual model to capture the formation process of internet public opinions.

2.1. Importance of Internet Public Opinions Risk Management to Sustainability of Society

The management of internet public opinions plays an important role in maintaining social stability and harmony. The benefit and the need of internet public opinion risk management can be summarized as the following.

(1).

Internet public opinion risk management is beneficial for the response to emergency events. The evolution process of internet public opinions reflects the process of emergencies. The internet has become an important channel for public emergencies, and it also makes public opinions spread widely and uncontrollably [16]. Meanwhile, internet public opinion has a great impact on the solutions of public emergency events, judicial decisions, and even government decisions.

(2).

Internet public opinion risk management is conducive to the sustainable development of society. The internet is a platform on which people express their demands and views. Due to the openness, timeliness and participation of the internet platform and the distortion of information, the risk of emergencies is expanded in the internet public platform, even with rumors. It is easy to cause social mood fluctuations, even social panic, and this affects social stability. Meanwhile, positive public opinion promotes the sustainable and harmonious development of society [36]. Therefore, it is particularly necessary to intervene, control, restrain and guide internet public opinion.

(3).

Internet public opinion risk management effectively weakens the negative impact of emergencies. Guidance by governments determines the development of internet public opinions. It is an effective measure to control the leader of public opinions, like a well-known blogger, who greatly affects and guides the public opinion.

2.2. Process of Internet Public Opinions Formation

The formation process of an internet public opinion is shown in Figure 1. The internet contains a collection of cognitions, attitudes, emotions and behavioral tendencies of people who spread opinions regarding emergency events. Studying these opinions’ formation processes, therefore, is of great significance.

The internet promotes quick information transmission when emergencies occur. An emergency event promotes internet public opinions, and the internet expands public opinions. However, users cannot obtain full information about emergency events because of information asymmetry. Meanwhile, users carry different educational backgrounds, information sources, and values; thus, their attitudes to emergency events are different. Negative public opinions, neutral public opinions and positive public opinions can all be produced at the occurrence of an emergency event. These opinions are then disseminated by the internet, making global internet public opinions. Meanwhile mainstream views influence global internet public opinions and play an important role in government decision making. The attitude and decision of governments to emergencies affect their credibility. A failure of an official response, improper media response or improper media reports can bring negative effects to public ecology and social harmony. Meanwhile, the internet speeds up the spread of mainstream ideas. Therefore, it is important to classify the risk levels of emergency internet public opinions in time, a classification which is also a good way to promote the governance of internet.

2.3. Conceptual Model of Internet Public Opinion Risk Grading

The conceptual model of internet public opinion risk grading is shown in Figure 2, and this model includes two-stages and double loops. The two-stages are the preparation stage and the field work stage. The double loops contain reference point update and information update.

In the preparation stage, the main target is to obtain the experience curve of every criterion. A group of experts are selected first, and then the criteria are selected by brainstorming and text analysis approaches. The weights of criteria are further determined by the experts. The public opinion about an emergency event is monitored in real time, which provides evaluation information for each criterion. In the fieldwork stage, the goal is to grade the risk of internet public opinion. We can determine the risk levels of internet public opinions based on the information of the emergency event and the experience curve of each criterion. Note that the risk levels of internet public opinions toward an emergency event can be evaluated in real time.

As shown in Figure 2, the updates of reference points and emergency information are interrelated. In the reference point update loop, reference points can be selected according to text analysis, expert discussions, historical case studies, brainstorming or the types of current emergencies. The emergency internet public opinion risk level grading system can deal with uncertainty and complexity. With advances of internet technology, the reference point and profile point can be adjusted belong to the emergency event.

The grading of internet public opinion risk levels is critical in developing early warnings of internet public opinion risk. The steps are as follows.

3. Internet Public Opinion Intelligent Grading Based on AHPSort II-DEMATEL

The Decision-Making Trial and Evaluation Laboratory (DEMATEL) is built on the basis of graph theory and is best suited for analyzing complex structural models [37,38]. It can effectively assess total relations and construct a map between system components (criterion). AHPSort II [39] is a new multi-criteria sorting method for a high number of alternatives which extends from the Analytic Hierarchy Process (AHP) method [40]. It is a good technique to reduce the complexity of calculation. According to the conceptual model of internet public opinion governance, we propose an intelligent grading approach whose steps are as follows.

3.1. Definition of Internet Public Opinion Grading Problem

Step 1. Define the risk levels of internet public opinions.

The risk levels are defined as very high risk ($L_1$), high risk ($L_2$), low risk ($L_3$), and very low risk ($L_4$), as shown in Figure 3, and the description of the risk levels is shown in

Table 1. Risk level of internet public opinion.

	Risk Level	Description
Very High (VH)	I	Public safety and social order pose a serious threat. The public opinion which is concerned by domestic and foreign media involves negative public opinion and has serious effect. The internet users of most countries know the emergency event.
High (H)	II	The emergency causes the public opinion which mainly refers to the harm and influence to the public safety and social order in a large scope. The domestic media and communication platform APPs involve a great deal of negative public opinion. Internet users in China and a few international countries know the emergency event.
Low (L)	III	Public safety and social order are threated in a certain scope. A small number of domestic media and websites follow the emergency event. The negative public opinion refers to the emergency event effecting the public emotion. The emergencies are well known in China.
Very Low (VL)	IV	This mainly refers to the general harm and influence to the public safety and social order in this county in a small scale. A few media and websites understand the emergency event, and the negative public opinion will be forgotten over time. The emergencies are known in an area.

. $\left\{l{p}_t^i|t=1,2,3;i=1,2,\cdots ,m\right\}$ is the set of limiting profiles of criterion $C_i$, which can be defined by experts based on the risk levels. If the emergency internet public opinion risk value $a_k$ is assigned to the level $L_i$, then

$$\begin{array}{lll}{p}_k\ge l{p}_1& \Rightarrow & a_k\in L_1;\\ l{p}_2\le p_k\le l{p}_1& \Rightarrow & a_k\in L_2;\\ l{p}_3\le p_k\le l{p}_2& \Rightarrow & a_k\in L_3;\\ p_k\le l{p}_3& \Rightarrow & a_k\in L_4.\end{array}$$

Let $\left\{E_x|x=1,2,\cdots ,p\right\}$ be the set of experts, and the weights of experts are given as $W_E=\left\{w_{E_1},w_{E_2},\cdots ,w_{E_p}\right\}$. $\left\{C_i|i=1,2,\cdots ,m\right\}$ are the sets of criteria with respect to the internet public opinion risk grading problem.

Step 2. Determine references points $r_j^i$ for each criterion.

AHPSort II requires experts to determine the reference points in advance; this is a vital step of the method. Different reference points can have a great influence on drawing the judgment curve. There are two ways to define the reference points $r_j^i$$(i=1,2,\cdots ,m;j=1,2,\cdots ,n)$ for each criterion $C_i$$(i=1,2,\cdots ,m)$.

(1).

Based on the similarity case of the emergency event, select the reference points for criterion $C_i$, as shown in Figure 4. The number of reference points N can be given by the top $t\%$ of similarity cases, or $N=N$ (the number of the case).

(2).

Determine the maximum and minimum values of each criterion, and the number of reference points N can be defined by experts. The reference points can be determined by equal distance (see Figure 5).

3.2. Definition of Internet Public Opinion Grading Problem

Step 3. The expert pairwise compares the criteria by linguistic term and transform the linguistic terms to their corresponding triangular fuzzy sets (TFS) based on

Table 2. Linguistic terms and their corresponding triangular fuzzy sets (TFS).

Linguistic Variables	Triangular Fuzzy Set
No influence (N)	(0,0,0)
Very Low influence (VL)	(0,0,0.3)
Low influence (L)	(0,0.25,0.5)
Moderate Influence (M)	(0.3,0.5,0.7)
High influence (H)	(0.5,0.75,1)
Very high influence (VH)	(0.7,1,1)

.

Step 4. Aggregate the experts’ points by Equation (1) and obtain the comprehensive evaluation matrix $Z$, when $i=j$, $z_{ij}=\left(\left(0,0,0\right),1,0\right)$.

$$z_{ij}={\displaystyle \sum _{\alpha =1}^m{w}_{\alpha }\left({\underset{\_}{y}}_{ij}^{\alpha },y_{ij}^{\alpha },{\overline{y}}_{ij}^{\alpha }\right)}$$

(1)

Step 5. Calculate the total relation matrix. Due to the comprehensive evaluation matrix $Z$, the total relation matrix can be derived by Equation (2), which contains direct and indirect relations between the factors.

$$T=\underset{k\to \infty }{\lim }\left(N+N^2+\cdots +N^k\right)=N{\left(1-N\right)}^{-1}$$

(2)

$$\begin{array}{ll}T& =\underset{k\to \infty }{\lim }(Z+Z^2+\cdots +Z^k)\hfill \\ & =\underset{k\to \infty }{\lim }\left({\displaystyle \sum _{n=1}^k{\underset{\_}{z}}_{ij}^n},{\displaystyle \sum _{n=1}^k{z}_{ij}^n},{\displaystyle \sum _{n=1}^k{\overline{z}}_{ij}^n}\right)\\ & =\left(\frac{{\underset{\_}{z}}_{ij}}{1-{\underset{\_}{z}}_{ij}},\frac{z_{ij}}{1-z_{ij}},\frac{{\overline{z}}_{ij}}{1-{\overline{z}}_{ij}}\right)\end{array}$$

(3)

Proof. 

$\underset{k\to \infty }{\lim }{\displaystyle \sum _{n=1}^k{z}_{ij}^n}=\underset{k\to \infty }{\lim }\left(z_{ij}^1+z_{ij}^2+\cdots +z_{ij}^k\right)=\underset{k\to \infty }{\lim }\frac{z_{ij}(1-z_{ij}^k)}{1-z_{ij}}$, where $z_{ij}\in \left(0,1\right)$ and $\underset{k\to \infty }{\lim z_{ij}^k}=0$. Therefore $\underset{k\to \infty }{\lim }{\displaystyle \sum _{n=1}^k{z}_{ij}^n}=\frac{z_{ij}}{1-z_{ij}}$. □

Step 6. Construct the DEMATEL map. Based on the total relation matrix, the sum of each row ($i=1,2,\cdots ,n$) and column $C_i$($i=1,2,\cdots ,n$) are calculated based on Equations (4) and (5). Then, $\left(R_i+C_i\right)$ and $\left(R_i-C_i\right)$ can be obtained by Equations (6) and (7). $p_i=R_i+C_i$ indicates the impact of i-th influential factor and its degree of being impacted. $q_i=R_i-C_i$ is defined as the importance of factors and classifies the i-th influential factor into the cause or effect category in the researching system.

$$R_i={\displaystyle \sum _{j=1}^n\left(T_{ij}\right)}={\displaystyle \sum _{j=1}^n\left({\underset{\_}{t}}_{ij},t_{ij},{\overline{t}}_{ij}\right)}$$

(4)

$$C_i={\displaystyle \sum _{i=1}^n\left(T_{ij}\right)}={\displaystyle \sum _{i=1}^n\left({\underset{\_}{t}}_{ij},t_{ij},{\overline{t}}_{ij}\right)}$$

(5)

$$p_i=R_i+C_i={\displaystyle \sum _{j=1}^n\left(T_{ij}\right)}+{\displaystyle \sum _{i=1}^n\left(T_{ij}\right)}={\displaystyle \sum _{j=1}^n\left({\underset{\_}{t}}_{ij},t_{ij},{\overline{t}}_{ij}\right)}+{\displaystyle \sum _{i=1}^n\left({\underset{\_}{t}}_{ij},t_{ij},{\overline{t}}_{ij}\right)}$$

(6)

$$q_i=R_i-C_i={\displaystyle \sum _{j=1}^n\left(T_{ij}\right)}-{\displaystyle \sum _{i=1}^n\left(T_{ij}\right)}={\displaystyle \sum _{j=1}^n\left({\underset{\_}{t}}_{ij},t_{ij},{\overline{t}}_{ij}\right)}-{\displaystyle \sum _{i=1}^n\left({\underset{\_}{t}}_{ij},t_{ij},{\overline{t}}_{ij}\right)}$$

(7)

Step 7. Obtain the weights of criteria by the value of R + C.

$$W_i=\frac{p_i}{{\displaystyle \sum _{i=1}^n{p}_i}}$$

(8)

3.3. Judgment Curve of Internet Public Opinion Grading

Step 8. Pairwise compare the limiting profiles with the reference points for each criterion $C_i$. In this step, we can reduce the number of pairwise comparisons using the method by Miccoli and Ishizaka [33]. Assume that the number of reference points is N and the references points are divided to $k$ clusters. The priorities are calculated for the $k$ clusters, as shown in Figure 6.

The priorities are calculated for the k clusters. Local priorities of the latter clusters are linked to the previous one by multiplying them by the ratio of the scores of joining points. The clusters are shown in Figure 7.

4. Application of DEMATEL-AHPSort II for Internet Public Opinion Grading

The internet public opinion can be classified into political public opinion, economic public opinion, cultural public opinion, and social public opinion [41]. The social public opinion is the most common internet public opinion, and we thus present a case on social public opinion risk grading. In this section, we first present an illustrative example of the performance of the proposed DEMATEL-AHPSort II for internet public opinion grading problem and then conduct further discussion about our proposed method.

4.1. Problem Description

China is particularly vulnerable to earthquakes and has suffered a high number of earthquake casualties [42]. From 2004 to 2018, 446 damaging earthquakes with Ms > 5.0 magnitude have occurred in China. Over the past 15 years, earthquakes have brought great economic losses. Sichuan Province of China is located in the southwestern seismic region, which is a prominent population and economic growth center. Meanwhile, Sichuan Province is an earthquake-stricken region, with one of the most dangerous cities in China. The regional distribution of earthquakes in Sichuan Province is shown in Figure 8. The red in the map of Sichuan Province represents the area where a fatal earthquake with $Ms\ge 7.0$ magnitude has occurred and caused serious economic losses. The orange represents the area where earthquakes have occurred with $5\le Ms<7.0$ magnitude, and the white represents the area where earthquakes have occurred with $Ms<5.0$ magnitude or no earthquake.

Yibin City is located at the junction of Sichuan Province, Yunnan Province and Guizhou Province. There are many mountains around Yibin City, like the Qixing, Longtou and Guandou Mountains. At 22:55 on 17 June 2019, an Ms = 6.0 magnitude earthquake occurred in Yibin City. The internet public opinions about the Yibin earthquake arose from 17 June to 19 June. We should take more effective and efficient measures to cope with internet public opinions propagation. The hot degree of internet public opinions with time is shown in Figure 9.

We can determine the risk level of internet public opinions at 2 p.m. on 18 June 2019. In this paper, the evaluation system was constructed by some previous studies [35], and the evaluation criteria were selected by expert’s brainstorming, as shown in

Table 3. Evaluation criteria.

Criteria	Description
Scale ($C_1$)	The time of the impact, the spread of the crowd and the range of areas involved.
Negative evaluation ($C_2$)	The more intense the negative evaluation of the public opinion of the incident network is, the faster the negative emotion public opinion spreads; the greater the influence spreads, the greater the influence scope is and the easier the network public opinion crisis happens.
Secondary hazards ($C_3$)	The secondary harm caused by the emergency network public opinion, such as the panic of the crowd, the harm to the personnel involved in the matter, and even the larger network public opinion crisis.
Moral perception ($C_4$)	Users feel an emotional response in posting or forwarding emergency information, such that they are meeting ethical expectations they have for themselves or toward others.

.

Next, we will introduce our method step by step, according to Section 4.

Step 1. The risk levels of internet public opinions are shown in Table 1. There are four experts participating in the evaluation $E=\left\{E_1,E_2,E_3,E_4\right\}$, and the weight of experts are given as $W_E=\left\{0.3,0.25,0.25,0.2\right\}$.

Step 2. Select the reference points by equal distance. Determine the minimum and maximum value of each criterion and then divide it by ten reference points, in which $r_0=minimum$, $r_{10}=maximum$. Take $C_1$ as an example, and the reference points can be obtained as shown in

Table 4. Reference points of $C_1$.

	Min	Max	${\mathit{r}}_1$	${\mathit{r}}_2$	${\mathit{r}}_3$	${\mathit{r}}_4$	${\mathit{r}}_5$	${\mathit{r}}_6$	${\mathit{r}}_7$	${\mathit{r}}_8$	${\mathit{r}}_9$	${\mathit{r}}_{10}$
$C_1$	0	1	0.1	0.2	0.3	0.4	0.5	0.6	0.7	0.8	0.9	1

.

4.2. Determination of Criteria Weights

Step 3. Establish the comparison matrices by the experts $E=\left\{E_1,E_2,E_3,E_4\right\}$, which are shown in

Table 5. Decision matrices of criteria by experts.

$E_1$	$C_1$	$C_2$	$C_3$	$C_4$	$E_2$	$C_1$	$C_2$	$C_3$	$C_4$
$C_1$	N	H	VL	L	$C_1$	N	H	M	VL
$C_2$	H	N	VH	M	$C_2$	L	N	L	H
$C_3$	L	M	N	H	$C_3$	L	M	N	M
$C_4$	VH	L	M	N	$C_4$	VL	VH	M	N
$E_3$	$C_1$	$C_2$	$C_3$	$C_4$	$E_4$	$C_1$	$C_2$	$C_3$	$C_4$
$C_1$	N	L	M	H	$C_1$	N	VH	H	VH
$C_2$	VL	N	L	VH	$C_2$	M	N	H	VL
$C_3$	H	H	N	M	$C_3$	L	VH	N	M
$C_4$	L	VH	M	N	$C_4$	M	H	L	N

. Then, transform the linguistic variable to triangular fuzzy number (TIFN).

Step 4. Obtain the comprehensive evaluation matrix.

$$\begin{array}{cccc}& & \begin{array}{cccc}{C}_1& C_2& C_3& C_4\end{array}& \\ \begin{array}{c}{C}_1\\ C_2\\ C_3\\ C_4\end{array}& |& \begin{array}{cccc}(0,0,0)& (0.71,2.08,7)& (0.33,0.67,1.78)& (0.36,0.86,2.08)\\ (0.27,0.63,1.78)& (0,0,0)& (0.45,1.35,3)& (0.64,1.42,3.35)\\ (0.14,0.6,1.67)& (0.75,1.96,5.06)& (0,0,0)& (0.56,1.35,3.76)\\ (0.37,0.86,1.78)& (0.82,2.64,5.67)& (0.32,0.82,1.94)& (0,0,0)\end{array}& |\end{array}$$

Step 5. Normalize the comprehensive evaluation matrix.

$$\begin{array}{cccc}& & \begin{array}{cccc}{C}_1& C_2& C_3& C_4\end{array}& \\ \begin{array}{c}{C}_1\\ C_2\\ C_3\\ C_4\end{array}& |& \begin{array}{cccc}(0,0,0)& (0.10,0.30,1)& (0.05,0.10,0.25)& (0.05,0.12,0.30)\\ (0.04,0.09,0.25)& (0,0,0)& (0.06,0.19,0.43)& (0.09,0.20,0.48)\\ (0.02,0.09,0.24)& (0.11,0.28,0.72)& (0,0,0)& (0.08,0.19,0.54)\\ (0.05,0.12,0.25)& (0.12,0.38,0.81)& (0.05,0.12,0.28)& (0,0,0)\end{array}& |\end{array}$$

Step 6. Calculate the value of R, C, R + C and R − C by Equations (22) and (23), as shown in

Table 6. Value of R, C, R + C and R − C.

	R	C	R + C	R − C
$C_1$	(0.20,0.51,1.55)	(0.11,0.30,0.75)	(0.72,0.81,2.30)	(0.09,0.22,0.80)
$C_2$	(0.19,0.49,1.16)	(0.33,0.95,2.53)	(0.68,1.44,3.69)	(−0.13,−0.47,−1.37)
$C_3$	(0.21,0.56,1.50)	(0.16,0.41,0.96)	(0.77,0.96,2.46)	(0.05,0.15,0.54)
$C_4$	(0.21,0.62,1.34)	(0.22,0.52,1.31)	(0.83,1.14,2.65)	(−0.01,0.10,0.03)

.

Step 7. Transform the TIFN to an accurate value by Equations (9) and (10), as shown in

Table 7. Decision matrices of criteria by experts.

	${\mathit{C}}_1$	${\mathit{C}}_2$	${\mathit{C}}_3$	${\mathit{C}}_4$
R + C	0.05	0.26	0.09	0.15
R − C	0.545	−0.12	0.36	0.198

. Then, the weights of criteria are obtained based on the value of R + C.

Then, we can get the weights of criteria $W=(0.09,0.47,0.17,0.27)$.

4.3. Acquirement of Judgment Curve of Internet Public Opinion Grading

Step 8. Pairwise compare the limiting profiles with reference points. Using $C_1$ as an example; the reference points are shown in Table 4. Cluster reference points and limiting profiles are shown in Figure 10. Three clusters of “Scale ($C_1$)” are shown in

Table 8. First cluster of “Scale ($C_1$)”.

	${\mathit{r}}_1^1$	${\mathit{r}}_2^1$	${\mathit{r}}_3^1$	${\mathit{r}}_4^1$	Local Priority
$r_1^1$	1	1/2	1/3	1/7	0.0654
$r_2^1$	2	1	1/5	1/6	0.0869
$r_3^1$	3	5	1	1/4	0.2427
$r_4^1$	7	6	4	1	0.6050

,

Table 9. Second cluster of “Scale ($C_1$)”.

	${\mathit{r}}_4^1$	${\mathit{r}}_5^1$	${\mathit{r}}_6^1$	${\mathit{r}}_7^1$	Local Priority
$r_4^1$	1	1/3	1/6	1/8	0.0476
$r_5^1$	3	1	1/4	1/7	0.0935
$r_6^1$	6	4	1	1/3	0.2716
$r_7^1$	8	7	3	1	0.5873

and

Table 10. Third cluster of “Scale ($C_1$)”.

	${\mathit{r}}_4^1$	${\mathit{r}}_5^1$	${\mathit{r}}_6^1$	${\mathit{r}}_7^1$	Local Priority
$r_4^1$	1	1/2	1/7	1/9	0.0449
$r_5^1$	2	1	1/4	1/6	0.0793
$r_6^1$	7	4	1	1/5	0.2359
$r_7^1$	9	6	5	1	0.6398

.

Step 9. Obtain the local priority and normalized local priorities. Taking the $C_1$ as an example, the result of normalized local priorities are shown in

Table 11. Integrated result of normalized local priorities of “$C_1$ Scale”.

Points	Local Priority	Normalized Local Priorities
$r_1^1$	0.0654	0.0006
$r_2^1$	0.0869	0.0008
$r_3^1$	0.2427	0.0023
$r_4^1$	0.6050/0.0476	0.0057
$r_5^1$	0.0935	0.0112
$r_6^1$	0.2716	0.0325
$r_7^1$	0.5873/0.0449	0.0702
$r_8^1$	0.0793	0.1239
$r_9^1$	0.2359	0.3687
$r_{10}^1$	0.6398	1.0000

.

Step 10. Derive the local priority of each criterion and then obtain the global priority. The local priority curve of $C_1$ is shown in Figure 11.

4.4. Risk Grading

Step 11. Collect information about the event and obtain the values of the criteria. According to Step 3 and Step 7, we can grade internet public opinion risk levels 2 p.m. 17 June 2019.

4.5. Managerial Suggestions

The results provide some suggestions to internet public opinion risk management for managers and government officials, detailed as the following:

(1).

Guide internet public opinion by opinion leaders. An opinion leader is the core of an internet public opinion who disseminates internet public opinions. They can be selected based on user influence and user activity. Policy makers can lead public opinion to the positive direction and reduce risk by controlling the opinion guidance of opinion leaders [43].

(2).

Construct and quantify a criteria system. A standard and systematic criteria system is necessary for risk grading. The quantitative evaluation of an internet public opinion criteria system is the basic premise of efficiently carrying out mass internet public opinion information monitoring and early warning.

(3).

Monitor active internet public opinions by machine learning. An active opinion is the basis for forming an internet public opinion or rumor. We should focus on the risk levels and pay attention to the threshold, and we should monitor the hot opinions in real time. Advance control and timely interventions are necessary when a hot opinion is beyond the threshold.

The suggestions are provided to internet users who are the main body of internet public opinion propagation as following:

(1).

Do not make a comment easily when you do not know the facts of an emergency event. Emergencies are prone to rumors which have a negative impact on emergency management. Secondary damage will be caused by rumors.

(2).

Do not follow the opinion leaders and obey their opinions about emergencies. You should have your own opinion when you know the facts of emergencies, and you should express your opinion in the right way.

5. Conclusions

The internet is an open platform with high liquidity, accessibility and concealment for the public to express their respective views on certain events. Therefore, it is vital to form an effective response and management system to classify and grade risk levels. In this paper, the mechanism of internet public opinion formation was proposed, and it was found that the internet plays a vital role in promoting the dissemination of public opinions. The attitude and decision of the government have important influences on the development of internet public opinion. Additionally, a conceptual model of internet public opinion, two stages and double loops, was built. Next the AHPSort II-DEMATEL approach was proposed to grade the risk levels of internet public opinions. Finally, an application of the proposed method to grade the risk levels of internet public opinions was provided to demonstrate the effectiveness of the proposed method. This research of internet public opinion risk management can greatly promote stability and harmony, both of which are of great significance to ensure the sustainable development of society.

However, there are also some limitations of this research work: (1) The decision-making situation in a variety of language environments was not considered. (2) The uncertainty of expert weight was not taken into account.

For future research, we aim to first reduce the impact of the approximation—we will pay more attention to the proportional relationship between the number of reference points and data scales to guarantee highest possible precision, along with fewer comparisons in AHPSort II. Then, in order to expand methods for grading the risk level of internet public opinion, we will expand the DEMATEL-AHPSort II to different linguistic environments. These aspect will all be considered in future research.