Constructing Fuzzy Hypothesis Methods to Determine Critical-To-Quality Service Items

Abstract

This paper constructs a performance evaluation matrix (PEM) with beta distribution. Beta is between zero and one, making it a suitable indicator to describe customer ratings of importance and satisfaction from 0% to 100%. According to the spirit of ceaseless improvement put forward by total quality management, the average ratings are set as the standard, and then the coordinates of each satisfaction and importance item is located in the performance areas. This makes it easy to identify critical-to-quality items that require improvement. However, the data collection method of questionnaires inevitably involves sampling error, and the opinions of customers are often ambiguous. To solve these problems, we constructed a fuzzy testing method based on confidence intervals. The use of confidence intervals decreases the chance of misjudgment caused by sampling errors, and more precisely gets closer to customers’ voices. This result is more reasonable than the traditional statistical testing principle. The proposed methods are applied to assessment of a computer-assisted language learning (CALL) system to display their competence.

1. Introduction

Lambert and Sharma [1] presented the performance evaluation matrix (PEM) for operating systems that collect users’ or customers’ perceptions. Compared with other assessment methods that need complicated data comparison, the PEM makes it easy to determine which service items most urgently require improvement, maintenance, or adjustment. This is achieved by locating the items according to (1) customer satisfaction with the items as they are and (2) how important the customers deem them. The PEM is widely applied for performance evaluation and improvement in a range of industries and institutions [2,3,4,5,6,7,8]. In the PEM, customer perception of the importance of an item is represented by the vertical axis, and customer satisfaction with the item itself is represented by the horizontal axis. Each axis is cut into three equal shares, forming a three by three matrix with nine performance blocks. The three performance areas on the diagonal are regarded as maintenance areas because their importance and satisfaction are equal; the three performance areas on the upper-left are regarded as improvement areas because their importance level is higher than customers’ satisfaction level; and the three areas on the lower-right are viewed as adjustment areas because their satisfaction rating is higher than their importance.

In order to make the PEM more applicable, Hung, Huang, and Chen [9] altered the position of the performance areas in the PEM to rationalize the evaluation principles. They divided the PEM into three equally-sized performance areas. The three performance areas are the following: the upper-left one is improvement, the middle one is maintenance, and the lower-right one is the adjustment area. Satisfaction and importance are viewed as random variables and submitted to the beta allocation. The importance and satisfaction indices are thereby standardized with values between zero and one. Hung, Huang, and Chen [9] also suggested that the conventional method of the PEM was subject to sampling error. They therefore proposed deriving the joint confidence interval of the importance and satisfaction indices based on the central limit theorem and replacing the point estimates of the PEM with these joint confidence intervals. Although this method overcomes sampling error, it is complicated and difficult to use [4,6,9].

In addition to sampling error, another limitation encountered by the PEM is that customer or user opinions are often fuzzy [6,10,11,12]. Therefore, Wang et al. [6] proposed a fuzzy semantic scale for practical application and also constructed a convenient and comprehensive calculation method. However, while this enables the collection of data that is closer to the feelings of the interviewees, it increases the quantity of data as well as the complexities of collection. Consequently, Yu, Chang, and Chen [13] as well as Chen et al. [14] revised and simplified the data collection and calculation of the fuzzy semantic scales. Their modifications include the Likert scale, which is easy to use, but data collection for the fuzzy semantic scale remains relatively complicated.

In order to overcome the above limitations, we check the fuzzy testing method of Buckley [15], Chen et al. [16], Lee et al. [17], and Sarkar et al. [18]. The method in this paper was extended from the ones in the stated-above three papers. It is based on calculated confidence intervals to construct the triangular shaped fuzzy number similar to the triangular fuzzy number mentioned by Sarkar et al. [18] and meets the requirements. However, values of the two points of the triangular fuzzy number are fixed, ones of the two points of triangular shaped fuzzy number are changeable. Besides, industries consider the cost benefit and control the timeliness and effectiveness, their samples are not too numerous [19]. The method in this paper utilizes confidence intervals as the foundation. It not only reduces the risk of misjudgment arising from sampling errors, but also makes the more accurate decision. First, the satisfaction index of each service item is checked to determine whether it is below the mean (i.e., requiring improvement). Second, the importance index of items requiring improvement is tested to determine whether they are above the mean. In cases of limited resources, service items that are the most important are prioritized for improvement. The strengths of Buckley’s fuzzy test are that it maintains the plain format of the Likert scale while deducing the confidence interval from the importance and satisfaction indices to create the fuzzy membership function of the two indexes; and through fuzzy hypothesis testing, it distinguishes items regarded critical to quality. We demonstrate the efficacy of the method through the presentation of a case study of a computer-assisted language learning (CALL) system.

The rest of this paper is organized as the following. Section 2 illustrates the theoretical model developed by Hung et al. [9], including definition of two evaluation indexes and their properties. Section 3 refers to the confidence interval of the satisfaction index in Section 2 to create a fuzzy membership function. A fuzzy testing criterion is proposed to evaluate whether the satisfaction index of each service item is below the mean. In Section 4, we construct the fuzzy membership function from the confidence interval of the importance index defined in Section 2 and propose a fuzzy testing criterion to evaluate whether the importance index is above the mean. A case study of a CALL system illustrates employment of the method in Section 5. In Section 6, we present our conclusions.

2. Performance Indices

For the sake of generality, we follow the example of Hung et al. [9] in this study and assume that there are q service items and each service item is measured by two questions (one for importance and one for satisfaction). This creates a total of 2 q questions. Hung et al. [9] also assumed that these ratings follow beta distributions. Therefore, this study takes random variables $X_i~Beta\left(a_i,b_i\right)$ and $Y_i~Beta\left(a_i^{\prime },b_i^{\prime }\right)$ respectively to represent the distributions associated with importance and satisfaction. The importance and satisfaction indices can be shown as the following:

$${\theta }_i=\frac{a_i}{a_i+b_i}\left(\mathrm{Satisfaction} \mathrm{Index}\right),$$

(1)

$${\theta }_i^{\prime }=\frac{a_i^{\prime }}{a_i^{\prime }+b_i^{\prime }}\left(\mathrm{Importance} \mathrm{Index}\right).$$

(2)

As noted by Hung et al. [9], index $\theta$ is between 0 and 1. More than half of customers are satisfied if index ${\theta }_i$ exceeds 0.50, when more than half of customers are dissatisfied if index ${\theta }_i$ is below 0.50. Thus, higher values for index $\theta$ reflect higher levels of customer satisfaction. Index ${\theta }_i^{\prime }$ behaves similarly. Index ${\theta }_i$ is used as the horizontal axis and index ${\theta }_i^{\prime }$ as the vertical axis to construct the performance evaluation matrix. Since the indices have unknown parameters, they must be estimated from sample data. If n customers are interviewed, the sample data of satisfaction and importance can be shown as the following:

Satisfaction sample matrix: $\left(\begin{array}{lllll}{X}_{1,1},& \cdots ,& X_{1,j},& \cdots ,& X_{1,n}\\ ⋮& & ⋮& & ⋮\\ X_{i,1},& \cdots ,& X_{i,j},& \cdots ,& X_{i,n}\\ ⋮& & ⋮& & ⋮\\ X_{q,1},& \cdots ,& X_{q,j},& \cdots ,& X_{q,n}\end{array}\right)$,

Importance sample matrix: $\left(\begin{array}{lllll}{Y}_{1,1},& \cdots ,& Y_{1,j},& \cdots ,& Y_{1,n}\\ ⋮& & ⋮& & ⋮\\ Y_{i,1},& \cdots ,& Y_{i,j},& \cdots ,& Y_{i,n}\\ ⋮& & ⋮& & ⋮\\ Y_{q,1},& \cdots ,& Y_{q,j},& \cdots ,& Y_{q,n}\end{array}\right)$.

The estimator of satisfaction index ${\theta }_i$ and the estimator of importance index ${\theta }_i^{\prime }$ can be shown separately, as the following:

$${\theta }_i^{*}=\frac{1}{n}\times {\displaystyle \sum _{j=1}^n{X}_{ij}}.$$

(3)

and

$${\theta }_i^{\prime }{}^{*}=\frac{1}{n}\times {\displaystyle \sum _{j=1}^n{Y}_{ij}}$$

(4)

In addition, the standard deviation of these two indices can be expressed respectively, as the following:

$$S_i=\sqrt{\frac{1}{n-1}\times {\displaystyle \sum _{j=1}^n{\left(X_{ij}-{\overline{X}}_i\right)}^2}}.$$

(5)

and

$$S_i^{\prime }=\sqrt{\frac{1}{n-1}\times {\displaystyle \sum _{j=1}^n{\left(Y_{ij}-{\overline{Y}}_i\right)}^2}}.$$

(6)

The expected values of ${\theta }_i$ and ${\theta }_i^{\prime }$ can be expressed respectively, as the following:

$$E\left({\theta }_i^{*}\right)=E\left({\overline{X}}_i\right)=\frac{1}{n}\times {\displaystyle \sum _{j=1}^{n}E\left(X_{ij}\right)}={\theta }_i,$$

(7)

where

$$\begin{array}{ll}E\left(X_{ij}\right)& ={\displaystyle \underset{0}{\overset{1}{\int }}xBeta\left(a_i,b_i\right)d_x}\\ & =\frac{\Gamma \left(a_i+b_i\right)\Gamma \left(a_i+1\right)}{\Gamma \left(a_i+b_i+1\right)}\times {\displaystyle \underset{0}{\overset{1}{\int }}Beta\left(a_i+1,b_i\right)d_x}\\ & ={\theta }_i. \end{array}$$

(8)

Similarly,

$$E\left({\theta }_i^{\prime }{}^{*}\right)=E\left({\overline{Y}}_i\right)=\frac{1}{n}\times {\displaystyle \sum _{j=1}^{n}E\left(Y_{ij}\right)}={\theta }_i^{\prime }$$

(9)

where

$$E\left(Y_{ij}\right)={\displaystyle \underset{0}{\overset{1}{\int }}yBeta\left(a_i^{\prime },b_i^{\prime }\right)d_y={\theta }_i^{\prime }.}$$

(10)

Obviously, ${\theta }_i^{*}$ and ${\theta }_i^{\prime }{}^{*}$ are unbiased estimators of ${\theta }_i$ and ${\theta }_i^{\prime }$ separately.

If we let

$$Z_i=\frac{{\theta }_i^{*}-{\theta }_i}{S_i/\sqrt{n}}.$$

(11)

and

$$Z_i^{\prime }=\frac{{\theta }_i^{\prime }{}^{*}-{\theta }_i^{\prime }}{{S^{\prime }}_i/\sqrt{n}},$$

(12)

then by the Central Limits Theorem (CLT), $Z_i$ and $Z_i^{\prime }$ are distributed as $N\left(0,1\right)$ for $n\to \infty$; that is

$$Z_i \underset{}{\overset{n\to \infty }{\to }}N\left(0,1\right) \mathrm{and} Z_i^{\prime } \underset{}{\overset{n\to \infty }{\to }}N\left(0,1\right).$$

3. Fuzzy Hypothesis Testing for Satisfaction Index

If we let $X_{ij}=x_{ij}$, then the data matrix of random variable $X_{ij}$ can be shown as

$$\left(\begin{array}{lllll}{X}_{1,1},& \cdots ,& X_{1,j},& \cdots ,& X_{1,n}\\ ⋮& & ⋮& & ⋮\\ X_{i,1},& \cdots ,& X_{i,j},& \cdots ,& X_{i,n}\\ ⋮& & ⋮& & ⋮\\ X_{q,1},& \cdots ,& X_{q,j},& \cdots ,& X_{q,n}\end{array}\right)=\left(\begin{array}{lllll}{x}_{1,1},& \cdots ,& x_{1,j},& \cdots ,& x_{1,n}\\ ⋮& & ⋮& & ⋮\\ x_{i,1},& \cdots ,& x_{i,j},& \cdots ,& x_{i,n}\\ ⋮& & ⋮& & ⋮\\ x_{q,1},& \cdots ,& x_{q,j},& \cdots ,& x_{q,n}\end{array}\right)$$

then the observation values for the means, standard deviations, and ${\theta }_i^{*}$ of each satisfaction service item can be expressed as the following:

$${\overline{x}}_i=\frac{1}{n}\times {\displaystyle \sum _{j=1}^n{x}_{ij}},$$

(13)

$$s_i=\sqrt{\frac{1}{n-1}\times {\displaystyle \sum _{j=1}^n{\left(x_{ij}-{\overline{x}}_i\right)}^2}}.$$

(14)

$${\theta }_{i0}^{*}={\overline{x}}_i=\frac{1}{n}\times {\displaystyle \sum _{j=1}^n{x}_{ij}}.$$

(15)

Chen et al. [14] drew on the concept of ceaseless improvement promoted by total quality management and set ${\theta }_0$ as the mean for all the satisfaction index as the following:

$${\theta }_0=\frac{1}{q}{\displaystyle \sum _{i=1}^q{\theta }_{i0}^{*}}.$$

(16)

When the satisfaction index of service item i is higher than the mean (${\theta }_i\ge {\theta }_0$), service item i does not require improvement. The following hypothesis test is equivalent:

$$\begin{gathered} H_0: {\theta }_i\ge {\theta }_0 (no improvement needed), \\ {H}_1: {\theta }_i<{\theta }_0 (improvement needed). \end{gathered}$$

This leads to the following test statistic:

$$Z_i=\frac{\sqrt{n}\left({\theta }_i^{*}-{\theta }_0\right)}{s_i}$$

(17)

The critical region is $C_i=\left\{{\theta }_i^{*}\le C_{i0}\right\}$ where $C_{i0}$ is determined by

$$P\left\{{\theta }_i^{*}\le C_{i0}|{\theta }_i={\theta }_0\right\}=P\left\{Z_i\le \frac{\left(C_{i0}-{\theta }_0\right)}{s_i/\sqrt{n}}\right\}=\beta$$

Thus, the critical value $C_{i0}$ can be shown as the following:

$$C_{i0}={\theta }_0-\frac{s_i}{\sqrt{n}}\times z_{\beta }$$

(18)

where $\beta$ is the significance level and the decision rule is as the following:

(1) Reject $H_0$ if ${\theta }_{i0}^{*}\le C_{i0}$ (i.e., service item i needs improvement),

(2) Do not reject $H_0$ if ${\theta }_{i0}^{*}>C_{i0}$ (i.e., service item i does not need improvement).

Since $Z_i$ is distributed as $N\left(0,1\right)$ for $n\to \infty$,

$$\begin{gathered} 1-\alpha =P\left\{-z_{\alpha /2}\le \frac{\left({\theta }_i^{*}-{\theta }_i\right)}{s_i/\sqrt{n}}\le z_{\alpha /2}\right\} \\ =P\left\{{\theta }_i^{*}-z_{\alpha /2}\times \frac{s_i}{\sqrt{n}}\le {\theta }_i\le {\theta }_i^{*}+z_{\alpha /2}\times \frac{s_i}{\sqrt{n}}\right\} \end{gathered}$$

(19)

Since the observed value of ${\theta }_i^{*}$ is ${\theta }_{i0}^{*}$, the observed value of $100\left(1-\alpha \right)\%$ confidence intervals of ${\theta }_i$ is

$$\left[{\theta }_{i0}^{*}-z_{\alpha /2}\times \frac{s_i}{\sqrt{n}},{\theta }_{i0}^{*}+z_{\alpha /2}\times \frac{s_i}{\sqrt{n}}\right]$$

(20)

where $z_{\alpha /2}$ is the upper $\alpha /2$ quintiles of $N(0,1)$. According to Buckley [15] and Chen et al. [16], the $\text{α-cuts}$ of triangular fuzzy number ${\tilde{\theta }}_{i0}^{*}$ is

$${\tilde{\theta }}_{i0}^{*}\left[\alpha \right]=\{\begin{array}{l}\left[{\theta }_{i1}^{*}(\alpha ),{\theta }_{i2}^{*}(\alpha )\right]=\left[{\theta }_{i0}^{*}-z_{\alpha /2}\times \frac{s_i}{\sqrt{n}},{\theta }_{i0}^{*}+z_{\alpha /2}\times \frac{s_i}{\sqrt{n}}\right], 0.01\le \alpha \le 1\\ \left[{\theta }_{i1}^{*}(\alpha ),{\theta }_{i2}^{*}(\alpha )\right]=\left[{\theta }_{i0}^{*}-z_{0.005}\times \frac{s_i}{\sqrt{n}},{\theta }_{i0}^{*}+z_{0.005}\times \frac{s_i}{\sqrt{n}}\right], 0\le \alpha \le 0.01\end{array}$$

(21)

Obviously, when $\alpha =1$, then ${\theta }_{i1}^{*}\left(1\right)={\theta }_{i2}^{*}\left(1\right)={\theta }_{i0}^{*}$. Thus, the triangular fuzzy number of ${\tilde{\theta }}_{i0}^{*}$ is ${\tilde{\theta }}_{i0}^{*}=\left[L{\theta }_i,{\theta }_{i0}^{*},R{\theta }_i\right]$ where

$$L{\theta }_i={\theta }_{i0}^{*}-z_{0.005}\times \frac{s_i}{\sqrt{n}},$$

(22)

$$R{\theta }_i={\theta }_{i0}^{*}+z_{0.005}\times \frac{s_i}{\sqrt{n}}.$$

(23)

Then the fuzzy membership function of ${\tilde{\theta }}_{i0}^{*}$ is

$${\eta }_i(x)=\{\begin{array}{l}0, if x<{\theta }_{i0}^{*}-z_{0.005}\times \frac{s_i}{\sqrt{n}}\\ 2\times \left(1-\Phi \left(\frac{{\theta }_{i0}^{*}-x}{s_i/\sqrt{n}}\right)\right), if {\theta }_i^{*}-z_{0.005}\times \frac{s_i}{\sqrt{n}}\le x<{\theta }_{i0}^{*}\\ 1, if x={\theta }_{i0}^{*}\\ 2\times \left(1-\Phi \left(\frac{x-{\theta }_{i0}^{*}}{s_i/\sqrt{n}}\right)\right), if {\theta }_{i0}^{*}<x\le {\theta }_{i0}^{*}+z_{0.005}\times \frac{s_i}{\sqrt{n}}\\ 1, if {\theta }_{i0}^{*}+z_{0.005}\times \frac{s_i}{\sqrt{n}}<x\end{array}.$$

(24)

Similarly to ${\tilde{\theta }}_{i0}^{*}$, the $\text{α-cuts}$ of triangular fuzzy number ${\tilde{C}}_{i0}$ is

$${\tilde{C}}_{i0}\left[\alpha \right]=\{\begin{array}{l}\left[C_{i1}(\alpha ),C_{i2}(\alpha )\right]=\left[C_{i0}-z_{\alpha /2}\times \frac{s_i}{\sqrt{n}},C_{i0}+z_{\alpha /2}\times \frac{s_i}{\sqrt{n}}\right], 0.01\le \alpha \le 1\\ \left[C_{i1}(\alpha ),C_{i2}(\alpha )\right]=\left[C_{i0}-z_{0.005}\times \frac{s_i}{\sqrt{n}},C_{i0}+z_{0.005}\times \frac{s_i}{\sqrt{n}}\right], 0\le \alpha \le 0.01\end{array}$$

(25)

The triangular shaped fuzzy number ${\tilde{C}}_{i0}$ is ${\tilde{C}}_{i0}=\left(L{C}_i, C_{i0}, R{C}_i\right)$ where

$$L{C}_i=C_{i0}-z_{0.005}\times \frac{s_i}{\sqrt{n}},$$

(26)

$$R{C}_i=C_{i0}+z_{0.005}\times \frac{s_i}{\sqrt{n}}.$$

(27)

Then the fuzzy membership function of ${\tilde{C}}_{i0}$ is

$${\eta }_0(x)=\{\begin{array}{l}0, if x<C_{i0}-z_{0.005}\times \frac{s_i}{\sqrt{n}}\\ 2\times \left(1-\Phi \left(\frac{C_{i0}-x}{s_i/\sqrt{n}}\right)\right), if C_{i0}-z_{0.005}\times \frac{s_i}{\sqrt{n}}\le x<C_{i0}\\ 1, if x=C_{i0}\\ 2\times \left(1-\Phi \left(\frac{x-{\theta }_{i0}^{*}}{s_i/\sqrt{n}}\right)\right), if C_{i0}<x\le C_{i0}+z_{0.005}\times \frac{s_i}{\sqrt{n}}\\ 1, if C_{i0}+z_{0.005}\times \frac{s_i}{\sqrt{n}}<x\end{array}.$$

(28)

Subsequently, we can express ${\eta }_i(x)$ and ${\eta }_0(x)$ graphically as the following (see Figure 1):

Figure 1. Membership functions of ${\eta }_i(x)$ and ${\eta }_0(x)$.

Obviously,

$${\theta }_{i0}^{*}-z_{a/2}\times \frac{s_i}{\sqrt{n}}=C_{i0}\Rightarrow z_{a/2}=\frac{{\theta }_{i0}^{*}-C_{i0}}{s_i/\sqrt{n}}\Rightarrow a=2\left[1-\Phi \left(\frac{{\delta }_i}{s_i/\sqrt{n}}\right)\right],$$

(29)

where ${\delta }_i=C_{i0}-{\theta }_{i0}^{*}$. We let $j=\left[100\alpha \right]$ and

$${\delta }_{ij}=\left({\theta }_{i0}^{*}+z_{0.005j}\times \frac{s_i}{\sqrt{n}}\right)-\left({\theta }_{i0}^{*}-z_{0.005j}\times \frac{s_i}{\sqrt{n}}\right)=2z_{0.005j}\times \frac{s_i}{\sqrt{n}}.$$

(30)

Among these, $[•]$ represents the biggest integer which is less than or equal to $•$. If we let $A_{Ti}$ represent the total area under the graph of ${\tilde{\theta }}_i^{*}$, then $j=0,1,\dots ,100$ for $0\le \alpha \le 1$. $A_{Ti}$ is divided into 100 blocks similar to trapezoids and block j can be shown as the following:

$$A_{Tj}=\left\{\left(x,\alpha \right)|{\theta }_{i0}^{*}-z_{\alpha /2}\times \frac{s_i}{\sqrt{n}}\le x\le {\theta }_{i0}^{*}+z_{\alpha /2}\times \frac{s_i}{\sqrt{n}},0.01\times \left(j-1\right)\le \alpha \le 0.01\times j\right\}.$$

(31)

We also let

$${\gamma }_{ij}=C_{i0}-{\theta }_{i0}^{*}+z_{0.005j}\times \frac{s_i}{\sqrt{n}}={\delta }_i+z_{0.005j}\times \frac{s_i}{\sqrt{n}}.$$

(32)

Suppose $j=h$ such that ${\gamma }_{ih}=0$, then

$$z_{0.005h}=\frac{{\delta }_i}{s_i/\sqrt{n}}\Rightarrow 0.005h=1-\Phi \left(\frac{{\delta }_i}{s_i/\sqrt{n}}\right).$$

(33)

For the purposes of practical manipulation, since $h_i$ is an integer,

$$h_i=200\times \left[1-\Phi \left(\frac{{\delta }_i}{s_i/\sqrt{n}}\right)\right]=\left[100a_i\right].$$

(34)

If we let $A_{Ri}$ represent the area under the graph of ${\tilde{\theta }}_i^{*}$ but to the right of the vertical line through point $C_0$, then $j=0,1,\dots ,h_i$. $A_{Ri}$ is divided into h trapezoid-like blocks and block j can be expressed as the following:

$$A_{Rij}=\left\{\left(x,\alpha \right)|{\theta }_{i0}^{*}-z_{\alpha /2}\times \frac{s_i}{\sqrt{n}}\le x\le C_{i0},0.01\times \left(j-1\right)\le \alpha \le 0.01\times j\right\}, j=1,\dots ,h_i.$$

(35)

Now, we compute the area of $A_{Tij}$ and $A_{Rij}$ as

$$A_{Tij}=\left(\frac{{\delta }_{i(j-1)}+{\delta }_{ij}}{2}\right)\times (0.01), j=1,\dots ,100.$$

(36)

$$A_{Rij}=\left(\frac{{\gamma }_{i(j-1)}+{\gamma }_{ij}}{2}\right)\times (0.01), j=1,\dots ,h_i.$$

(37)

$A_T$ and $A_R$ can be shown as the following:

$$A_{Ti}={\displaystyle \sum _{j=1}^{100}{A}_{Tij}}$$

(38)

and

$$A_{Ri}={\displaystyle \sum _{j=1}^h{A}_{Rij}}.$$

(39)

As noted by Buckley [15], we may employ two numbers ($0 < {\varphi }_1<{\varphi }_2<0.5$) as the following:

(1) If $A_{Ri}/A_{Ti}\le {\varphi }_1$, then do not reject $H_0$ and infer that service item i does not need improvement (${\theta }_i\ge {\theta }_0$).

(2) If ${\varphi }_1\le A_{Ri}/A_{Ti}\le {\varphi }_2$, then make no decision on whether to reject/not reject.

(3) If $A_{Ri}/A_{Ti}\ge {\varphi }_2$, then reject $H_0$ and infer that service item i needs improvement (${\theta }_i<{\theta }_0$).

According to the above-mentioned evaluation rules, q service items can be subjected to fuzzy hypothesis testing to find all service items requiring improvement.

4. Fuzzy Hypothesis Testing for Importance Index

If we let $Y_{ij}=y_{ij}$, then the data matrix of random variable $Y_{ij}$ can be shown as

$$\left(\begin{array}{lllll}{Y}_{1,1},& \cdots ,& Y_{1,j},& \cdots ,& Y_{1,n}\\ ⋮& & ⋮& & ⋮\\ Y_{i,1},& \cdots ,& Y_{i,j},& \cdots ,& Y_{i,n}\\ ⋮& & ⋮& & ⋮\\ Y_{q,1},& \cdots ,& Y_{q,j},& \cdots ,& Y_{q,n}\end{array}\right)=\left(\begin{array}{lllll}{y}_{1,1},& \cdots ,& y_{1,j},& \cdots ,& y_{1,n}\\ ⋮& & ⋮& & ⋮\\ y_{i,1},& \cdots ,& y_{i,j},& \cdots ,& y_{i,n}\\ ⋮& & ⋮& & ⋮\\ y_{q,1},& \cdots ,& y_{q,j},& \cdots ,& y_{q,n}\end{array}\right)$$

Then the observation values for the means, standard deviations, and ${\theta }_i^{\prime }{}^{*}$ of each importance service item can be expressed as the following:

$${\overline{y}}_i=\frac{1}{n}\times {\displaystyle \sum _{j=1}^n{y}_{ij}},$$

(40)

$$s_i^{\prime }=\sqrt{\frac{1}{n-1}\times {\displaystyle \sum _{j=1}^n{\left(y_{ij}-{\overline{y}}_i\right)}^2}},$$

(41)

and

$${\theta }_{i0}^{\prime *}={\overline{y}}_i=\frac{1}{n}\times {\displaystyle \sum _{j=1}^n{y}_{ij}}.$$

(42)

All service items requiring improvement are represented by the Set “SI”. For example, if $SI=\left\{3,7,8,11\right\}$, service items 3, 7, 8, and 11 require improvement. These service items are then checked for importance. If their importance index is higher than the mean, they are prioritized for improvement in cases of limited resources. This can be represented by hypothesis testing as the following:

$$\begin{gathered} H_0: {\theta }_i^{\prime }\ge {\theta }_0^{\prime } (\mathrm{prioritize} \mathrm{improvement}) \\ {H}_1: {\theta }_i^{\prime }<{\theta }_0^{\prime } (\mathrm{Non-prioritize} \mathrm{improvement}) \end{gathered}$$

This gives the following test statistic:

$$Z_i=\frac{\sqrt{n}\left({\theta }_i^{*}-{\theta }_0^{\prime }\right)}{s_i}$$

(43)

where

$${\theta }_0^{\prime }=\frac{1}{q}{\displaystyle \sum _{i=1}^q{\theta }_{i0}^{\prime *}}.$$

(44)

The critical region is $C^{\prime }=\left\{{\theta }_i^{\prime }{}^{*}\le C_{i0}^{\prime }\right\}$ where $C_{i0}^{\prime }$ is determined by

$$P\left\{{\theta }_i^{\prime }{}^{*}\le C_{i0}^{\prime }|{\theta }_i^{\prime }={\theta }_0^{\prime }\right\}=P\left\{Z_i\le \frac{\left(C_{i0}^{\prime }-{\theta }_0^{\prime }\right)}{s_i/\sqrt{n}}\right\}=\beta$$

(45)

$$\Rightarrow C_{i0}^{\prime }={\theta }_0^{\prime }-z_{\beta }\times \frac{s_i}{\sqrt{n}}$$

(46)

where $\beta$ is the significance level and the decision rule is

$$\begin{gathered} \left(1\right) \mathrm{Reject} H_0 \mathrm{if} {\theta }_i^{\prime }{}^{*}\le C_{i0}^{\prime }, \\ \left(2\right) \mathrm{Do} \mathrm{not} \mathrm{reject} H_0 \mathrm{if} {\theta }_i^{\prime }{}^{*}>C_{i0}^{\prime }. \end{gathered}$$

Similar to the satisfaction index ${\theta }_i$, the $100\left(1-\alpha \right)\%$ confidence intervals of ${\theta }_i^{\prime }$ is

$$\left[{\theta }_{i0}^{\prime *}-z_{\alpha /2}\times \frac{s_i}{\sqrt{n}},{\theta }_{i0}^{\prime *}+z_{\alpha /2}\times \frac{s_i}{\sqrt{n}}\right].$$

(47)

According to Buckley [15], the $\text{α-cuts}$ of triangular fuzzy number ${\theta }_i^{\prime }{}^{*}$ is

$${\tilde{\theta }}_i^{\prime }{}^{*}\left[\alpha \right]=\{\begin{array}{l}\left[{\theta }_{i1}^{\prime *}(\alpha ),{\theta }_{i2}^{\prime *}(\alpha )\right]=\left[{\theta }_{i0}^{\prime *}-z_{\alpha /2}\times \frac{s_i^{\prime }}{\sqrt{n}},{\theta }_{i0}^{\prime *}+z_{\alpha /2}\times \frac{s_i^{\prime }}{\sqrt{n}}\right], 0.01\le \alpha \le 1\\ \left[{\theta }_{i1}^{\prime *}(\alpha ),{\theta }_{i2}^{\prime *}(\alpha )\right]=\left[{\theta }_{i0}^{\prime *}-z_{0.005}\times \frac{s_i^{\prime }}{\sqrt{n}},{\theta }_{i0}^{\prime *}+z_{0.005}\times \frac{s_i^{\prime }}{\sqrt{n}}\right], 0\le \alpha \le 0.01\end{array}.$$

(48)

Obviously, when $\alpha =1$, then ${\theta }_{i1}^{\prime *}\left(1\right)={\theta }_{i2}^{\prime *}\left(1\right)={\theta }_i^{\prime }{}^{*}$. Thus, the triangular shaped fuzzy number of ${\theta }_i^{\prime }{}^{*}$ is ${\tilde{\theta }}_{i0}^{\prime *}=\left(L{\theta }_i^{\prime }, {\theta }_{i0}^{\prime }, R{\theta }_i^{\prime }\right)$ where

$$L{\theta }_i^{\prime }={\theta }_{i0}^{\prime *}-z_{0.005}\times \frac{s_i^{\prime }}{\sqrt{n}},$$

(49)

$$R{\theta }_i^{\prime }={\theta }_{i0}^{\prime *}+z_{0.005}\times \frac{s_i^{\prime }}{\sqrt{n}}.$$

(50)

The membership function of triangular fuzzy number ${\tilde{\theta }}_i^{\prime *}$ is

$${\eta }_i^{\prime }(x)=\{\begin{array}{l}0, if x<{\theta }_{i0}^{\prime *}-z_{0.005}\times \frac{s_i^{\prime }}{\sqrt{n}}\\ 2\times \left(1-\Phi \left(\frac{{\theta }_{i0}^{\prime *}-x}{s_i^{\prime }/\sqrt{n}}\right)\right), if {\theta }_{i0}^{\prime *}-z_{0.005}\times \frac{s_i^{\prime }}{\sqrt{n}}\le x<{\theta }_{i0}^{\prime *}\\ 1, if x={\theta }_{i0}^{\prime *}\\ 2\times \left(1-\Phi \left(\frac{x-{\theta }_{i0}^{\prime *}}{s_i^{\prime }/\sqrt{n}}\right)\right), if {\theta }_{i0}^{\prime *}<x\le {\theta }_{i0}^{\prime *}+z_{0.005}\times \frac{s_i^{\prime }}{\sqrt{n}}\\ 1, if {\theta }_{i0}^{\prime *}+z_{0.005}\times \frac{s_i^{\prime }}{\sqrt{n}}<x\end{array}.$$

(51)

Similarly to ${\theta }_i^{*}$, the $\text{α-cuts}$ of triangular fuzzy number of $C_0^{\prime }$ is

$${\tilde{C}}_0^{\prime }\left[\alpha \right]=\{\begin{array}{l}\left[C_{01}^{\prime }(\alpha ),C_{02}^{\prime }(\alpha )\right]=\left[C_0^{\prime }-z_{\alpha /2}\times \frac{s_i^{\prime }}{\sqrt{n}},C_0^{\prime }+z_{\alpha /2}\times \frac{s_i^{\prime }}{\sqrt{n}}\right], 0.01\le \alpha \le 1\\ \left[C_{01}^{\prime }(\alpha ),C_{02}^{\prime }(\alpha )\right]=\left[C_0^{\prime }-z_{0.005}\times \frac{s_i^{\prime }}{\sqrt{n}},C_0^{\prime }+z_{0.005}\times \frac{s_i^{\prime }}{\sqrt{n}}\right], 0\le \alpha \le 0.01\end{array}.$$

(52)

The triangular fuzzy number of $C_0^{\prime }$ is ${\tilde{C}}_{i0}^{\prime }=\left(L{C}_i^{\prime }, C_{i0}^{\prime }, R{C}_i^{\prime }\right)$ where

$$L{C}_i^{\prime }=C_{i0}^{\prime }-z_{0.005}\times \frac{s_i^{\prime }}{\sqrt{n}},$$

(53)

$$R{C}_i^{\prime }=C_{i0}^{\prime }+z_{0.005}\times \frac{s_i^{\prime }}{\sqrt{n}}.$$

(54)

Then the fuzzy membership function of ${\tilde{C}}_{i0}^{\prime }$ is

$${\eta }_0^{\prime }(x)=\{\begin{array}{l}0, if x<C_{i0}^{\prime }-z_{0.005}\times \frac{s_i^{\prime }}{\sqrt{n}}\\ 2\times \left(1-\Phi \left(\frac{C_{i0}^{\prime }-x}{s_i^{\prime }/\sqrt{n}}\right)\right), if C_{i0}^{\prime }-z_{0.005}\times \frac{s_i^{\prime }}{\sqrt{n}}\le x<C_{i0}^{\prime }\\ 1, if x=C_{i0}^{\prime }\\ 2\times \left(1-\Phi \left(\frac{x-C_{i0}^{\prime }}{s_i^{\prime }/\sqrt{n}}\right)\right), if C_{i0}^{\prime }<x\le C_{i0}^{\prime }+z_{0.005}\times \frac{s_i^{\prime }}{\sqrt{n}}\\ 1, if C_{i0}^{\prime }+z_{0.005}\times \frac{s_i^{\prime }}{\sqrt{n}}<x\end{array}.$$

(55)

Subsequently, ${\eta }_i^{\prime }(x)$ and ${\eta }_0^{\prime }(x)$ can be represented graphically as the following (see Figure 2):

Figure 2. Membership functions of ${\eta }_i^{\prime }(x)$ and ${\eta }_o^{\prime }(x)$.

Obviously,

$${\theta }_{i0}^{\prime *}-z_{a^{\prime }/2}\times \frac{s_i^{\prime }}{\sqrt{n}}=C_0^{\prime }\Rightarrow z_{a^{\prime }/2}=\frac{{\theta }_{i0}^{\prime *}-C_0^{\prime }}{s_i^{\prime }/\sqrt{n}}\Rightarrow a^{\prime }=2\left[1-\Phi \left(\frac{{\delta }_i^{\prime }}{s_i^{\prime }/\sqrt{n}}\right)\right],$$

(56)

where ${\delta }_i^{\prime }=C_0^{\prime }-{\theta }_{i0}^{\prime *}$. Similarly to ${\delta }_{ij}$, if we let $j=\left[100\alpha \right]$,

$${\delta }_{ij}^{\prime }=\left({\theta }_{i0}^{\prime *}+z_{0.005j}\times \frac{s_i^{\prime }}{\sqrt{n}}\right)-\left({\theta }_{i0}^{\prime *}-z_{0.005j}\times \frac{s_i^{\prime }}{\sqrt{n}}\right)=2z_{0.005j}\times \frac{s_i^{\prime }}{\sqrt{n}},$$

(57)

and $A_{Ti}^{\prime }$ represent the total area under the graph of ${\tilde{\theta }}_i^{\prime }{}^{*}$, then $j=0,1,\dots ,100$. $A_{Ti}^{\prime }$ is divided into 100 trapezoid-like blocks and block j therein can be shown as the following:

$$A_{Tij}^{\prime }=\left\{\left(x,\alpha \right)|{\theta }_{i0}^{\prime *}-z_{\alpha /2}\times \frac{s_i^{\prime }}{\sqrt{n}}\le x\le {\theta }_{i0}^{\prime *}+z_{\alpha /2}\times \frac{s_i^{\prime }}{\sqrt{n}},0.01\times \left(j-1\right)\le \alpha \le 0.01\times j\right\}.$$

(58)

We also let

$${\gamma }_{ij}^{\prime }=C_0^{\prime }-\left({\theta }_{i0}^{\prime *}-z_{0.005j}\times \frac{s_i^{\prime }}{\sqrt{n}}\right)=z_{0.005j}\times \frac{s_i^{\prime }}{\sqrt{n}}-{\delta }_i^{\prime }.$$

(59)

Suppose $j=$$h_i^{\prime }$ such that ${\gamma }_{i{h}^{\prime }}=0$. Then

$$z_{0.005h_i^{\prime }}=\frac{{\delta }_i^{\prime }}{s_i^{\prime }/\sqrt{n}}\Rightarrow 0.005h_i^{\prime }=1-\Phi \left(\frac{{\delta }_i^{\prime }}{s_i^{\prime }/\sqrt{n}}\right)\Rightarrow h_i^{\prime }=\left[100a_i^{\prime }\right].$$

(60)

If we let $A_{Ri}^{\prime }$ represent the area under the graph of ${\tilde{\theta }}_i^{\prime }{}^{*}$ but to the right of the vertical line through point $C_0^{\prime }$, then $j=0,1,\dots ,h_i^{\prime }$. $A_{Ri}^{\prime }$ is cut by h′ into h′ trapezoid-like blocks and block j can be expressed as the following:

$$A_{Rij}^{\prime }=\left\{\left(x,\alpha \right)|{\theta }_{i0}^{\prime *}-z_{\alpha /2}\times \frac{s_i^{\prime }}{\sqrt{n}}\le x\le C_0^{\prime },0.01\times \left(j-1\right)\le \alpha \le 0.01\times j\right\}.$$

(61)

Now, we compute areas $A_{Tij}^{\prime }$ and $A_{Rij}^{\prime }$ as

$$A_{Tij}^{\prime }=\left(\frac{{\delta }_{i(j-1)}^{\prime }+{\delta }_{ij}^{\prime }}{2}\right)\times (0.01), j=1,\dots ,100.$$

(62)

$$A_{Rij}^{\prime }=\left(\frac{{\gamma }_{i(j-1)}+{\gamma }_{ij}}{2}\right)\times (0.01), j=1,\dots ,h_i^{\prime }.$$

(63)

$A_{Tij}^{\prime }$ and $A_{Rij}^{\prime }$ can be shown as the following:

$$A_{Ti}^{\prime }={\displaystyle \sum _{j=1}^{100}{A}_{Tij}^{\prime }}$$

(64)

and

$$A_{Ri}^{\prime }={\displaystyle \sum _{j=1}^{h^{\prime }}{A}_{Rij}^{\prime }}.$$

(65)

As noticed by Buckley [15], we may cogitate employing two numbers ($0 < {\varphi }_1<{\varphi }_2<0.5$) as the following:

(1). If $A_{Ri}^{\prime }/A_{Ti}^{\prime }\le {\varphi }_1$, then do not reject $H_0$ and infer that service item i must be prioritized for improvement,

(2). If ${\varphi }_1\le A_{Ri}^{\prime }/A_{Ti}^{\prime }\le {\varphi }_2$, then make no decision on whether to reject/not reject,

(3). If $A_{Ri}^{\prime }/A_{Ti}^{\prime }\ge {\varphi }_2$, then reject $H_0$ and conclude that service item i is not a priority.

5. Case Study

E-learning represents a rapidly growing trend in education. According to the 2018 survey report on digital industries by the Industrial Development Bureau of the Ministry of Economic Affairs, the worth of digital services output in Taiwan in 2017 was as high as NT $5.01 trillion, representing an increase of 19.6% compared to 2016. E-learning is often utilized for language teaching [20,21]. It is forecasted that digital English learning in the Asia-Pacific area will increase by as much as 25.83% per year from 2018 to 2022 [22]. We therefore selected a CALL system operating at three private universities in central Taiwan as a case study to manifest the presented method.

This paper applied the web-based e-learning system (WELS) questionnaire proposed by Shee and Wang [23] to explore students’ satisfaction and perception of importance when using the selected CALL system. There are 13 questions (shown in Table 1) in the WELS questionnaire. Five-point Likert scales were applied as the following: for satisfaction, (1) strongly disagree, (2) disagree, (3) average, (4) agree, and (5) strongly agree; for importance, (1) strongly disagree, (2) disagree, (3) average, (4) agree, and (5) strongly agree. The sample was students at three private universities using the CALL system. A total of 507 questionnaires were distributed and 433 questionnaires were recovered, representing a recovery rate of 85%. Among these, 18 questionnaires were deemed invalid, making the effective recovery rate 82%. In this paper, we establish a calculation process to complete the evaluation procedure based on the equation in Section 3.

Table 1. Satisfaction Indices for CALL System.

Dimensions	Items	$\mathit{L}{\mathit{\theta }}_{\mathit{i}}$	${\mathit{\theta }}_{\mathit{i}0}^{*}$	$\mathit{R}{\mathit{\theta }}_{\mathit{i}}$	${\mathit{C}}_{\mathit{i}0}$	${\mathit{A}}_{\mathit{R}\mathit{i}}/{\mathit{A}}_{\mathit{T}\mathit{i}}$
Learner interface	1. Ease of use	0.4932	0.6029	0.7126	0.4535	0.0000
	2. User-friendliness	0.4833	0.5949	0.7064	0.4521	0.0000
	3. Ease of understanding	0.4673	0.5809	0.6944	0.4506	0.0000
	4. Operational stability	0.4505	0.5670	0.6835	0.4483	0.0000
Learning community	5. Ease of discussion with other Learners	0.3647	0.4631	0.5615	0.4570	0.4106*
	6. Ease of discussion with teachers	0.3533	0.4612	0.5691	0.4549	0.4101*
	7. Ease of accessing shared data	0.3520	0.4610	0.5700	0.4540	0.4018*
	8. Ease of exchanging learning with the others	0.3513	0.4601	0.5689	0.4542	0.4161*
System content	9. Up-to-date content	0.4378	0.5488	0.6598	0.4525	0.0019
	10. Sufficient content	0.4538	0.5633	0.6727	0.4537	0.0083
	11. Useful content	0.4494	0.5627	0.6759	0.4508	0.0074
Personalization	12. Capability of controlling learning progress	0.4398	0.5512	0.6626	0.4522	0.0026
	13. Capability of recording learning performance	0.4561	0.5639	0.6717	0.4550	0.0091

Note: When values of $A_{Ri}/A_{Ti}$ are marked *, it indicates $A_{Ri}/A_{Ti}>{\varphi }_2=0.4$, which also means that service item i needs improvement.

Step 1: The mean and standard deviation of each service item are calculated: ${\theta }_{i0}^{*}$ is calculated by Equation (15) and $L{\theta }_i$ and $R{\theta }_i$ are calculated by Equations (22) and (23). The results are input to Table 1. Then we take service item 1 (i = 1) for an example to describe the calculation process of these 3 statistics as the following:

$$\begin{gathered} {\theta }_{10}^{*}=\frac{1}{n}\times {\displaystyle \sum _{j=1}^n{x}_{ij}}=\frac{1}{415}\times {\displaystyle \sum _{j=1}^{415}{x}_{1j}}=0.6029 \\ L{\theta }_1={\theta }_{10}^{*}-z_{0.005}\times \frac{s_1}{\sqrt{n}}=0.6029-2.576\times \frac{0.867}{\sqrt{415}}=0.4932 \\ R{\theta }_1={\theta }_{10}^{*}+z_{0.005}\times \frac{s_1}{\sqrt{n}}=0.6029+2.576\times \frac{0.876}{\sqrt{415}}=0.7126 \end{gathered}$$

Step 2: Based on Equation (16), we calculate ${\theta }_0$$={\displaystyle {\sum }_{i=1}^{13}{\theta }_{i0}^{*}}/13$ = 0.5370 and set up hypothesis testing as the following:

$H_0: {\theta }_i\ge 0.5370 (no improvement needed)$,

$H_1: {\theta }_i<0.5370 (improvement needed)$.

Step 3: We let the significance level $\beta =0.025$ and according to Equation (18) calculate $C_{i0}$ and fill in the result in Table 1. We take service item 1 (i = 1) for an instance to state the calculation process of $C_{i0}$ as the following:

$$C_{10}={\theta }_0-z_{0.025}\times \frac{s_1}{\sqrt{n}}=0.5370-1.96\times \frac{0.876}{\sqrt{415}}=0.4535$$

Step 4: Based on Equations (38) and (39), we calculate $A_{Ti}$ and $A_{Ri}$ respectively, then calculate $A_{Ri}/A_{Ti}$ and fill the result in Table 1. Because service items 5, 6, 7, and 8 of dimension 2 all need improvement, we take these 4 items for example to describe the calculation process of $A_{Ri}/A_{Ti}$ as the following:

$$\begin{gathered} A_{T5}={\displaystyle \sum _{j=1}^{100}{A}_{T5j}}=2.6206, A_{R5}={\displaystyle \sum _{j=1}^{h_5}{A}_{R5j}}=6.3824, A_{R5}/A_{T5}=0.4106 \\ {A}_{T6}={\displaystyle \sum _{j=1}^{100}{A}_{T6j}}=2.6865, A_{R6}={\displaystyle \sum _{j=1}^{h_6}{A}_{R6j}}=6.5513, A_{R6}/A_{T6}=0.4101 \\ {A}_{T7}={\displaystyle \sum _{j=1}^{100}{A}_{T7j}}=2.6598, A_{R7}={\displaystyle \sum _{j=1}^{h_7}{A}_{R7j}}=6.6204, A_{R7}/A_{T7}=0.4018 \\ {A}_{T8}={\displaystyle \sum _{j=1}^{100}{A}_{T8j}}=2.7495, A_{R8}={\displaystyle \sum _{j=1}^{h_8}{A}_{R8j}}=6.6073, A_{R8}/A_{T8}=0.4161 \end{gathered}$$

Step 5: We set ${\varphi }_1=0.2$ and ${\varphi }_2=0.4$. According to the evaluation rules presented in Section 3, $A_{Ri}/A_{Ti}\ge {\varphi }_2=0.4$ (marked ^*). Therefore, we reject $H_0$ and conclude that service item i needs improvement. The $A_{Ri}/A_{Ti}$ values of items 5, 6, 7, and 8 are larger than 0.4, which means that service items 5, 6, 7, and 8 need improvement.

As stated above, according to the statistical testing principle (i.e., if ${\theta }_{i0}^{*}>C_{i0}$, do not reject $H_0$ and service item i does not need improvement), service items 5, 6, 7, and 8 do not need improvement. However, the ${\theta }_{i0}^{*}$ values of service item 5, 6, 7, 8 is much smaller than ${\theta }_0$. In practices, they should be listed as improvement items. Therefore, they are revised to be improved after applying the fuzzy hypothesis testing method constructed in this paper. Obviously, this result is more reasonable than the traditional statistical testing principle.

The improvement items are confirmed by fuzzy hypothesis testing. In cases of limited resources, the items with an importance index above the mean will be prioritized for improvement. The calculation process is constructed as described in Section 4.

Step 1: The mean and standard deviation of each service item are calculated: ${\theta }_{i0}^{\prime *}$ is calculated by Equation (42) and $L{\theta }_i^{\prime }$ and $R{\theta }_i^{\prime }$ are calculated by Equations (49) and (50). The results are input to Table 2. We take service item 5 (i = 5) for an instance to describe the calculation process of these 3 statistics as the following:

$$\begin{gathered} {\theta }_{50}^{\prime *}=\frac{1}{415}\times {\displaystyle \sum _{j=1}^{415}{y}_{1j}}=0.7108 \\ L{\theta }_5^{\prime }={\theta }_{50}^{\prime *}-z_{0.005}\times \frac{s_5^{\prime }}{\sqrt{n}}=0.7108-2.576\times \frac{0.8294}{\sqrt{415}}=0.6060 \\ R{\theta }_5^{\prime }={\theta }_{50}^{\prime *}+z_{0.005}\times \frac{s_5^{\prime }}{\sqrt{n}}=0.7108+2.576\times \frac{0.8294}{\sqrt{415}}=0.8157 \end{gathered}$$

Table 2. Importance Indices for CALL system.

Dimensions	Items	$\mathit{L}{{\mathit{\theta }}^{\prime }}_{\mathit{i}}$	${{\mathit{\theta }}^{\prime }}_{\mathit{i}0}^{*}$	$\mathit{R}{{\mathit{\theta }}^{\prime }}_{\mathit{i}}$	${{\mathit{C}}^{\prime }}_{\mathit{i}0}$	${{\mathit{A}}^{\prime }}_{\mathit{R}\mathit{i}}/{{\mathit{A}}^{\prime }}_{\mathit{T}\mathit{i}}$
Learning community	5. Ease of discussion with other Learners	0.6060	0.7108	0.8157	0.6411	0.0181
	6. Ease of discussion with teachers	0.6082	0.7145	0.8208	0.6401	0.0138
	7. Ease of accessing shared data	0.6159	0.7205	0.8251	0.6414	0.0086
	8. Ease of exchanging learning with the others	0.5959	0.7006	0.8053	0.6413	0.0357

Note: When values of $A_{Ri}^{\prime }/A_{Ti}^{\prime }$ are marked *, it indicates $A_{Ri}^{\prime }/A_{Ti}^{\prime }$$>$${\varphi }_2=0.4$, which also means that service item i should not be prioritized.

Step 2: Based on Equation (44), we calculate ${\theta }_0^{\prime }$ $=$ ${\displaystyle {\sum }_{i=1}^{13}{\theta }_{i0}^{\prime *}}/13$ $=$ 0.7209 and set up hypothesis testing as the following:

$H_0: {\theta }_i^{\prime }\ge 0.7209 (\mathrm{prioritize} \mathrm{improvement})$

$H_1: {\theta }_i^{\prime }<0.7209 (\mathrm{Non-prioritize} \mathrm{improvement})$

Step 3: We let the significance level $\beta =0.025$ and according to Equation (46) calculate $C_{i0}^{\prime }$. The results are input to Table 2. Then we take service item 5 (i = 5) for an example to describe the calculation process of $C_{50}^{\prime }$ as the following:

$$C_{50}^{\prime }={\theta }_0^{\prime }-z_{0.025}\times \frac{s_1^{\prime }}{\sqrt{n}}=0.7209-1.96\times \frac{0.8294}{\sqrt{415}}=0.6411$$

Step 4: Based on Equations (64) and (65), we calculate $A_{Ti}^{\prime }$ and $A_{Ri}^{\prime }$ respectively, then calculate $A_{Ri}^{\prime }/A_{Ti}^{\prime }$ and fill in Table 2. We take service items 5, 6, 7, and 8 of dimension 2 all for example to state the calculation process of $A_{Ri}/A_{Ti}$ as the following:

$$\begin{gathered} A_{T5}^{\prime }={\displaystyle \sum _{j=1}^{100}{A}_{T5j}^{\prime }}=0.1154, A_{R5}^{\prime }={\displaystyle \sum _{j=1}^{h_5^{\prime }}{A}_{R5j}^{\prime }}=6.3689, A_{R5}^{\prime }/A_{T5}^{\prime }=0.0181 \\ {A}_{T6}^{\prime }={\displaystyle \sum _{j=1}^{100}{A}_{T6j}^{\prime }}=0.0891, A_{R6}^{\prime }={\displaystyle \sum _{j=1}^{h_6^{\prime }}{A}_{R6j}^{\prime }}=6.4554, A_{R6}^{\prime }/A_{T6}^{\prime }=0.0138 \\ {A}_{T7}^{\prime }={\displaystyle \sum _{j=1}^{100}{A}_{T7j}^{\prime }}=0.0545, A_{R7}^{\prime }={\displaystyle \sum _{j=1}^{h_7^{\prime }}{A}_{R7j}^{\prime }}=6.3509, A_{R7}^{\prime }/A_{T7}^{\prime }=0.0086 \\ {A}_{T8}^{\prime }={\displaystyle \sum _{j=1}^{100}{A}_{T8j}^{\prime }}=0.2268, A_{R8}^{\prime }={\displaystyle \sum _{j=1}^{h_8^{\prime }}{A}_{R8j}^{\prime }}=6.3574, A_{R8}^{\prime }/A_{T8}^{\prime }=0.0357 \end{gathered}$$

Step 5: We let ${\varphi }_1=0.2$ and ${\varphi }_2=0.4$. According to the evaluation rules of Section 4, if $A_{Ri}^{\prime }/A_{Ti}^{\prime }\le {\varphi }_1$, then do not reject $H_0$ and infer that service item i must be prioritized. The $A_{Ri}^{\prime }/A_{Ti}^{\prime }$ values of items 5, 6, 7, and 8 are smaller than 0.2, which means that service items 5, 6, 7, and 8 must be prioritized.

6. Conclusions

Hung et al. [9] indicated that as beta distribution is between zero and one, it is a suitable indicator of the degree of importance and satisfaction from 0% to 100%. We applied the proposed methods to the evaluation of a CALL system. We found that according to statistical testing, service items 5, 6, 7, and 8 did not need improvement. However, the ${\theta }_{i0}^{*}$ values of service items 5, 6, 7, and 8 were much smaller than ${\theta }_0$. In practice, these items would therefore be considered as requiring improvement. Following application of the proposed fuzzy hypothesis testing method, these items were re-categorized. The proposed methods were then applied to determine the importance of these service items, and all four were deemed as a priority. The major managerial insights of this study are:

(1) We have constructed a performance evaluation matrix, on the basis of the spirit of continuous improvement promoted by total quality management. Through the fuzzy hypothesis testing method presented by Buckley [15], we have identified the characteristics considered critical to quality and determined which items should be prioritized for improvement in cases of limited resources.

(2) According to Chen et al. [16], fuzzy hypothesis testing brings more reasonable results for those in practice than in traditional statistics. In fact, fuzzy hypothesis testing mentioned in this paper combines the statistical inference method with experts’ experiences to make accurate decision. Therefore, service item 5, 6, 7, and 8 are not missed to improve and the loss of the industries can be reduced to the lowest [16,24,25,26,27].

(3) Based on the work of Buckley [15] and Chen et al. [16], we introduced confidence intervals to decreases the chance of misjudgment arising from sampling errors.

(4) This method of data collection preserves the voice of the customer, and it is relatively simple to apply, thereby increasing customers’ willingness to participate.

(5) This method only requires a small quantity of the sample size. It not only meets the industries cost benefit but also controls the timeliness and effectiveness.