Fuzzy Evaluation Model for Enhancing E-Learning Systems

Abstract

As the environment and information-technology conditions of the Internet of Things matured, various applications were launched. In education, e-learning is promoted so that students’ learning is no longer restricted to the classroom. E-learning schedules are flexible, and learners’ commuting costs are low. Apparently, improving the quality of e-learning systems can enhance learners’ learning effectiveness, satisfaction, engagement, and learning efficacy. A performance evaluation matrix is a useful tool for collecting users’ opinions to assess the performance of an operating system, and it is widely used to evaluate and improve performance in numerous industries and organizations. Therefore, this study used this matrix to construct a model for evaluation and analysis, providing suggestions on improving e-learning systems. This approach maintained the simple response model of Likert scales, which increases the efficiency and accuracy of data collection. Furthermore, the fuzzy membership function of the discriminant index was constructed based on the confidence interval, thereby solving the problems of sampling error and the complexity of collecting fuzzy linguistic data. Besides, we simplified calculations by standardizing test statistics to increase evaluation efficiency. As a result, this study improved the quality of e-learning system, enhanced users’ learning effectiveness, satisfaction, and engagement, and achieved the goal of sustainability.

1. Introduction

Product process quality and service quality performance are two critical axes which explore quality performance. Good product quality is the best backing for service quality. The Taguchi loss function and the corresponding process capability indicators are the best tools for exploring process quality [1,2,3,4,5,6,7,8]. The best tool for service quality performance evaluation is the performance evaluation matrix (PEM). As the environment of the Internet of Things gradually matures and data-collection technology improves, both types of quality performance evaluations are more immediate and accurate [9,10,11,12,13,14,15]. Thus, various applications have enabled the launch of sharing economies, such as Uber, bicycle-sharing, and ridesharing. In education, e-learning is promoted so that student learning is no longer restricted to the classroom. With e-learning, learning schedules are flexible, and the costs of learners’ traveling to and from their learning institutions are lower. Also, learners can learn at any time and place, without being restricted by the environment, which will improve their learning efficiency. As a result, constructing a comprehensive e-learning system can benefit the learning effectiveness and satisfaction of students or learners and certainly increase the number of people engaged in e-learning, thereby creating a profitable commercial model.

Lambert and Sharma (1990) proposed the performance evaluation matrix (PEM) to define the importance index (Y-axis) and the satisfaction index (X-axis) for the evaluation of the operating system by means of collecting users’ perceptions through the questionnaire. Hung, Huang, and Chen (2003) revised the position of the PEM performance block to make the evaluation rules more rational. Subsequently, many scholars conducted relevant revised studies to make the evaluation model more complete [9,10,16,17,18,19]. Obviously, performance evaluation method or performance evaluation matrix (PEM) is a useful tool that collects the opinions of customers or users to assess the performance of an operating system. Compared to other evaluation methods that require complex data comparisons, PEMs have an advantage in terms of ease of analysis. PEMs are thus widely applied by various researchers to evaluate and improve performance in a variety of industries and organizations. Such researchers include Chen and Chen [16], Hossain and Ahmed [20], Markovic’ and Jankovic’ [21], Wang, et al. [17], Wong and Szeto [22], Basso and Funari [23], Wu et al. [24], and Zhou et al. [25]. In view of this, this study used PEMs to construct a model for the assessment, analysis, and provision of recommendations for e-learning systems, so as to improve the quality of these systems.

As mentioned earlier, some scholars also use fuzzy methods to explore the performance evaluation of online learning and provide good suggestions for improvement. Those scholars include Çelikbilek et al. [26], Lange [27], Ma et al. [28], Shanmugam et al. [29], and Wong et al. [30]. In order to improve the data collection of fuzzy numbers, which is complicated and may reduce customers’ willingness to answer questions, Chen et al. [18] developed a discriminate index to determine whether a service item requires improvement by having customers rank how important a service item is to them and how satisfactory they found that service item to be. They referred to the fuzzy test methods used by Buckley [31], Wang [32], Chen et al. [33], and Chen et al. [34] to propose a discriminant index and develop a fuzzy test method to identify the service items that needed improvement in PEMs. The advantage of this approach is that it maintains the simple response model of Likert scales, which increases the efficiency and accuracy of data collection. Furthermore, the fuzzy membership function of the discriminant index was constructed based on the confidence interval, which solves the problems of sampling error and the complexity of collecting fuzzy linguistic data. Although this approach simplifies many calculation models, multiple question items must be evaluated simultaneously, which still presents some complexity in practical applications. This study thus simplified calculations by standardizing test statistics and developed fuzzy evaluation criteria based on the fuzzy evaluation method proposed by Chen et al. [18] to identify the service items that need improvement in a PEM. Finally, we used a case study of a computer-assisted language learning system (CALL system) to demonstrate the efficacy of the proposed approach.

The remainder of this paper is organized as follows. Section 2 defines the performance indices, PEM, discriminant index, and performance zones in the PEM. Section 3 uses statistical inference to derive the confidence interval of the discriminant index, following which the confidence interval is used to construct the fuzzy membership function of the discriminant index. Then, Section 3 presents the fuzzy test method and fuzzy evaluation criteria based on the fuzzy membership function to identify items considered critical to quality (CTQ). Section 4 contains the case study involving the computer-assisted language learning system (CALL system) to demonstrate the efficacy of the proposed approach. Conclusions are summarized in Section 5.

2. Performance Indices and Performance Evaluation Matrix

Similar to Chen et al. [18] and Yu, Chang, and Chen [19], for the sake of generality, this paper investigates q service items on the satisfaction scale; each item is associated with one question related to importance and one related to satisfaction, thereby totaling 2q questions. This paper uses the web-based e-learning system (WELS) questionnaire developed by Shee and Wang [35] to collect information on user satisfaction with the CALL system. The WELS questionnaire comprises 4 dimensions (learner interface, learning community, system content, and personalization) and the 13 items (q = 13) listed in

Table 1. Four dimensions and 13 items on the web-based e-learning system (WELS) questionnaire.

Dimensions	Items
Learner interface	1.  Ease of use 2.  User-friendliness 3.  Ease of understanding 4.  Operational stability
Learning community	5.  Ease of discussion with other learners 6.  Ease of discussion with teachers 7.  Ease of accessing shared data 8.  Ease of exchanging learning with the others
System content	9.  Up-to-date content 10. Sufficient content 11. Useful content
Personalization	12. Capability of controlling learning progress 13. Capability of recording learning performance

.

Based on Chen et al. [18], we use random variable $Y_i$ to represent the importance of service item i, and random variable $X_i$ to represent the user’s satisfaction with service item i. For a questionnaire using a k-point scale, let R = $k-1$. If we let $Y_{ti}=\left(Y_i-1\right)/R$, the expectation $E\left(Y_{ti}\right)={\theta }_i^{\prime }$ of random variable $Y_{ti}$ results in the following importance index:

$$\mathrm{importance}\mathrm{index}:{\theta }_i^{\prime }=\frac{{\mu }_i^{\prime }-1}{R}$$

(1)

where ${\mu }_i^{\prime }=E\left(Y_i\right)$. Similarly, if we let $X_{ti}=\left(X_i-1\right)/R$, the expectation $E\left(X_{ti}\right)={\theta }_i$ of random variable $X_{ti}$ results in the following satisfaction index:

$$\mathrm{satisfaction}\mathrm{index}:{\theta }_i=\frac{{\mu }_i-1}{R}$$

(2)

where ${\mu }_i=E\left(X_i\right)$. Obviously, $0\le {\theta }_i^{\prime }\le 1$ and $0\le {\theta }_i\le 1$, i = 1,…, 13. Based on Hung, et al. [16], the importance index ${\theta }_i^{\prime }$ is the vertical coordinate, and the satisfaction index ${\theta }_i$ is the horizontal coordinate. They revised the method employed by Lambert and Sharma [9] and constructed a PEM divided into three zones that are equal in area (Figure 1):

• Upper left corner (to improve): Area $B=\left\{\left({\theta }_i,{\theta }_i^{\prime }\right)|{\theta }_i^{\prime }>a+{\theta }_i,{\theta }_i\ge 0,{\theta }_i^{\prime }\le 1\right\}$
• Middle zone (to maintain): Area $M=\left\{\left({\theta }_i,{\theta }_i^{\prime }\right)|-a+{\theta }_i\le {\theta }_i^{\prime }\le a+{\theta }_i,0\le {\theta }_i\le 1,0\le {\theta }_i^{\prime }\le 1\right\}$
• Lower right corner (to reduce resource allocation): Area $T=\left\{\left({\theta }_i,{\theta }_i^{\prime }\right)|{\theta }_i^{\prime }<-a+{\theta }_i,{\theta }_i\le 1,{\theta }_i^{\prime }\ge 0\right\}$

Each zone covers a third of the matrix; therefore, $0.5{(1-a)}^2=1/3$. Thus, we can easily obtain the result, $a=1-\sqrt{6}/3=0.1835$. Hung, Huang, and Chen [10] pointed out that when $\left({\theta }_i,{\theta }_i^{\prime }\right)\in B$, the service items in question need to be improved. To determine whether service item i falls in Area B, we let $D_i=Y_{ti}-X_{ti}$, whereby the expected value of random variable $D_i$ is $E\left(D_i\right)={\delta }_i$ and serves as the following discriminant index:

$$\mathrm{discriminant}\mathrm{index}:{\delta }_i={\theta }_i^{\prime }-{\theta }_i$$

(3)

When discriminant index ${\delta }_i>0.1835$, then ${\theta }_i^{\prime }>0.1835+{\theta }_i$, which means that service item i falls in Area B and its quality must be improved. Similarly, when ${\delta }_i\le 0.1835$, then ${\theta }_i^{\prime }\le 0.1835+{\theta }_i$, which means that service item i does not fall in Area B. In this case, its quality should be maintained, or it should be discussed whether resources should be reallocated from this area to another for better overall quality.

3. Fuzzy Hypothesis Testing

Let $\left(X_{ti1},X_{ti2},\dots ,X_{tin}\right)$ be a random sample of satisfaction with service item i and $\left(Y_{ti1},Y_{ti2},\dots ,Y_{tin}\right)$ be a random sample of the importance of service item i, and then

$$\left(D_{i1},D_{i2},\dots ,D_{in}\right)=\left(Y_{ti1}-X_{ti1},Y_{ti2}-X_{ti2},\dots ,Y_{tin}-X_{tin}\right)$$

(4)

The sample mean and sample standard deviation of $X_{ti}$ can be shown as follows:

$${\overline{X}}_{ti}=\frac{1}{n}\times {\displaystyle \sum _{j=1}^n{X}_{tij}}\mathrm{and}{S}_{ti}=\sqrt{\frac{1}{n-1}\times {\displaystyle \sum _{j=1}^n{\left(X_{tij}-{\overline{X}}_{ti}\right)}^2}}$$

(5)

Similarly, the sample mean and sample standard deviation of $Y_{ti}$ can be shown as follows:

$${\overline{Y}}_{ti}=\frac{1}{n}\times {\displaystyle \sum _{j=1}^n{Y}_{tij}}\mathrm{and}{S}_{ti}^{\prime }=\sqrt{\frac{1}{n-1}\times {\displaystyle \sum _{j=1}^n{\left(Y_{tij}-{\overline{Y}}_{ti}\right)}^2}}$$

(6)

In addition, let ${\overline{D}}_i$ and $S_i$ denote the sample mean and sample standard deviation of $D_i$, respectively, as follows:

$${\overline{D}}_i=\frac{1}{n}\times {\displaystyle \sum _{j=1}^n{D}_{ij}}$$

(7)

$$S_i=\sqrt{\frac{1}{n-1}\times {\displaystyle \sum _{j=1}^n{\left(D_{ij}-{\overline{D}}_i\right)}^2}}$$

(8)

Based on the above data, satisfaction index estimator is ${\theta }_i^{*}$. Importance index estimator ${{\theta }_i^{\prime }}^{*}$ and improvement index estimator ${\delta }_i^{*}$ can therefore be shown as follows:

$${\theta }_i^{*}={\overline{X}}_{ti}$$

(9)

$${{\theta }_i^{\prime }}^{*}={\overline{Y}}_{ti}$$

(10)

$${\delta }_i^{*}={{\theta }_i^{\prime }}^{*}-{\theta }_i^{*}={\overline{Y}}_{ti}-{\overline{X}}_{ti}={\overline{D}}_i$$

(11)

Obviously, $E\left[{\theta }_i^{*}\right]=E\left[{\overline{X}}_{ti}\right]={\theta }_i$, $E\left[{{\theta }_i^{\prime }}^{*}\right]=E\left[{\overline{Y}}_{ti}\right]={\theta }_i^{\prime }$, and $E\left[{\delta }_i^{*}\right]=E\left[{\overline{D}}_i\right]={\delta }_i$. Using the satisfaction index ${\delta }_i$ to determine whether service item i requires improvement is equivalent to the following hypothesis test:

$$\begin{gathered} H_0:{\delta }_i\le 0.1835\left(\mathrm{no}\mathrm{improvement}\mathrm{needed}\right); \\ {H}_1:{\delta }_i>0.1835\left(\mathrm{improvement}\mathrm{needed}\right). \end{gathered}$$

From the random sample, we compute the test statistic ${\delta }_i^{*}={\overline{D}}_i$. First, let

$$Z_i=\frac{\sqrt{n}\left({\delta }_i^{*}-0.1835\right)}{S_i}$$

(12)

Then $Z_i$ is distributed as $N\left(0,1\right)$ for $n\to \infty$; that is

$$Z_i\underset{H_{0}istrue}{\overset{n\to \infty }{\to }}N\left(0,1\right)$$

(13)

Let $\left(y_{ti1},y_{ti2},\dots ,y_{tin}\right)$ be the observed value of $\left(Y_{ti1},Y_{ti2},\dots ,Y_{tin}\right)$ and $\left(x_{ti1},x_{ti2},\dots ,x_{tin}\right)$ be the observed value of $\left(X_{ti1},X_{ti2},\dots ,X_{tin}\right)$. Then the observed value of $\left(D_{i1},D_{i2},\dots ,D_{in}\right)$ is $\left(d_{i1},d_{i2},\dots ,d_{in}\right)$ and the observed value of ${\delta }_i^{*}$ and $S_i$ is

$${\delta }_{i0}^{*}=\frac{1}{n}\times {\displaystyle \sum _{j=1}^n{d}_{ij}}\mathrm{and}{s}_i=\sqrt{\frac{1}{n-1}\times {\displaystyle \sum _{j=1}^n{\left(d_{ij}-{\delta }_{i0}^{*}\right)}^2}}$$

(14)

Therefore, the observed value of $Z_i$ is

$$z_i=\frac{\sqrt{n}\left({\delta }_{i0}^{*}-0.1835\right)}{s_i}$$

(15)

The critical region $C=\left\{z_i\ge z_{\beta }\right\}$, where $\beta$ is the significance level of the test and usually $\beta$ are 0.01, 0.05, and 0.10. When the null hypothesis holds, the decision rule is as follows:

(1)

Reject $H_0$ if $z_i\ge z_{\beta }$;

(2)

Do not reject $H_0$ if $z_i<z_{\beta }$.

In fact, $P\left\{-z_{\alpha /2}\le Z_i\le z_{\alpha /2}\right\}=1-\alpha$, based on Buckley (2005), and the $\alpha$-cuts of triangular-shaped fuzzy number ${\tilde{z}}_i$ is

$${\tilde{z}}_i\left[\alpha \right]=\{\begin{array}{l}\left[z_{i1}(\alpha ),z_{i2}(\alpha )\right]=\left[z_i-z_{\alpha /2},z_i+z_{\alpha /2}\right],0.01\le \alpha \le 1\\ \left[z_{i1}(\alpha ),z_{i2}(\alpha )\right]=\left[z_i-z_{0.005},z_i+z_{0.005}\right],0\le \alpha \le 0.01\end{array}$$

(16)

Obviously, when $\alpha =1$, then $z_{\alpha /2}=z_{0.5}=0$, $z_{i1}(1)=z_{i2}(1)=z_i$. Thus, the triangular-shaped fuzzy number of ${\tilde{z}}_i$ is as follows:

$${\tilde{z}}_i=\left(z_i-z_{0.005},z_i,z_i+z_{0.005}\right)$$

(17)

Then, the membership function of triangular-shaped fuzzy number ${\tilde{z}}_i$ is

$${\eta }_i(x)=\{\begin{array}{l}0,ifx\le z_i-z_{0.005}\\ 2\times \left(1-\Phi (z_i-x)\right),if{z}_i-z_{0.005}<x<z_i\\ 1,ifx=z_i\\ 2\times \left(1-\Phi (x-z_i)\right),if{z}_i<x<z_i+z_{0.005}\\ 0,if{z}_i+z_{0.005}\le x\end{array}$$

(18)

where $\Phi (·)$ denotes the cumulative distribution function of the standard normal distribution. Similarly to $z_i$, the $\alpha$-cuts of triangular-shaped fuzzy number ${\tilde{z}}_{\beta }$ is

$${\tilde{z}}_{\beta }\left[\alpha \right]=\{\begin{array}{l}\left[z_{\beta 1}(\alpha ),z_{\beta 2}(\alpha )\right]=\left[z_{\beta }-z_{\alpha /2},z_{\beta }+z_{\alpha /2}\right],0.01\le \alpha \le 1\\ \left[z_{\beta 1}(\alpha ),z_{\beta 2}(\alpha )\right]=\left[z_{\beta }-z_{0.005},z_{\beta }+z_{0.005}\right],0\le \alpha \le 0.01\end{array}$$

(19)

Obviously, when $\alpha =1$, then $z_{\alpha /2}=z_{0.5}=0$, $z_{i1}(1)=z_{i2}(1)=z_i$. Thus, the triangular-shaped fuzzy number of ${\tilde{z}}_{\beta }$ is ${\tilde{z}}_{\beta }=\left(z_{\beta }-z_{0.005},z_{\beta },z_{\beta }+z_{0.005}\right)$. Then the membership function of triangular-shaped fuzzy number ${\tilde{z}}_{\beta }$ is

$${\eta }_{\beta }(x)=\{\begin{array}{l}0,ifx\le z_{\beta }-z_{0.005}\\ 2\times \left(1-\Phi (z_{\beta }-x)\right),if{z}_{\beta }-z_{0.005}<x<z_{\beta }\\ 1,ifx=z_{\beta }\\ 2\times \left(1-\Phi (x-z_{\beta })\right),if{z}_{\beta }<x<z_{\beta }+z_{0.005}\\ 0,if{z}_{\beta }+z_{0.005}\le x\end{array}$$

(20)

Subsequently, the diagram of ${\eta }_i(x)$ and ${\eta }_{\beta }(x)$ is presented below (Figure 2).

If set $A_T$ is the area in the graph of ${\eta }_{\beta }(x)$, then

$$A_T=\left\{\left(x,\alpha \right)|z_{\beta }-z_{\alpha /2}\le x\le z_{\beta }+z_{\alpha /2},0.01\le \alpha \le 1\right\}$$

(21)

If $d_T$ is the length of the bottom of $A_T$, then

$$d_T=\left(z_{\beta }+z_{0.005}\right)-\left(z_{\beta }-z_{0.005}\right)=2\times z_{0.005}=5.15$$

(22)

On the other hand, if set $A_T$ is the area in the graph of ${\eta }_{\beta }(x)$ to the left of the vertical line $x=z_i$, then

$$A_i=\left\{\left(x,\alpha \right)|z_{\beta }-z_{\alpha /2}\le x\le z_i,0.01\le \alpha \le b\right\}$$

(23)

where $b=2\times {\Phi }^{-1}\left(z_{\beta }-z_i\right)$. If $d_i$ is the length of the bottom of $A_i$, then

$$d_i=z_i-\left(z_{\beta }-z_{0.005}\right)=z_i+0.93$$

(24)

with $\beta =0.05$. Obviously,

$$d_i/d_T=\{\begin{array}{l}0,z_i\le -0.93\\ \frac{z_i+0.93}{5.15},-0.93<z_i<4.22\\ 1,4.22\le z_i\end{array}$$

(25)

and $d_i/d_T$ = 0.5 with $z_i=z_{\beta }$. Based on Chen et al. (2019), we can use $d_i/d_T$ to replace the ratio $A_i/A_T$. If we let $0<{\varphi }_1<{\varphi }_2<0.5$, then the fuzzy test rule can be show as follows:

(1)

If $d_i/d_T\le {\varphi }_1$, then do not reject $H_0$ and conclude that ${\delta }_i\le 0.1835$.

(2)

If ${\varphi }_1<d_i/d_T<{\varphi }_2$, then make no decision.

(3)

If $d_i/d_T\ge {\varphi }_2$, then reject $H_0$ and conclude that ${\delta }_i>0.1835$.

Clearly, the calculations for $d_i/d_T$ are relatively easy compared to those for $A_i/A_T$, so this approach has an advantage in practical application.

4. Case Study

As mentioned previously, this study developed a fuzzy evaluation model using a PEM for an e-learning system and formulated suggestions for system improvement. According to Hwang and Tsai [36], e-learning applications rank at the top of language learning. We conducted a case study using a CALL system that is commonly used in universities in Taiwan to demonstrate the fuzzy test method and evaluation criteria proposed in this study.

4.1. Samples

Respondents reported their perceptions regarding their satisfaction with and the importance of each item of the CALL system using Shee and Wang’s [35] web-based e-learning system (WELS) questionnaire, which is often used to collect satisfaction data regarding e-learning systems [26,27,28,29,30]. The WELS questionnaire contains 13 items under the four following dimensions: learner interface, learning community, system content, and personalization. Response options for importance are not important at all, not important, neutral, important, and very important. Response options for satisfaction are very dissatisfied, dissatisfied, neutral, satisfied, and very satisfied. Respondents were asked to express how important they felt each item was and how satisfied they felt with each item. The respondents in this study comprised students who use the CALL system at four universities in Central Taiwan. We distributed a total of 405 questionnaires and recovered 367questionnaires, representing a recovery rate of 90.6%. Twelve of the questionnaires were invalid, so the valid recovery rate was 87.7%. Among the 355 valid samples, male and female subjects occupied 34.9% and 65.1% of the subject population, respectively. The majority of the respondents were second-year students, accounting for 45.1% of the valid samples. First-year students accounted for 35.2%, while third- and fourth-year students accounted for 13.5% and 6.2%, respectively.

4.2. Results and Confirmatory Factor Analyses

With regard to reliability, SPSS analysis revealed the overall Cronbach’s $\alpha$ of the importance items to be 0.946 and the overall Cronbach’s $\alpha$ of the satisfaction items to be 0.973, which indicates that the questionnaire in this study has good reliability (DeVellis [37]). With regard to validity, confirmatory factor analysis (CFA) using LISREL for importance resulted in ${\chi }^2\left(59\right)=217.06$ ($p$-value $=$ 0.000), CFI = 0.98, GFI = 0.91, and SRMR = 0.057; and for satisfaction ${\chi }^2\left(59\right)=212.92$ ($p$-value $=$ 0.000), CFI = 0.99, GFI = 0.92, and SRMR = 0.023. These values are all acceptable (Hooper, Coughlan, and Mullen [38]). This analysis showed that the sample data had good reliability and validity and were suitable for this study.

The observed values of ${\delta }_i^{*}$, $S_i$, and $Z_i$ were computed as follows:

$${\delta }_{i0}^{*}=\frac{1}{n}\times {\displaystyle \sum _{j=1}^n{d}_{ij}},s_i=\sqrt{\frac{1}{n-1}\times {\displaystyle \sum _{j=1}^n{\left(d_{ij}-{\delta }_{i0}^{*}\right)}^2}}\mathrm{and}{z}_i=\frac{\sqrt{n}\left({\delta }_{i0}^{*}-0.1835\right)}{s_i},i=1,\dots ,13$$

Furthermore, based on these three observed values, ${\delta }_{i0}^{*}$, $s_i$, and $z_i$, we can compute the $d_i/d_T$ value shown in

Table 2. Importance and satisfaction survey for computer-assisted language learning (CALL) system.

Dimensions	Items	${\mathit{s}}_{\mathit{i}}$	${\mathit{\delta }}_{\mathit{i}0}^{*}$	${\mathit{z}}_{\mathit{i}}$	${\mathit{d}}_{\mathit{i}}/{\mathit{d}}_{\mathit{T}}$
Learner interface	1.  Ease of use	0.2782	0.2028	1.3083	0.4346 *
	2.  User-friendliness	0.2632	0.2049	1.5339	0.4784 *
	3.  Ease of understanding	0.2816	0.1704	−0.8751	0.0107
	4.  Operational stability	0.2933	0.1775	−0.3877	0.1053
Learning community	5.  Ease of discussion with other learners	0.2672	−0.1049	−20.3365	0
	6.  Ease of discussion with teachers	0.3276	−0.2268	−23.5943	0
	7.  Ease of accessing shared data	0.2711	−0.0965	−19.4619	0
	8.  Ease of exchanging learning with the others	0.2756	−0.1162	−20.4865	0
System content	9.  Up-to-date content	0.3239	0.0472	−7.9287	0
	10. Sufficient content	0.3312	0.0556	−7.2734	0
	11. Useful content	0.3275	0.0556	−7.3568	0
Personalization	12. Capability of controlling learning progress	0.3355	0.0507	−7.4585	0
	13. Capability of recording learning performance	0.3200	0.0592	−7.3205	0

* Remarks: 1. Let ${\varphi }_1=0.2$; ${\varphi }_2=0.4$; 2. When value of $d_i/d_T$ is marked *, it indicates $d_i/d_T>{\varphi }_2$.

.

4.3. Discussion and Management Implications

As shown in Table 2, $z_1=1.3083$ and $z_2=1.5339$, both of which are less than $z_{\beta }=1.645$ with $\beta =0.05$. Based on statistical testing rules, neither requires improvement. However, ${\delta }_{10}^{*}=0.2028$ and ${\delta }_{20}^{*}=0.2049$, both of which are clearly greater than 0.1835. We therefore suggest further fuzzy hypothesis testing, the results of which are $d_1/d_T=0.4346$ and $d_2/d_T=0.4784$. Both are greater than ${\varphi }_2=0.4$. Based on the previous fuzzy decision principles, Items 1 and 2 require improvement.

The model shows that when the importance index $({\theta }_i^{\prime })$ is not significantly higher than the satisfaction index $({\theta }_i)$, $({\theta }_i,{\theta }_i^{\prime })$ falls in Area M or Area T. This in turn means that $H_0$ is not rejected and that the service item in question does not need improvement. In contrast, a greater ${\delta }_{i0}^{*}$ means a greater $z_i$ as well, which means that ${\theta }_i^{\prime }>{\theta }_i$; furthermore, $H_0$ is rejected, which means that $({\theta }_i,{\theta }_i^{\prime })$ falls in Area B. In this case, the satisfaction index $({\theta }_i^{\prime })$ is significantly greater than the satisfaction index, so the service item in question needs improvement. From a practical-implications perspective, it is clear that the results of fuzzy hypothesis testing are more reasonable. The managerial implications are as follows:

(1)

Items 1 and 2 both involve the learner interface, which must be easy for users to use in order to enhance their satisfaction and attract more users.

(2)

More users mean more commercial profits, thereby enabling corporate sustainability.

(3)

Similarly, more users mean that more corporations will be encouraged to invest in e-learning systems, which will reduce the amount of carbon emissions produced by vehicle transportation even further and promote sustainable development.

(4)

More corporations will be encouraged to promote sustainable development and their joint existence and prosperity with the environment.

(5)

Increased commercial profits will also make corporations more willing to fulfill their social responsibility, create good corporate image, and form a virtuous cycle.

5. Conclusions

A PEM is a useful tool that collects the opinions of customers or users to assess the performance of an operating system. PEMs are widely applied to evaluate and improve performance in a variety of industries and organizations. Compared to other evaluation methods that require complex data comparisons, PEMs have an advantage in the ease of analysis they offer. This is also the main background and purpose of this paper—to propose a fuzzy evaluation model for e-learning. We referred to the methods used by Chen et al. [11] to construct a PEM for an e-learning system using importance and satisfaction indices. In addition, we referred to the fuzzy test methods utilized by Buckley [12] and Chen et al. [13] and simplified calculations by standardizing test statistics. We also developed fuzzy evaluation criteria based on the fuzzy evaluation method proposed by Chen et al. [11] and identified items considered critical to quality (CTQ) that need improvement using the PEM. This approach maintains the simple response model of Likert scales, which increases the efficiency and accuracy of data collection. Furthermore, the fuzzy membership function of the discriminant index was constructed based on the confidence interval, which solves the problems of sampling error and the complexity of collecting fuzzy linguistic data. We also presented a case study of a CALL system to demonstrate the efficacy of the proposed approach. In fact, the model proposed in this paper can be applied to the performance evaluations of various service industries, such as educational services, cultural creativity, and food service industries, except for the case of a CALL system. However, when applying this model in the above-mentioned industries, a suitable questionnaire needs to be made first, and then the method proposed in this paper can be applied. That is a research limitation and can also be regarded as an important topic for future research.

In e-learning, learning schedules are flexible, and the costs for learners to travel to and from their learning institutions are lower, including the carbon emissions of vehicle transportation. Constructing a comprehensive e-learning system can benefit the learning effectiveness and satisfaction of students or learners and increase the number of people engaged in e-learning, thereby creating a profitable commercial model.