1. Introduction
Artificial neural networks (ANNs) become more and more important in many fields, such as science, medicine, and industry [
1,
2,
3,
4,
5,
6] since they can approximate a system based on only data, without the knowledge of concrete requirements or behavior model of the system [
7]. For example, in an intelligent self-adaptive software system, there usually have two kinds of components: the subsystem components describing the system’s behavior and the adaptation component making self-adaptation decisions; we usually cannot describe the behavior of the adaptation component due to the uncertain and unpredictable environment. So, ANNs are usually applied to describe the adaptation components [
7].
Coming with their advantages, however, ANNs are usually black boxes and inherently have the inability to explain the process of decision making and output generation in a comprehensible form. The lack of interpretability limits their extensive usage, especially in safety-critical domains, such as unmanned aerial vehicles, unmanned ground vehicles, and autonomous vehicles. Hence, except the successful applications of machine learning models, there has been a growing demand for explainable artificial intelligence (XAI) [
8,
9,
10]. Interpretable local surrogates, occlusion analysis, gradient-based techniques, and layerwise relevance propagation are four kinds of post hoc methods for XAI [
8]. However, these methods cannot naturally explain system behavior. The rule-based explanation provides a way to reveal the hidden knowledge of a network intuitively and approximate the network’s predictive behavior [
11]. Especially, fuzzy rules can not only provide an intuitive explanation but also support rigorous semantic analysis and prediction inference [
12]. Hence, in this paper, we focus on XAI via fuzzy rules.
Three kinds of techniques have been proposed for rule extraction from ANNs: decomposition techniques which work on the neuron-level, pedagogical techniques which regard the whole network as a black box, and eclectics techniques that combine both decomposition and pedagogical approaches [
13]. These techniques usually focus on crisp rule extraction [
14,
15,
16,
17]. In Reference [
14], rules are extracted based on (continuous) Boolean function approximation of each perceptron. Decision trees are widely used in rule extraction from neural networks. For example, in Reference [
15], a decision tree is generated from a trained neural network, and rules are then extracted from the decision tree. This method is extended to extract rules from ensemble neural networks [
18] and deep neural networks [
19,
20].
However, crisp rules are lack of flexibility, adaptation, and cognitive understanding. Compared with crisp rules, fuzzy rules provide a reasonable and cognitive format to explain ANNs with the following advantages.
Combining numeric and linguistic, a fuzzy inference system is not only understandable but also has well-defined semantics. It can be used in a complementary way with other model languages for property analysis.
Fuzzy rules provide an interface between data and variable symbols, and the variables can be reconfigured in the running time. Thus, the running data can be incorporated into the model. Via fuzzy inference reasoning, fuzzy rules show great adaptation.
If ANNs are to be integrated within traditional software systems that need to be verified, then ANNs must meet requirements. Rule extraction algorithms provide a mechanism for either partially or completely decompiling a trained ANN. This allows us to compare extracted rules and the software specification if we have.
There are a few works on the ANN explanation by fuzzy rules. The authors in Reference [
21] proposed a method to generated Sugeno fuzzy rules from sample data through learning; the methods in References [
22,
23] extracted Sugeno fuzzy rules based on the continuous activation functions of ANNs. However, for the interpretability purpose, Mamdani fuzzy rules are preferred since they are more intuitive and interpretable and have widespread acceptance [
24]. The first method is to generate fuzzy rules from data directly [
25,
26,
27]. For example, in Reference [
25], an algorithm is proposed to generate Mamdani fuzzy rules directly from data, which has been widely applied to design fuzzy controllers [
28,
29]. Note that this method can also be regarded as a pedagogical technique to explain an ANN since we can generate fuzzy rules from the training data of an ANN or the sample data computed by the ANN, which, in turn, can approximate the whole network. Fuzzy rules can also be extracted from fuzzy neural networks (FNNs) [
30,
31,
32]. For example, in Reference [
30], a fuzzy neural network, combining with the fuzzification layer and a three-layer ANN, is proposed to extract fuzzy rules according to casual index. Even though the methods in References [
14,
15] are designed for the extraction of crisp rules, they can also be applied to generate fuzzy rules from ANNs via fuzzification.
Even though some methods have been proposed for fuzzy rule extraction, they are evaluated on some specific data sets. There is no study on the evaluation of explanation accuracy with various data sets, which is important for their further applications. That is: which method can deliver an accurate fuzzy rule-based explanation for ANNs trained from various data sets? In this paper, we will exam three typical decomposition methods, i.e., References [
14,
15,
30], to show their quality. The first method [
30] extracts fuzzy rules from an FNN. In the network, the data of each input variable is first transformed to a membership value vector related to its fuzzy numbers; taking the membership value vectors as the input layer, an ANN is trained, where the output layer is the membership value vectors of the output variables. Then, by computing causal index for each pair of input neurons and output neurons, i.e., the partial derivations of output neurons with respect to input neurons, fuzzy rules can be extracted to approximate the neural network. Rather than using a fuzzy neural network, the second method [
14] and the third method [
15] extract rules from a well-trained ANN, which can be translated to fuzzy rules. In Reference [
14], the behavior of each perceptron is approximated by (continuous) Boolean functions with respect to its input neurons, and a set of logical expressions are generated to approximate the whole network, from which we can generate fuzzy rules with two fuzzy numbers for each variable. In Reference [
15], after dividing the continuous domain of each continuous output variable into discrete intervals by clustering, a set of decision trees are built for each type of output cluster. Based on the trees, a set of continuous/discrete rules are extracted, and we can generate fuzzy rules by designing proper fuzzy numbers for each variable.
To compare their qualities, focusing on discrete, continuous, and hybrid data sets, we compare the generated fuzzy rules with a standard pedagogical method [
25], as this method can be used to generate a benchmark of fuzzy rules to show the knowledge learned by a convergent NN. The results show that the causal index-based method [
30] is not robust and cannot generate proper fuzzy rules to describe the data sets; the logical expression-based method [
14] can generate proper fuzzy rules on discrete data sets, but induce high errors on continuous or hybrid data sets due to the 2-fuzzy-number fuzzification; the decision tree-based method [
15] can usually generate approximate fuzzy rules on discrete, continuous, and hybrid data sets, but cannot guarantee the accuracy for data sets with poor separability. Hence, there are no general methods to extract fuzzy rules that can explain different ANNs accurately. It guides us to develop new methods to extract fuzzy rules from ANNs for explanation.
The paper is organized as follows.
Section 2 gives a brief review of the three rule extraction methods.
Section 3 introduces the data sets, the trained ANNs, and the measures for comparison of different fuzzy rules.
Section 4,
Section 5 and
Section 6 perform the detailed evaluations on the three methods, respectively. The conclusion is shown in
Section 7.
5. Evaluation on iDRF-2
In this section, we first generate fuzzy rules using iDRF-2. For comparison, with the membership functions computed from iDRF-2, we generate fuzzy rules based on DFR. Finally, we give an evaluation on iDRF-2.
5.1. (Continuous) Logical Expressions and Fuzzy Rule Generation
As described in
Section 2.2, given the trained ANNs shown in
Figure 5, each discrete variable is approximated by two values (i.e., 0 or 1), and each continuous variable
x is approximated by two functions (i.e., x or 1 − x). Hence, we can obtain the (continuous) logical expressions based on iDRF-2. The results are shown in
Table 7, where “
” represents “
”, "
" represents “
”, and
represents the negative of
. Note that, in the continuous logical expression,
means
, while
means
.
Based on the logical expression, we can obtain the related fuzzy rules. Take the logical expression for the nonlinear function as an example. If and , , so we have ; if and , , so we have ; similarly, if and , ; if and , . Hence, we have the following fuzzy rules:
IF is S and is S, THEN y is S,
IF is L and is S, THEN y is L,
IF is S and is L, THEN y is L,
IF is L and is L, THEN y is S.
Similarly, we can generate fuzzy rules for other data sets.
5.2. Fuzzy Rule Generation Using DFR
Now, we apply DFR to generate fuzzy rules from data directly. Based on the logical expression, we can fuzzify each variable into two fuzzy numbers: S (small) and L (large), whose membership functions are given in
Figure 7. Hence, based on DFR, we find that the rules for the discrete data sets are the same as those given in
Table 2, and the rules for the linear function are:
IF is S and is S, THEN y is S;
IF is S and is L, THEN y is S;
IF is L and is S, THEN y is S;
IF is L and is L, THEN y is L.
The rules for the iris data set are:
IF is S and is S, THEN y is Setosa;
IF is S and is L, THEN y is Versicolor;
IF is L and is S, THEN y is Versicolor;
IF is L and is L, THEN y is Virginica.
Note that we cannot generate proper rules for the nonlinear function based on DFR since each potential IF part has multiple THEN parts with the same rule degree.
5.3. Analysis and Evaluation on iDRF-2
Table 7 also gives a comparison of the generated rules based on DFR and iDRF-2. In the table, the third column shows the number of rules generated by DFR where each variable is associated with two fuzzy numbers (i.e., S and L), and the fourth column shows the number of rules generated by iDRF-2 where the first and the second items represent the numbers of rules in and not in the set of rules generated by DFR, respectively. We can find that except for the nonlinear function, with 2-fuzzy-number fuzzification, the accuracy of iDRF-2 is 100%, and inaccuracy is 0, which means the rules generated by iDRF-2 are the same as those generated by DFR. Hence, iDRF-2 performs well if each variable is fuzzified with two fuzzy numbers.
However, the main limitation of iDRF-2 is that it can only deal with the situation that each variable can be fuzzified with two fuzzy numbers directly or after some transformation. Hence, iDRF-2 can be used and perform well for data with discrete domains since the discrete domains can be reduced to
domain by dummy variables, which can be well fuzzified with two fuzzy numbers. For the continuous domains, if a variable is fuzzified with only two fuzzy numbers, it may induce large errors or inaccuracy. For example,
Figure 8 shows the data distribution of the linear function data set, where the points show the distribution of the fuzzy numbers of
y with respect to the input variables, and the alphabets show the output fuzzy numbers of the generated rules with respect to the combinations of the input fuzzy numbers. For the 2-fuzzy-number fuzzification, there are 25% data whose relationship between the input and output variables cannot be represented by the generated rules (
Figure 8a), while, for 3-fuzzy-number fuzzification, 22% data cannot be described correctly by the generated rules (
Figure 8b). Using DFR, we can get higher accuracy with more fuzzy numbers, which cannot be achieved by iDRF-2.
In conclusion, iDRF-2 can generate accurate fuzzy rules to describe the relation of discrete data sets but does not perform well in continuous and hybrid data sets. Hence, it can be used to explain ANNs which are applied in discrete domains but is not suitable to explain ANNs trained in continuous domains.
6. Evaluation on iDRF-3
In this section, we evaluate the performance of iDRF-3.
6.1. Clustering and Fuzzy Rule Generation
Based on iDRF-3, we first generate the hidden-output and input-hidden decision trees. To generate the hidden-output decision tree, we need to compute the values of the hidden neurons based on the training data. In the sequel, we give the detailed procedure for each data set.
6.1.1. Discrete Data Sets
Given the training data
and the well-trained ANNs, we can compute the values of the four hidden neurons, denoted as
. Based on the data set
, we can build the decision tree for the hidden-output layer. Take the “XOR” data set as an example, whose trained ANN is shown in
Figure 5c, and the values of the input, hidden, and output neurons are shown in
Table 8.
Based on the hidden and output data, we can build a hidden-output decision tree for each target output.
Figure 9a shows the hidden-output decision tree for the target
. Based on the decision tree, we have an intermediate rule:
. Then, we can build the input-hidden decision tree with the target
, which is shown in
Figure 9b. Based on this decision tree, we have
and
. Hence, we can extract the following rules for the target
: “IF
and
, THEN
” and “IF
and
, THEN
”. Similarly, for the target
, the decision trees are shown in
Figure 10, and we can obtain the rules: “IF
and
, THEN
” and “IF
and
, THEN
”. For the data sets of “AND” and “OR”, we can do the same procedure and generate the rules.
Based on the generated rules, we can fuzzify each variable with any number of fuzzy numbers, e.g., those shown in
Figure 6 and
Figure 7, and obtain the related fuzzy rules, which are the same with those shown in
Table 2.
6.1.2. Continuous Data Sets
For the continuous data sets, we first use k-mean to cluster the output values into three categories. For each category, we then build the related hidden-output and input-hidden decision trees. For example, in the data set of the linear function, the output is clustered into three discrete ranges:
,
, and
.
Figure 11a shows the decision tree whose target is that the output is in the first discrete range, and we can extract a rule “IF
&
→
”.
Figure 11b represents the input-hidden tree with the target
&
, where each rectangle represents a path in the tree that leads to the target leaf, and the number in it denotes the number of samples belonging to this path. For example, the largest rectangle represents the path “
&
→
&
”, and there are 64 samples satisfying this path. For simplicity and to reduce over-fitting, this decision tree is then pruned by omitting the rectangles with a few samples and approximating some adjacent rectangles with one rectangle. Hence, the decision tree in
Figure 11b is pruned to three rectangles, i.e., the three large blue rectangles, which can extract three rules: “
&
→ target”, “
&
→ target”, and “
&
→ target”. Finally, we can generate the following input-output rules:
& →;
& →;
& →.
Similarly, the decision trees for the range
is shown in
Figure 12. We can extract the following rules:
& →;
& →;
& →.
Figure 13 shows the decision trees for the middle range. For the input-hidden tree, it is hard to find proper pruning directly. However, based on the results of the other two ranges, we can find that we can partition each input variable into
,
, and
. Based on such a partition, as shown in
Figure 13b, we can obtain the following rules:
& →;
& →;
& →.
Based on the above extracted rules, we can generate fuzzy rules. We first fuzzify each variable using the membership functions shown in
Figure 6 with
and
. Thus, the above nine rules can be transformed into a set of fuzzy rules, whose formulas are the same as those given in
Table 3.
Similarly, for the data set from the nonlinear function, based on the decision trees and with proper pruning and approximation, we can fuzzify each variable using the same membership functions in
Figure 6 with
and
. Then, the generated fuzzy rules are with the same formulas of those in
Table 3.
6.1.3. iris Data Set
In the
iris data set, the output variable is discrete, so we can build decision trees for each discrete value. First, for the type “Setosa”, the decision trees are shown in
Figure 14. Based on the decision trees, we can extract the following rule:
. Second, for the type “Versicolor”, the pruned decision trees are shown in
Figure 15, and we can extract a rule:
. Based on the decision trees, shown in
Figure 16, for the target Virginica, we can extract rules:
, and
.
Based on these rules, we fuzzify either input variable into S, M, and L, whose membership functions are shown in
Figure 6 with
and
, and generate fuzzy rules:
IF is S, THEN y is Setosa;
IF is M and is S, THEN y is Versicolor;
IF is M and is M, THEN y is Versicolor;
IF is L, THEN y is Virginica.
6.2. Fuzzy Rule Generation Using DFR
For the discrete data sets, no matter what kinds of membership functions applied, the generated fuzzy rules using iDRF-3 are the same as those generated with DFR, consisting of those given in
Table 2. For the continuous data sets, if the membership functions are determined with the results of data clustering, then the fuzzy rules generated by DFR and iDRF-3 are the same, which have the same formula with those in
Table 3. Based on the new membership functions, we can obtain the fuzzy rules for the
iris data set using DFR, which is shown as follows.
IF is S and is S, THEN y is Setosa;
IF is M and is M, THEN y is Versicolor;
IF is M and is L, THEN y is Virginica;
IF is L and is M, THEN y is Virginica;
IF is L and is L, THEN y is Virginica.
6.3. Analysis and Evaluation of iDRF-3
Based on the generated fuzzy rules from DFR and iDRF-3, the discrete and continuous data sets, DFR and iDRF-3 are consistent. For the hybrid data set, comparing the two sets of fuzzy rules by DFR and iDRF-3, we can find that, in the set of fuzzy rules generated by iDRF-3, there are four rules that DFR does not generate: “IF is S and is M, THEN y is Setosa”, “IF is S and is L, THEN y is Setosa”, “IF is M and is S, THEN y is Versicolor”, and “IF is L and is S, THEN y is Virginica”, while the rule “IF is L and is M, THEN y is Virginica” does not exist in the rules generated by iDRF-3.
The reason for the missing of the four rules using DFR is that there is no data related to these four rules, which means that their degrees are 0, so DFR does not generate these rules, while iDRF-3 focuses on the boundaries that can separate different kinds of data without considering the region without any data. This means the extracted rules may contain empty regions using the generated boundary. Moreover, to reduce over-fitting, we prune some branches of the decision trees since there are a few samples in the leaf nodes with large depth. This will result in the missing of some fuzzy rules generated by DFR since they only contain very few samples. However, one drawback of iDRF-3 is that, sometimes, it is not easy to extract proper rules from the decision trees and compute a unified membership function for a variable among different output targets, especially for data with poor separability.
In conclusion, both DFR and iDRF-3 can generate proper fuzzy rules for discrete, continuous, and hybrid data sets, but may induce errors. Hence, iDRF-3 can be applied to explain ANNs trained by discrete, continuous, and hybrid data sets, and cannot guarantee the accuracy if we have no knowledge of the data relations (e.g., separability).