Fidex and FidexGlo: From Local Explanations to Global Explanations of Deep Models^†

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

Review of the article

«Fidex and FidexGlo: From Local Explanations to Global Explanations of Deep Models»

 

The paper discusses the problem of constructing explainable artificial intelligence for a class of neural network models, including deep neural network models.

Two versions of the algorithm for explaining the decision made by the model based on a set of constructed production rules have been developed. A detailed analysis of the work in the field of approaches to improving the interpretability of black-box models is carried out.

The main results (algorithms and analysis model) have practical significance and scientific novelty.

The proposed estimates of the results of the series of experiments are justified and described in detail.

The interpretation of the results of the computational experiment and the reliability of their analysis are sufficient.

The provided list of references to bibliographic sources reflects the depth of the research problem.

The abstract, introduction and conclusions are presented correctly.

 

Main questions and remarks on the work:

1. The review does not pay attention to such a direction of constructing explainable neural network models as ANFIS – hybrid neural network and “fuzzy” models. Training fuzzy logic models using backpropagation variations and extracting a base of production rules from such models can be the basis for building expert systems.

2. Figures 5 and 6 in the form of bar histograms have low information content for the reader, due to the close values in 3 of the 4 variants.

3. When describing a series of computational experiments, it would be highly desirable to estimate both the total time of model construction and the amount of RAM, in order to confirm the O() estimate theoretically formulated in the work.

4. What are the limitations of applicability of the proposed algorithms? Is it possible to apply them to neural network models with feedback loops, such as RNN?

5. In the Conclusions section, it might be worth adding the resulting numerical performance estimates (metrics) for comparison with other models.

 

The work may be published in its current form.

Author Response

Thank you very much for taking the time to review this manuscript. Please find the detailed responses of the author below. The reviewer comments are in italic and the author’s answers are in bold.

The paper discusses the problem of constructing explainable artificial intelligence for a class of neural network models, including deep neural network models.

Two versions of the algorithm for explaining the decision made by the model based on a set of constructed production rules have been developed. A detailed analysis of the work in the field of approaches to improving the interpretability of black-box models is carried out.

The main results (algorithms and analysis model) have practical significance and scientific novelty.

The proposed estimates of the results of the series of experiments are justified and described in detail.

The interpretation of the results of the computational experiment and the reliability of their analysis are sufficient.

The provided list of references to bibliographic sources reflects the depth of the research problem.

The abstract, introduction and conclusions are presented correctly.

Main questions and remarks on the work:

1. The review does not pay attention to such a direction of constructing explainable neural network models as ANFIS – hybrid neural network and “fuzzy” models. Training fuzzy logic models using backpropagation variations and extracting a base of production rules from such models can be the basis for building expert systems.

In this work we focused on propositional rules based on Boolean logic. We added a sentence at the beginning of the Explainability Techniques Section: “In this work, we focus on propositional rules using Boolean logic but note that the field of neuro-fuzzy systems has been extensively studied, consisting of the insertion and extraction of rules based on fuzzy logic.”

2. Figures 5 and 6 in the form of bar histograms have low information content for the reader, due to the close values in 3 of the 4 variants.

We think that these close values are still informative. For example, the small change in average fidelity with variation of the drop-out parameters is a good result, because increasing these parameters makes Fidex run faster for almost the same fidelity. The same applies to the average accuracy.

3. When describing a series of computational experiments, it would be highly desirable to estimate both the total time of model construction and the amount of RAM, in order to confirm the O() estimate theoretically formulated in the work.

Unfortunately, we did not record the exact execution time or RAM usage. It would take too long to re-run the experiments as the CPUs available to us are very busy at the moment. We are very sorry for this!

4. What are the limitations of applicability of the proposed algorithms? Is it possible to apply them to neural network models with feedback loops, such as RNN?

That is an open research question. In principle, the same technique could be applied to RNNs such as Jordan/Elman networks by inserting a DIMLP layer after the input layer. In this way, the generated rules will have in the rule conditions the input variables of the input data with the addition of hidden states corresponding to the neuron states of the feedback loops.  Several sentences were added in the Discussion Section.

5. In the Conclusions section, it might be worth adding the resulting numerical performance estimates (metrics) for comparison with other models.

See above (answer to 3).

The work may be published in its current form.

Thank you!

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

This paper applies existing algorithms to explain the CNN-VGG architecture. However, the contribution is not clearly defined. At times, it was confusing whether FidexGlo was the authors' own contribution. The quality of the paper could be improved by refining the research design and explicitly highlighting the novel aspects of the work. Given that the paper focuses on complex rule-based explanations, it is crucial to clearly articulate what the rule-based approach entails and how it is applied in this context.

Comments on the Quality of English Language

The clarity of the paper is hindered by its structure rather than just language quality. While the English is understandable, the way the content is organized makes it difficult to follow the key contributions. Restructuring the paper to clearly distinguish existing work from the authors' contributions would significantly improve readability.

Author Response

Thank you very much for taking the time to review this manuscript. Please find the detailed responses of the author below. The reviewer comments are in italic and the author’s answers are in bold.

This paper applies existing algorithms to explain the CNN-VGG architecture. However, the contribution is not clearly defined. At times, it was confusing whether FidexGlo was the authors' own contribution. The quality of the paper could be improved by refining the research design and explicitly highlighting the novel aspects of the work. Given that the paper focuses on complex rule-based explanations, it is crucial to clearly articulate what the rule-based approach entails and how it is applied in this context.

It has been clearly mentioned in the last paragraph of the introduction that FidexGlo is the authors’ own contribution. Furthermore, the key contribution of this article is the use of FidexGlo with CNNs. This is mentioned in the last paragraph of the introduction.

 

Comments on the Quality of English Language

The clarity of the paper is hindered by its structure rather than just language quality. While the English is understandable, the way the content is organized makes it difficult to follow the key contributions. Restructuring the paper to clearly distinguish existing work from the authors' contributions would significantly improve readability.

All the DIMLP framework is the first author’s contribution. The DIMLP model is explained in the Materials and Methods Section. The description of DIMLP is essential for understanding the Fidex algorithm, which requires the precise location of axis-parallel hyperplanes. We do not really understand how to restructure this paper better, as all the models are described in a separate subsection.

Reviewer 3 Report

Comments and Suggestions for Authors

This work introduces two algorithms, Fidex and FidexGlo, that explain black-box machine learning model responses. Fidex constructs local explanations for individual samples, while FidexGlo extends this to global rule sets. Both algorithms use propositional rules with hyperplane boundaries parallel to input variables for high fidelity and accuracy. Their applications to ensembles, SVMs, and CNNs demonstrate performance on benchmark datasets and accurate explanations for image classification problems. 

The manuscript does not address the behavior of the rules or explanations extracted by the methods in adversarial conditions or with noisy data inputs, which will be crucial for actual deployment in critical applications.

Alternative techniques, like Grad-CAM, LIME, and Shapley values, are discussed very briefly without going deeper into a comparative analysis of the trade-offs of both Fidex and FidexGlo against the competing methods.

These experiments consider all standard datasets: MNIST, FER, and Cracks. However, the manuscript does not discuss other more challenging datasets with which complex model explainability would be at stake in a time-series or multimodal setting.

Though the manuscript argues for the superiority of propositional rules in explaining and predicting, it does not discuss scenarios in which other forms of explanation—for example, visual or probabilistic—can be more effective.

While the manuscript does present a comparison of performance metrics such as fidelity and accuracy, it does not benchmark runtime efficiency or memory usage, which is necessary for practical adoption.

Though propositional rules are touted as interpretable, the manuscript scantly discusses how users and domain experts perceive and utilize them in making decisions.

The manuscript contains all the details about implementation but does not explain how others can reproduce these experiments. Key elements, such as the data preprocessing steps and random seed initialization, are missing or unclear.

Most of the figure resolution and text readability are poor.

 

Author Response

Thank you very much for taking the time to review this manuscript. Please find the detailed responses of the author below. The reviewer comments are in italic and the author’s answers are in bold.

This work introduces two algorithms, Fidex and FidexGlo, that explain black-box machine learning model responses. Fidex constructs local explanations for individual samples, while FidexGlo extends this to global rule sets. Both algorithms use propositional rules with hyperplane boundaries parallel to input variables for high fidelity and accuracy. Their applications to ensembles, SVMs, and CNNs demonstrate performance on benchmark datasets and accurate explanations for image classification problems. 

The manuscript does not address the behavior of the rules or explanations extracted by the methods in adversarial conditions or with noisy data inputs, which will be crucial for actual deployment in critical applications.

Yes, this is a very interesting research question. Note that in a perfectly secure environment (for example, with a computer that is not connected to the network), there will be no attack, and so our framework is correct. Now suppose that there is noise in the data (without adversarial attacks), then a model will be able to be robust up to a given noise threshold. The rules will also be robust if noise has been taken into account during training, as FidexGlo tends to generate rules with high fidelity.

Our main aim here was to determine the characteristics of the rules extracted from the benchmark datasets, and also to find out whether the rules are understandable with the centroids represented (and indeed they are). Noisy data inputs or adversarial conditions are left for the future.  A new research programme will address them.

We added in the Discussion:

This work does not address the behavior of the rules or explanations extracted by the methods under adversarial conditions or with noisy data inputs. Our main goal was to determine the characteristics of the rules extracted from several benchmark datasets and to find out whether the rules are understandable with the centroids represented as images. In the future, it will be very important to investigate the behavior of the rules under adversarial conditions to determine whether they are robust to attacks.

 

Alternative techniques, like Grad-CAM, LIME, and Shapley values, are discussed very briefly without going deeper into a comparative analysis of the trade-offs of both Fidex and FidexGlo against the competing methods.

We added the following text in the Discussion:

Grad-CAM, LIME, and SHAP are local methods. Grad-CAM can be used with neural networks, but not with decision trees, while LIME and SHAP are agnostic. Fidex is local but not agnostic, although it is currently used with MLPs, DTs, SVMs and CNNs. For SHAP, the global scores of the variables for each class can be obtained by averaging the Shapley scores of all samples. Therefore, SHAP can be used as a global method. FidexGlo is global and its computational complexity scales quadratically with the number of training samples, whereas SHAP scales exponentially with the number of input variables. Furthermore, LIME learns a new linear model in the neighborhood of a target sample, but there is no guarantee that the new local model matches the black-box model. This is in contrast to Fidex, which does not perform any new learning, but searches for the best hyperplanes, according to the fidelity on the training samples. Finally, while Grad-CAM, LIME, and SHAP can highlight the most important regions in an image, they may not provide detailed insights into why a specific prediction was accomplished. This can be a drawback in cases where fine-grained understanding is required.

 

These experiments consider all standard datasets: MNIST, FER, and Cracks. However, the manuscript does not discuss other more challenging datasets with which complex model explainability would be at stake in a time-series or multimodal setting.

The main limitation of FidexGlo is its squared algorithmic complexity with respect to the number of samples. With time series containing a reasonable number of samples (not millions), we can imagine a CNN learning the data. Propositional rules can then be extracted and visualized, as it was accomplished in this work. In a multimodal setting, e.g. with time series and tabular data as input variables, the same applies, except that the extracted rules will have both temporal and tabular data in the antecedents. Similar sentences have been added in the Discussion.

 

Though the manuscript argues for the superiority of propositional rules in explaining and predicting, it does not discuss scenarios in which other forms of explanation—for example, visual or probabilistic—can be more effective.

We have never claimed the superiority of propositional rules (it is not our intention). Can you tell us where the superiority is mentioned? In terms of visual explanations, our propositional rules are also visual for images, as the rule antecedents emphasize relevant areas. For the probabilistic approach, can you be more specific? Which method? Naive Bayes? Without more details, we do not really know how to explain that a probabilistic technique is more effective in a particular context.

 

While the manuscript does present a comparison of performance metrics such as fidelity and accuracy, it does not benchmark runtime efficiency or memory usage, which is necessary for practical adoption.

Unfortunately, we did not record the exact execution time or RAM usage. It would take too long to re-run the experiments, as the CPUs available to us are currently very busy. We are really sorry for that !

 

Though propositional rules are touted as interpretable, the manuscript scantly discusses how users and domain experts perceive and utilize them in making decisions.

In this paper, we used public domain datasets (including a medical dataset on breast cancer diagnosis). We do not know exactly how domain experts perceive and use propositional rules in decision making and this question deserves a new article on the subject. We have the intuition that the variables in the conditions of the rules are a key factor in decision making.

 

The manuscript contains all the details about implementation but does not explain how others can reproduce these experiments. Key elements, such as the data preprocessing steps and random seed initialization, are missing or unclear.

The only data preprocessing is normalization, which is already mentioned. See Sect. 4.2: “To train the DIMLPs, the data were normalized using Gaussian normalization. Specifically, during cross-validation trials, the means and standard deviations of each variable were calculated for the training datasets, and then these values were used to normalize the training data and the testing data. Normalization was not applied to the training of ensembles of DTs (RF and GB) as this is not necessary. Finally, for the QSVMs, the normalization was encoded in the weight values of the first layer (see Sect.~\ref{qsvm}), which was frozen during training.”

For CNNs the normalization is already explained in corresponding sections (5.1, 5.2. and 5.3); essentially the DIMLP layer makes the normalization. Regarding seed values, this was not mentioned; so, we added in the text (in 4,3): “For each training phase, the seed value was that of the process assigned by the Linux Operating system.”

 

Most of the figure resolution and text readability are poor.

From Figures 7 to 14, the low resolution of the images is normal, because the data learned by the CNNs represent images of resolution 28x28, 48x48 and 64x64. The texts in Figures 2, 3, 4, 5 and 6 are now easier to read, as the Figures have been enlarged.