An Artificial Intelligence (AI) Framework to Predict Operational Excellence: UAE Case Study

Round 1

Reviewer 1 Report

The paper discusses the European Foundation for Quality Management (EFQM) and how it can be applied to various sectors. There is a lot of discussion regarding the entities involved with EFQM and how data is shared among them. However, it barely has any AI implemented to make this a better system. Integrating k-means clustering with no proper reasoning is not an acceptable model for this issue. Authors would benefit from submitting this to a non-CS journal. Some suggestions:

1. There is no need to include such a big table for a literature survey when the papers discussed are not even related to EFQM.

2. Provide a system architecture with all the entities involved rather than flowcharts

3. Phases in AI - data acquisition, preparation, ... . These are standard phases that have existed in AI for a very long time. You don't need to talk about all this.

No comments

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

The abstract could be improved by focusing at first on the general context and more describing what is the problem, and how it will be solved. The readers need to understand clearly, what is the added value of the paper.

A small paragraph at the beginning of the introduction section to explain in which cases it is important to use the standard of quality management will increase the quality of the paper.

At the end of the introduction section, a sentence describing the structure of the paper could increase the quality of the paper.

I think that the literature review has to be more structured at the beginning of the section 2. May be a section on this part will increase the quality of the paper. Indeed, it is difficult to clearly know what has been done by others and what you propose in you put both in the same section. The section 2.1 could be divided in sub-sections to focus on different keywords such as quality management or AI.

A sentence at the end or at the beginning of each section to link the sections between them is required. Please read slowly the paper once again and pay attention to mistakes in the paper (space for instance).

The last sub-sections of the section 2 are well structured.

The section 3 corresponds to what is expected in the paper structure.

The conclusion needs to be improved. This part is too short regarding what has been presented in the paper. Indeed, you have to show what you have done and what it will change. I’m not sure it is important to have two sub-sections in the conclusion even if the global structuration will focus on what has been done and perspectives.

The global structure of the paper could be increased for well showing the interest of this research.

Good quality of language

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 3 Report

Some claims on AI are entirely wrong: claiming that "Machine learning allows machines to develop their programs" entails that machines can solve the halting problem, which is impossible. Still, most training-based AI learn by approximating a total (i.e., always terminating) function (universal approximation theorem), mainly by fitting a problem by parameter tuning and space search. So, they mostly simplify reality by assuming the data will fit a specific distribution. Furthermore, k-means clustering is an unsupervised algorithm, so no "training" is actually happening under the hood! ("will train the algorithm", 2nd paragraph of 2.2). So, the authors should better clarify what they mean by training using an unsupervised approach.

On the other hand, k-means clustering is mainly related to mining algorithms, which do not necessarily fit a model but use data distance or similarity measures for grouping the data together. Also, k-means can be run with k>2: the authors should clarify why they think that 2 is the best parameter so far (e.g, the problem always involves 2 classes to discriminate?). Furthermore, the traditional k-means clustering algorithm is not necessarily able to associate a class label with a specific data element, as it mainly separates the data space. The authors should better reflect on whether they discarded any training-based algorithm (which would have also provided a classification label to the elements within the cluster).

Overall, the overall formulation of the training problem seems rather limited: how many classes are associated to each data point? If 2, then using any value except from k=2 will make little sense, as you are dealing with a binary classification problem (in fact, it seems that the two classes of interest are 'High' and 'Low', but this is not made explicit from the context). As it seems you are also using sci-kit learn, have you tried comparing k-means clustering with other classification methods, such as decision trees, logit models, or neural networks? By doing this, you can clearly see whether the data is merely randomly distributed (i.e., training-based solutions will yield very low precision/recall values). E.g., Decision trees will also partition the data space in multiple sub-clasters, while also guaranteeing to keep the information as compact as possible; you might the compare the k value with the tree's height/depth.

Overall, the authors provide very little if no reflection on the reasons why the same approach will provide unsuitable accuracy for the new model: the authors are therefore warmly advised to re-run the experiment with other simple training-based models easily accessible from scikit learn, for then comparing the outcome of different algorithms, both supervised and unsupervised. This would help the paper to be in a stronger position for better substaining the overall project and idea.

In section 1.1, the authors acknowledge the different versions of the quality protocol with their subsequent extensions. Still, the authors should have elaborated on whether there was any previous attempt at automating this for the prior versions and, if so, the challenges for extending any previous implementation to the updated requirements.

When referring to the other published papers in Sect. 2.1. the authors forgot to mention how they attempted to solve the problem and did not explain whether AI was in any way also used in previous approaches. This discussion is mandatory in journal papers, as they need to be self-contained.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 4 Report

The manuscript addresses an important issue: the integration of artificial intelligence into the European Foundation for Quality Management (EFQM) business excellence model. It is hard to disagree with the authors that this is a promising approach to improving the efficiency and effectiveness of excellence in organizations. It is also a good idea to discuss the results obtained for a case study involving a UAE government organization serving as a sample to train the AI framework. The unsupervised machine learning technique k-means clustering (with k=2) was used to evaluate the organization. The thesis that if the predicted EFQM score is not high enough, then the AI framework provides feedback to decision makers regarding the criteria that need reconsideration is also correct. I also agree with the statement that this approach can be considered as an innovative contribution and enhancement to knowledge body and organizational practices. Well, there are a few shortcomings in the manuscript that need to be removed.

1. The Introduction lacks the purpose of the work. There is no description of the organization of the sections in the manuscript.

2. The description of the data for which accuracy is presented in the manuscript is not clear. In this article, data is the basis. The description of the techniques used is very general and does not add anything. This description should be extended and several mathematical relationships should be provided.

3. Minor typos and editing errors:

a) Most figures extend beyond the right margin;

b) page 1: Figure.1 => Figure 1

Figure is always written with a capital letter;

c) page 2: Figure1 => Figure 1

To sum up, after making the necessary corrections that I have indicated, the manuscript will meet the criteria for publication.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

All comments have been addressed

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 3 Report

I appreciate the authors' efforts in implementing most of the comments. Still, the authors should clarify on the the last point of the following, after which the paper might be accepted:

* Which were the hyperparameters providing the results listed in the final comparison table between machine-learning techniques?

* Clearly state which are the two classes of interest.

* It seems that the authors' accuracy score is currently not the best practice for determining k-means precision. «The problem is that "cluster labels" have arbitrary values assigned by the clustering algorithm, for example, all samples labelled as 0 could have been labelled as 1, 2 or 3 and vice versa. They could also be A, B, C, D, or blue, green, red, yellow, they just need to be 4 distinct groups (clusters).» So, this might bias the precision analysis as originally provided by the authors. The best practice in these cases is to use Rand index adjusted for chance (https://datascience.stackexchange.com/questions/118374/how-to-compare-labels-from-clustering-analysis-and-original-ones).  Given that you were not re-implementing precision to use similarity-distance based evaluation, you need to re-consider the metric for the evaluation into an appropriate one. This is because clustering is not actually a prediction algorithm.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf