A Quali-Quantitative Analysis Model Integrating Fuzzy Analytical Hierarchy Process and Cost–Benefit Analysis for Optimizing KPI Implementation: Insights from a Practical Case Study Application

Round 1

Reviewer 1 Report (Previous Reviewer 1)

Comments and Suggestions for Authors

Dear,

Thank you very-very much for taking into account most of my previous comments, so that the offered framework makes much more sense to me.

Below some other suggestios. Please take into account formatting details, as there is a lot of room for improvement. See, for example:

The places with several references together are not correctly formatted, in terms of citing. E.g.

• Line 154: The references should be cited this way [17-18]
• Similar in lines 100, 114, 245, etc. Please review all lines with more than one reference together.

 

The list of references does not correspond to the text: e.g. in line 197 Ortega is cited, but the 11^th reference corresponds to Thengane et al. Same happens to ref [7], which stands for both, the explanation of the Saaty matrix and a combination of CBA and MCDM.

 

Line 154: “financial, and social perspectives [17] [18]ultimately facilitating optimal resource alloca”,

• A space between 18] and ultimately is needed
• It seems that a comma is missing (the sentence is too hard to understand, and it is an important sentence…)

Line 197: Delete the space between the 19] and the full stop in “project alternatives [19] .”

Line 197: I think that the “2” should be deleted in “Ortega 2[11] reports”

Thus, a deep review of the citations and the references is needed.

Finally, in my opinion, the impact of the article is rather limited, but the development you propose is actual and fruitful. Thus, if you try to increase the impact it can be more usable for both the industrial and the academic community.

Hope you do it.

The reviewer

Author Response

Please see the attachement

Author Response File: Author Response.docx

Reviewer 2 Report (Previous Reviewer 3)

Comments and Suggestions for Authors

Dear Authors, 

Even though some issues are solved (entirely or partially) still I would like to underline some other issues that require to be treated accordingly.

The most important issue is related to the methodology. So, in my opinion, instead of including and describing the steps for CBA and FAHP respectively, it is better to design a graphical representation of your approach that contains all the major elements and their flow in your particular research. This could be considered as an original part of the article. Many things are well known, including the steps of those two methods, already mentioned. That's why I recommended to remove the Figure 1. 

Then, I would like from you to insert few ideas based on the extent to which the research answered the two research questions:

RQ1: How can qualitative and quantitative aspects of KPI selection be integrated 
through FAHP and CBA in constrained industrial environments?
RQ2: What benefits does this integrated approach offer over traditional, single-
method approaches?

These things could be treated in the last part of the article, lets say in the Conclusions section.

Last but not least, in the idea of ​​highlighting the utility of the research, I request that there be a clarification regarding how such an approach can be generalized, with adequate results, in the case of other entities with a similar or complementary profile. Through these clarifications, I believe that limitations of the research carried out in this case can be extracted (and not through the perspective of the research proposed in the future).

For sure, these issues need to be addressed adequately to clarify some sensitive aspects in the analysis carried out. 

Comments on the Quality of English Language

The paper needs some proofreading sessions... (spelling, grammar...)

Author Response

Please see the attachement

Author Response File: Author Response.docx

Reviewer 3 Report (Previous Reviewer 4)

Comments and Suggestions for Authors

Dear authors 

The article can be considered after a major revision. Please consider all the comments below: 

1. The abstract claims that the model provides a "flexible and adaptable framework for KPI selection," but it does not provide evidence or examples to support this claim.
2. The abstract briefly mentions the integration of CBA and FAHP but does not explain how these methods were combined or why they were chosen.
3. The introduction states that existing methods for KPI selection "predominantly focus on either quantitative or qualitative evaluations," but this claim is not supported by specific examples or citations.
4. The introduction mentions that "some literature suggests that integrated frameworks combining qualitative and quantitative methodologies already exist," but it does not provide specific examples or citations.
5. The introduction briefly mentions a case study of a Cambodian charcoal factory but does not explain why this specific case was chosen or how it is representative of broader industrial challenges. All the authors are from Italy, in particular. 
6. The introduction claims that the model's structure ensures "universal applicability," but this claim is not supported by evidence or examples beyond the single case study.
7. The literature review underscores the limitations of both CBA and FAHP in isolation and highlights the need for an integrated framework that combines quantitative and qualitative approaches. Addressing these gaps could lead to more effective KPI selection and implementation, particularly in resource-constrained environments. Future research should focus on developing simplified, adaptable models that can bridge the divide between financial and strategic considerations.
8. CBA focuses primarily on quantifiable financial metrics, making it difficult to incorporate qualitative aspects that are crucial for long-term success and stakeholder satisfaction. This limitation can lead to decisions that are economically sound but strategically misaligned.
9. CBA focuses primarily on quantifiable financial metrics, making it difficult to incorporate qualitative aspects that are crucial for long-term success and stakeholder satisfaction. This limitation can lead to decisions that are economically sound but strategically misaligned.
10. Although the combination of CBA and FAHP aims to balance financial and qualitative factors, the methodology does not provide a seamless integration framework, potentially leaving room for misalignment between the two approaches.
11. The methodology focuses on FAHP without thoroughly considering other modern MCDM methods that might provide better scalability, reduced complexity, or improved handling of interdependencies among criteria.
12. Relying on only two experts for interviews and pairwise comparisons in the FAHP process may introduce bias and reduce the robustness of the results, as their perspectives may not fully represent the organization's diverse needs.
13. Even though interviews were conducted separately, the close working relationship between the CEO and CFO could still lead to aligned or influenced responses, reducing the independence of their inputs.
14. The KPIs were selected based on discussions between the authors and top management, which may reflect the management's priorities rather than a comprehensive assessment of the company's needs.
15. The discussion dismisses Availability as economically unviable but does not delve into the underlying reasons for its poor performance, such as operational constraints or data limitations, which could provide valuable insights for future improvements.
16. While the conclusion mentions the model's potential applicability to other industries and contexts, it does not provide concrete examples or evidence to support this claim, leaving its scalability and adaptability largely unproven.
17. While the study identifies top-performing KPIs, it does not provide detailed guidance on how the company (or others) should implement these KPIs, such as steps for data collection, monitoring, or addressing potential challenges.
18. The discussion and conclusion do not critically assess the potential biases or limitations of relying on expert opinions for both CBA and FAHP, which could undermine the objectivity of the results.

Author Response

Please see the attachement

Author Response File: Author Response.docx

Reviewer 4 Report (Previous Reviewer 2)

Comments and Suggestions for Authors

The authors have significantly improved the manuscript compared to the first version. In particular, a clear rationale for the choice of the combination of methods used is provided. The essence of the proposed indicator and the combined method of its calculation became clearer. The authors described the case study in more detail. Thus, the presented methodology can be scaled and implemented in other organizations.

Author Response

Please see the attachement

Author Response File: Author Response.docx

Round 2

Reviewer 3 Report (Previous Reviewer 4)

Comments and Suggestions for Authors

Dear authors 

The article looks much better in its revised version. Therefore, I can recommend that your article be accepted. 

This manuscript is a resubmission of an earlier submission. The following is a list of the peer review reports and author responses from that submission.

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

Dear editors and authors,

Thank you for giving me a chance for reviewing your manuscript. It is an interesting application combining the FAHP and the CBA. The equation applied in line 366 is clever, it is an imaginative approach, and the application itself adds value to the approach.

As the article is so focused on the FAHP and the CBA, I would change the title and the formulation of the hypotheses to better delimit the scope of the research, and to align the already existing literature review with what the paper would need. Thus, the title should contain the FAHP and CBA terms, and the hypotheses should mention them too.

I do not agree with the gap proposed in lines 40 to 43 because not having a link between the two tools proposed, FAHP and CBA, does not mean that integrated frameworks that combine quantitative and qualitative methodologies are not available. In fact, Most production systems based on the Toyota Production System have a combination qualitative and quantitative tools for the optimization KPI implementation (see, e.g. Goti, A., De la Calle, A., Gil, M. J., Errasti, A., Bom, P. R., & García-Bringas, P. (2018). Development and application of an assessment complement for production system audits based on data quality, IT infrastructure, and sustainability. Sustainability, 10(12), 4679.).

In my view, the current hypotheses can be tackled by using other techniques, so other integration methods and other benefits can flourish if these other techniques are applied.

Regarding the literature review, a higher depth is needed. Just 21 references for such know topics is clearly insufficient. Please make an effort to try to find papers where both are combined directly or indirectly (e.g. Mahmoodzadeh, S., Shahrabi, J., Pariazar, M., & Zaeri, M. S. (2007). Project selection by using fuzzy AHP and TOPSIS technique. International Journal of Industrial and Manufacturing Engineering, 1(6), 270-275).

Although simple, the research is clear, and well presented.

Thus, I feel that there is still room for improvement for having an FAHP and CBA Model for Optimizing KPI Implementation.

Kind regards

 

Reviewer 2 Report

Comments and Suggestions for Authors

The main merit of the manuscript is the rather interesting combination of Cost-Benefit Analysis (CBA) and Fuzzy Analytic Hierarchy Process (FAHP) to evaluate the effectiveness of KPIs.

However, I have a few suggestions to improve the manuscript.

1. I propose to justify the choice of FAHP. This is a rather old method. Its main drawback is the dramatic increase in the labor intensity of pairwise comparisons as the number of alternatives increases. What happens if the number of indicators of alternatives is larger than the case study considered in the manuscript? Would it be possible to use more modern MCDM?

2. I propose to describe in more detail the multi-criteria evaluation procedure in the case study. What form of questionnaire was used? How many experts of what qualification participated in the survey?

3. The manuscript does not present the results of sensitivity analysis. I realize that it may be difficult to perform a sensitivity analysis of the authors' proposed combined assessment. However, it would be interesting to compare the results obtained by the proposed combined method with the results of the current MCDM without combining it with Cost-Benefit Analysis.

Reviewer 3 Report

Comments and Suggestions for Authors

Your approach is based two methods:

1. Cost-Benefit Analysis (CBA)

2. Fuzzy Analytic Hierarchy Process (FAHP)

These methods are quite very well known and applied in many cases. By far, the most important thing is to highlight the originality of the approach and the elements that bring added value to the field. In what way combining these two methods it was generated an optimized model, new/original and with contribution in the field?

In my opinion, the figure 1 is not useful (being a general approach, known) and can be removed. 

Regarding the case study for the Cambodian charcoal manufacturing company some questions could arise: 

- Why was this company chosen?

- How were selected the KPIs?

- Who decided for those five KPIs? 

- Is was a discussion between the authors and company' management on this matter?

- Explain the process for the interviews? 

- Besides CEO and CFO of the company which other person are interviewed in pairwise comparisons? How many people? 

- Insert some other data regarding your study (When was carried out?, By which means?, Who conducted the entire process?, Was there previously a system for highlighting performance at the company level? How was the level of operational success analyzed in this case? 

- Can theoretical and practical implications be identified based on the research analyzed?

Reviewer 4 Report

Comments and Suggestions for Authors

Dear authors

The study lacks originality and does not demonstrate significant methodological or practical advancements.

The justification for the research and its case study application is weak, limiting its contribution to the field.

The proposed model, while conceptually appealing, is inadequately validated and lacks scalability or applicability beyond the presented context.

The conclusions claim significant implications for industries in both developed and developing nations without providing evidence or discussing the differences between these contexts.

The interpretation of results from FAHP lacks depth, particularly in terms of how the qualitative judgments are derived and validated.

The results emphasize financial metrics (e.g., NPV, BCR) over qualitative insights, which contradicts the claim of a "balanced" approach.

The integration of CBA and FAHP is described but does not appear methodologically innovative. Similar hybrid approaches are prevalent in the literature.

The choice of a Cambodian charcoal factory as the case study is not adequately justified in terms of its relevance or importance for broader industrial applications.

Terms like "comprehensive understanding" and "informed decision-making" are overused without concrete evidence or elaboration.