A Review of Methods for Assessing the Quality of Measurement Systems and Results

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

The idea for the papers is very good and it could be an important literature position in the future, if you address remarks that I list below:

1. Page 4, below Table 1, please separate the footnote no. 2 to the Table 1 and the main text of section 1. Introduction and terminological framework, as now the continuation of footnote no. 2 seems to be a part of main text of this section.
2. Page 6 and 7, lines 123 – 147. There is quite a lot of information given about MSA method and ISO 5725 standard, while GUM method, Monte Carlo method and Bayesian approach are just summarized in one short paragraph containing few sentences for each method. Are the latter methods less important than those mentioned earlier?
3. Page 9, line 181, you write: “According to %Tolerance, the Gage R&R accounts for 12.90% of the tolerance range, which is also well within acceptable limits.” It is within acceptable limits but Table 5 says: “10%-30% Marginally acceptable”. Some comment on why measurement system may be accepted in this particular case is necessary.
4. Subsections 3.1 and 3.2 and Appendices A and B. My general remark is that you do not give any details of these examples. For example presented in section 3.1 and appendix A there are some results but we do not know what was measured, what was the measurement equipment used, what is the measurement unit, etc. Similar case for second example. Here the alternative assessment is presented but also details on the example are missing. I know that those are general examples but they are not convincing for me if I do not know the context of the experiment/usage. Please add more details about these examples.
5. Subsection 3.3 – you call it an example but in fact in this subsection only the reference to other paper is given and some interpretation of results is presented. It does not contain any example.
6. Subsection 3.5 – I believe that examples presented in review papers like the one that is subject of this review should present the procedure of application of some method. From example presented in subsection 3.5 it is impossible to understand how Monte Carlo Method should be applied for uncertainty determination and what steps should be done to do that.
7. Page 12, both equations on this page are numbered as (5). Please correct.
8. Page 12, symbols used in equations on this page are not explained (only minority of them). Please add explanations of all symbols.
9. At the beginning of section ‘4. Conclusion’ you write: “This review has presented key methodologies for evaluating the quality of measurement systems and results, including Measurement System Analysis (MSA), ISO 5725, and GUM-based uncertainty evaluation. Each approach has its strengths and is suitable for different measurement contexts from industrial production control to high-precision scientific measurements …”
   The strengths and weaknesses of presented approaches are not discussed. There are also no recommendations on which method should be used in certain usages. Such discussion and recommendations would increase value of this paper very much.
10. Later in the same paragraph you write: “… Through comparative analysis and practical examples, it has been demonstrated that a clear understanding of concepts such as repeatability, reproducibility, trueness, accuracy, bias, and measurement uncertainty is essential for selecting and correctly applying these methods.”
    Methods presented in this paper were not presented in relation to concepts that are mentioned here. It would be also with benefit to manuscript’s value if you prepare an overview saying: if you are going to investigate repeatability or reproducibility you should use this method and if you want to check the trueness of results it is good to use another method, etc. 
11. This review paper mainly focuses on international standards, industry manuals and previous papers published by the authors of this review. Some of references are just cited without explaining their content in the context of the review. There are also a lot of important publications from all around the world and different disciplines of engineering and applied sciences treating about uncertainty of measurement determination using GUM, Monte Carlo and Bayesian approaches that were not included in this review. Situation is similar regarding use of MSA and ISO 5725 for assessing the quality of measurement systems and results. It is strongly recommended to add them to references list and discuss them within the manuscript text.  

Author Response

We sincerely appreciate the reviewer’s time and effort in evaluating our manuscript. The comments have been extremely helpful in improving the clarity, quality, and relevance of the paper. Please find our point-by-point responses below. All changes made in the manuscript are highlighted in yellow.

Comment 1
Page 4, below Table 1, please separate the footnote no. 2 to the Table 1 and the main text of section 1. Introduction and terminological framework, as now the continuation of footnote no. 2 seems to be a part of main text of this section.
Response to Comment 1
We thank the reviewer for pointing out this formatting issue. In the revised manuscript, the footnote no. 2 has been clearly separated from the main text of Section 1 and is now appropriately placed in relation to Table 1.

Comment 2
Page 6 and 7, lines 123 – 147. There is quite a lot of information given about MSA method and ISO 5725 standard, while GUM method, Monte Carlo method and Bayesian approach are just summarized in one short paragraph containing few sentences for each method. Are the latter methods less important than those mentioned earlier?
Response to Comment 2
We thank the reviewer for this pertinent observation. In response, the descriptions of the GUM method, the Monte Carlo method, and the Bayesian approach have been expanded to provide a more balanced and comprehensive overview relative to the MSA method and ISO 5725 standard. Furthermore, a Discussion section has been added to the manuscript, which also addresses these methods in a broader context.

Comment 3
Page 9, line 181, you write: “According to %Tolerance, the Gage R&R accounts for 12.90% of the tolerance range, which is also well within acceptable limits.” It is within acceptable limits but Table 5 says: “10%-30% Marginally acceptable”. Some comment on why measurement system may be accepted in this particular case is necessary.
Response to Comment 3
Thank you for your valuable observation.
We agree that the %Tolerance value of 12.90% formally falls within the “Marginally acceptable” category (10%–30%) as commonly defined in MSA guidelines. However, it is widely accepted in industrial practice that results up to 20% are generally considered acceptable without further justification. Values between 20% and 30% are usually subject to an economic or cost-benefit analysis to determine whether improving the measurement system is justified or whether the current system is sufficient for the intended application.
In this particular case, the value of 12.90% is closer to the lower end of the marginal range and is considered acceptable based on the specific application context and industry tolerance standards. We have added a clarification in the revised manuscript to reflect this interpretation and provide rationale for accepting the measurement system in this instance.
Addition to the manuscript (suggested placement after the sentence mentioning 12.90%):
Although a %Tolerance value of 12.90% technically falls within the "Marginally acceptable" range (10%–30%) according to standard MSA guidelines, values below 20% are generally considered acceptable in most industrial applications. In practice, the interval between 20% and 30% is subject to further economic justification, evaluating whether the cost and complexity of improving the measurement system outweigh the benefits. In this case, the result of 12.90% indicates a stable and usable measurement system without the need for modification.

Comment 4
Subsections 3.1 and 3.2 and Appendices A and B. My general remark is that you do not give any details of these examples. For example presented in section 3.1 and appendix A there are some results but we do not know what was measured, what was the measurement equipment used, what is the measurement unit, etc. Similar case for second example. Here the alternative assessment is presented but also details on the example are missing. I know that those are general examples but they are not convincing for me if I do not know the context of the experiment/usage. Please add more details about these examples.

Response to Comment 4
We thank the reviewer for this very helpful comment. We agree that providing additional context for the presented examples can significantly enhance the clarity and credibility of the analysis. Accordingly, we have revised Sections 3.1 and 3.2, as well as Appendices A and B, by including more detailed descriptions of each example.
We believe these additions will enable a clearer interpretation of the results and improve the transparency of the methodological distinctions presented.

Comment 5
Subsection 3.3 – you call it an example but in fact in this subsection only the reference to other paper is given and some interpretation of results is presented. It does not contain any example.
Response to Comment 5
We appreciate the reviewer’s observation. We agree that the previous version of Subsection 3.3 relied heavily on referencing an external study without providing sufficient detail to qualify as a proper example. In response, we have revised the subsection to include specific numerical results from the referenced work and clarified the measurement context. This now provides a concrete and informative example of how ISO 5725 methodology is applied in practice.
We believe this enhancement improves the clarity and credibility of the discussion and aligns the subsection more closely with the expectations for an illustrative example.

Comment 6
Subsection 3.5 – I believe that examples presented in review papers like the one that is subject of this review should present the procedure of application of some method. From example presented in subsection 3.5 it is impossible to understand how Monte Carlo Method should be applied for uncertainty determination and what steps should be done to do that.
Response to Comment 6
We thank the reviewer for this helpful remark. We agree that review papers should present the application procedure of each method to enhance clarity and practical value. In response, we have revised Subsection 3.5 to include a more detailed description of the Monte Carlo method for uncertainty evaluation. The revised section now outlines the key steps of the simulation process, including the selection of input parameters, definition of their probability distributions, execution of a large number of iterations, and statistical analysis of the output results.
We believe that these additions will help readers better understand the practical implementation of the Monte Carlo approach in the context of measurement uncertainty evaluation.
Comment 7
Page 12, both equations on this page are numbered as (5). Please correct.
Response to Comment 7
We thank the reviewer for noticing this oversight. The equation numbering on page 12 has been corrected in the revised manuscript so that each equation now has a unique and accurate number.

Comment 8
Page 12, symbols used in equations on this page are not explained (only minority of them). Please add explanations of all symbols.
Response to Comment 8
We appreciate the reviewer’s suggestion. In response, we have added explanations for all physical quantities and symbols used in the equations on page 12. These clarifications are now included directly below the equations to ensure better readability and understanding.

Comment 9
At the beginning of section ‘4. Conclusion’ you write: “This review has presented key methodologies for evaluating the quality of measurement systems and results, including Measurement System Analysis (MSA), ISO 5725, and GUM-based uncertainty evaluation. Each approach has its strengths and is suitable for different measurement contexts from industrial production control to high-precision scientific measurements …” 
The strengths and weaknesses of presented approaches are not discussed. There are also no recommendations on which method should be used in certain usages. Such discussion and recommendations would increase value of this paper very much.
Response to Comment 9
We sincerely thank the reviewer for this valuable suggestion. In response to the comment regarding the need to discuss the strengths and weaknesses of the presented approaches and to provide guidance on method selection, we have substantially revised the manuscript. 

A new section entitled “Discussion: Strengths, Limitations, and Method Selection” has been added. This section offers a comparative overview of the practical applications, advantages, and limitations of MSA, ISO 5725, and uncertainty evaluation methods based on GUM, Monte Carlo simulations, and Bayesian approaches. We also provide recommendations on the suitability of each method depending on the measurement context (e.g., industrial control, interlaboratory studies, scientific research).
We believe this addition significantly improves the usefulness and applicability of the review and better supports readers in understanding when and how to apply each approach.
Thank you again for helping us strengthen the manuscript.

Comment 10
Later in the same paragraph you write: “… Through comparative analysis and practical examples, it has been demonstrated that a clear understanding of concepts such as repeatability, reproducibility, trueness, accuracy, bias, and measurement uncertainty is essential for selecting and correctly applying these methods.” Methods presented in this paper were not presented in relation to concepts that are mentioned here. It would be also with benefit to manuscript’s value if you prepare an overview saying: if you are going to investigate repeatability or reproducibility you should use this method and if you want to check the trueness of results it is good to use another method, etc. 

Response to Comment 10
We thank the reviewer for this helpful comment. We fully agree that clearly linking metrological concepts to methodological frameworks adds value to the manuscript. However, we would like to clarify that terms such as repeatability, reproducibility, and trueness are not universal in meaning rather, their definitions and evaluation procedures differ substantially across various standards and methodologies. To avoid the impression that each concept can be neatly mapped to a single method, we have revised the paragraph in question as follows:
"Through comparative analysis and practical examples, it is demonstrated that a clear understanding of concepts such as repeatability, reproducibility, trueness, accuracy, bias, and measurement uncertainty is essential for correctly interpreting results and applying methods within each specific metrological framework. Since these terms are defined and evaluated differently across various standards (e.g., MSA, ISO 5725, GUM), the paper examines these concepts in the context of each framework, highlighting their methodological distinctions."

Comment 11
This review paper mainly focuses on international standards, industry manuals and previous papers published by the authors of this review. Some of references are just cited without explaining their content in the context of the review. There are also a lot of important publications from all around the world and different disciplines of engineering and applied sciences treating about uncertainty of measurement determination using GUM, Monte Carlo and Bayesian approaches that were not included in this review. Situation is similar regarding use of MSA and ISO 5725 for assessing the quality of measurement systems and results. It is strongly recommended to add them to references list and discuss them within the manuscript text.  
Response to Comment 11
We appreciate the reviewer’s valuable comment regarding the need to expand the reference base beyond standards and the authors’ prior publications. In response, we have thoroughly revised the manuscript to include a broader range of international peer-reviewed literature. We have added 45 new references that represent relevant applications of  MSA, GUM, Monte Carlo, and Bayesian approaches across various engineering and scientific domains. These references have been integrated into the manuscript text and discussed in the context of their methodological contribution. We believe this addition strengthens the scientific foundation of the review and broadens its relevance to a wider academic and professional audience.

Reviewer 2 Report

Comments and Suggestions for Authors

The paper presents an overview of key methods for assessing the quality of measurement systems and results, performing a comparative analysis based on different metrological standards. The comparative study conducted accompanied by practical examples demonstrated that established concepts Such as repeatability, reproducibility, trueness, accuracy. bias and measurement uncertainty are essential for the correct  selection and application of these methods. The research also highlighted some inconsistencies in the definitions and use of key terms in different standards and methodological frameworks. Also, ue se objective of the work, the research attempts to align terminology across standards such as ISO 5725 and GUM. The article is well written, the research methodology being well systematized (also using adequate bibliographic references), highlighting relatively clear results for the performed analysis Comments, questions and recommendations for the manuscript improvement: 

-For all equations and relations mentioned in the paper, the notations of all variables must be explained in the text (eq. 1-6) An equation numbering error: after line 205 the mathematical relationship referred to should be numbered (6) (equation (5) already exists)  - In the context in which the subject addressed is treated in various works (in the specialized literature what is the novelty of this research? think this aspect should be highlighted in conclusion.  -What solutions do you think are possible (viable) to remedy the reported shortcomings?  

Author Response

We sincerely appreciate the reviewer’s time and effort in evaluating our manuscript. The comments have been helpful in improving the clarity, quality, and relevance of the paper. Please find our point-by-point responses below. All changes made in the manuscript are highlighted in yellow.

Comment 1
The paper presents an overview of key methods for assessing the quality of measurement systems and results, performing a comparative analysis based on different metrological standards. The comparative study conducted accompanied by practical examples demonstrated that established concepts Such as repeatability, reproducibility, trueness, accuracy. bias and measurement uncertainty are essential for the correct  selection and application of these methods. The research also highlighted some inconsistencies in the definitions and use of key terms in different standards and methodological frameworks. Also, ue se objective of the work, the research attempts to align terminology across standards such as ISO 5725 and GUM. The article is well written, the research methodology being well systematized (also using adequate bibliographic references), highlighting relatively clear results for the performed analysis Comments, questions and recommendations for the manuscript improvement: 
For all equations and relations mentioned in the paper, the notations of all variables must be explained in the text (eq. 1-6) 

Response to Comment 1
We thank the reviewer for this important observation. In the revised manuscript, we have ensured that all variables and symbols used in Equations (1)–(6) are clearly explained in the accompanying text. These explanations have been added either directly below the equations or in the surrounding narrative to enhance clarity and support reader understanding throughout the paper.

Comment 2
An equation numbering error: after line 205 the mathematical relationship referred to should be numbered (6) (equation (5) already exists)  

Response to Comment 2
We appreciate the reviewer’s careful reading and for pointing out the equation numbering error. The numbering has been corrected in the revised manuscript. The mathematical model following line 205 is now properly labeled as Equation (6).

Comment 3
In the context in which the subject addressed is treated in various works (in the specialized literature what is the novelty of this research? think this aspect should be highlighted in conclusion. 

Response to Comment 3

Thank you for this valuable comment. In response, we have revised the Conclusion section to explicitly highlight the novelty of the research. A new paragraph has been added at the end of the section, emphasizing the original contribution of this review, namely, the structured comparison of MSA, ISO 5725, and GUM frameworks from both theoretical and applied perspectives. We believe this addition clarifies the unique value of the manuscript. The newly added paragraph is highlighted in yellow in the revised manuscript.

The novelty of this review lies in its integrated and structured comparison of the MSA, ISO 5725, and GUM frameworks, an analysis that, to the best of the authors’ knowledge, has not been systematically presented in the literature to date. By combining theoretical definitions, methodological procedures, and practical examples from real measurement applications, the paper provides a unified perspective that bridges industrial and scientific approaches to measurement quality. This contribution aims to support both practitioners and researchers in choosing appropriate evaluation strategies and in understanding the implications of terminological and methodological differences across standards.

Comment 4
What solutions do you think are possible (viable) to remedy the reported shortcomings?  
Response to Comment 4
Thank you for this important question. In response, we have added a new section titled Discussion: Strengths, Limitations, and Method Selection, where we address possible solutions to the identified shortcomings. The additions are highlighted in yellow in the revised manuscript.

Reviewer 3 Report

Comments and Suggestions for Authors

The paper presents a review of methods for assessing the quality of measurement systems and results.

From a technical and scientific point of view, this paper does not present novelty and lacks clear critical evaluation and new insight within the text. It is good for a class but not for pubblication

Author Response

Comment 
The paper presents a review of methods for assessing the quality of measurement systems and results.
From a technical and scientific point of view, this paper does not present novelty and lacks clear critical evaluation and new insight within the text. It is good for a class but not for pubblication

Response to Comment
Thank you for your comment and for taking the time to review our manuscript. We appreciate your perspective and understand the concern regarding the level of scientific contribution. However, we would like to clarify that the aim of this review paper is not to present a new experimental method but to provide a structured comparison of existing methodologies for evaluating measurement quality, namely MSA, ISO 5725, and GUM.
To our knowledge, no previous review has systematically analyzed the conceptual, methodological, and terminological distinctions between these frameworks, nor provided practical examples to demonstrate their application across both industrial and scientific contexts.
In response to reviewer feedback, we have further expanded the Discussion section to include a detailed critical comparison of the strengths and limitations of each method, along with practical guidance for method selection. We hope this improves the scientific relevance and usefulness of the manuscript. All new additions are highlighted in yellow in the revised version.

Reviewer 4 Report

Comments and Suggestions for Authors

This paper provides an overview of key methods for assessing the quality of measurement systems and measurement results. The paper also presents practical examples of how these methods are applied to real data. However, there are still some aspects that require further improvement. I can recommend publication in Tribology International if the authors respond properly to the issues as follow:

1. For a measurement system, the accurate of the measurement results should be affected by many factors, such vibration, noise, and other interference factors. Those issue should also be clearly discussed.
2. Some figure are suggested to be added for showing the structures of different measurement systems.
3. If it is okay, the authors can find some examples from some published references to compare the measurement results.
4. For a review paper, only 54 references are used. It is not enough. More references are suggested to be added.

Author Response

We sincerely appreciate the reviewer’s time and effort in evaluating our manuscript. The comments have been helpful in improving the clarity, quality, and relevance of the paper. Please find our point-by-point responses below. All changes made in the manuscript are highlighted in yellow.

Comment 1

This paper provides an overview of key methods for assessing the quality of measurement systems and measurement results. The paper also presents practical examples of how these methods are applied to real data. However, there are still some aspects that require further improvement. I can recommend publication in Tribology International if the authors respond properly to the issues as follow:

For a measurement system, the accurate of the measurement results should be affected by many factors, such vibration, noise, and other interference factors. Those issue should also be clearly discussed.

Response to Comment 1
We thank the reviewer for this valuable comment. We agree that various external factors, such as vibration, noise, and other interference, can significantly influence the accuracy of measurement results and should be considered in uncertainty evaluation. In Example 3.6, we have emphasized the importance of such influencing factors on the measurement outcome. The model includes several relevant variables: h_x represents the measured value (indication), r the scan rate, t the scanning time, l the scan length, d the nominal step height, α the temperature expansion coefficient, Δϑ the temperature difference, δr the repeatability component, δR the reproducibility component, and δprobe the probe-related uncertainty. These parameters account for key sources of variability, including those arising from environmental and mechanical influences. We have clarified this aspect further in the revised manuscript.

Comment 2
Some figure are suggested to be added for showing the structures of different measurement systems.

Response to Comment 2

Thank you for the helpful suggestion. As noted, Figure 1 in the original manuscript already illustrates the structure and key components of the three main approaches for measurement uncertainty evaluation (GUM, MCS, and Bayesesian). In response to your comment, we have added two additional figures: one showing the structure and key components of the MSA method for GR&R studies, and another illustrating the ISO 5725 framework for evaluating the accuracy of measurement methods. These additions aim to enhance the clarity and completeness of the methodological comparison. All changes are highlighted in yellow in the revised manuscript.

Comment 3
If it is okay, the authors can find some examples from some published references to compare the measurement results.

Response to Comment 3

Thank you for the suggestion. We would like to clarify that the original version of the manuscript already included an example based on published data. Specifically, in Section 3.4, the application of the GUM method for uncertainty evaluation was demonstrated using real data from the literature. In Section 3.5, the same dataset was used to apply the Monte Carlo Method, enabling a comparative view of the two approaches. These examples are based on the methodology described in JCGM 100:2008 — Evaluation of Measurement Data: Guide to the Expression of Uncertainty in Measurement (GUM). We hope this satisfies the reviewer’s request for examples derived from published references.

Comment 4
For a review paper, only 54 references are used. It is not enough. More references are suggested to be added.

Response to Comment 4

We appreciate the reviewer’s valuable comment regarding the need to expand the reference base beyond standards and the authors’ prior publications. In response, we have thoroughly revised the manuscript to include a broader range of international peer-reviewed literature. We have added 45 new references that represent relevant applications of  MSA, GUM, Monte Carlo, and Bayesian approaches across various engineering and scientific domains. These references have been integrated into the manuscript text and discussed in the context of their methodological contribution. We believe this addition strengthens the scientific foundation of the review and broadens its relevance to a wider academic and professional audience.

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

Thank you for your corrections and responses. I have no more negative remarks and recommend publishing of this paper.

Author Response

We sincerely thank you for your thoughtful and constructive comments during the review process. Your detailed suggestions in the first round were instrumental in helping us improve the clarity, structure, and overall quality of our manuscript. We truly appreciate the time and effort you invested in reviewing our work. We are especially grateful for your positive assessment in the second round and for your recommendation to publish the paper. Your feedback has been both encouraging and valuable to us.

Reviewer 3 Report

Comments and Suggestions for Authors

I haven't noticed any substantial changes in the text that would justify a significant change in the scientific landscape. A discussion comparing Vim, Gum, and a technical guide that has no metrological value cannot be considered publishable. In the GUM itself, when discussing uncertainties, the use of statistical methods is foreseen. Unfortunately, the authors' responses didn't convince me.

Author Response

We thank the reviewer for their time and comments. We would like to note that no specific revisions were requested in either the first or the second round of review. We are therefore resubmitting the manuscript in its current form, with full respect for the review process and the feedback received.