Measures of Effectiveness Analysis of an Advanced Air Mobility Post–Disaster Response System
Round 1
Reviewer 1 Report
Comments and Suggestions for AuthorsThe paper introduces a hybrid method (MC-AHHP) combining Monte Carlo simulation and AHP for MOE analysis, addressing subjectivity limitations in traditional methods like the Pugh Matrix. The method was applied to AAMPDR system to demonstrate the usefulness. This paper shows potential, but only if substantial improvements are made in both methodology and presentation.
1) The integration of Monte Carlo simulation with AHP in Section 2.3 is not clearly demonstrated, the process is vague.
2) Title of Section 3.1, what do you mean by “Measures of Effectiveness (SysML)”? Actually, what’s the role of SysML in this paper?
3) In Section 3, how did you get these distributions? Pls explain clearly.
4) Figure1, Figure2, Figure3, Figure4, Figure5, Figure6, and Figure8 all have poor visibility, pls fix them.
5) In table2, ellipsis in the first column is misplaced.
6) There are some grammar errors, pls double check. E.g., Lines 161-162; Lines 204-205; Lines 318-319.
Author Response
Reviewer Comments and Author's Responses:
The paper introduces a hybrid method (MC-AHHP) combining Monte Carlo simulation and AHP for MOE analysis, addressing subjectivity limitations in traditional methods like the Pugh Matrix. The method was applied to AAMPDR system to demonstrate the usefulness. This paper shows potential, but only if substantial improvements are made in both methodology and presentation.
Comments 1: The integration of Monte Carlo simulation with AHP in Section 2.3 is not clearly demonstrated, the process is vague.
Response 1: The integration of Monte Carlo (MC) may be found in the preprocessing stage which involved curating the data samples. That is, first and foremost to refine the selected data in such a way that pseudo-randomizes the selection process. By so doing, effects of the researcher’s bias is greatly reduced in the selection of the best average, viz a viz, the best mean operating weight, the best operating cost, and so on.
Furthermore, Taylor (2019) argued the significance of integrating pseudo-randomness to the probability distribution as critical to artificially reconstruct a natural process, which will otherwise not have been possible to perform this type of simulation.
Moreover, Momani and Ahmed (2011) have noted that the role of MC simulation is to obtain pseudorandom numbers based on some known distributions (in this case, normal distribution) for numerical experimentation.
Finally, the integration of MC in AHP technique is well-established and documented as demonstrated in works such as by Rosenbloom (1996), albeit not in the exact manner as utilized in our work. That is, our work involves a specialized use case specifically towards concept selection of SAR – purposed vehicles. By contrast, Rosenbloom (1996) had only recommended the generalized integration of a Monte Carlo simulation experiment in AHP such that the pairwise comparisons be considered as random variables whose distribution is bounded by a minima and maxima as specified by the 9-point standard preference level similar to that shown on Table 9. The keyword being “random variables”. Thus, it follows that their specific values be taken from the random numbers generated through a prior MC process.
Comment 2: Title of Section 3.1, what do you mean by “Measures of Effectiveness (SysML)”? Actually, what’s the role of SysML in this paper?
Response 2: The choice of title is to distinguish this section by emphasizing the application of SysML effort to computing the Measures of Effectiveness (MOE). Simply put, the SysML effort demonstrated in Section 3.1 is the Parametric diagrams presented on Figures 3 and 4. For succinct note on the role and application of SysML with respect to computation of the MOE for the AAMPDR System, the reviewer may refer to line 214 and ff of the manuscript. A portion of the line 214 reads as follows: “Furthermore, it is possible to develop a general purpose parametric diagram in SysML that aids the MOE analysis for the proposed AAMPDR System model as well as apply the same to other candidate alternatives to the AAM as presented in Figure 3…”
Comment 3: In Section 3, how did you get these distributions? Pls explain clearly.
Response 3:
A summary of the steps taken by the researchers is enumerated as follows:
- Data used in this study was curated into a sample distribution for each system of interest, namely, SAR AAM, SAR Boat and SAR Swimmer, from verified data sources. Such authoritative sources include the US Navy and Airforce for some data on the SAR AAM, the National Academies for some information on the SAR Boat, US Coast Guard for some information on the SAR Swimmers, to mention but a few.
- Using MS Excel, the data for each system of interest were grouped into their corresponding parameters of interest including operating weight, operating range and acquisition cost.
- Subsequently, from the data entered into Excel, their corresponding mean and standard deviation (S.D.) was obtained.
- Following the assumption that the respective data is normally distributed, the Monte Carlo simulation is performed. This assumption is based on a number of factors. Firstly, the researcher’s experience with (or otherwise, expert knowledge on) the data, having administered survey questionnaires to stakeholders and intended beneficiaries of the AAMPDR System during the Stakeholder Expectation Discovery Process (SEDP) (Nasa et al., 2017). Such key stakeholders and beneficiaries include NASA AAM engineer, local medical practitioners, supply chain / delivery logistics personnel, Aerospace and Systems Engineering academics, to name but a few. Secondly, owing to the fact that normal distribution exhibits mathematical properties that makes it easy to work with as stated by Taylor (2018). Thirdly, normal distribution constitutes a reasonable approximation of the continuous probability distribution of a number of natural phenomena such as those treated in this study including operating weight of SAR Swimmers (US Coast Guard, 2020; Taylor, 2019).
- Generate a random number with reference to the corresponding mean and S.D. for each of the parameters aforementioned. That is, operating weight, operating range and acquisition cost. NOTE: The r – number for each simulation was set to 500 runs (or trials).
- The weighted outcome for each parameter is then computed against the 500-simulation run with the aid of the ‘What-If’ table function in Excel.
- The probability distribution of the weighted outcome is then generated using the histogram plot function in Excel as illustrated on Figures 7 – 9.
- The foregoing steps 5 - 7 is then repeated for other respective systems of interest.
Please kindly note that while the foregoing steps were not enumerated in the manuscript to this level of detail, the researchers succinctly highlighted the overarching process with the aid of a flow chart as depicted in Figure 1. Some researchers might consider the former approach to be verbose, too simplistic or didactic. Especially if such subject may be more fittingly covered in a textbook as opposed to scientific article.
Comment 4: Figure 1, Figure 2, Figure 3, Figure 4, Figure 5, Figure 6 and Figure 8 all have poor visibility, pls fix them.
Response 4: The authors have taken this into consideration and will work with the editors / production team to improve the visibility of the said figures as much as possible. However, kindly note that these figures (such as Figure 8) were directly generated from their respective software. Thus, the authors have little control over the way they are rendered or exported in picture format.
Comment 5: In Table 3, ellipsis in the first column is misplaced.
Response 5: The vertical ellipsis(-es) have been added and placed, accordingly. Thank you very much.
Comment 6: There are some grammar errors, pls double check. E.g., Lines 161 – 162; Lines 204 – 205; Lines 318 – 319.
Response 6: Thank you for pointing out some of the grammar errors. Considerable effort has been made to correct the errors including the ones identified on Lines 161 – 162; 204 – 205; 318 – 319. Other grammatical error checks include on Line 24, 42, 47, 55, 80, 105, 113, 138, 142 – 143, 144, 145, 147 – 148, 152, 155 – 156, 172, 205 – 206, 217, 227, 235, 241, 251, 255, 272, 274, 300, 303, 323, 326,
REFERENCES
Momani, A.M., Ahmed, A.A., (2011). Material handling equipment selection using hybrid Monte Carlo simulation and analytic hierarchy process. International Journal of Industrial and Manufacturing Engineering, 5(11).
NASA, Hirshorn, S.R., Voss, L.D., Bromley, L.K., (2017). NASA Systems Engineering Handbook, NASA, 6th Ed.
Rosenbloom, E.S., (1996). A probabilistic interpretation of the final rankings in AHP. European Journal of Operations Research, 96(2).
Taylor, B.W., (2019). Introduction to Management Science, Pearson, 13th Ed.
US Coast Guard, Rob V., (2020). Coast Guard Requirements, https://web.archive.org/web/20240424210441/https:/www.operationmilitarykids.org/coast-guard-requirements/. Accessed: May 2025.
Reviewer 2 Report
Comments and Suggestions for AuthorsThis is a well written and appropriate to the journal audience article. My only question is that it appears this has been submitted (Version April 24, 2025) to the journal Geomatics? If this is the case, and it is the same article, it should be withdrawn from Geomatics should it be accepted.
Author Response
Comment 1: This is a well written and appropriate to the journal audience article. My only question is that it appears this has been submitted (Version April 24, 2025) to the journal Geomatics? If this is the case, and it is the same article, it should be withdrawn from Geomatics should it be accepted.
Response 1: Your comments and observation are well taken, thank you. Yes, we have requested that it be withdrawn from Geomatics journal. We will also follow—up on this request. Thank you.
Reviewer 3 Report
Comments and Suggestions for AuthorsThe paper presents an analysis of measures of effectiveness (MOE) for an Advanced Air Mobility Post-Disaster Response (AAMPDR) System, proposing the Monte Carlo–Analytical Hierarchical Hybrid Process (MC–AHHP) as an alternative to the traditional Pugh Matrix. The study contributes to the field by offering a less subjective and more data-driven approach to evaluating candidate systems. However, the reviewer believes the following issues must be addressed before the paper is suitable for publication:
(1)While the paper outlines the MC–AHHP method in detail, it falls short in providing a comprehensive comparison with existing MOE evaluation methods. Could the authors expand on how the MC–AHHP method offers superior results compared to other established techniques in terms of accuracy and reliability?
(2)Moreover, some sentences lack subject-verb agreement and proper tense usage. For example, in the sentence "The results obtained from the application of both approaches demonstrated that the MC–AHHP is less subjective, more objective, data–driven and a quantitative–based measure for MOE analysis compared to the erstwhile Pugh Matrix method," the verb "demonstrated" should be in the past tense to match the context.
(3)There are also instances of inconsistent terminology and notation throughout the paper. For instance, the term "weights" is used in different contexts without clear definition each time, which can lead to confusion. The authors should ensure that all technical terms are consistently defined and used.
(4)The discussion section could be expanded to include a more thorough analysis of the implications of the findings for both the AAMPDR System and the broader field of systems engineering. This would help to contextualize the study's contributions and highlight its significance.
Author Response
Reviewer Comments and Author's Responses:
The paper presents an analysis of measures of effectiveness (MOE) for an Advanced Air Mobility Post-Disaster Response (AAMPDR) System, proposing the Monte Carlo–Analytical Hierarchical Hybrid Process (MC–AHHP) as an alternative to the traditional Pugh Matrix. The study contributes to the field by offering a less subjective and more data-driven approach to evaluating candidate systems. However, the reviewer believes the following issues must be addressed before the paper is suitable for publication:
Comment 1: While the paper outlines the MC–AHHP method in detail, it falls short in providing a comprehensive comparison with existing MOE evaluation methods. Could the authors expand on how the MC–AHHP method offers superior results compared to other established techniques in terms of accuracy and reliability?
Response 1: While other methods for MOE analysis exist including (but not limited to) the Score ranking model, Cost or utility function models, Economic Order Quantity (EOQ) model, Weighted Utility Addictive Theory (WUTA) (Taylor (2018); Friedenthal et al. (2014); Karande and Chakraborty (2013)), the aim and objectives of this paper are to evaluate the MOE for the proposed AAMPDR System using two methods, only, namely Pugh matrix and MC—AHHP techniques.
Notwithstanding, the authors duly welcome this reviewer’s feedback and will consider undertaking a comprehensive comparison between the MOE evaluation methods as aforementioned in a future study.
In anticipation, here is a snippet contrasting between the Pugh matrix and MC—AHHP techniques which have been applied in this present study:
Depending on the aim and objectives of the decision maker, the Pugh Matrix represents a relatively rapid and straightforward approach for computing the MOE using the score ranking model. Albeit, it is not the most accurate in terms of being prone to subjectivity of the practitioner who’s performing the analysis. Note that even experts may also introduce subjective bias (Porter et al. (2004)).
For other scenarios where the decision maker’s aim is for accuracy and precision, then the researchers recommend that the MC—AHHP technique be adopted and performed in place of the Pugh matrix method. However, the cons of the MC—AHHP technique may include factors such as being time—consuming, requiring additional computational resources depending on the number of simulation runs that the analyst intended to perform or how large the data distribution is.
Comment 2: Moreover, some sentences lack subject-verb agreement and proper tense usage. For example, in the sentence "The results obtained from the application of both approaches demonstrated that the MC–AHHP is less subjective, more objective, data–driven and a quantitative–based measure for MOE analysis compared to the erstwhile Pugh Matrix method," the verb "demonstrated" should be in the past tense to match the context.
Response 2: Thank you for pointing this out. The authors have confirmed that the above sentence is indeed in the past tense form. Further, other similar grammatical errors have been checked and rectified to the best of the authors’ knowledge. For example, on lines 24, 42, 47, 55, 80, 105, 113, 138, 142 – 143, 144, 145, 147 – 148, 152, 155 – 156, 172, 205 – 206, 217, 227, 235, 241, 251, 255, 272, 274, 300, 303, 323, 326,
Comment 3: There are also instances of inconsistent terminology and notation throughout the paper. For instance, the term "weights" is used in different contexts without clear definition each time, which can lead to confusion. The authors should ensure that all technical terms are consistently defined and used.
Response 3: This observation is noted. The term “weight” as used within the text and context of the MC—AHHP, refers to the relative importance given to a set of criteria. That is, a value assigned to the criteria in the order of most to least importance (Taylor (2019)). Moreover, Porter et al. (2004) describes weights (or weighting factors) as a means of identifying relative importance that seeks a closer problem—to—solution matching.
Thus, the foregoing definition and usage has been clarified for “weight” within the text (Please refer to the footnote #1 on page 4 of the updated manuscript).
Comment 4: The discussion section could be expanded to include a more thorough analysis of the implications of the findings for both the AAMPDR System and the broader field of systems engineering. This would help to contextualize the study's contributions and highlight its significance.
Response 4: Additional material has been added to the manuscript to provide further context regarding the study’s contribution. Please refer to lines 321—331 on page 14 as well as lines 343—348 on page 15 for additional notes highlighting the significance of the Pugh matrix and MC—AHHP techniques, the corresponding results and their respective application.
REFERENCES
Friedenthal, S., Moor, A., Steiner, R. (2014), A practical guide to SysML: the systems modeling language. 3 ed.
Karande, P. and Chakraborty, S. Material handling Equipment Selection Using Weighted Utility Additive Theory. Journal of 369 Industrial Engineering 2013, 2013(2), pp. 1–9. https://dx.doi.org/10.1155/2013/268708.
Porter, B., Crabtree, B. and Kern, J. (2004), SMC Systems Engineering Primer & Handbook – Concepts, Processes and Techniques, Space & Missile Systems Center U.S. Airforce, 2 ed.
Taylor, B.W., (2019). Introduction to Management Science, Pearson, 13th Ed.
Reviewer 4 Report
Comments and Suggestions for AuthorsWhen using MC simulation, do you have a hypothesis drawn?
Have you done sensitivity analysis?
Figure 3 and 4 text is too small to read. Please make it larger?
What is the ramification of choosing one alternative over another in your Pugh Matrix or MC-AHHP method? Are there scenarios where one will select an alternative over the best scored option?
Please discuss?
Author Response
Reviewer Comments and Author's Responses:
Comment 1: When using MC simulation, do you have a hypothesis drawn?
Response 1: Yes, we have a hypothesis drawn. Although, this work is culled from part of a larger project/body of work containing the description of the overarching hypotheses. As such, that portion of the dissertation portfolio is not included in the article. However, here is an excerpt as related to the hypothesis for this article as follows:
Hypothesis 2: AAM model will demonstrate a better measure of effectiveness (MOEs) factor than conventional SAR intervention methods.
Null Hypothesis 2: There is no significant difference between MOEs of the AAM platform and other conventional SAR methods.
Following the results obtained from MOEs analysis, the AAMPDR System model demonstrated a better score than six (6) other potential alternative SAR agents when a Pugh Matrix analysis was applied. However, the AAMPDR System was ranked second to the SAR swimmer candidate, followed by the SAR Boat when a Monte Carlo Analytic Hierarchical Hybrid Process (MC—AHHP) was applied. The same results were also reported in the conclusion of the study.
Comment 2: Have you done sensitivity analysis?
Response 2: Not yet, since it falls outside of the scope of this present study. However, there are plans to perform a sensitivity analysis which we have indicated as part of the future works in the conclusion section.
Comment 3: Figure 3 and 4 text is too small to read. Please make it larger?
Response 3: Duly noted and thank you for your observation. To resolve these figure sizes, the researcher intend to reach out to the production team of Systems MDPI to make Figures 3 and 4 larger as well as meet the publisher’s formatting style requirement in terms of the figures matching the appropriate margin width and placement within the body of the text. Work is in progress on this matter.
Comment 4: What is the ramification of choosing one alternative over another in your Pugh Matrix or MC-AHHP method? Are there scenarios where one will select an alternative over the best scored option?
Please discuss.
Response 4: Yes, there are ramifications for choosing one alternative over the other as discussed in the Results section of the article and presented here:
Depending on the aim and objectives of the decision maker, the Pugh Matrix represents a relatively rapid and straightforward way to compute the MOE using the score ranking model. Albeit, it is not the most accurate in terms of being prone to subjectivity of the person performing the analysis. Even experts may also introduce subjective bias.
For other scenarios where the decision maker’s aim is for accuracy and precision, then the researchers recommend that the MC—AHHP technique be adopted and performed in place of the Pugh matrix method. However, the cons of the MC—AHHP technique may include factors such as being time—consuming, requiring additional computational resources depending on the number of simulation runs that the analyst intended to perform or how large the data distribution is.
