Providing a User Extensible Service-Enabled Multi-Fidelity Hybrid Cloud-Deployable SoS Test and Evaluation (T&E) Infrastructure: Application of Modeling and Simulation (M&S) as a Service (MSaaS)

Round 1

Reviewer 1 Report (Previous Reviewer 2)

The authors have addressed the issues raised in the previous submission round, except for the API structure; the latter is understandable, since the authors report on an implemented system and it may not be easy (or possible) to change the implementation for a number of reasons. 

The literature review has been improved and the newly introduced discussion section is useful. 

Reviewer 2 Report (Previous Reviewer 1)

I am very grateful that the authors put substantial effort into framing their valuable work and results into the appropriate and extensively described background. 
As mentioned in previous review, this work is evidently valuable to the field (even in the first submission), with the only concern being how it was introduced and described with respect to current context and state of the art. 
The authors clearly addressed all the raised points in a very appropriate and satisfactory manner, and I think that the manuscript is ready for publication.

On an additional positive note, I also personally appreciated the use of change tracking, which made it very easy to perform this additional review.

This manuscript is a resubmission of an earlier submission. The following is a list of the peer review reports and author responses from that submission.

Round 1

Reviewer 1 Report

The authors tackle a very important gap in the world of System of Systems, both from a software perspective and from an operational perspective, expanding on their previous conference presentation of the SEAT framework. The work is meant to address a few needs that are widespread in current practice: Modeling and Simulation on demand, representing and integrating models and data from different sources, at different hierarchical levels, and with different degree of complexity and detail. 

The second part of the paper is well structured and detailed (in further comments below, some suggestion to make this paper a little more agile and flowing), and it clearly shows the use of the framework and applies it to a case study. For this part, I have only minor comments and suggestions.

The first part of the paper, however, brings a major missing point. While I personally think that the work is valuable and effectively addresses the gap in the field and brings novelty and significance, in comparison with current practice, this fundamental point is barely touched upon. Therefore, readers who are not expert in this specific aspect of the state-of-the-art will fail to see how this work relates to current practice.

Since a journal paper needs to include a description of why filling the gaps is valuable (this is present), a description of the proposed model (this is present), and clear description how state-of-the-art and how the work performs "better" than current practice, the lack of the last of these points results in a paper that apparently is only a "glorified description and instruction manual of SEAT and other MITRE products, such as Symphony". On the contrary, I think that this paper has much more value than this, and to show that properly I suggest the following changes:

Self-citations: while they are proper (this paper clearly shows more than the 2020 Winter Simulation Conference, though the structure of SEAT has not been modified from that publication), the authors should make clear throughout the paper what parts are actually new and what parts are just extension of previously published work.

Citations: the Winter Simulation Conference appears in the text as Mittal et al., 2019 and in the list of references as 2020. In general, the list of references is extremely scarce for a journal publication (which is related to the last point, which is the only major change I suggest for this paper): out of 19 references, 6 are links to software, 4 are appropriate self-citations of the predecessors of this work, and 3 are related to MITRE's Symphony software. This means that the entire literature survey and comparison with state-of-the-art has only 6 sources. Vast blocks of existing literature and research are not included, for example methodologies to deal with hierarchy and heterogeneity described in "System of Systems Modeling and Analysis" (DeLaurentis et al., 2022), and various pieces of work from SERC, including some in collaboration with MITRE or the DoD (Integration of M&S and DoDAF, https://sercuarc.org/serc-programs-projects/project/29, Multi-level modeling frameorks, https://sercuarc.org/publication/?id=13&pub-type=Technical-Report&publication=SERC-2013-TR-020-2-Multi-Level+Modeling+of+Socio-Technical+Systems+%E2%80%93+Phase+1).The last comment should also address this issue.

Comparison with state-of-the-art: this is the major missing piece in this otherwise very valuable work. Section 2 states that an exhaustive investigation of these threads are (typo: is. The subject is investigation) beyond the scope of this paper, which is in part true. But the entire claims about the needs and the value of the proposed work cannot be represented by just two references. I suggested a few sources in the previous comment, probably there are many more that should be cited in support of the claims. I want to repeat once more that I personally think that the claims are valid, but they need to be properly supported with literature review.
Similarly, assumptions about the identified driving requirements (4.1) and user types (4.2) need to be justified against state-of-the-art, or at least with some logical support, and not just stated as "a user-focused Journey-Map development effort led to this", which is just stating that in the end the team decided that there are the appropriate number and type of requirements/user types: do other frameworks or approaches to M&S have different requirements? If so, why are this assumptions made in SEAT. Is there some work in literature with similar requirements? 
Finally, the same approach occurs in the demo problem: I see the outcome which shows how to use SEAT and what are the results with SEAT, but I do not see a clear and direct comparison with what could (or could not) be achieved with some of the many existing approaches.
Major changes are needed for what concerns literature review and comparison of this product with current practice, with more extensive review and references, and explicit comparison of assumptions and results for the proposed framework to state-of-the-art.

English language is good, some typos here and there, but style can be improved in three ways:
1) Avoid repetition. The first sentence of the abstract is copied as the first sentence of the paper. The second sentence of the abstract is copied as the fourth sentence of the paper. I would suggest to either rewrite the introduction, avoiding repetition, or to rewrite the abstract, which needs to be a summary and justification for the paper, rather than an extract of sentences from the paper itself.
2) Reduce verbosity. Since in some of the comments I suggested to expand some sections, other parts should be reduced. There are some repetitions throughout the paper, and (for example) there is no need to describe the organization of the paper at the end of the introduction, nor to give a full description of existing products, such as Symphony, which is used as part of SEAT but it is not the original content which is the scope of this paper.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

Acronyms "M&S" and "T&E" should be fully spelled out on first use. Acronyms appearing in images (e.g. tevv in fig 4.1) should be also defined before or upon first use.

The paper content spans many areas of computing, yet the number of references is relatively small (19). The authors are prompted to review related work and enrich the reference list accordingly.  

REST API URLs could be refactored to comply with the prescribed guidelines and be more object-oriented, e.g.

/KillVehicle --> PUT /vehicle/{vehicleId}/kill

/VehicleSpeed --> GET /vehicle/{vehicleId}/speed

Author Response

Please see the attachment.

Author Response File: Author Response.pdf