RV4JaCa—Towards Runtime Verification of Multi-Agent Systems and Robotic Applications

Round 1

Reviewer 1 Report

In this paper authors present an interesting approach for integrating Runtime Verification (RV) into the JaCaMo framework. Overall the proposal is sound and worth being published as a journal paper. There are, however, some important aspects to be addressed by authors towards making a proper manuscript for publication. 

Regarding the abstract, the part starting in line 5 (MAS have been...) does not seem properly related with the previous one - it sounds like a very different work. It should be improved. Besides, the abstract lacks a real claim, i.e., a problem to be solved, as well as an overview of authors' solution.

In respect to to the introduction, the paragraph on lines 57-63 presents authors' contribution. However, there should be some motivations associated with it, i.e., explain why is it important to have RV with JaCaMo? What can be done thereby that was not possible before?

I suggest modifying the components presentation at the beginning of Sec.3, so that first a more general overview of each component is presented, and only latter more technical details are highlighted. This is specially important for the RV4JaCa artefact, as in the current version there is too much technical info. 

It is unclear what one would need to download/install in order to use the proposed system. For instance, in the github repository provided by the authors, there is no link to RML (or how it is combined with RV4JaCa.

Fig.2 should be improved. Why RV4JaCa appears twice? Does the Belief Base belongs exclusively to the Monitor Agent?  Maybe include Jason's part responsible for MAS-communication, and associate it with the .send(...) method.

Last paragraph sentence about the monitor (lines 238-239) explains the obvious. What is not clear in the "send" command are the parameters. BTW: the print in listing.1 would only help it it includes all the parameters. Finally, it is worth mentioning that a detailed presentation of the agent could be done latter in this section.

Section 4 is poorly introduced. It should be mentioned what are the objectives of the proposed study, and how the results are evaluated.

Something I missed in the study description of sec.4 has to do with the monitor Agent. How does it looks like? 
I also think section 4 lacks more relevant information in regarding the results of the application of the RV in the system under study. For how long did it run? How many faults were observed? And so on. 

Figure 5 should be edited to show the messages directly in English instead of Portuguese+English, it would increase the image readability. 

Minors: 

- What is "resp." on line 151?
- Authors should properly define "safety net" on line 161.
- The sentence from line 173 to 175 is unclear, please rephrase. 
- It is also confusing the sentence from line 177 to 179, please rephrase. 
- Suggest changing the position of "Listing 1", as it currently creates difficulty for a smooth reading of the PDF.

- Line 237: CHANGE: It informs the agent that... TO It informs an interested agent (receiver) that...
- Line 349: the Monitor agent is the only other agent => the Monitor agent is the single agent
- Line 519: The second one is to => The second purpose of this study is to
- There are two Listing.1 (pg. 6 and 14). 
- Section 5 should be named "Related Works" (it currently misses an 's')

Author Response

We thank the reviewer for his/her comments and suggestions.

We integrated all of them in the paper. We used a different colour, so it is easier to detect the new additions.

In this paper authors present an interesting approach for integrating Runtime Verification (RV) into the JaCaMo framework. Overall the proposal is sound and worth being published as a journal paper. 

Thank you very much, we appreciate it.

There are, however, some important aspects to be addressed by authors towards making a proper manuscript for publication.

Regarding the abstract, the part starting in line 5 (MAS have been...) does not seem properly related with the previous one - it sounds like a very different work. It should be improved. Besides, the abstract lacks a real claim, i.e., a problem to be solved, as well as an overview of authors' solution.

Yes, the reviewer is right, we updated the abstract and a claim.

In respect to to the introduction, the paragraph on lines 57-63 presents authors' contribution. However, there should be some motivations associated with it, i.e., explain why is it important to have RV with JaCaMo? What can be done thereby that was not possible before?

Yes, the introduction lacked a better positioning of the work. We added more motivations and better explained the novelty of RV4JaCa.

It is unclear what one would need to download/install in order to use the proposed system. For instance, in the github repository provided by the authors, there is no link to RML (or how it is combined with RV4JaCa.

Thanks for noticing it, we added additional information in the GitHub repository

Fig.2 should be improved. Why RV4JaCa appears twice?

We have a label showing the part of the system that belongs to RV4JaCa. We also have RV4JaCa Artefact, which is the artefact created for RV4JaCa. So, the two labels are actually for two different things.

Does the Belief Base belongs exclusively to the Monitor Agent?  

Yes, the Belief Base belongs exclusively to the Monitor Agent. Each agent has its own belief base.

Last paragraph sentence about the monitor (lines 238-239) explains the obvious. What is not clear in the "send" command are the parameters. BTW: the print in listing.1 would only help it it includes all the parameters. Finally, it is worth mentioning that a detailed presentation of the agent could be done latter in this section.

Yes, this part needed to be improved. We added additional information in the paragraph.

Section 4 is poorly introduced. It should be mentioned what are the objectives of the proposed study, and how the results are evaluated.

Thanks, this part was a bit short. We introduced it better.

Something I missed in the study description of sec.4 has to do with the monitor Agent. How does it looks like?

It can be seen in Section 3, Listing 1.

I also think section 4 lacks more relevant information in regarding the results of the application of the RV in the system under study. For how long did it run? How many faults were observed? And so on.

These are all relevant questions. We added additional information on this in the introduction. However, it was not the objective of this work to carry out experimental analysis on the tool, but to present its engineering and possible integration/use in a robotic domain. 

Figure 5 should be edited to show the messages directly in English instead of Portuguese+English, it would increase the image readability.

We thank the reviewer for pointing this out, but we preserved the original figure because it corresponds to the real one. It denotes an actual screenshot of the process. 

Minors:

Thanks for pointing out all the minor points. 

All the minor points have been tackled and integrated in the paper.

Reviewer 2 Report

This paper presents a Runtime Verification (RV) approach for Multi-Agent Systems (MAS) using the JaCaMo framework. It is very interesting, and more details would be provided to improve the quality of the manuscript.

1. It should be discussed about the relationship between the JaCaMo framework and RV.

2. The Label for Listing 1: MAS Log in Page 14 should be added.

3. English grammar and representation should be check throughout the manuscript.

Author Response

We thank the reviewer for his/her comments and suggestions.

We integrated all of them in the paper. We used a different colour, so it is easier to detect the new additions.

This paper presents a Runtime Verification (RV) approach for Multi-Agent Systems (MAS) using the JaCaMo framework. It is very interesting, and more details would be provided to improve the quality of the manuscript.

We thank the reviewer.

1. It should be discussed about the relationship between the JaCaMo framework and RV.

Yes, the reviewer is right. We added additional description on the framework in the introduction.

2. The Label for Listing 1: MAS Log in Page 14 should be added.

Yes, the label has been added.

3. English grammar and representation should be check throughout the manuscript.

We re-read the manuscript and corrected all the typos and minor issues we could find.

Round 2

Reviewer 1 Report

I acknowledge authors' efforts to address the issues that I highlighted in the previous review round. To my extend, there is not other issues to be tackled by authors.