Coloured Petri Nets-Based Modeling and Validation of Insulin Infusion Pump Systems

Round 1

Reviewer 1 Report

I congratulate the authors for their work. It is evident that a great deal of effort has gone into compiling a study of this nature.

The authors present an MBA focused on CPN reference models of insulin infusion pump systems, aiming to assist manufacturers in assessing quality, and described a case study on ACCU-CHEK Spirit. The research question is also clearly presented and answered based the case study results and the empirical evaluation of models.

There are some aspects that I believe would improve the paper:

- Some references are missing (lines 28 or 32)

- The Figures, or Tables should be inserted into the main text close to their first citation. This is not done.

Author Response

Dear reviewer,

We would like to thank you for the constructive comments. We present below a point-by-point response to your comments.

Best regards.

1. There are some aspects that I believe would improve the paper:

1.1. - Some references are missing (lines 28 or 32)

Response to 1.1. We apologize for such reference mistakes. We have corrected the references by removing some incorrect citations and by including:

[3] Woodcock, J., Larsen, P. G., Bicarregui, J., Fitzgerald, J. Formal methods: Practice and experience. ACM Comput Surv 2009, 41, 1–36.

[4] Chen, Y., Lawford, M., Wang, H., Wassyng, A. Insulin Pump Software Certification. In Proceedings of the International Symposium on Foundations of Health Informatics Engineering and Systems, Macau, China, 87–106, 2013.

[5] Sivakumar, M.S., Casey, V., McCaffery, F., Coleman, G. Improving Verification Validation in the Medical Device Domain. In Proceedings of the European Conference on Software Process Improvement, Krems, Austria, 61–71, 2011.

1.2. - The Figures, or Tables should be inserted into the main text close to their first citation. This is not done.

Response to 1.2. We carefully revised the figures and tables to place them close to the first citation.

Author Response File: Author Response.pdf

Reviewer 2 Report

To whom it may concern,

I think that this paper should be revised carefully. I suggest the following comments:

• The authors should polish the paper suitably. The whole paper should be reviewed carefully, in order to correct all the typing and text formatting errors.
• Something is wrong with the citations in this article. There are many (question marks) in the introduction that does not allow us to check the correctness of the citation.
• In the introduction/related works, it is not enough to state the current work. Including the motivation, the main difficulties, the main work and the improvements compared with previous related works should be emphasized in this section. The novelty of the proposed approach (MBA/CPN) should be highlighted carefully.
• As far as I have been interested, this publication is not the first for the authors on this topic (for example https://ieeexplore.ieee.org/document/8927111). Therefore, I think the authors should provide more detailed explanations of the ongoing work and its differences from previous works.
• In my opinion, one figure format should be followed, because there are now many different formats (for example Figure 10, Figure 11, Figure 12, Figure 13): different sizes, different fonts, different font sizes and so on. A uniform figure format would make this article much better and easier to understand.
• Both in the conclusions and perhaps in the abstract, I lack a quantitative assessment of the extent to which the applied methods have made it possible to improve the basic parameters in comparison with the works of other authors. Because the conclusions presented now are very abstract.

Overall merit:  The paper covers a very important medical / engineering topic, which could be interesting to both industry and the scientific community. I'd suggest accepting this paper with major revisions after addressing the comments and suggestions above.

Author Response

Dear reviewer,

We would like to thank you for the constructive comments. We present below a point-by-point response to your comments.

Best regards.

2. I think that this paper should be revised carefully. I suggest the following comments:

2.1 - The authors should polish the paper suitably. The whole paper should be reviewed carefully, in order to correct all the typing and text formatting errors.

Response to 2.1. We carefully revised the article to correct typing and text formatting errors.

2.2 - Something is wrong with the citations in this article. There are many (question marks) in the introduction that does not allow us to check the correctness of the citation.

Response to 2.2. We apologize for such reference mistakes. We have corrected the references by removing some incorrect citations and by including:

[3] Woodcock, J., Larsen, P. G., Bicarregui, J., Fitzgerald, J. Formal methods: Practice and experience. ACM Comput Surv 2009, 41, 1–36.

[4] Chen, Y., Lawford, M., Wang, H., Wassyng, A. Insulin Pump Software Certification. In Proceedings of the International Symposium on Foundations of Health Informatics Engineering and Systems, Macau, China, 87–106, 2013.

[5] Sivakumar, M.S., Casey, V., McCaffery, F., Coleman, G. Improving Verification Validation in the Medical Device Domain. In Proceedings of the European Conference on Software Process Improvement, Krems, Austria, 61–71, 2011.

2.3 - In the introduction/related works, it is not enough to state the current work. Including the motivation, the main difficulties, the main work and the improvements compared with previous related works should be emphasized in this section. The novelty of the proposed approach (MBA/CPN) should be highlighted carefully.

Response to 2.3. For motivations, in the introduction, we emphasized that handling formal methods usually requires expert knowledge and increased costs and development time, which is one of the main motivations to conduct this study. In lines 46, 47, and 48, we also described our previous research [14] to improve clarity. Lines 68, 69, and 70 relate to requirements required by regulatory agencies to certify insulin infusion pump systems and a large number of  reported recalls, that are also motivations to conduct this research. In lines 81, 82, and 83, we highlighted examples of challenges (main difficulties): (1) the integration of requirements specification and assurance cases to provide re-use and (2) the re-use of a CPN reference model in a time-effective and cost-effective manner. We also included the explanations from line 101 to line 109, aiming to highlight improvements compared to related works and the novelty of our proposed approach. The last paragraph of related works also highlights improvements compared with previous related works (from line 212 to line 228).

2.4 - As far as I have been interested, this publication is not the first for the authors on this topic (for example https://ieeexplore.ieee.org/document/8927111). Therefore, I think the authors should provide more detailed explanations of the ongoing work and its differences from previous works.

Response to 2.4. In the introduction (from line 90 to line 96), we stated that, in our previous study [7,14], we did not address (1) the integration of requirements specification and assurance cases to provide re-use (goal-oriented requirements engineering) and (2) the re-use of a CPN reference model in a time-effective and cost-effective manner.

2.5 - In my opinion, one figure format should be followed, because there are now many different formats (for example Figure 10, Figure 11, Figure 12, Figure 13): different sizes, different fonts, different font sizes and so on. A uniform figure format would make this article much better and easier to understand.

Response to 2.5. We have improved Figures 10, 11, 12, and 13, considering size and font. We also adjusted all figure sizes to be as readable as possible. However, due to the margins of the latex template from MDPI, we were not able to increase the fonts of some figures, such as Figure 8 and Figure 9. As Figures 8 and 9 are exported from the CPN/Tools software, readers are able to download the same CPN models from our repository and access such subpages using a GUI.   

2.6 - Both in the conclusions and perhaps in the abstract, I lack a quantitative assessment of the extent to which the applied methods have made it possible to improve the basic parameters in comparison with the works of other authors. Because the conclusions presented now are very abstract.

Response to 2.6. In the conclusions (from line 845 to line 850), we highlighted that, as there is a lack of studies providing generic, parametric, and timed insulin infusion pump systems models, the methods applied in this study enabled the improvement of available specifications of such systems. Thus, such research gap also resulted in the lack of studies providing evaluations of productivity and reusability of models when reusing insulin infusion pump systems models. Our study addressed such limitations improving the state-of-the art. During the empirical evaluation, time showed to be a relevant metric to measure productivity (computing the time required by each group to finish the problems that we asked modelers to solve during the evaluation phase). Two factors (understandability and adaptability) also showed to be relevant to measure reusability. Regarding formal techniques (e.g., CPN and model checking), as stated in lines 110-113, although it is a consensus that the proposal and the use of formal methods to validate the behavior of safety-critical systems is valuable, our proposal also works as a guide, along with reusable project artifacts, for developers of insulin infusion pump systems (lines 852-859).

Author Response File: Author Response.pdf

Round 2

Reviewer 2 Report

The concerns have been addressed.