Artificial Intelligence-Driven Composition and Security Validation of an Internet of Things Ecosystem

Round 1

Reviewer 1 Report

This paper presents an event-based model-checking framework for IoT systems' design and management, called CompoSecReasoner, which can measure the security, privacy, and dependability (SPD) properties of a composed system and administrate it automatically based on a federated artificial intelligence setting.

The topic is very interesting. However, there exist many things to be improved as follows:

- The paper should be rewritten carefully. The organization of the paper is not consistent. For instance, the problem, motivation, challenges, and contributions are not clear. Moreover, the type of paper seems like a technical report rather than a research paper.

- In the introduction part of the paper, authors shortly describe the difficulties of estimating the SPD in a dynamically composed system, without a detailed analysis of the existing researches. That is, a more detailed problem definition is required in this part or a new section based on the analysis of existing approaches.

- The authors need to show the contribution of the paper compactly and clearly. In Section 5, the authors showed the implementation and analysis results of the proposal without any comparison with other approaches. That is, to clearly present the contributions of the author's approach, experimental or comparison results (e.g., quantitative or qualitative analysis) with other approaches should be discussed more in detail.

- In Section 6.1, the literature review is not efficient. That is, most references are so old, especially related to the topic of this paper. The authors should search for the youngest similar works. And then, they should show clearly the limitations of the existing approaches and how your approach overcomes these limitations. I can find some recent papers in major publishers (e.g., MDPI, IEEE, ACM, ELSEVIER, SPRINGER, ...).

Author Response

Manuscript No # applsci-834121 entitled "AI-driven composition and security validation of an IoT ecosystem".

These are the authors’ responses to the reviewers’ comments. The changes in the manuscript have been highlighted with red color.

Associate Editor
Comments to the Author (Required):

It has been reviewed by experts in the field and we request that you make major revisions before it is processed further.

We want to thank all reviewers for their effort and the feedback that they provided to us. We confirm that we addressed all their comments in the revisited version of the paper as it is clarified below. The main changes include:

• A new Section 2, where we are referring to more recent studies and conduct a qualitative comparative analysis, highlighting the limitations of the existing works and the advantages of the proposed CompoSecReasoner
• We add the Section 7.1, where we validated our method against standardized and widely-used methods for risk assessment proposed by NIST and HIPAA.
• We better explain in the text all formula terms.
• We better clarify the runtime performance of CompoSecReasoner as well as other issues that were mentioned by the reviewers.
• We made minor corrections in the text.

Reviewer(s)' Comments to Author:

Reviewer #1

This paper presents an event-based model-checking framework for IoT systems' design and management, called CompoSecReasoner, which can measure the security, privacy, and dependability (SPD) properties of a composed system and administrate it automatically based on a federated artificial intelligence setting.

The topic is very interesting. However, there exist many things to be improved as follows:

• The paper should be rewritten carefully. The organization of the paper is not consistent. For instance, the problem, motivation, challenges, and contributions are not clear. Moreover, the type of paper seems like a technical report rather than a research paper.

We revise the text to better clarify the examined problem, motivation, challenges, and the overall contribution.

• In the introduction part of the paper, authors shortly describe the difficulties of estimating the SPD in a dynamically composed system, without a detailed analysis of the existing researches. That is, a more detailed problem definition is required in this part or a new section based on the analysis of existing approaches.

We add a new section as Section 2, where we are better defining the scope of the paper and discuss the related works.

• The authors need to show the contribution of the paper compactly and clearly. In Section 5, the authors showed the implementation and analysis results of the proposal without any comparison with other approaches. That is, to clearly present the contributions of the author's approach, experimental or comparison results (e.g., quantitative or qualitative analysis) with other approaches should be discussed more in detail.

At the end of Section 2 (Related Works), we are also comparing our proposal with other works. A qualitative comparative analysis is conducted, highlighting the advantages of CompoSecReasoner.

• In Section 6.1, the literature review is not efficient. That is, most references are so old, especially related to the topic of this paper. The authors should search for the youngest similar works. And then, they should show clearly the limitations of the existing approaches and how your approach overcomes these limitations. I can find some recent papers in major publishers (e.g., MDPI, IEEE, ACM, ELSEVIER, SPRINGER, ...).

The discussion of the previous Section 6,1 is now moved under the new Section 2 (Section 2.4 Comparison). In the overall Section we have included in the discussion newer studies. We clearly state the limitations of the existing approaches and how CompoSecReasoner overcomes them.

Reviewer #2

In this paper, the authors implement what they call CompoSecReasoner – a methodology for the description of the SPD (security, privacy, and dependability) aspects of composed systems, and the effects of changes in the state-architecture. They state that CompoSecReasoner is appropriate for SPD verification, composition validation, comparison between different system configurations, impact assessment of changes in the system, and materialization of automated reactive strategies. Furthermore, the developed CompoSecReasoner framework is utilized as middleware of IoT applications, providing real-time monitoring and administration.

• • Page 2 - line 79, 80 - the authors need to define what JADE and OSGi are for the reader - as like me - some may not be familiar with them.

The two terms have been defined in the test: i) Java Agent DEvelopment framework (JADE) and ii) Open Services Gateway initiative (OSGi).

• Page 3 -line 106 - define these terms - CERT and CVE.

The two terms have been defined in the test: i) Computer Emergency Response Team (CERT) and ii) Common Vulnerabilities and Exposures (CVE).

• Figure 1 - is the Surface here the Attack Surface? If then say this in the Figure. If not, then what does Surface refer to?

Yes, it is the same term. The figure has been revised as suggested.

• In Table 1 - I understand that a scoring system is used but it needs to be clarified how this scoring system works - for ex. what does a score of 5, 4, 3 etc. mean. Is 5 the best/greatest effort and 1 the worst/least effort? Clarify.

We better clarify in the text the values’ meaning: “The damage potential in case of a successful exploit on the legitimate system ranges from 1 (low damage) to 5 (high damage). The effort that the attacker has to devote to perform an attack ranges from 1 (low effort) to 4 (high effort)”.

• The mention of Figure 1 in the text is a full-page after the image - I recommend that all figures appear after they are mentioned in the Text and not before. The authors need to be consistent when they do this - currently, it varies from figure to figure in the paper.

We think that the Figure should be included early in the text. Therefore, we leave the figure where it was, and we move the text accordingly with slight changes: “Figure 1 illustrates the main SPD multi-metric concepts, which are detailed below.”.

• Line 425: The Statement "(device’s security is reduced when it connects Internet)" in the paper is not something I would agree with. The device's security is not reduced but more attack vectors are now open/possible due to the device being networked. Can the authors comment on this?

We see the point that is raised by the reviewer and revise the text accordingly. In general, we think that the reviewer’s point of view is more accurate than our original statement.

The revised text is: “a device’s security could be reduced when it connects to Internet, as more attack vectors are now open/possible due to the device being networked”.

• Line 518 - provide a reference for the nSHIELD project you refer to here.

Two references are provided for the nSHIELD project:

- [29] EU funded project – nSHIELD: new embedded Systems arcHItecturE for multi-Layer Dependable solutions, https://artemis-ia.eu/project/34-nshield.html .

- [30] M. Cesena, et al., "SHIELD Technology Demonstrators," CRC Press, Book for Measurable and Composable Security, Privacy, and Dependability for Cyberphysical Systems, pp. 381-434, 2017.

• Figure 5 is not clear to me - what does the y scale indicate? there is no unit?

The figure has been updated and now includes the axis titles. The y scale indicates the individual S, P, and D values, which range from 0 to 100.

• Line 735 - RETE algorithm - what is this - reference for it?

A brief explanation for the RETE algorithm is mentioned: “The RETE algorithm (one of the main and widely-utilized pattern matching algorithms for implementing rule-based systems)”.

Also, a reference has been provided:

- [41] B. Berstel, “Extending the RETE algorithm for event management,” 9^th International Symposium on Temporal Representation and Reasoning, IEEE, Manchester, UK, 7-9 July, pp. 49-51, 2002.

• Line 823 - We use CompoSecReasoner to urge if the composition is feasible - what do the authors mean here?

One of the features of CompoSecReasoner is that it can reason if the composition of a complex IoT setting is feasible or not. If the setting can be instantiated as in the case of the demonstrated example, then it also evaluates the SPD status of the composed system.

We change the word ‘argue’ in the text to better clarify this issue: “We use CompoSecReasoner to reason if the composition is feasible and figure out the total outcome for SPD metrics”.

• Line 744 - The author's comment: Nonetheless, the framework exhibits acceptable delay, even for a real-time environment is not justifiable given the previous statement on line 738 - The reasoning operations need on average 1.6 seconds - 1.6 seconds delay is not a suitable delay for real-time applications - this delay should be in the low ms range.

We better clarify this issue. “The whole reasoning framework need on average 1.6 seconds, 45MB RAM, and 1.87MB for the code. Nevertheless, this is expected to be done once, when an agent is started. After that, when the rule engine is up and running, it takes around 0.002s to process a theory with a few hundreds of facts [42]. Therefore, this is the actual real-time delay for applications. The code size is not affected while the additional RAM is minimal”.

- [42] Y. Malcolm, “A Federated Agent-Based Crowd Simulation Architecture,” 21^st European Conference on Modelling and Simulation (ECMS), Prague, Czech Republic, June 4-6, 2007, pp. 1-7.

Reviewer #3
The paper is interesting and well-written. It addresses a very popular and promising topic, as the combination of AI and IoT. The proposal, I think, is sound and technically relevant.

However, some problems should be corrected before the manuscript is accepted.

•  
•  
• • First, some equations are hard to understand, as variables are not well described. For example, some variables in expressions (1) and (2) are not introduced.

We revised the text accordingly and explain all equations’ terms.

• On the other hand, algorithms are not clearly presented. Do they contain any novelty or thy are just the implementation of the above description? I'm not sure and this point must be clarified.

The algorithms refer the exact implementation of the procedures that are described in the text. Minor changes in the text have been made.

• Nevertheless, the main problem of the manuscript is the validation section. Figure 8 must be improved. Besides, additional performance indicators must be evaluated, focused on security. NIST's test or other similar standardized experiments should be employed to evaluate if the provided security level is good enough, similar to other existing technologies, etc.

We compare the results of the SPD assessment procedure with similar standardized or widely-used methodologies proposed by the National Institute of Standards and Technology (NIST) and the Health Insurance Portability and Accountability Act (HIPAA). At first, we compare the SPD analysis for the individual system components against the Common Vulnerability Scoring System (CVSS) – a standardized methodology by NIST for the evaluation of distinct system modules. Then, we compare the SPD evaluation of the composed system against the Security Risk Assessment (SRA) tool – a widely-used methodology proposed by NIST and HIPPA for the assessment of integrated systems in the healthcare domain. We noticed that our systematic analysis method resulted similar outcomes and adequately captured the protection posture. The overall comparative study is documented in the subsection 7.1.

• Besides, when AI is integrated into IoT solutions some relevant AI indicators must be described: confusion matrix, precision, training delay, etc.

In general, we do not perform any machine learning procedures in this paper. Thus, no classification precision or training are applicable. The agents evaluate the SPD level based on a deterministic method and react to runtime events based on pre-defined policies. Thus, the underlying intrusion detection (or other security) mechanisms that might raise an alarm that an attack against the system is ongoing, can be subject of machine learning and produce true positives or negatives. Nevertheless, the agents of the proposed framework trust these events and do not perform further processing, as the lay in a higher management layer.

On the other hand, as the agents exchange their individual points of view, inconsistencies may arise when they try to figure out the global state of the system. Thus, for the main reasoning we are based on a previous work [36], where we have implemented a negotiation and conflict resolution mechanism which guarantees that the multi-agent reasoning results will always be at least coherent (if consistency cannot be achieved with the current pieces of knowledge).

[36] G. Hatzivasilis, "Multi-agent distributed epistemic reasoning in ambient intelligence environments," Master Thesis, University of Crete, Greece – FORTH-ICS, November 2011.

• Please, consider to modify Section 4 and 5 to include all these additional results.

We thank the reviewers for their fruitful feedback. We incorporated most of the results in the revisited version. As aforementioned in R2.3, we added a new section where we validated our method against standardized and widely-used methods for risk assessment proposed by NIST and HIPAA. Regarding R2.4, machine learning is not included in this study.

Author Response File: Author Response.pdf

Reviewer 2 Report

In this paper, the authors implement what they call CompoSecReasoner – a methodology for the description of the SPD (security, privacy, and dependability) aspects of composed systems, and the effects of changes in the state-architecture. They state that CompoSecReasoner is appropriate for SPD verification, composition validation, comparison between different system configurations, impact assessment of changes in the system, and materialization of automated reactive strategies. Furthermore, the developed CompoSecReasoner framework is utilized as middleware of IoT applications, providing real-time monitoring and administration. 

Page 2 - line 79, 80 - the authors need to define what JADE and OSGi are for the reader - as like me - some may not be familiar with them.

Page 3 -line 106 - define these terms - CERT and CVE.

Figure 1 - is the Surface here the Attack Surface? If then say this in the Figure. If not then what does Surface refer to?

In Table 1 - I understand that a scoring system is used but it needs to be clarified how this scoring system works - for ex. what does a score of 5, 4, 3 etc mean. Is 5 the best/greatest effort and 1 the worst/least effort? Clarify.

The mention of Figure 1 in the text is a full-page after the image - I recommend that all figures appear after they are mentioned in the Text and not before. The authors need to be consistent when they do this - currently, it varies from figure to figure in the paper.

Line 425:
The Statement "(device’s security is reduced when it connects Internet)" in the paper is not something I would agree with. The device's security is not reduced but more attack vectors are now open/possible due to the device being networked. Can the authors comment on this?

Line 518 - provide a reference for the nSHIELD project you refer to here.

Figure 5 is not clear to me - what does the y scale indicate? there is no unit?

Line 735 - RETE algorithm - what is this - reference for it?

Line 823 - We use CompoSecReasoner to urge if the composition is feasible - what do the authors mean here?

Line 744 - The author's comment: Nonetheless, the framework exhibits acceptable delay, even for a real-time environment is not justifiable given the previous statement on line 738 - The reasoning operations need on average
1.6 seconds - 1.6 seconds delay is not a suitable delay for real-time applications - this delay should be in the low ms range. 

Author Response

Manuscript No # applsci-834121 entitled "AI-driven composition and security validation of an IoT ecosystem".

These are the authors’ responses to the reviewers’ comments. The changes in the manuscript have been highlighted with red color.

Associate Editor
Comments to the Author (Required):

It has been reviewed by experts in the field and we request that you make major revisions before it is processed further.

We want to thank all reviewers for their effort and the feedback that they provided to us. We confirm that we addressed all their comments in the revisited version of the paper as it is clarified below. The main changes include:

• A new Section 2, where we are referring to more recent studies and conduct a qualitative comparative analysis, highlighting the limitations of the existing works and the advantages of the proposed CompoSecReasoner
• We add the Section 7.1, where we validated our method against standardized and widely-used methods for risk assessment proposed by NIST and HIPAA.
• We better explain in the text all formula terms.
• We better clarify the runtime performance of CompoSecReasoner as well as other issues that were mentioned by the reviewers.
• We made minor corrections in the text.

Reviewer(s)' Comments to Author:

Reviewer #1

This paper presents an event-based model-checking framework for IoT systems' design and management, called CompoSecReasoner, which can measure the security, privacy, and dependability (SPD) properties of a composed system and administrate it automatically based on a federated artificial intelligence setting.

The topic is very interesting. However, there exist many things to be improved as follows:

• The paper should be rewritten carefully. The organization of the paper is not consistent. For instance, the problem, motivation, challenges, and contributions are not clear. Moreover, the type of paper seems like a technical report rather than a research paper.

We revise the text to better clarify the examined problem, motivation, challenges, and the overall contribution.

• In the introduction part of the paper, authors shortly describe the difficulties of estimating the SPD in a dynamically composed system, without a detailed analysis of the existing researches. That is, a more detailed problem definition is required in this part or a new section based on the analysis of existing approaches.

We add a new section as Section 2, where we are better defining the scope of the paper and discuss the related works.

• The authors need to show the contribution of the paper compactly and clearly. In Section 5, the authors showed the implementation and analysis results of the proposal without any comparison with other approaches. That is, to clearly present the contributions of the author's approach, experimental or comparison results (e.g., quantitative or qualitative analysis) with other approaches should be discussed more in detail.

At the end of Section 2 (Related Works), we are also comparing our proposal with other works. A qualitative comparative analysis is conducted, highlighting the advantages of CompoSecReasoner.

• In Section 6.1, the literature review is not efficient. That is, most references are so old, especially related to the topic of this paper. The authors should search for the youngest similar works. And then, they should show clearly the limitations of the existing approaches and how your approach overcomes these limitations. I can find some recent papers in major publishers (e.g., MDPI, IEEE, ACM, ELSEVIER, SPRINGER, ...).

The discussion of the previous Section 6,1 is now moved under the new Section 2 (Section 2.4 Comparison). In the overall Section we have included in the discussion newer studies. We clearly state the limitations of the existing approaches and how CompoSecReasoner overcomes them.

Reviewer #2

In this paper, the authors implement what they call CompoSecReasoner – a methodology for the description of the SPD (security, privacy, and dependability) aspects of composed systems, and the effects of changes in the state-architecture. They state that CompoSecReasoner is appropriate for SPD verification, composition validation, comparison between different system configurations, impact assessment of changes in the system, and materialization of automated reactive strategies. Furthermore, the developed CompoSecReasoner framework is utilized as middleware of IoT applications, providing real-time monitoring and administration.

• • Page 2 - line 79, 80 - the authors need to define what JADE and OSGi are for the reader - as like me - some may not be familiar with them.

The two terms have been defined in the test: i) Java Agent DEvelopment framework (JADE) and ii) Open Services Gateway initiative (OSGi).

• Page 3 -line 106 - define these terms - CERT and CVE.

The two terms have been defined in the test: i) Computer Emergency Response Team (CERT) and ii) Common Vulnerabilities and Exposures (CVE).

• Figure 1 - is the Surface here the Attack Surface? If then say this in the Figure. If not, then what does Surface refer to?

Yes, it is the same term. The figure has been revised as suggested.

• In Table 1 - I understand that a scoring system is used but it needs to be clarified how this scoring system works - for ex. what does a score of 5, 4, 3 etc. mean. Is 5 the best/greatest effort and 1 the worst/least effort? Clarify.

We better clarify in the text the values’ meaning: “The damage potential in case of a successful exploit on the legitimate system ranges from 1 (low damage) to 5 (high damage). The effort that the attacker has to devote to perform an attack ranges from 1 (low effort) to 4 (high effort)”.

• The mention of Figure 1 in the text is a full-page after the image - I recommend that all figures appear after they are mentioned in the Text and not before. The authors need to be consistent when they do this - currently, it varies from figure to figure in the paper.

We think that the Figure should be included early in the text. Therefore, we leave the figure where it was, and we move the text accordingly with slight changes: “Figure 1 illustrates the main SPD multi-metric concepts, which are detailed below.”.

• Line 425: The Statement "(device’s security is reduced when it connects Internet)" in the paper is not something I would agree with. The device's security is not reduced but more attack vectors are now open/possible due to the device being networked. Can the authors comment on this?

We see the point that is raised by the reviewer and revise the text accordingly. In general, we think that the reviewer’s point of view is more accurate than our original statement.

The revised text is: “a device’s security could be reduced when it connects to Internet, as more attack vectors are now open/possible due to the device being networked”.

• Line 518 - provide a reference for the nSHIELD project you refer to here.

Two references are provided for the nSHIELD project:

- [29] EU funded project – nSHIELD: new embedded Systems arcHItecturE for multi-Layer Dependable solutions, https://artemis-ia.eu/project/34-nshield.html .

- [30] M. Cesena, et al., "SHIELD Technology Demonstrators," CRC Press, Book for Measurable and Composable Security, Privacy, and Dependability for Cyberphysical Systems, pp. 381-434, 2017.

• Figure 5 is not clear to me - what does the y scale indicate? there is no unit?

The figure has been updated and now includes the axis titles. The y scale indicates the individual S, P, and D values, which range from 0 to 100.

• Line 735 - RETE algorithm - what is this - reference for it?

A brief explanation for the RETE algorithm is mentioned: “The RETE algorithm (one of the main and widely-utilized pattern matching algorithms for implementing rule-based systems)”.

Also, a reference has been provided:

- [41] B. Berstel, “Extending the RETE algorithm for event management,” 9^th International Symposium on Temporal Representation and Reasoning, IEEE, Manchester, UK, 7-9 July, pp. 49-51, 2002.

• Line 823 - We use CompoSecReasoner to urge if the composition is feasible - what do the authors mean here?

One of the features of CompoSecReasoner is that it can reason if the composition of a complex IoT setting is feasible or not. If the setting can be instantiated as in the case of the demonstrated example, then it also evaluates the SPD status of the composed system.

We change the word ‘argue’ in the text to better clarify this issue: “We use CompoSecReasoner to reason if the composition is feasible and figure out the total outcome for SPD metrics”.

• Line 744 - The author's comment: Nonetheless, the framework exhibits acceptable delay, even for a real-time environment is not justifiable given the previous statement on line 738 - The reasoning operations need on average 1.6 seconds - 1.6 seconds delay is not a suitable delay for real-time applications - this delay should be in the low ms range.

We better clarify this issue. “The whole reasoning framework need on average 1.6 seconds, 45MB RAM, and 1.87MB for the code. Nevertheless, this is expected to be done once, when an agent is started. After that, when the rule engine is up and running, it takes around 0.002s to process a theory with a few hundreds of facts [42]. Therefore, this is the actual real-time delay for applications. The code size is not affected while the additional RAM is minimal”.

- [42] Y. Malcolm, “A Federated Agent-Based Crowd Simulation Architecture,” 21^st European Conference on Modelling and Simulation (ECMS), Prague, Czech Republic, June 4-6, 2007, pp. 1-7.

Reviewer #3
The paper is interesting and well-written. It addresses a very popular and promising topic, as the combination of AI and IoT. The proposal, I think, is sound and technically relevant.

However, some problems should be corrected before the manuscript is accepted.

•  
•  
• • First, some equations are hard to understand, as variables are not well described. For example, some variables in expressions (1) and (2) are not introduced.

We revised the text accordingly and explain all equations’ terms.

• On the other hand, algorithms are not clearly presented. Do they contain any novelty or thy are just the implementation of the above description? I'm not sure and this point must be clarified.

The algorithms refer the exact implementation of the procedures that are described in the text. Minor changes in the text have been made.

• Nevertheless, the main problem of the manuscript is the validation section. Figure 8 must be improved. Besides, additional performance indicators must be evaluated, focused on security. NIST's test or other similar standardized experiments should be employed to evaluate if the provided security level is good enough, similar to other existing technologies, etc.

We compare the results of the SPD assessment procedure with similar standardized or widely-used methodologies proposed by the National Institute of Standards and Technology (NIST) and the Health Insurance Portability and Accountability Act (HIPAA). At first, we compare the SPD analysis for the individual system components against the Common Vulnerability Scoring System (CVSS) – a standardized methodology by NIST for the evaluation of distinct system modules. Then, we compare the SPD evaluation of the composed system against the Security Risk Assessment (SRA) tool – a widely-used methodology proposed by NIST and HIPPA for the assessment of integrated systems in the healthcare domain. We noticed that our systematic analysis method resulted similar outcomes and adequately captured the protection posture. The overall comparative study is documented in the subsection 7.1.

• Besides, when AI is integrated into IoT solutions some relevant AI indicators must be described: confusion matrix, precision, training delay, etc.

In general, we do not perform any machine learning procedures in this paper. Thus, no classification precision or training are applicable. The agents evaluate the SPD level based on a deterministic method and react to runtime events based on pre-defined policies. Thus, the underlying intrusion detection (or other security) mechanisms that might raise an alarm that an attack against the system is ongoing, can be subject of machine learning and produce true positives or negatives. Nevertheless, the agents of the proposed framework trust these events and do not perform further processing, as the lay in a higher management layer.

On the other hand, as the agents exchange their individual points of view, inconsistencies may arise when they try to figure out the global state of the system. Thus, for the main reasoning we are based on a previous work [36], where we have implemented a negotiation and conflict resolution mechanism which guarantees that the multi-agent reasoning results will always be at least coherent (if consistency cannot be achieved with the current pieces of knowledge).

[36] G. Hatzivasilis, "Multi-agent distributed epistemic reasoning in ambient intelligence environments," Master Thesis, University of Crete, Greece – FORTH-ICS, November 2011.

• Please, consider to modify Section 4 and 5 to include all these additional results.

We thank the reviewers for their fruitful feedback. We incorporated most of the results in the revisited version. As aforementioned in R2.3, we added a new section where we validated our method against standardized and widely-used methods for risk assessment proposed by NIST and HIPAA. Regarding R2.4, machine learning is not included in this study.

Author Response File: Author Response.pdf

Reviewer 3 Report

The paper is interesting and well-written. It addresses a very popular and promising topic, as the combincation of AI and IoT. The proposal, I think, is sound and technically relevant.

However, some problems should be corrected before the manuscript is accepted.

First, some equations are hard to understand, as variables are not well described. For example, some variables in expressions (1) and (2) are not introduced.

On the other hand, algorithms are not clearly presented. Do they contain any novelty or thy are just the implementation of the above description? I'm not sure and this point must be clarified.

Nevertheless, the main problem of the manuscript is the validation section. Figure 8 must be improved. Besides, additional performance indicators must be evaluated, focused on security. NIST's test or other similar standarized experiments should be employed to evalute if the provided security level is good enough, similar to other existing technologies, etc. 

Besides, when AI is integrated into IoT solutions some relevant AI indicators must be described: confusion matrix, precision, trainning delay, etc.

Please, consider to modify Section 4 and 5 to include all these additional results.

Author Response

Manuscript No # applsci-834121 entitled "AI-driven composition and security validation of an IoT ecosystem".

These are the authors’ responses to the reviewers’ comments. The changes in the manuscript have been highlighted with red color.

Associate Editor
Comments to the Author (Required):

It has been reviewed by experts in the field and we request that you make major revisions before it is processed further.

We want to thank all reviewers for their effort and the feedback that they provided to us. We confirm that we addressed all their comments in the revisited version of the paper as it is clarified below. The main changes include:

• A new Section 2, where we are referring to more recent studies and conduct a qualitative comparative analysis, highlighting the limitations of the existing works and the advantages of the proposed CompoSecReasoner
• We add the Section 7.1, where we validated our method against standardized and widely-used methods for risk assessment proposed by NIST and HIPAA.
• We better explain in the text all formula terms.
• We better clarify the runtime performance of CompoSecReasoner as well as other issues that were mentioned by the reviewers.
• We made minor corrections in the text.

Reviewer(s)' Comments to Author:

Reviewer #1

This paper presents an event-based model-checking framework for IoT systems' design and management, called CompoSecReasoner, which can measure the security, privacy, and dependability (SPD) properties of a composed system and administrate it automatically based on a federated artificial intelligence setting.

The topic is very interesting. However, there exist many things to be improved as follows:

• The paper should be rewritten carefully. The organization of the paper is not consistent. For instance, the problem, motivation, challenges, and contributions are not clear. Moreover, the type of paper seems like a technical report rather than a research paper.

We revise the text to better clarify the examined problem, motivation, challenges, and the overall contribution.

• In the introduction part of the paper, authors shortly describe the difficulties of estimating the SPD in a dynamically composed system, without a detailed analysis of the existing researches. That is, a more detailed problem definition is required in this part or a new section based on the analysis of existing approaches.

We add a new section as Section 2, where we are better defining the scope of the paper and discuss the related works.

• The authors need to show the contribution of the paper compactly and clearly. In Section 5, the authors showed the implementation and analysis results of the proposal without any comparison with other approaches. That is, to clearly present the contributions of the author's approach, experimental or comparison results (e.g., quantitative or qualitative analysis) with other approaches should be discussed more in detail.

At the end of Section 2 (Related Works), we are also comparing our proposal with other works. A qualitative comparative analysis is conducted, highlighting the advantages of CompoSecReasoner.

• In Section 6.1, the literature review is not efficient. That is, most references are so old, especially related to the topic of this paper. The authors should search for the youngest similar works. And then, they should show clearly the limitations of the existing approaches and how your approach overcomes these limitations. I can find some recent papers in major publishers (e.g., MDPI, IEEE, ACM, ELSEVIER, SPRINGER, ...).

The discussion of the previous Section 6,1 is now moved under the new Section 2 (Section 2.4 Comparison). In the overall Section we have included in the discussion newer studies. We clearly state the limitations of the existing approaches and how CompoSecReasoner overcomes them.

Reviewer #2

In this paper, the authors implement what they call CompoSecReasoner – a methodology for the description of the SPD (security, privacy, and dependability) aspects of composed systems, and the effects of changes in the state-architecture. They state that CompoSecReasoner is appropriate for SPD verification, composition validation, comparison between different system configurations, impact assessment of changes in the system, and materialization of automated reactive strategies. Furthermore, the developed CompoSecReasoner framework is utilized as middleware of IoT applications, providing real-time monitoring and administration.

• • Page 2 - line 79, 80 - the authors need to define what JADE and OSGi are for the reader - as like me - some may not be familiar with them.

The two terms have been defined in the test: i) Java Agent DEvelopment framework (JADE) and ii) Open Services Gateway initiative (OSGi).

• Page 3 -line 106 - define these terms - CERT and CVE.

The two terms have been defined in the test: i) Computer Emergency Response Team (CERT) and ii) Common Vulnerabilities and Exposures (CVE).

• Figure 1 - is the Surface here the Attack Surface? If then say this in the Figure. If not, then what does Surface refer to?

Yes, it is the same term. The figure has been revised as suggested.

• In Table 1 - I understand that a scoring system is used but it needs to be clarified how this scoring system works - for ex. what does a score of 5, 4, 3 etc. mean. Is 5 the best/greatest effort and 1 the worst/least effort? Clarify.

We better clarify in the text the values’ meaning: “The damage potential in case of a successful exploit on the legitimate system ranges from 1 (low damage) to 5 (high damage). The effort that the attacker has to devote to perform an attack ranges from 1 (low effort) to 4 (high effort)”.

• The mention of Figure 1 in the text is a full-page after the image - I recommend that all figures appear after they are mentioned in the Text and not before. The authors need to be consistent when they do this - currently, it varies from figure to figure in the paper.

We think that the Figure should be included early in the text. Therefore, we leave the figure where it was, and we move the text accordingly with slight changes: “Figure 1 illustrates the main SPD multi-metric concepts, which are detailed below.”.

• Line 425: The Statement "(device’s security is reduced when it connects Internet)" in the paper is not something I would agree with. The device's security is not reduced but more attack vectors are now open/possible due to the device being networked. Can the authors comment on this?

We see the point that is raised by the reviewer and revise the text accordingly. In general, we think that the reviewer’s point of view is more accurate than our original statement.

The revised text is: “a device’s security could be reduced when it connects to Internet, as more attack vectors are now open/possible due to the device being networked”.

• Line 518 - provide a reference for the nSHIELD project you refer to here.

Two references are provided for the nSHIELD project:

- [29] EU funded project – nSHIELD: new embedded Systems arcHItecturE for multi-Layer Dependable solutions, https://artemis-ia.eu/project/34-nshield.html .

- [30] M. Cesena, et al., "SHIELD Technology Demonstrators," CRC Press, Book for Measurable and Composable Security, Privacy, and Dependability for Cyberphysical Systems, pp. 381-434, 2017.

• Figure 5 is not clear to me - what does the y scale indicate? there is no unit?

The figure has been updated and now includes the axis titles. The y scale indicates the individual S, P, and D values, which range from 0 to 100.

• Line 735 - RETE algorithm - what is this - reference for it?

A brief explanation for the RETE algorithm is mentioned: “The RETE algorithm (one of the main and widely-utilized pattern matching algorithms for implementing rule-based systems)”.

Also, a reference has been provided:

- [41] B. Berstel, “Extending the RETE algorithm for event management,” 9^th International Symposium on Temporal Representation and Reasoning, IEEE, Manchester, UK, 7-9 July, pp. 49-51, 2002.

• Line 823 - We use CompoSecReasoner to urge if the composition is feasible - what do the authors mean here?

One of the features of CompoSecReasoner is that it can reason if the composition of a complex IoT setting is feasible or not. If the setting can be instantiated as in the case of the demonstrated example, then it also evaluates the SPD status of the composed system.

We change the word ‘argue’ in the text to better clarify this issue: “We use CompoSecReasoner to reason if the composition is feasible and figure out the total outcome for SPD metrics”.

• Line 744 - The author's comment: Nonetheless, the framework exhibits acceptable delay, even for a real-time environment is not justifiable given the previous statement on line 738 - The reasoning operations need on average 1.6 seconds - 1.6 seconds delay is not a suitable delay for real-time applications - this delay should be in the low ms range.

We better clarify this issue. “The whole reasoning framework need on average 1.6 seconds, 45MB RAM, and 1.87MB for the code. Nevertheless, this is expected to be done once, when an agent is started. After that, when the rule engine is up and running, it takes around 0.002s to process a theory with a few hundreds of facts [42]. Therefore, this is the actual real-time delay for applications. The code size is not affected while the additional RAM is minimal”.

- [42] Y. Malcolm, “A Federated Agent-Based Crowd Simulation Architecture,” 21^st European Conference on Modelling and Simulation (ECMS), Prague, Czech Republic, June 4-6, 2007, pp. 1-7.

Reviewer #3
The paper is interesting and well-written. It addresses a very popular and promising topic, as the combination of AI and IoT. The proposal, I think, is sound and technically relevant.

However, some problems should be corrected before the manuscript is accepted.

•  
•  
• • First, some equations are hard to understand, as variables are not well described. For example, some variables in expressions (1) and (2) are not introduced.

We revised the text accordingly and explain all equations’ terms.

• On the other hand, algorithms are not clearly presented. Do they contain any novelty or thy are just the implementation of the above description? I'm not sure and this point must be clarified.

The algorithms refer the exact implementation of the procedures that are described in the text. Minor changes in the text have been made.

• Nevertheless, the main problem of the manuscript is the validation section. Figure 8 must be improved. Besides, additional performance indicators must be evaluated, focused on security. NIST's test or other similar standardized experiments should be employed to evaluate if the provided security level is good enough, similar to other existing technologies, etc.

We compare the results of the SPD assessment procedure with similar standardized or widely-used methodologies proposed by the National Institute of Standards and Technology (NIST) and the Health Insurance Portability and Accountability Act (HIPAA). At first, we compare the SPD analysis for the individual system components against the Common Vulnerability Scoring System (CVSS) – a standardized methodology by NIST for the evaluation of distinct system modules. Then, we compare the SPD evaluation of the composed system against the Security Risk Assessment (SRA) tool – a widely-used methodology proposed by NIST and HIPPA for the assessment of integrated systems in the healthcare domain. We noticed that our systematic analysis method resulted similar outcomes and adequately captured the protection posture. The overall comparative study is documented in the subsection 7.1.

• Besides, when AI is integrated into IoT solutions some relevant AI indicators must be described: confusion matrix, precision, training delay, etc.

In general, we do not perform any machine learning procedures in this paper. Thus, no classification precision or training are applicable. The agents evaluate the SPD level based on a deterministic method and react to runtime events based on pre-defined policies. Thus, the underlying intrusion detection (or other security) mechanisms that might raise an alarm that an attack against the system is ongoing, can be subject of machine learning and produce true positives or negatives. Nevertheless, the agents of the proposed framework trust these events and do not perform further processing, as the lay in a higher management layer.

On the other hand, as the agents exchange their individual points of view, inconsistencies may arise when they try to figure out the global state of the system. Thus, for the main reasoning we are based on a previous work [36], where we have implemented a negotiation and conflict resolution mechanism which guarantees that the multi-agent reasoning results will always be at least coherent (if consistency cannot be achieved with the current pieces of knowledge).

[36] G. Hatzivasilis, "Multi-agent distributed epistemic reasoning in ambient intelligence environments," Master Thesis, University of Crete, Greece – FORTH-ICS, November 2011.

• Please, consider to modify Section 4 and 5 to include all these additional results.

We thank the reviewers for their fruitful feedback. We incorporated most of the results in the revisited version. As aforementioned in R2.3, we added a new section where we validated our method against standardized and widely-used methods for risk assessment proposed by NIST and HIPAA. Regarding R2.4, machine learning is not included in this study.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

I think the authors have carefully revised the manuscript based on the comments. The revised manuscript could be accepted for publication.

Author Response

Thank you for your comments and your effort.

Author Response File: Author Response.pdf

Reviewer 2 Report

With regards to my comment:

• Figure 5 is not clear to me - what does the y scale indicate? there is no unit?

The Authors answered:

The figure has been updated and now includes the axis titles. The y scale indicates the individual S, P, and D values, which range from 0 to 100.

Query: 0 to 100 - what units - is it % etc..

I commend the authors on their additions to the paper and I think that the additions have improved the work significantly. 

Author Response

In Figure 5, we indicate the unit for measuring the individual S, P, and D values as the percentage (%) of pore coverage, ranging from 0%-100% for no to optimal protection, respectively.

Author Response File: Author Response.pdf

Reviewer 3 Report

In general, all my previous concerns have ben addressed, and the paper may be accepted.  

Author Response

Thank you for your comments and your effort.

Author Response File: Author Response.pdf