Automated Penetration Testing Framework for Smart-Home-Based IoT Devices
Round 1
Reviewer 1 Report
This article presents an automated penetration testing framework for Smart-Home-based IoT devices using Wi-Fi wireless technology.
All the sections of this text are well written, well balanced, but not well presented with very bad text formatting. The authors use descriptive language to present the problem, related works, and their proposed framework (related to past research findings). The only missing part in the introduction is to explain in more detail (lines 60-61) how they believe such kinds of IoT devices will be tested for vulnerabilities (non-Wi-Fi devices).
Another comment is that the authors refer to a “project” (ex. lines 187-188), but this work presents a framework, please make all corrections in the text.
The results section is poor but acceptable considering the difficulty of the availability of various devices in a laboratory.
Finally, please improve the presentation and coherence of the Discussion section.
Minor comments:
Figure 2. poor quality.
Figure 3. quality could be improved.
Follow mdpi formatting guidelines to enhance your article.
When to refer to Python 3.0, do you mean Python 3.x? because Python 3.0 Release date was back to 03-Dec-2008
Explain the acronym CVSS in the text (not only in keywords)
Author Response
Reviewer#1, Comment # 1: This article presents an automated penetration testing framework for Smart-Home-based IoT devices using Wi-Fi wireless technology.
Author response: We would like to thank the reviewer for acknowledging this.
Author action: NA
Reviewer#1, Comment # 2: All the sections of this text are well written, well balanced, but not well presented with very bad text formatting. The authors use descriptive language to present the problem, related works, and their proposed framework (related to past research findings).
Author response: We would like to thank the reviewer for his comment.
Author action: We have edited the presentation of the sections and the text formatting aspects in the manuscript.
Reviewer#1, Comment # 3: The authors use descriptive language to present the problem, related works, and their proposed framework (related to past research findings). The only missing part in the introduction is to explain in more detail (lines 60-61) how they believe such kinds of IoT devices will be tested for vulnerabilities (non-Wi-Fi devices).
Author response: Currently, the scope of our research only deals with testing the devices that are accessible over a network, i.e. The framework uses the IP Address of the device (as one of the major inputs) to interact with the device during the penetration testing. Therefore, this aspect has been purposefully omitted from the introduction to avoid the readers from deviating (from the current topic).
Author action: We have mentioned this in the manuscript's scope and future work sections.
Reviewer#1, Comment # 4: The authors refer to a “project” (ex. lines 187-188), but this work presents a framework, please make all corrections in the text. We have used this word to refer to the research as a whole
Author response: We would like to thank the reviewer for his comment. We have used this word to refer to the research (as a whole).
Author action: We have updated the manuscript by replacing the word ‘project’ with the words ‘research work’ and ‘research’ (according to the context) everywhere in the manuscript.
Reviewer#1 Comment # 5: The results section is poor but acceptable considering the difficulty of the availability of various devices in a laboratory.
Author response: We would like to thank the reviewer for his comment. Some of the details related to the results section were present in the conclusion in detail. We have modified that.
Author action: We have modified the results section by adding more findings that were initially present in the conclusion to make the section complete and more presentable.
Reviewer#1 Comment # 6: Please improve the presentation and coherence of the Discussion section
Author response: We would like to thank the reviewer for his comment. We have made the necessary changes to the section to make it more coherent and presentable as requested.
Author action: We have added, deleted and restructured this section in the manuscript to improve the coherence of the discussion.
Reviewer#1 Comment # 7: Figure 2. poor quality; Figure 3. quality could be improved.
Author response: We would like to thank the reviewer for his comment. We have updated the figures.
Author action: We have updated Figure 2’ and ‘Figure 3’ (as requested by Reviewer 2) and the quality of the images has also been subsequently improved.
Reviewer#1 Comment # 8: Follow MDPI formatting guidelines to enhance your article.
Author response: We would like to thank the reviewer for his comment. We did follow the guidelines initially, but we noticed that some minor changes can be done (especially in the references section).
Author action: So, as requested, we made all the necessary changes we could to enhance the article.
Reviewer#1 Comment # 8: When you refer to Python 3.0, do you mean Python 3.x? because Python 3.0 Release date was back to 03-Dec-2008
Author response: We would like to thank the reviewer for his comment. We are sorry for the misunderstanding. When we referred to Python 3.0, we meant it as ‘Python3’. We have designed the code in Kali Linux 2019 version which comes with Python 3.6 preinstalled.
Author action: We have changed ‘Python 3.0’ to ‘Python 3.6’ everywhere necessary in the manuscript.
Reviewer#1 Comment # 9: Explain the acronym CVSS in the text (not only in keywords)
Author response: We would like to thank the reviewer for his comment. We updated the manuscript with the necessary changes.
Author action: As requested, the acronym ‘CVSS’ was changed to ‘Common Vulnerability Scoring System (CVSS)’ (wherever present in the manuscript).
Author Response File: 
Author Response.pdf
Reviewer 2 Report
The contribution of the manuscript is the automation of penetration testing. The framework considers a subset of vulnerabilities for selected IoT devices.
The article is structured correctly and the content is presented in a logically consistent order.
However, several elements of the article require significant improvement.
In the "Abstract" there is no summary of the obtained results and main conclusion.
In my opinion, the entire fragment from lines 62-74 from the "Introduction" section is highly desirable. However, figure 1 should be corrected. Not all components are named. Besides, the architectural view would be better when applying layers. The authors may draw components from left to right starting with those responsible for the user interface.
On lines 100-111, the authors represent the benefits rather than describe the contribution. Rather, this section fits in the "Discussion and limitations." Please highlight the contribution.
The authors claim (lines 155-157) that "usage of automated techniques (for the security assessment of IoT devices) would lead to a prominent increase in the level of security of IoT devices". Testing alone does not increase the level of security. It allows you to detect vulnerabilities and what we do with this knowledge may have an impact on increasing the level of security.
Figures 2 and 3 are needed. Unfortunately, they have many inaccuracies. If possible, use the Unified Modeling Language Activity diagram to represent the flow of events/activities. Please remember about pairs of fork/join synchronization bars (Figure 2), and decision point merges (Figure 3). Please also use the start activity and activity final in diagrams.
Unfortunately, the architectural description in the manuscript is missing. In the design of the framework, there are not shown diagrams. The issue of architectural description is described in the standard: ISO/IEC/IEEE 42010: 2011, Systems and software engineering - Architecture description, https://www.iso.org/standard/50508.html. Well known in the community is the 4+1 model by Kruchten (http://dx.doi.org/10.1109/52.469759). But, the authors should also look at the up-to-date article discussing various aspects of building the architecture of complex systems "The 1+5 Architectural Views Model in Designing Blockchain and IT System Integration Solutions" (10.3390/sym13112000). The authors should use standard modeling means like Unified Modeling Language (UML) to present the software architecture of their framework. In my view, the authors should use the UML Component and UML Deployment diagrams.
In the "Discussion and limitations" section, please do not use as many sub-sections. In this section, please comment on the possibility of using software patterns to increase the security of IoT devices, especially connected with blockchain technology.
Please consider the following papers:
- "The design of secure IoT applications using patterns: State of the art and directions for research" (https://doi.org/10.1016/j.iot.2021.100408),
- "Leveraging blockchain for ensuring trust in IoT: A survey" (https://doi.org/10.1016/j.jksuci.2021.09.004),
- "Reconfigurable Smart Contracts for Renewable Energy Exchange with Re-Use of Verification Rules" (https://doi.org/10.3390/app12115339),
- "Reconfigurable Security Architecture (RESA) Based on PUF for FPGA-Based IoT Devices" (https://doi.org/10.3390/s22155577).
It should give a broader perspective.
The "Conclusions" section should be more supported by the results. In the section, it is worth emphasizing conclusions from the work.
In the "References" section the authors should add DOIs for papers. Besides, the description of the 11th bibliographic item is not complete. Generally, there are too few articles from journals with Impact Factor.
The source code is not available.
The English used in the manuscript requires a lot of effort to correct as far as the style and punctuation are concerned. The text should be justified. Please also avoid using etc. The authors should be more specific. Please use formal language. Do not use abbreviated forms like "doesn't". Please check and correct the manuscript thoroughly.
Author Response
Reviewer#2 Comment #1: The contribution of the manuscript is the automation of penetration testing. The framework considers a subset of vulnerabilities for selected IoT devices.
Author response: We would like to thank the reviewer for acknowledging this.
Author action: NA
Reviewer#2 Comment # 2: The article is structured correctly, and the content is presented in a logically consistent order.
Author response: We would like to thank the reviewer for his comment.
Author action: NA
Reviewer#2 Comment # 3: In the "Abstract" there is no summary of the obtained results and main conclusion.
Author response: We would like to thank the reviewer for his comment. We have updated the abstract accordingly.
Author action: We have added the summary of the results obtained and the main conclusion to the abstract.
Reviewer#2 Comment # 4: In my opinion, the entire fragment from lines 62-74 from the "Introduction" section is highly desirable. However, figure 1 should be corrected. Not all components are named. Besides, the architectural view would be better when applying layers.
Author response: We would like to thank the reviewer for his comment. We have updated ‘Figure 1’ accordingly.
Author action: We have named all components in the figure and added names to the layers in the figure too (i.e.: ‘User Interface’, ‘Hardware’ and ‘Network’).
Reviewer#2 Comment # 5: On lines 100-111, the authors represent the benefits rather than describe the contribution. Rather, this section fits in the "Discussion and limitations." Please highlight the contribution.
Author response: We would like to thank the reviewer for his comment. We have made changes to the benefits part as requested. We have also updated the contribution part.
Author action: We have updated the manuscript with the reformulated contribution section and added the ‘benefits’ part in the discussion section.
Reviewer#2 Comment # 6: The authors claim (lines 155-157) that "usage of automated techniques (for the security assessment of IoT devices) would lead to a prominent increase in the level of security of IoT devices". Testing alone does not increase the level of security. It allows you to detect vulnerabilities and what we do with this knowledge may have an impact on increasing the level of security
Author response: Firstly, we would like to clarify that this statement was a deduction that was referenced from the paper “Rak, M.; Salzillo, G.; Granata, D. ESSecA: An Automated Expert System for Threat Modelling and Penetration Testing for IoT Ecosystems. Computers and Electrical Engineering 2022, 99, 107721, doi:10.1016/J.COMPELECENG.2022.107721.” Yes, we agree with the reviewer that testing alone does not increase the level of security. It allows you to detect vulnerabilities and what we do with this knowledge may have an impact on increasing the level of security. But, since testing is necessary to gain that knowledge, and the knowledge helps in improving the security, it is therefore implied that the automated techniques can improve the level of security for the IoT devices. A more detailed explanation of this is provided in the paper referenced and the deduced conclusion from the paper has been taken to support our cause in the literature review section of our manuscript
Author action: NA
Reviewer#2 Comment # 7: Figures 2 and 3 are needed. Unfortunately, they have many inaccuracies. If possible, use the Unified Modeling Language Activity diagram to represent the flow of events/activities. Please remember about pairs of fork/join synchronization bars (Figure 2), and decision point merges (Figure 3). Please also use the start activity and activity final in diagrams.
Author response: We would like to thank the reviewer for his comment. We have made the necessary changes as requested.
Author action: We have followed the instruction accordingly and restructured both the flowcharts into UML activity diagrams.
Reviewer#2 Comment # 8: Unfortunately, the architectural description in the manuscript is missing. In the design of the framework, there are not shown diagrams. The issue of architectural description is described in the standard: ISO/IEC/IEEE 42010: 2011, Systems and software engineering - Architecture description, https://www.iso.org/standard/50508.html. Well known in the community is the 4+1 model by Kruchten (http://dx.doi.org/10.1109/52.469759). But the authors should also look at the up-to-date article discussing various aspects of building the architecture of complex systems "The 1+5 Architectural Views Model in Designing Blockchain and IT System Integration Solutions" (10.3390/sym13112000).
Author response: We would like to thank the reviewer for his comment. We have provided the architectural description in the design section by specifying the design of our framework (i.e., the algorithm) through The UML activity diagram.
Author action: The UML activity diagram explaining the flow of the algorithm of the framework has been added to the Design section.
Reviewer#2 Comment # 9: In the "Discussion and limitations" section, please do not use as many sub-sections. In this section, please comment on the possibility of using software patterns to increase the security of IoT devices, especially connected with blockchain technology.
Author response: We would like to thank the reviewer for his comment. We have tried to cut down and made changes as much as possible to the ‘Discussion’ section. We believe that after the edits, the sub-sections (and their details) in their current state are very much relevant and necessary for the understanding of the reader and the completeness of our paper. We also appreciate the reviewer for giving us a (broader perspective that is relevant to our topic). We have made the necessary additions and changes to the discussions section as requested.
Author action: We have restructured the “discussion and limitations” section completely to make it more presentable. We have also added the ‘Future work’ and the ‘Benefits’ part to this section. We have also added a paragraph where we talk about the use of blockchain in the IoT environment and its impact on IoT security and cited references suggested by the reviewer.
Reviewer#2 Comment # 10: The "Conclusions" section should be more supported by the results. In the section, it is worth emphasizing conclusions from the work
Author response: We would like to thank the reviewer for his comment. This was done in our final draft of the manuscript, but we edited it out to make the conclusion more concise.
Author action: We have added the results from the manuscript in the conclusion and emphasized the same.
Reviewer#2 Comment # 11: In the "References" section the authors should add DOIs for papers. Besides, the description of the 11th bibliographic item is not complete.
Author response: We would like to thank the reviewer for his comment. We have made all the necessary changes as requested.
Author action: We have added the DOIs for all the reference papers that had them except for one. We replaced one paper “Pederson, P. Penetration Testing of Industrial Control Systems. Security 2005, 5.” as we could no longer find the paper online. We replaced the paper with the reference “Duggan, D.P. Penetration Testing of Industrial Control Systems; Albuquerque, New Mexico 87185 and Livermore, California 94550, 2005;” as it had the same contents that we needed to reference. But, unfortunately, even this paper had no DOI. We have also ensured that the description of the 11th Bibliographic item is complete.
Reviewer#2 Comment # 12: The source code is not available.
Author response: We would like to thank the reviewer for his comment. We have added a link to the source code.
Author action: We have provided a link to the public GitHub repository of the source code (of the framework) as a footnote in the implementation section.
Reviewer#2 Comment # 13: The English used in the manuscript requires a lot of effort to correct as far as the style and punctuation are concerned. The text should be justified. Please also avoid using etc. The authors should be more specific. Please use formal language. Do not use abbreviated forms like "doesn't". Please check and correct the manuscript thoroughly
Author response: We would like to thank the reviewer for his comment. We have made all the necessary changes as requested.
Author action: We have checked and edited the manuscript as much as possible to make sure that the style and punctuation of the language used in the document are accurate. We edited all the abbreviated forms and words like ‘etc.’ (we replaced them with other formal words according to context) to make the language more formal as requested by the reviewer.
Author Response File: Author Response.pdf
Round 2
Reviewer 1 Report
The authors responded satisfactorily to all my comments.
One minor comment concerning the CVSS acronym: please explain it only when it first appears in the main text, not on all occurences.
Author Response
Response to Reviewer 1 Comments
Point 1: The authors responded satisfactorily to all my comments.
Response 1: We would like to thank the reviewer for this comment.
Point 2: One minor comment concerning the CVSS acronym: please explain it only when it first appears in the main text, not on all occurrences.
Response 2: We would like to thank the reviewer for this comment. We have changed the acronym explanation in all the other places except the main text. Now, the acronym is only explained in the main text.
Reviewer 2 Report
The contribution of the manuscript is the automation of penetration testing. The framework considers a subset of vulnerabilities for selected IoT devices.
The article is structured correctly and the content is presented in a logically consistent order.
I confirm that the authors have addressed a few of my concerns. However, the manuscript still has many weaknesses. It needs further clarification and significant improvement.
The authors should further clarify the contribution. Is the framework really integrated with CVSS? There is no such description in section 3.1 "Design of framework".
In the "Introduction" section figure 1 should be corrected. The architectural view would be better when applying layers. The authors should draw components from left to right starting with those responsible for the user interface.
I confirm that figure 2 has been corrected and the appropriate UML diagram used.
But, the architectural description of the framework has significant faults. Figure 3 still needs refinement. Please remember about decision points, merges, and activity final in the diagram. In the diagram, the authors should use partitions to underline responsibility for the actions of components of the framework.
Figure 3 shows only the flow of events of the framework that should be situated in the Use case view of the architectural description (the 4+1 model by Kruchten (http://dx.doi.org/10.1109/52.469759), and the 1+5 model by Górski (https://doi.org/10.3390/sym13112000)).
However, the architectural description in the manuscript is still missing. In my view, the authors should use the UML Component and UML Deployment diagrams.
Besides, using UML diagrams is not the authors' idea. So, the authors should cite papers by Kruchten and Górski. The authors used those works to refine their manuscript and broaden their perspective.
The "Discussion and limitations" section has been changed. But it requires further improvement. The authors used two of the suggested papers. In this section, please comment on the possibility of using software patterns to increase the security of IoT devices, especially those connected with blockchain technology. It is not enough to state that saves time and raises ease of use. The authors did not mention the efficiency of test design and execution. They have not mentioned the number of test cases used in their tool. In addition, the results of the runtime of these tests are not presented anywhere in this manuscript.
The "Conclusions" section should be more supported by the results. In the section, it is worth emphasizing conclusions from the work.
The source code repository should be added to the "References" section.
There are still not enough current articles from reputable journals in the "References" section. Besides, the authors should not add their own work that is not related to the subject. This is the case for the item [25] which should be deleted.
The English used in the manuscript requires a lot of effort to correct as far as the style and punctuation are concerned. The text should be justified. Please also avoid using etc. The authors should be more specific. Please use formal language. Do not use abbreviated forms like "it's" (line 593). Please check and correct the manuscript thoroughly.
Author Response
Response to Reviewer 2 Comments
Point 1: The contribution of the manuscript is the automation of penetration testing. The framework considers a subset of vulnerabilities for selected IoT devices. The article is structured correctly and the content is presented in a logically consistent order. I confirm that the authors have addressed a few of my concerns. However, the manuscript still has many weaknesses. It needs further clarification and significant improvement.
Response 1: We would like to thank the reviewer for these comments.
Point 2: The authors should further clarify the contribution. Is the framework really integrated with CVSS? There is no such description in section 3.1 "Design of framework".
Response 2: Firstly, we are sorry for the confusion. In our current implementation of the framework, we have not integrated it with CVSS for automatic calculation of CVSS score. Rather currently the CVSS score is calculated manually based on the finding of the automatic framework. However, we keep this as one of our future work.
Point 3: In the "Introduction" section figure 1 should be corrected. The architectural view would be better when applying layers. The authors should draw components from left to right starting with those responsible for the user interface.
Response 3: We acknowledge the reviewer’s comments. We hve updated Figure 1 based on the feedback.
Point 4: I confirm that figure 2 has been corrected and the appropriate UML diagram used.
Response 4: We would like to thank the reviewer for this comment.
Point 5: But, the architectural description of the framework has significant faults. Figure 3 still needs refinement. Please remember about decision points, merges, and activity final in the diagram. In the diagram, the authors should use partitions to underline responsibility for the actions of components of the framework.
Response 5: We acknowledge the reviewer’s comments. We have checked the decision points and merges in the diagram. Also, partitions have been added and Figure 3 has been corrected accordingly.
Point 6: Figure 3 shows only the flow of events of the framework that should be situated in the Use case view of the architectural description (the 4+1 model by Kruchten (http://dx.doi.org/10.1109/52.469759), and the 1+5 model by Górski (https://doi.org/10.3390/sym13112000)). Besides, using UML diagrams is not the authors' idea. So, the authors should cite papers by Kruchten and Górski. The authors used those works to refine their manuscript and broaden their perspective
Response 6: We acknowledge the reviewer’s comments. The papers by Kruchten and Gorski have been cited in our paper (in places where the UML diagrams are added).
Point 7: However, the architectural description in the manuscript is still missing. In my view, the authors should use the UML Component and UML Deployment diagrams.
Response 7: We acknowledge the reviewer’s comments. We have added the UML component and deployment diagrams for the framework to the paper.
Point 8: The "Discussion and limitations" section has been changed. But it requires further improvement. The authors used two of the suggested papers. In this section, please comment on the possibility of using software patterns to increase the security of IoT devices, especially those connected with blockchain technology
Response 8: We acknowledge the reviewer’s comments. We have already added these papers and commented on the possibility of using the software patterns (in paragraph number 2 of the discussions section).
Point 9: It is not enough to state that saves time and raises ease of use. The authors did not mention the efficiency of test design and execution. They have not mentioned the number of test cases used in their tool. In addition, the results of the runtime of these tests are not presented anywhere in this manuscript
Response 9: We acknowledge the reviewer’s comments. We understand the reviewer’s query and we would like to clarify that when we stated that our framework saves time, we meant that when in comparison to using the same tools/techniques individually (i.e., manually), the automation of these tools/techniques take less time to assess smart hombe devices. Secondly, when we said that our framework is easier to use, we implied that the user does not have any major inputs to give to the framework apart from the IP address (if not discovered from the NetDiscover part) and the framework does all the execution of tools on the device and reports the vulnerabilities present in them. Whereas, when we do the same execution of the tools manually (one after the other), the user is required to provide multiple inputs for each technique which is cumbersome. In addition, non-technical users might find it difficult to understand and provide these inputs.
We would also like to mention that we had several test cases during the implementation stage of the framework, to check whether each part of the framework is working or not. We have added a table in the results section which contains an approximation of the execution time of the tests. The execution time is an approximate range as it was difficult to calculate the actual time due to some manual inputs during the execution time. In summary, we have observed that the execution of the framework saves consideraable amount to time compared to manual penetration testing.
Point 9: The "Conclusions" section should be more supported by the results. In the section, it is worth emphasizing conclusions from the work
Response 9: We acknowledge the reviewer’s comments. We would like to clarify that we have changed the conclusion section based on the comments from round one of corrections. The conclusions section is supported by the results from the results section of our work.
Point 10: The "Conclusions" section should be more supported by the results. In the section, it is worth emphasizing conclusions from the work
Response 10: We acknowledge the reviewer’s comments. We would like to clarify that we have already changed the conclusion section based on the comments from round one of corrections. The conclusions section is supported by the results from the results section of our work
Point 11: The source code repository should be added to the "References" section.
Response 11: We acknowledge the reviewer’s comments. We have added the link to the source code repository to the references section.
Point 12: There are still not enough current articles from reputable journals in the "References" section. Besides, the authors should not add their own work that is not related to the subject. This is the case for the item [25] which should be deleted.
Response 12: We acknowledge the reviewer’s comments. We have added a few more references from reputable sources. Also, item [25] has been deleted, we have only added that reference because we found that it was relevant to the topic under discussion. But, to avoid discrepancies in the future, we addressed the reviewer’s comments and a new reference that conveys the same detail has been added in its place.
Point 13: The English used in the manuscript requires a lot of effort to correct as far as the style and punctuation are concerned. The text should be justified. Please also avoid using etc. The authors should be more specific. Please use formal language. Do not use abbreviated forms like "it's" (line 593). Please check and correct the manuscript thoroughly.
Response 13: We acknowledge the reviewer’s comments. We would like to clarify that we have already made these changes in round one of the corrections when the reviewer mentioned the same. The paper now does not contain any abbreviated forms like it’s, etc. These words have been replaced by more professional and formal phrases. Furthermore, we have checked the document again based on the reviewer’s comments and made corrections wherever necessary.
Round 3
Reviewer 2 Report
I confirm that the authors have addressed virtually all of my concerns. Besides, the authors have augmented the manuscript with pointed topics.
The manuscript has been improved substantially.
I am really glad that the authors have used UML diagrams in the architectural description. The authors' contribution has been additionally broadened.
The "References" section has also been corrected according to all comments.
Minor stylistic or punctuation errors should be corrected but they do not diminish the value of the manuscript. Besides, the text should be justified. But, it can be done at the author's proofreading stage.
I recommend accepting the paper in its present form.
