Blockchain-Based Evidence Trustworthiness System in Certification

Round 1

Reviewer 1 Report

Please make a clear case of what the novelty of the work is compared to existing literature.

For example, 

Pu, Shuyi, and Jasmine Siu Lee Lam. "The benefits of blockchain for digital certificates: A multiple case study analysis." Technology in Society 72 (2023): 102176.

Please see the papers that cite this work as well from Google Scholar.

Lines 59-66 in the introduction is a repetition of what was said earlier about evidence

Tables 3 and 5 appear to be the same

Lines 83-88 does not provide a description of.what Section 2 is about. 

Section 2 seems to be a list of well known issues. Its not clear how the article addresses these. 

Author Response

Thank you very much for your comments.

Comments 1: Please make a clear case of what the novelty of the work is compared to existing literature. For example, Pu, Shuyi, and Jasmine Siu Lee Lam. "The benefits of blockchain for digital certificates: A multiple case study analysis." Technology in Society 72 (2023): 102176. Please see the papers that cite this work as well from Google Scholar.

Thank you for the suggestion. It is true that the innovation was not clearly defined in the paper. That is why we have included a new paragraph in the "Validation and Results" section explaining the main benefits of this proposal (page 18, lines:507-516). We have also added Table 7 explaining how the use of the trustworthiness system helps to mitigate the risks identified for evidence in certification.

We have also checked the provided reference and, although it highlights the use of Blockchain for certification due to several inherent security features (and we have included it for this reason), the objective is different. The analyzed solutions are more related to the certificate itself. In our case, we deal with evidence considered in a certification process. Anyway, it was true that the analysis of the state of the art needed to be extended. We have identified the main limitations of existing solutions from the literature in section 3 (page 10 lines from 261-269).

Comments 2: 

Lines 59-66 in the introduction is a repetition of what was said earlier about evidence.

Thank you for your comment. It is not really a repetition, as the first paragraph introduces why evidence is important for a certification process, while the second paragraph details the main security issues evidence currently deals with. It is true that they may seem repetitive. However, we consider it is better to keep both paragraphs separate to better understand the context of the paper.

Tables 3 and 5 appear to be the same

Thank you for the comment. It is true that the "likelihood" column is the same int the two indicated tables. However:

• Table 3 refers to the justification of the considered likelihood values.
• Table 4 refers to the justification of the considered impact values.
• Table 6 (previous Table 5) refers to the identification of the risk level based on the likelihood values from Table 3 and impact values from table 4.

We think it is better to keep all the tables separate for better understanding of the risk assessment process.

Lines 83-88 does not provide a description of.what Section 2 is about. 

You are totally right. We have added the explanation of the purpose of Section 2 in page 3 lines 86-93.

Section 2 seems to be a list of well known issues. Its not clear how the article addresses these. 

You are totally right. The link between section 2 and section 3 was not clear enough. That is why we have decided to add some explanations that justify the use of a Blockchain based trustworthiness system in certification in page 10 lines 244-259.

Reviewer 2 Report

1. The focus on Blockchain as a solution is under-contextualized against other technological alternatives that could be implemented, such as traditional databases with replication or sophisticated security protocols.

2. This study adapts Blockchain technology to meet the specific needs of the certification process, offering a system that ensures data integrity, transparency, and traceability. While the contribution is strong on the technical side, the lack of discussion on real-world applications in scenarios outside of cybersecurity certification limits the generalizability of the study results.

3. The risk assessment does not include a risk scale or weight that could strengthen the mitigation priority analysis.

4. The use of Blockchain has high implementation and maintenance costs. Discussion on cost efficiency compared to non-Blockchain solutions is not explored.

5. The selection of Quorum as a Blockchain implementation is not accompanied by an in-depth discussion of why it is more suitable than other Blockchain technologies.

6. Validation is limited to the cybersecurity domain, making it difficult to assess its effectiveness in other certification scenarios.

7. There is no quantitative evaluation of increased efficiency or risk reduction, so the claims are qualitative.

8. This solution in the paper is designed for cloud security certification, but the lack of case studies outside of that context limits its relevance to other certification needs, such as industrial or healthcare environments.

9. This paper makes a significant contribution to the discussion of Blockchain applications to support trust in certification processes. However, to increase its impact, broader validation, deeper comparative analysis, and quantitative measurement of the benefits of the proposed system are needed.

1. Please add the risk weights to the assessment to provide clearer prioritization of mitigation.

2. The improvement can be done through conducting quantitative experiments to quantify the benefits in terms of speed, cost, and security delivered by this system.

3. Also, the author should compare the Blockchain solution with alternative technologies to provide insights into cost efficiency and scalability.

4. It is important to conduct validation in other domains beyond cloud security to assess the flexibility and effectiveness of the solution more broadly.

Author Response

Thank you for your comments.

Comment 1. The focus on Blockchain as a solution is under-contextualized against other technological alternatives that could be implemented, such as traditional databases with replication or sophisticated security protocols.

You are right. It was not clearly explained how we considered some of the main disadvantages of Blockchain against traditional or replicated databases. For this reason, in addition to the already existing comparison in pages 8 and 9, we have added some additional details in page 10 lines 244-259, with focus on the performance and costs.

Comment 2. This study adapts Blockchain technology to meet the specific needs of the certification process, offering a system that ensures data integrity, transparency, and traceability. While the contribution is strong on the technical side, the lack of discussion on real-world applications in scenarios outside of cybersecurity certification limits the generalizability of the study results.

Thank you for your suggestion. In fact, although we have validated the proposal for the cloud services cybersecurity certification, it is also applicable to other certification domains by simply adapting the data model as needed (contextual information and data/hash values). As we consider this is one o the main advantages of the proposal, we have added this explanation with some examples in page 19 lines 531-546.

Comment 3. The risk assessment does not include a risk scale or weight that could strengthen the mitigation priority analysis.

You are right. The scale definition was missing. We have added it in new Table 5.

Comment 4. The use of Blockchain has high implementation and maintenance costs. Discussion on cost efficiency compared to non-Blockchain solutions is not explored.

You are totally right. In fact, this comment is related to the first one, as costs seem to be one of the main drawbacks of using Blockchain. A discussion about this topic has been included in page 10 lines 254-259. 

Comment 5. The selection of Quorum as a Blockchain implementation is not accompanied by an in-depth discussion of why it is more suitable than other Blockchain technologies.

It is true that the selection of Quorum is not explained in the paper but it is not a design decision of this paper but of the paper considered as basis. Anyway, in this case, the specific Blockchain technology to be considered is not important provided that it is a permissioned one (as already explained in the paper in page 10 lines 270-282) and it is interoperable/compatible with EBSI for a future deployment (as already explained in page 10 lines 283-300). Quroum is a suitable technology that fulfills both requirements, although other options, such as for example Hyperledger Besu, could be also considered. 

Comment 6. Validation is limited to the cybersecurity domain, making it difficult to assess its effectiveness in other certification scenarios.

You are right. We have not mentioned the possibility of using the proposed system in other certification domains by simply adapting the data models (contextual information and data/hash values). As we consider this is one o the main advantages of the proposal, we have added this explanation with some examples in page 19 lines 531-546.

Comment 7. There is no quantitative evaluation of increased efficiency or risk reduction, so the claims are qualitative.

To be honest, it is not possible to make quantitative measures of the efficiency or risk reduction, mainly because the deployment has not been done in a real Blockchain network but on a proof of concept one. Anyway, we consider that you are right and more analysis related to these issues are needed to better understand the benefits of the solution.

• Regarding the performance, we have added some explanations about what it depends on in page 19 lines 516-530.
• Regarding the risks reduction, we have added Table justifying how risks identified in section 2 have been mitigated with the use of the proposed trustworthiness system.

Comment 8. This solution in the paper is designed for cloud security certification, but the lack of case studies outside of that context limits its relevance to other certification needs, such as industrial or healthcare environments.

You are right. We have not mentioned the possibility of using the proposed system in other certification domains by simply adapting the data models (contextual information and data/hash values). As we consider this is one o the main advantages of the proposal, we have added this explanation with some examples in page 19 lines 531-546.

Comment 9. This paper makes a significant contribution to the discussion of Blockchain applications to support trust in certification processes. However, to increase its impact, broader validation, deeper comparative analysis, and quantitative measurement of the benefits of the proposed system are needed.

Thank you for your feedback. We think you are right and we have added several new details concerning the topics you are mentioning:

• Extension to other certificaton domains in page 19 lines 531-546.
• Performance and risk mitigation analysis in Table 7 and page 19 lines 516-530.
• More comparative analysis in page 18 lines 507-515 regarding extended state of the art in page 10 lines 261-269.

Comment 10. Please add the risk weights to the assessment to provide clearer prioritization of mitigation.

We have already included the risk matrix description in Table 5.

Comment 11. The improvement can be done through conducting quantitative experiments to quantify the benefits in terms of speed, cost, and security delivered by this system.

Quantitative experiments do not really make sense in this paper as the validation has been done using a proof of concept for the Blockchain network, which is the fomponent which will mostly influence the speed and costs. Anyway, we consider a better analysis of these two aspects is important and that is why we have included them as follows:

• We have included an analysis of the costs in page 10 lines 254-259.
• We have included an analysis of the performance in page 19 lines 516-528.
• We have included and analysis of the security identifying how the trustworthiness system mitigates the risks identified for evidence in certification in Table 7.

Comment 12. Also, the author should compare the Blockchain solution with alternative technologies to provide insights into cost efficiency and scalability.

Although the comparison was already included in section 2.2, we have extended it with a deeper analysis of the main drawbacks of Blockchain in page 10 lines 249-259.

Comment 13. It is important to conduct validation in other domains beyond cloud security to assess the flexibility and effectiveness of the solution more broadly.

You are right. We have not mentioned the possibility of using the proposed system in other certification domains by simply adapting the data models (contextual information and data/hash values). As we consider this is one o the main advantages of the proposal, we have added this explanation with some examples in page 19 lines 531-546.

Round 2

Reviewer 1 Report

The

23The paperadsProvided above.
certification for the following reasons T

Provided above. On Line 558: "The trustworthiness system is based on storing hashes of evidence on the Blockchain while keeping the war evidence in a local and private storage." – The word "war" should be corrected to "raw."

Author Response

Thank you very much for your comments.

Introduction Line 22 - Evidence is essential for 23 certification for the following reasons... Introduction Line 58 - such as evidence Barbara Guttman (2022). Thus, they play a crucial role in a certification process across various industries for the following reasons: Seems like a repetition to me.

We understand why you consider both list of bullets like repetition as they are related. The first list of bullets refers to the reasons why evidence is needed in a certification process, while the second refers to the reasons why a digital audit trail system is needed for evidence. Anyway, to avoid repetition of ideas as suggested, we have reduced the details of the second list of points (lines 54-61).

Figure 6 is hard to read Line 499, Figure 8 shows an example in which two different filters have been created for the 499 contextual information of the evidence (cloud service identifier it refers to and the identifier 500 of the tool used to gather the evidence) from Figure 5 I don't see how Figure 8 demonstrates this.

Regarding the first part of the comment, I totally agree with you that Figure 6 is a bit difficult to read. That is because the dashboard includes much information to be shown. Anyway, we have updated it as much as possible (mainly the titles sizes), keeping the same information as required in the example (take into account that it is based on Kibana and there are many limitations in terms of format).

Regarding the second part of the comment, I think the problem is the different considered names. We have updated the names in the webapp of Figure 8 in order to better understand the relation between the implemented filters and the suggested data models. Moreover, the explanation in lines 483-493 has been slightly updated for better understanding.

On Line 558: "The trustworthiness system is based on storing hashes of evidence on the Blockchain while keeping the war evidence in a local and private storage." – The word "war" should be corrected to "raw."

We have corrected the spelling mistake.

Reviewer 2 Report

The authors have made laudable efforts to attend to the comments of the reviewer, especially in extending the discussion on the limitations of Blockchain and its comparison with other technologies. Also, the cost analysis included on page 10, lines 254-259, and the more extensive review of alternative solutions serve to place the role of Blockchain within certification processes in context. However, while the response focused on the potential to apply such a system to other areas of certification, it would be useful if the authors could make this potential more salient with at least one practical example or even a case study outside cloud security. Such a clarification would have made the document more generalizable and relevant in other scenarios of certification. The author has also identified and suggested mitigations against some security risks.

The introduction of a risk scale in Table 5, and the inclusion of a justification over how risks are mitigated in Table 7, improves things. However, to shed better light on the priority, the authors should present greater detail on the methods utilized to define the risk weight, and how these weights affect their choice of mitigation strategy. This risk analysis would be more crystal clear and applicable if a better visualization or explanation of how the prioritization was conducted is given. In addition, whereas the authors did their best to explain that efficiency and risk reduction quantifications could not be effectively done using the proof-of-concept Blockchain network, inclusion of simulated performance results or projected quantitative benefits should, therefore, be considered as such information provides readers with a greater feel for the real-world potential of the system. The explanations related to choosing Quorum as a Blockchain technology and interoperability with EBSI are welcome. However, it would be good to have an explicit rationale for the selection of Quorum over other feasible alternatives, such as Hyperledger Besu.

Although the authors acknowledge that the particular Blockchain choice is not crucial as long as it is permissioned and interoperable, a short comparative rationale for Quorum would clarify its selection. Further, while the authors' response to the request for broader validation across certification domains is noted, the additional explanation on page 19 (lines 531-546) remains theoretical. A stronger validation effort, such as a pilot implementation in a domain like healthcare or industry certification, would substantially improve the impact and generalizability of this study's results. 

Refer to the major comment, some revision needs to be fulfilled accordingly.

Author Response

Thank you very much for your comments.

Please make it clear the method, it could be through the visualization of step by step systematically.

This is a really good suggestion to better understand how our research work was planned. We have added a new section 2 describing the research methodology followed.

Provide with proper research question and match them with the answer accordingly.

Thanks for the suggestion. This comment is related to the previous one. We have defined the research question as the starting point for our research methodology (lines 78-81). We have also linked the results with the identified research gap in lines 526-527.

Add more relevant reference to increase the quality of the paper.

We have added some additional references related to the new added information from the two revisions. Additionally, we have made a new analysis and study of research works related to Blockchain based evidence and we have added two new references (references Sakshi et al. (2024) and Tian et al. (2024)).

While the response focused on the potential to apply such a system to other areas of certification, it would be useful if the authors could make this potential more salient with at least one practical example or even a case study outside cloud security. Such a clarification would have made the document more generalizable and relevant in other scenarios of certification. 

We totally agree that extending the application to other certification domains will be beneficial for the impact of the trustworthiness system. However, we consider it would be another research work. For this reason, we have included this suggestion as future work (lines 625-626). Anyway, we have also created the new subsection “Applicability to other domains” inside the “Validation and Results” section. Here, we have included some of the information already available related to the application of the trustworthiness system to other domains, as well as some examples about how to adapt the system to the health certification domain (lines 579-600).

To shed better light on the priority, the authors should present greater detail on the methods utilized to define the risk weight, and how these weights affect their choice of mitigation strategy. This risk analysis would be more crystal clear and applicable if a better visualization or explanation of how the prioritization was conducted is given.

Thank you for the suggestion. We have extended the explanation of the risk matrix in lines 163-169 for better understanding.

Whereas the authors did their best to explain that efficiency and risk reduction quantifications could not be effectively done using the proof-of-concept Blockchain network, inclusion of simulated performance results or projected quantitative benefits should, therefore, be considered as such information provides readers with a greater feel for the real-world potential of the system.

Thanks for the comment. As you mention, the quantification of the efficiency cannot be effectively done as it mainly depends on the underlaying Blockchain performance, and we have just considered a dedicated proof of concept Blockchain network for validation purposes (for example, no congestion is expected, default values have been considered in the configuration), so performance measurements are not representative. For a real deployment, a correct configuration depending on the specific Blockchain technology should be done. For that reason, we consider it is more useful to explain how the different parameters can affect the performance in order to help to define appropriate configuration parameters. In this sense, we have extended the existing explanation including more details and some useful references and examples (lines 540-567).

Regarding the projected quantitative benefits, we have extended the theoretical explanation in lines 526-532 with some more details. However, it is very difficult to quantify the benefits as this should be measured/provided by the potential users in real scenarios (certification auditors, certified systems). Anyway, we consider it is a good suggestion as future work of the paper. We have included it in lines 623-625.  

The explanations related to choosing Quorum as a Blockchain technology and interoperability with EBSI are welcome. However, it would be good to have an explicit rationale for the selection of Quorum over other feasible alternatives, such as Hyperledger Besu. Although the authors acknowledge that the particular Blockchain choice is not crucial as long as it is permissioned and interoperable, a short comparative rationale for Quorum would clarify its selection. 

We have not made a direct selection of the Quorum technology for this research study. We considered an already existing solution from the literature, which was based in Quorum. As we suggest EBSI as the potential ecosystem to deploy the evidence trustworthiness system, Quorum based solutions are valid as they are Ethereum-like. Anyway, we have also included a brief explanation of some reasons why Quorum improves Hyperledger Besu (lines 340-346) with some useful references including comparisons between both technologies.

Further, while the authors' response to the request for broader validation across certification domains is noted, the additional explanation on page 19 (lines 531-546) remains theoretical. A stronger validation effort, such as a pilot implementation in a domain like healthcare or industry certification, would substantially improve the impact and generalizability of this study's results. 

I totally agree that a second pilot validation will generalize the paper. However, we consider it would be another research work. For this reason, we have included this suggestion as future work (lines 625-626). Anyway, we have also created the new subsection “Applicability to other domains” inside the “Validation and Results” section. Here, we have included some of the information already available related to the application of the trustworthiness system to other domains, as well as some examples about how to adapt the system to the health certification domain (lines 579-600).