Improving Trust in a (Trans)National Invoicing System: The Performance of Crash vs. Byzantine Fault Tolerance at Scale

Round 1

Reviewer 1 Report

The contribution of this article is not clear.

The authors argue that the main contribution is to quantify how much faster CFT is compared to BFT. But the specific quantitative measure is not given.

My concern is what is the practical benefit or implication of this comparison?

Another argument of authors is that BFT protocols are not sufficiently performant on a EU scale. This sentence needs description of an alternative solution for EU scale.

I would recommend more specific and summarized guildlines of blockchin designs for national scale and transnational scale.

This article include grammatical errors. So the proof-reading is required.

Author Response

The contribution of this article is not clear.

The contribution in a nutshell is “The paper measures the additional performance requirements for the improved trust in e-business trading that results from Byzantine fault tolerant, compared to Crash fault tolerance, at national and transnational scale.”

The authors argue that the main contribution is to quantify how much faster CFT is compared to BFT. But the specific quantitative measure is not given

From the paper, “Apache Kafka (CFT) is on average more than 298 times faster than Quorum, 437 times faster than Hyperledger Fabric and 654 times faster than Tendermint.” Further, Table 6 shows the performance of Quorum, Tendermint and Hyperledger Fabric, a fair and direct comparison.

My concern is what is the practical benefit or implication of this comparison?

An EU-wide e-business ecosystem, like the Blockchain-based Service Network in China, would have enormous practical benefit. Can this benefit be achieved using open source blockchain frameworks?  With respect to the use-case, Byzantine fault tolerant protocols, running on standardised cloud infrastructure, with realistic transaction volumes and sizes, are currently insufficient on an EU scale, and only just adequate to work at a national scale for a smaller member state.

Another argument of authors is that BFT protocols are not sufficiently performant on an EU scale. This sentence needs description of an alternative solution for EU scale.

Table 1 reports throughput and response time differently in each case, making it difficult to draw meaningful comparisons between competing solutions. The approach we adopt is intended to be a recipe for how emerging blockchain frameworks can be tested in a real and transparent way.

I would recommend more specific and summarized guidelines of blockchain designs for national scale and transnational scale.

It is not our ambition to be prescriptive about national and (trans)national e-business infrastructure requirements,  we leave that to regulators and standards bodies. We also don't claim that Blockchain will never scale to become a transnational e-business platform for the EU but specify what it will take for a framework to work at the EU scale. It is possible that improved consensus protocols (and other technical innovations such as compression algorithms, caching, and parallel processing) will result in a blockchain e-business solution scalable to the EU.

Comments on the Quality of English Language

This article includes grammatical errors. So, the proof-reading is required.

The paper has been re-written and proof-read to improve grammar, structure flow and readability. 

Submission Date

15 April 2023

Date of this review

20 Apr 2023 09:32:10

Author Response File: Author Response.pdf

Reviewer 2 Report

1. Novelties of the proposed method should explain elaborately in the introduction part.

2. The language of the paper is quite weak, and considerable amendments are required.

3. Introduction needs to explain the main contributions of the work more clearly.

4. It seems it is a novel work and where is the novelty? If it is not novel, where is the literature survey section?

5. I suggest the authors should add some tables to demonstrate the experimental results. And how about the time/space complexity of proposed algorithm?

6. The authors should give some future work in the part of conclusion.

7. The system model is inadequately explained. It is really difficult to adhere to. It should be entirely rebuilt. It is suggested that the authors define the notations used in the model first. It is not understandable in its current form. There are numerous errors in it.

8. The author should depict the flow graph to illustrate the need of the proposed approach.

9.  In the references in the Introduction section, the authors cite some works. However, they have not indicated the advantage or disadvantage and their relations to this paper. It's a little confusing.

  The language usage throughout this paper need to be improved, the author should do some proofreading on it. Give the article a rigorous language revision to get rid of a few complex sentences that hinder readability and eradicate typo errors.

Author Response

1. Novelties of the proposed method should explain elaborately in the introduction part.

Added the following to the Introduction.

“In-the-wild testing, as described by Zhuang et al.~\cite{6362954}, involves conducting experiments or evaluations in real-world settings, capturing realistic conditions, actual usage scenarios, and various environmental factors. By deploying and testing systems under authentic conditions, valuable insights are gained regarding their performance, behaviour, and interaction with the environment.

Utilizing an in-the-wild testing methodology, this research goes beyond traditional laboratory-based synthetic evaluations and examines how blockchain technologies perform under realistic and dynamic conditions. This approach provides a more accurate representation of the system's behaviour and its interaction with the environment, offering a comprehensive understanding of capabilities and limitations.

Therefore, the novelty of this work lies in the validation of the selected blockchain technologies' performance claims and in its pioneering use of an in-the-wild testing methodology for blockchains. This enhances the validity and relevance of the findings.”

2. The language of the paper is quite weak, and considerable amendments are required.

The paper has been re-written, the language improved, the presentation restructured and, in cases where sentences were overly complicated, they have been simplified.

3. Introduction needs to explain the main contributions of the work more clearly.

See the response to 1.

4. It seems it is a novel work and where is the novelty? If it is not novel, where is the literature survey section?

See response to 1.

5. I suggest the authors should add some tables to demonstrate the experimental results. And how about the time/space complexity of proposed algorithm?

 

A consolidated table with all results in now included on page 15, Table 6. This was requested by other reviewers and is a useful idea. It's important to note that analysing the time and space complexity of distributed systems is highly dependent on the specific architecture, algorithms, and design choices made within the system. In a practical sense we have measured space and time complexity in terms of performance, the size of the messages (2-MB or 10-MB) and the system throughput in transactions per second.  A theoretical treatment of space/time complexity is a useful tool in the analysis of various consensus algorithms (and their variants) but is not considered a suitable tool in the scope of this comparison.

6. The authors should give some future work in the part of conclusion

A future work section has been added on page 16.

7. The system model is inadequately explained. It is really difficult to adhere to. It should be entirely rebuilt. It is suggested that the authors define the notations used in the model first. It is not understandable in its current form. There are numerous errors in it.

We added the following text

To summarize, the test parameters encompass the following aspects in the model:

• number of transacting nodes: this refers to the count of nodes actively participating in the transactions in the system.
• size of individual transactions: denotes the data size or complexity of each transaction processed within the system.
• number of nodes participating in consensus/ordering: this parameter pertains to the quantity of nodes involved in the consensus or ordering process within the tested platforms.
• volume of transactions per unit time: this represents the rate at which transactions are generated and processed within the e-business system, typically measured as the number of transactions occurring within a specific time interval.

 

We think this provides a good overall high-level description of the test parameters. A lot of effort (3 pages worth) is spent in Section 3 trying to explain the parameters for each framework that map to these general parametric features. This is a little dry (Section 3.3.1 to 3.3.4) but these sections are necessary to declare so that the researchers that follow can reproduce our results.

 

8. The author should depict the flow graph to illustrate the need of the proposed approach.

Figure 2 addresses this comment, it shows an overview of the testing environment in both BFT (top) and CFT (bottom). The accompany text in Section 3.2 describes the test set-up.

9. In the references in the Introduction section, the authors cite some works. However, they have not indicated the advantage or disadvantage and their relations to this paper. It's a little confusing.

Table 1 informs our paper and was an important influence in its production. Our research origins was the need to get an overview of the performance blockchain frameworks and when these were examined, we discovered anomalies in the claims. In summary, performance tests were mostly artificial, in the sense that the topology of the network, its geographic distribution, message lengths and transaction volume, were not realistic in terms of how the system might be deployed.

 

Comments on the Quality of English Language

  The language usage throughout this paper need to be improved, the author should do some proofreading on it. Give the article a rigorous language revision to get rid of a few complex sentences that hinder readability and eradicate typo errors.

The language has been improved: awkward, long and overly complex sentences have been simplified and reworked.

Submission Date

15 April 2023

Date of this review

Author Response File: Author Response.pdf

Reviewer 3 Report

I am happy to provide a brief summary of the paper's main objectives and research questions.

The novelty and significance of the research are convincing as it,  advances the current state of the art in the field.

The writing style and organization of the paper, including the clarity and coherence of the arguments was well presented.

The methodology used in the paper was novel, including the data collection and analysis methods, and also I appreciate how well they address the research questions.

The strengths of the paper are the approach they have used and the weaknesses of the paper are still the authors failed to point it out the importance and the impact of this research in other domains so I assume that these limitations may have affected the results.

The authors need to provide suggestions for future improvements to the current study.

Overall, evaluation and the contribution of the paper to the field bring the potential impact of research

It is better to add some practical applications.

The writing style and organization of the paper, including the clarity and coherence of the arguments was well presented.

Author Response

I am happy to provide a summary of the paper's main objectives and research questions.

The novelty and significance of the research are convincing as it, advances the current state of the art in the field.

The writing style and organization of the paper, including the clarity and coherence of the arguments was well presented.

The methodology used in the paper was novel, including the data collection and analysis methods, and also I appreciate how well they address the research questions.

The strengths of the paper are the approach they have used and the weaknesses of the paper are still the authors failed to point it out the importance and the impact of this research in other domains so I assume that these limitations may have affected the results.

Our use-case aligns closely with the requirement of an e-business system at the EU scale. However, we added the comment in the future work section “the in-the-wild methodology we pursue maybe an inspiration for blockchain testing in other large-scale applications, such as IoT and Internet of vehicles (IoV), and supply chain management, where performance at scale, and trust, maybe limiting factors.”

The authors need to provide suggestions for future improvements to the current study.

A future work section has been added on page 16.

Overall, evaluation and the contribution of the paper to the field bring the potential impact of research

It is better to add some practical applications.

These are now mentioned as described above, namely “large-scale applications, such as IoT and Internet of vehicles (IoV), and supply chain management”.

 

Comments on the Quality of English Language

The writing style and organization of the paper, including the clarity and coherence of the arguments was well presented.

Thanks, but not everyone thought this, so we spent a lot of time re-working the order of the presentation and the quality of the language.

Submission Date

15 April 2023

Date of this review

 

 

Author Response File: Author Response.pdf

Reviewer 4 Report

This review evaluated the manuscript entitled "Crash vs Byzantine fault tolerance at scale: the cost of distributing trust in a (trans)national invoicing system". The study aims to investigate whether blockchain platforms can scale to support national and transnational e-commerce.

The authors present a study that is generally well structured and academically sound. However, there are a few areas of the manuscript that require improvement. First, the introduction does not follow the generally accepted outline, which may make it difficult for the reader to follow the argument. I suggest that the authors revise the introduction by providing a clear and concise overview of the research question and objectives, which are currently spread across a mixture of background, description of the technology, and outline of the paper.

Second, the title is somewhat misleading as it includes the word "cost" which is ambiguous as the paper is not a comparison of the technologies in terms of (monetary) cost.

Also, the paper contains numerous acronyms that make it difficult to read and understand. It is important to limit the use of acronyms and clearly explain each acronym when it is introduced, even if the nature of the research technology requires extensive use. Some of the acronyms have even been explained in the abstract. This will help readers better understand the approach and results of the study.

In addition, the authors did not adequately describe the methodology and summarize the results. The paper would benefit from a more detailed and comprehensive explanation of the methodology in the form of a workflow with the most important aspects. This would help to illustrate the work done and to understand the significance of the research. The results (e.g., Tables 3-5) could be presented more succinctly if a common table was created to show the key differences and thresholds at the national and global scale.

Overall, I find the research sound and the conclusions acceptable. However, we suggest that the authors address the above issues to improve the overall quality and readability of the manuscript.

Although the authors' use of English is generally good, some sentences could be improved for clarity and readability. Some words are misspelled and some sentences are awkward. I recommend that the authors revise the manuscript for grammatical errors and readability to ensure that the study is accessible to a wider audience.

Author Response

This review evaluated the manuscript entitled "Crash vs Byzantine fault tolerance at scale: the cost of distributing trust in a (trans)national invoicing system". The study aims to investigate whether blockchain platforms can scale to support national and transnational e-commerce.

The authors present a study that is generally well structured and academically sound. However, there are a few areas of the manuscript that require improvement. First, the introduction does not follow the generally accepted outline, which may make it difficult for the reader to follow the argument. I suggest that the authors revise the introduction by providing a clear and concise overview of the research question and objectives, which are currently spread across a mixture of background, description of the technology, and outline of the paper.

The introduction has been restructured and rewritten. The order of the presentation, its structure and language quality have been improved.

Second, the title is somewhat misleading as it includes the word "cost" which is ambiguous as the paper is not a comparison of the technologies in terms of (monetary) cost.

Good point! We agree and changed the title of the paper accordingly to “Improving trust in a (trans)national invoicing system: the performance of Crash vs Byzantine fault tolerance at scale” to eliminate any ambiguity in relation to monetary cost.

Also, the paper contains numerous acronyms that make it difficult to read and understand. It is important to limit the use of acronyms and clearly explain each acronym when it is introduced, even if the nature of the research technology requires extensive use. Some of the acronyms have even been explained in the abstract. This will help readers better understand the approach and results of the study.

We carefully re-worked the paper and tried to minimise the use of acronyms throughout. B2B and B2G still appear in the abstract but these terms should be well understood to Applied Science readers.

In addition, the authors did not adequately describe the methodology and summarize the results. The paper would benefit from a more detailed and comprehensive explanation of the methodology in the form of a workflow with the most important aspects. This would help to illustrate the work done and to understand the significance of the research. The results (e.g., Tables 3-5) could be presented more succinctly if a common table was created to show the key differences and thresholds at the national and global scale.

The authors feel that Figure 3 conveys succinctly the experimental workflow. In relation to a common result table, we agree and have provided exactly this, a combined result table is now presented in Table 6.

Overall, I find the research sound and the conclusions acceptable. However, we suggest that the authors address the above issues to improve the overall quality and readability of the manuscript.

Done, thank you.

 

Comments on the Quality of English Language

Although the authors' use of English is generally good, some sentences could be improved for clarity and readability. Some words are misspelled and some sentences are awkward. I recommend that the authors revise the manuscript for grammatical errors and readability to ensure that the study is accessible to a wider audience.

The language has been extensively re-worked, improved and checked. Where is has been overly technical, it has been simplified. The order of the presentation of the clarity of the findings have been improved.

Submission Date

15 April 2023

Date of this review

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

Most of my concerns are solved in the revised article.

So I recommend the publication of this article.