Empirical Performance Analysis of Hyperledger LTS for Small and Medium Enterprises
Abstract
:1. Introduction
- The impact of system configurations (total number of transactions, number of nodes) on the performance with respect to blockchain scalability.
- The performance analysis results as a practical reference for SMEs practitioners in selecting the Hyperledger Fabric LTS version for their business applications.
2. Background
2.1. Membership Service Provider (MSP)
2.2. Client
2.3. Peer
2.4. Channel
2.5. Orderer
2.5.1. Solo
2.5.2. Kafka
2.5.3. Raft
2.6. Chaincode
2.7. Transaction Flow in Hyperledger Fabric
2.7.1. Phase 1: Proposal (Execute)
- The transaction proposal is structured correctly.
- It is not duplicating an already existing transaction.
- The issuer’s signature is valid.
- The transaction issuer is permitted to execute the proposed operation.
2.7.2. Phase 2: Ordering and Packaging (Order)
2.7.3. Phase 3: Validation
- VSCC validation: A validation system chaincode is responsible for comparing the list of endorsements in the transactions with the endorsement policy (listed for the chaincode). If it is noted that the endorsing policy is not followed during the process, then that transaction is declared invalid.
- MVCC validation: MVCC is also known as a read–write dispute search because it guarantees that the versions of keys read or written during the execution process match the actual ledger state. This MVCC is applied sequentially to all transactions in a block. Transactions are flagged as invalid if the versions do not align.
3. Related Work
4. Methodology
4.1. Experiments
- Experiment 1: To evaluate the performance by having the workload as a variable. This includes the number of transactions and the simultaneous requests by the same number of nodes.
- Experiment 2: Evaluate the scalability by having the number of nodes as a variable. The threshold was set to 20 nodes, with the same/constant workload.
4.2. Application: Simulated Application and Smart Contracts
4.3. Hyperledger Fabric Deployment/Blockchain Platforms
4.4. Hyperledger Benchmark Tool
4.5. Evaluation Metrics:
- Throughput is described as the number of successful transactions per second.
- Average latency is specified as the average time interval between the initialization of the transaction and the actual execution of the transaction.
- The success rate of a blockchain is determined by the number of successful transactions performed out of the total transactions.
5. Results and Discussion
5.1. Performance Assessment
- Success rate: The experiment evaluation reveals that both the (open and query) functions attained a 100% success on 50–1000 simultaneous transactions on same number of nodes.
- Throughput: Figure 4 illustrates the transaction throughput after executing the open and query functions using 50 to 1000 simultaneous transactions. The blue bars show the open function, and brown bars highlight the query function. It is observed that the throughput on the query function is slightly higher than the open function. Initially, the throughputs on both functions are almost equal. As the number of transactions increases up to 600, there is slight growth observed in the throughput in the query function, whereas the open function shows consistency in the throughput. The continuous consistency observed in the throughput reflects the reliability and availability of Hyperledger. In the following figure, the X-axis represents the number of transactions, and the Y-axis represents the throughput.
- Average latency: Figure 5 shows the average latency after executing the open and the query functions, using 50 to 1000 simultaneous transactions. The blue bars show the open function, and brown bars highlight the query function. It is noticed that there is continuous growth in the average latency as the number of transactions are increasing for both the query and the open functions. However, the query function continuously has more latency than the open function, and it achieves more growth after 600 transactions. However, the average latency of the query function is higher than the open function. In the following figure, the X-axis represents the number of transactions, and the Y-axis represents the seconds.
5.2. Scalability Assessment
- Success rate: The experimental analysis reveals that both the open and query functions attended 100% success up to 20 nodes, with respect to 500 and 1000 transactions.
- Throughput: Figure 6 and Figure 7 demonstrate the throughput for executing the open and the query functions up to 20 nodes, based on 500 and 1000 transactions respectively. The blue bars show the open function, and brown bars highlight the query function. In Figure 6, it can be observed that the query function obtains higher throughput than the open function. However, the throughput of the query function decreases with the increasing number of nodes, and it also slightly decreases in the open function. In Figure 7, on 1000 transactions, the throughout in the query function is higher than the open function, but there is a continuous increase in throughput on query function as the number of nodes increases. Although the open function obtains lower throughput, it shows consistency. In the following figures, the X-axis represents the number of nodes, and the Y-axis represents the throughput.
- Average latency: Figure 8 and Figure 9 demonstrate the average latency on executing the open and query functions on the 20 nodes, using 500 and 1000 transactions, respectively. The blue bars show the open function, and the brown bars highlight the query function. In Figure 8, overall, the query function obtains lower latency than the open function, and as the number of nodes increases, the latency on both of the functions remains consistent. However, there is slight growth observed after the nodes are more than 10, whereas in Figure 9, initially the latency of the open function is quite high but as the number of nodes increases, the latency decreases, and the query function has more latency than the open function. However, after crossing the 8 nodes, there is a bit of consistency noticed in the query function. Overall, there is consistency noticed in the latency on the open function. In the following figure, the X-axis represents the number of nodes and the Y-axis represents the time in seconds.
6. Conclusions
7. Future Work
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Nakamoto, S. Bitcoin: A peer-to-peer electronic cash system. Decentralized Bus. Rev. 2008, 21260. [Google Scholar] [CrossRef]
- Dorri, A.; Kanhere, S.S.; Jurdak, R.; Gauravaram, P. Blockchain for IoT security and privacy: The case study of a smart home. In Proceedings of the 2017 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), Kona, HI, USA, 13–17 March 2017; pp. 618–623. [Google Scholar]
- Biswas, K.; Muthukkumarasamy, V. Securing smart cities using blockchain technology. In Proceedings of the 2016 IEEE 18th International Conference on High Performance Computing and Communications and IEEE 14th International Conference on Smart City, IEEE 2nd International Conference on Data Science and Systems (HPCC/SmartCity/DSS), Sydney, Australia, 12–14 December 2016; pp. 1392–1393. [Google Scholar]
- Yue, X.; Wang, H.; Jin, D.; Li, M.; Jiang, W. Healthcare data gateways: Found healthcare intelligence on blockchain with novel privacy risk control. J. Med. Syst. 2016, 40, 1–8. [Google Scholar] [CrossRef] [PubMed]
- Ekblaw, A.; Azaria, A.; Halamka, J.D.; Lippman, A. A Case Study for Blockchain in Healthcare:“MedRec” prototype for electronic health records and medical research data. In Proceedings of the IEEE Open & Big Data Conference, Washington, DC, USA, 5–8 December 2016; p. 13. [Google Scholar]
- Fatokun, T.; Nag, A.; Sharma, S. Towards a blockchain assisted patient owned system for electronic health records. Electronics 2021, 10, 580. [Google Scholar] [CrossRef]
- Christidis, K.; Devetsikiotis, M. Blockchains and smart contracts for the internet of things. IEEE Access 2016, 4, 2292–2303. [Google Scholar] [CrossRef]
- Dabbagh, M.; Kakavand, M.; Tahir, M. Towards integration of blockchain and IoT: A bibliometric analysis of state-of-the-Art. In Proceedings of the International Congress on Blockchain and Applications, Ávila, Italy, 25–28 June 2019; pp. 27–35. [Google Scholar]
- Tahir, M.; Ashraf, Q.M.; Dabbagh, M. Towards enabling autonomic computing in IoT ecosystem. In Proceedings of the IEEE Intl Conf on Dependable, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence and Computing, Intl Conf on Cloud and Big Data Computing, Intl Conf on Cyber Science and Technology Congress (DASC/PiCom/CBDCom/CyberSciTech), Fukuoka, Japan, 5–8 August 2019; pp. 646–651. [Google Scholar]
- Saberi, S.; Kouhizadeh, M.; Sarkis, J.; Shen, L. Blockchain technology and its relationships to sustainable supply chain management. Int. J. Prod. Res. 2019, 57, 2117–2135. [Google Scholar] [CrossRef] [Green Version]
- Tahir, M.; Habaebi, M.H.; Dabbagh, M.; Mughees, A.; Ahad, A.; Ahmed, K.I. A review on application of blockchain in 5G and beyond networks: Taxonomy, field-trials, challenges and opportunities. IEEE Access 2020, 8, 115876–115904. [Google Scholar] [CrossRef]
- Shalaby, S.; Abdellatif, A.A.; Al-Ali, A.; Mohamed, A.; Erbad, A.; Guizani, M. Performance evaluation of hyperledger fabric. In Proceedings of the 2020 IEEE International Conference on Informatics, IoT, and Enabling Technologies (ICIoT), Doha, Qatar, 2–5 February 2020; pp. 608–613. [Google Scholar]
- Thakkar, P.; Nathan, S.; Viswanathan, B. Performance benchmarking and optimizing hyperledger fabric blockchain platform. In Proceedings of the 2018 IEEE 26th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS), Milwaukee, WI, USA, 25–28 September 2018; pp. 264–276. [Google Scholar]
- Kuzlu, M.; Pipattanasomporn, M.; Gurses, L.; Rahman, S. Performance analysis of a hyperledger fabric blockchain framework: Throughput, latency and scalability. In Proceedings of the 2019 IEEE international conference on blockchain (Blockchain), Seoul, Korea, 14–17 May 2019; pp. 536–540. [Google Scholar]
- Khan, D.; Jung, L.T.; Hashmani, M.A.; Waqas, A. A Critical Review of Blockchain Consensus Model. In Proceedings of the 2020 3rd International Conference on Computing, Mathematics and Engineering Technologies (iCoMET), Sindh, Pakistan, 29–30 January 2020; pp. 1–6. [Google Scholar]
- Javaid, H.; Hu, C.; Brebner, G. Optimizing validation phase of hyperledger fabric. In Proceedings of the 2019 IEEE 27th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS), Rennes, France, 22–24 October 2019; pp. 269–275. [Google Scholar]
- Buterin, V. Ethereum white paper. GitHub Repos. 2013, 1, 22–23. [Google Scholar]
- Androulaki, E.; Barger, A.; Bortnikov, V.; Cachin, C.; Christidis, K.; De Caro, A.; Enyeart, D.; Ferris, C.; Laventman, G.; Manevich, Y. Hyperledger fabric: A distributed operating system for permissioned blockchains. In Proceedings of the Thirteenth EuroSys Conference, Porto, Portugal, 23–26 April 2018; pp. 1–15. [Google Scholar]
- Razdan, S.; Sharma, S. Internet of Medical Things (IoMT): Overview, Emerging Technologies, and Case Studies. IETE Tech. Rev. 2021, 1–14. [Google Scholar] [CrossRef]
- Junejo, A.Z.; Hashmani, M.A.; Memon, M.M. Empirical Evaluation of Privacy Efficiency in Blockchain Networks: Review and Open Challenges. Appl. Sci. 2021, 11, 7013. [Google Scholar] [CrossRef]
- Dabbagh, M.; Kakavand, M.; Tahir, M.; Amphawan, A. Performance Analysis of Blockchain Platforms: Empirical Evaluation of Hyperledger Fabric and Ethereum. In Proceedings of the 2020 IEEE 2nd International Conference on Artificial Intelligence in Engineering and Technology (IICAIET), Sabah, Malaysia, 26–27 September 2020; pp. 1–6. [Google Scholar]
- Pilkington, M. Blockchain technology: Principles and applications. In Research Handbook on Digital Transformations; Edward Elgar Publishing: Cheltenham, UK, 2016. [Google Scholar]
- Khan, D.; Jung, L.T.; Hashmani, M.A. Systematic Literature Review of Challenges in Blockchain Scalability. Appl. Sci. 2021, 11, 9372. [Google Scholar] [CrossRef]
- Yli-Huumo, J.; Ko, D.; Choi, S.; Park, S.; Smolander, K. Where is current research on blockchain technology?—A systematic review. PLoS ONE 2016, 11, e0163477. [Google Scholar]
- Caliper, H. Hyperledger Caliper Architecture. Electronic Article. Available online: https://hyperledger.github.io/caliper/docs/2_Architecture.html (accessed on 3 October 2019).
- Caliper, H. Available online: https://www.hyperledger.org/projects/caliper (accessed on 15 January 2019).
- Dinh, T.T.A.; Wang, J.; Chen, G.; Liu, R.; Ooi, B.C.; Tan, K.-L. Blockbench: A framework for analyzing private blockchains. In Proceedings of the 2017 ACM International Conference on Management of Data, Singapore, 6–10 November 2017; pp. 1085–1100. [Google Scholar]
- Jiang, L.; Chang, X.; Liu, Y.; Mišić, J.; Mišić, V.B. Performance analysis of Hyperledger Fabric platform: A hierarchical model approach. Peer-Peer Netw. Appl. 2020, 13, 1014–1025. [Google Scholar] [CrossRef]
- Baliga, A.; Subhod, I.; Kamat, P.; Chatterjee, S. Performance evaluation of the quorum blockchain platform. arXiv 2018, arXiv:1809.03421. [Google Scholar]
- Wang, C.; Chu, X. Performance characterization and bottleneck analysis of hyperledger fabric. In Proceedings of the 2020 IEEE 40th International Conference on Distributed Computing Systems (ICDCS), Singapore, 8–10 July 2020; pp. 1281–1286. [Google Scholar]
- Sousa, J.; Bessani, A.; Vukolic, M. A byzantine fault-tolerant ordering service for the hyperledger fabric blockchain platform. In Proceedings of the 2018 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN), Baltimore, ML, USA, 25–28 June 2018; pp. 51–58. [Google Scholar]
- Sukhwani, H.; Wang, N.; Trivedi, K.S.; Rindos, A. Performance modeling of hyperledger fabric (permissioned blockchain network). In Proceedings of the 2018 IEEE 17th International Symposium on Network Computing and Applications (NCA), Cambridge, MA, USA, 1–3 November 2018; pp. 1–8. [Google Scholar]
- Navroop, S.; Nathalie, S.; Alexandra, G.; Robert, S.; Arianna, G. Blockchain for Business-An Introduction to Hyperledger Technologies. Available online: https://courses.edx.org/courses/course-v1:LinuxFoundationX+LFS171x+3T2017/course/ (accessed on 4 June 2018).
- Pongnumkul, S.; Siripanpornchana, C.; Thajchayapong, S. Performance analysis of private blockchain platforms in varying workloads. In Proceedings of the 2017 26th International Conference on Computer Communication and Networks (ICCCN), Vancouver, BC, Canada, 31 July–3 August 2017; pp. 1–6. [Google Scholar]
- Xu, X.; Sun, G.; Luo, L.; Cao, H.; Yu, H.; Vasilakos, A.V. Latency performance modeling and analysis for hyperledger fabric blockchain network. Inf. Process. Manag. 2021, 58, 102436. [Google Scholar] [CrossRef]
- Munoz, D.-J.; Constantinescu, D.-A.; Asenjo, R.; Fuentes, L. Clinicappchain: A low-cost blockchain hyperledger solution for healthcare. In Proceedings of International Congress on Blockchain and Applications; Springer: Berlin/Heidelberg, Germany, 2016; pp. 36–44. [Google Scholar]
- Nasir, Q.; Qasse, I.A.; Abu Talib, M.; Nassif, A.B. Performance analysis of hyperledger fabric platforms. Secur. Commun. Netw. 2018, 2018, 3976093. [Google Scholar] [CrossRef] [Green Version]
- Sukhwani, H.; Martínez, J.M.; Chang, X.; Trivedi, K.S.; Rindos, A. Performance modeling of PBFT consensus process for permissioned blockchain network (hyperledger fabric). In Proceedings of the 2017 IEEE 36th Symposium on Reliable Distributed Systems (SRDS), Hong Kong, China, 26–29 September 2017; pp. 253–255. [Google Scholar]
- Hao, Y.; Li, Y.; Dong, X.; Fang, L.; Chen, P. Performance analysis of consensus algorithm in private blockchain. In Proceedings of the 2018 IEEE Intelligent Vehicles Symposium (IV), Suzhou, China, 30 June–1 July 2018; pp. 280–285. [Google Scholar]
Article | Year | Cite | Title | Comments |
---|---|---|---|---|
[35] | 2021 | 44 | Latency performance modeling and analysis for Hyperledger Fabric blockchain network. | Focused on latency of Hyperledger Fabric. It also proposed a new framework to measure the latency. |
[28] | 2020 | 12 | Performance analysis of Hyperledger Fabric platform: A hierarchical model approach. | Concentrated on two important factors: ignored block timeout and transaction endorsement failure. It also introduced a hierarchical model for the transaction mechanism in Hyperledger Fabric v1.4. |
[29] | 2018 | 38 | Performance evaluation of the quorum blockchain platform. | Analyzed Quorum’s performance analysis. The throughput and latency of various workloads and consensus algorithms were taken into consideration. |
[30] | 2020 | 5 | Performance characterization and bottleneck analysis of Hyperledger Fabric. | A thorough performance assessment of Hyperledger Fabric in accordance with the new architecture. Each process was assessed with respect to the execute, request, and validate phases. |
[27] | 2017 | 644 | Blockbench: A framework for analyzing private blockchains. | Evaluated performance of 3 major platforms (Ethereum, Parity, and Hyperledger Fabric). Results showed that none of them came close in displaying performance comparable to the existing database systems. |
[7] | 2016 | 3203 | Blockchains and smart contracts for the internet of things. | Evaluated the HLF by applying and benchmarking a digital currency HPL to produce a higher throughput in some common implementation setups with sub-second latency. |
[38] | 2017 | 245 | Performance modeling of PBFT consensus process for permissioned blockchain network (Hyperledger Fabric). | Explored the impact of a consensus mechanism based on PBFT on peer evaluation performance with a wide number of peers when running an IoT system. |
[39] | 2018 | 70 | Performance analysis of consensus algorithm in private blockchain. | Investigated the impact of consensus protocol in HLF performance evaluation. Proposed a novel method to evaluate the performance of consensus algorithms in the permissioned blockchain. |
[12] | 2020 | 12 | Performance evaluation of Hyperledger Fabric. | Investigated the possibilities of customizing the blockchain networks for the needs of the applications. |
This study | 2021 | - | Empirical performance analysis of Hyperledger LTS for small and medium enterprises. | Performed the analysis of Hyperledger LTS version, focusing on the real-world implementation of blockchain (HLF). Considered 3 critical metrics (success and fail rate, throughout, and latency) for SMEs businesses to select HLF. To serve as reference for SMEs to select suitable blockchain platform for their respective business in considering the scale up demands in coming years which are missing in all above articles. |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Khan, D.; Jung, L.T.; Hashmani, M.A.; Cheong, M.K. Empirical Performance Analysis of Hyperledger LTS for Small and Medium Enterprises. Sensors 2022, 22, 915. https://doi.org/10.3390/s22030915
Khan D, Jung LT, Hashmani MA, Cheong MK. Empirical Performance Analysis of Hyperledger LTS for Small and Medium Enterprises. Sensors. 2022; 22(3):915. https://doi.org/10.3390/s22030915
Chicago/Turabian StyleKhan, Dodo, Low Tang Jung, Manzoor Ahmed Hashmani, and Moke Kwai Cheong. 2022. "Empirical Performance Analysis of Hyperledger LTS for Small and Medium Enterprises" Sensors 22, no. 3: 915. https://doi.org/10.3390/s22030915
APA StyleKhan, D., Jung, L. T., Hashmani, M. A., & Cheong, M. K. (2022). Empirical Performance Analysis of Hyperledger LTS for Small and Medium Enterprises. Sensors, 22(3), 915. https://doi.org/10.3390/s22030915