Dynamic-EC: an efficient dynamic erasure coding method for permissioned blockchain systems

Abstract

Blockchain as a decentralized storage technology is widely used in many fields. It has extremely strict requirements for reliability because there are many potentially malicious nodes. Generally, blockchain is a chain storage structure formed by interconnecting blocks^†

A block is consist of a block header and a block body. The metadata is stored in block header and data is stored in block body.

, which are stored by full replication method, where each node stores a replica of all blocks and the data consistency is maintained by the consensus protocol. To decrease the storage overhead, previous approaches such as BFT-Store and Partition Chain store blocks via erasure codes. However, existing erasure coding based methods utilize static encoding schema to tolerant f malicious nodes, but in the typical cases, the number of malicious nodes is much smaller than f as described in previous literatures. Using redundant parities to tolerate excessive malicious nodes introduces unnecessary storage overhead.

To solve the above problem, we propose Dynamic-EC, which is a Dynamic Erasure Coding method in permissioned blockchain systems. The key idea of Dynamic-EC is to reduce the storage overhead by dynamically adjusting the total number of parities according to the risk level of the whole system, which is determined by the number of perceived malicious nodes, while ensuring the system reliability. To demonstrate the effectiveness of Dynamic-EC, we conduct several experiments on an open source blockchain software Tendermint. The results show that, compared to the state-of-the-art erasure coding methods, Dynamic-EC reduces the storage overhead by up to 42%, and decreases the average write latency of blocks by up to 25%, respectively.

Graphical abstract

Keywords

blockchain / Byzantine Fault Tolerance (BFT) / erasure coding / consensus / reputation evaluation

Mizhipeng Zhang received the BE degree in computer science from Shanghai Jiao Tong University, China in 2021, where he is currently a candidate for the master degree in computer science, working in the field of blockchain and storage systems

Chentao Wu received the BS, ME, and PhD degrees in computer science from Huazhong University of Science and Technology, China in 2004, 2006, and 2010, respectively, and the PhD degree in electrical and computer engineering from Virginia Commonwealth University, USA in 2012. He is currently a professor with the Department of Computer Science and Engineering, Shanghai Jiao Tong University, China. His research interests include computer architecture and data storage systems

Jie Li received the BE degree from Zhejiang University, China in 1982, the ME degree from China Academy of Posts and Telecommunications, China in 1985, and the Dr.Eng. degree from The University of Electro-Communications, Japan in 1993. He is a professor with the Department of Computer Science and Engineering, Shanghai Jiao Tong University, China. His current research interests are in big data, IoT, blockchain, edge computing, OS, and modeling and performance evaluation of information systems

Minyi Guo received his PhD degree in computer science from the University of Tsukuba, Japan. He is currently Zhiyuan Chair Professor, Shanghai Jiao Tong University, China. His research interests include parallel/distributed computing, compiler optimizations, cloud computing and big data. He is now on the editorial board of IEEE Transactions on Parallel and Distributed Systems and Journal of Parallel and Distributed Computing. Dr. Guo is IEEE fellow and CCF fellow

References

[1] Wood G. Ethereum: a secure decentralised generalised transaction ledger. Ethereum Project Yellow Paper, 2014, 151(2014): 1

[2] Armknecht F, Karame G O, Mandal A, Youssef F, Zenner E. Ripple: overview and outlook. In: Proceedings of the 8th International Conference on Trust and Trustworthy Computing. 2015, 163−180

[3] Qi X, Zhang Z, Jin C, Zhou A. BFT-Store: storage partition for permissioned blockchain via erasure coding. In: Proceedings of the 36th IEEE International Conference on Data Engineering (ICDE). 2020, 1926−1929

[4] Barborak M, Dahbura A, Malek M . The consensus problem in fault-tolerant computing. ACM Computing Surveys, 1993, 25( 2): 171–220

[5] Zheng Z, Xie S, Dai H, Chen X, Wang H. An overview of blockchain technology: architecture, consensus, and future trends. In: Proceedings of 2017 IEEE International Congress on Big Data (BigData Congress). 2017, 557−564

[6] Castro M, Liskov B. Practical byzantine fault tolerance. In: Proceedings of the 3rd Symposium on Operating Systems Design and Implementation. 1999, 173−186

[7] Du Z, Pang X, Qian H . PartitionChain: a scalable and reliable data storage strategy for permissioned blockchain. IEEE Transactions on Knowledge and Data Engineering, 2023, 35( 4): 4124–4136

[8] Reed I S, Solomon G . Polynomial codes over certain finite fields. Journal of the Society for Industrial and Applied Mathematics, 1960, 8( 2): 300–304

[9] Alvisi L, Malkhi D, Pierce E, Reiter M K, Wright R N. Dynamic Byzantine quorum systems. In: Proceedings of Proceeding International Conference on Dependable Systems and Networks. 2000, 283−292

[10] Xia M, Saxena M, Blaum M, Pease D A. A tale of two erasure codes in HDFS. In: Proceedings of the 13th USENIX Conference on File and Storage Technologies. 2015, 213−226

[11] Qiu H, Wu C, Li J, Guo M, Liu T, He X, Dong Y, Zhao Y. EC-fusion: An efficient hybrid erasure coding framework to improve both application and recovery performance in cloud storage systems. In: Proceedings of 2020 IEEE International Parallel and Distributed Processing Symposium (IPDPS). 2020, 191−201

[12] Stathakopoulou C, Rüsch S, Brandenburger M, Vukolić M. Adding fairness to order: preventing front-running attacks in BFT protocols using TEEs. In: Proceedings of the 40th International Symposium on Reliable Distributed Systems (SRDS). 2021, 34−45

[13] Guo B, Lu Z, Tang Q, Xu J, Zhang Z. Dumbo: faster asynchronous BFT protocols. In: Proceedings of 2020 ACM SIGSAC Conference on Computer and Communications Security. 2020, 803−818

[14] Miller A, Xia Y, Croman K, Shi E, Song D. The honey badger of BFT protocols. In: Proceedings of 2016 ACM SIGSAC Conference on Computer and Communications Security. 2016, 31−42

[15] Duan S, Reiter M K, Zhang H. BEAT: asynchronous BFT made practical. In: Proceedings of 2018 ACM SIGSAC Conference on Computer and Communications Security. 2018, 2028−2041

[16] Androulaki E, Barger A, Bortnikov V, Cachin C, Christidis K, De Caro A, Enyeart D, Ferris C, Laventman G, Manevich Y, Muralidharan S, Murthy C, Nguyen B, Sethi M, Singh G, Smith K, Sorniotti A, Stathakopoulou C, Vukolić M, Cocco S W, Yellick J. Hyperledger fabric: a distributed operating system for permissioned blockchains. In: Proceedings of the 13th EuroSys Conference. 2018, 30

[17] Buchman E. Tendermint: Byzantine fault tolerance in the age of blockchains. University of Guelph, Dissertation, 2016

[18] Ghemawat S, Gobioff H, Leung S T. The Google file system. In: Proceedings of the 19th ACM Symposium on Operating Systems Principles. 2003, 29−43

[19] Shvachko K, Kuang H, Radia S, Chansler R. The hadoop distributed file system. In: Proceedings of the 26th IEEE Symposium on Mass Storage Systems and Technologies (MSST). 2010, 1−10

[20] Dai X, Xiao J, Yang W, Wang C, Jin H. Jidar: a jigsaw-like data reduction approach without trust assumptions for bitcoin system. In: Proceedings of the 39th IEEE International Conference on Distributed Computing Systems (ICDCS). 2019, 1317−1326

[21] Sukhwani H, Wang N, Trivedi K S, Rindos A. Performance modeling of hyperledger fabric (permissioned blockchain network). In: Proceedings of the 17th IEEE International Symposium on Network Computing and Applications (NCA). 2018, 1−8

[22] Weatherspoon H, Kubiatowicz J D. Erasure coding vs. replication: a quantitative comparison. In: Proceedings of the 1st International Workshop on Peer-to-Peer Systems. 2002, 328−337

[23] Yang G, Xue H, Gu Y, Wu C, Li J, Guo M, Li S, Xie X, Dong Y, Zhao Y. XHR-code: an efficient wide stripe erasure code to reduce cross-rack overhead in cloud storage systems. In: Proceedings of the 41st International Symposium on Reliable Distributed Systems (SRDS). 2022, 273−283

[24] Hu Y, Cheng L, Yao Q, Lee P P C, Wang W, Chen W. Exploiting combined locality for Wide-Stripe erasure coding in distributed storage. In: Proceedings of the 19th USENIX Conference on File and Storage Technologies. 2021, 233−248

[25] Ye L, Feng D, Hu Y, Liu Q. Hybrid-RC: Flexible erasure codes with optimized recovery performance and low storage overhead. In: Proceedings of the 36th IEEE Symposium on Reliable Distributed Systems (SRDS). 2017, 124–133

[26] Wu C, Wan S, He X, Cao Q, Xie C. H-Code: a hybrid MDS array code to optimize partial stripe writes in RAID-6. In: Proceedings of 2011 IEEE International Parallel & Distributed Processing Symposium. 2011, 782−793

[27] Kong L, Manohar D J, Subbiah A, Sun M, Ahamad M, Blough D M. Agile store: Experience with quorum-based data replication techniques for adaptive Byzantine fault tolerance. In: Proceedings of the 24th IEEE Symposium on Reliable Distributed Systems (SRDS’05). 2005, 143−154

[28] Bahsoun J P, Guerraoui R, Shoker A. Making BFT protocols really adaptive. In: Proceedings of 2015 IEEE International Parallel and Distributed Processing Symposium. 2015, 904−913

[29] Silva D S, Graczyk R, Decouchant J, Völp M, Esteves-Verissimo P. Threat adaptive byzantine fault tolerant state-machine replication. In: Proceedings of the 40th International Symposium on Reliable Distributed Systems (SRDS). 2021, 78−87

[30] Shostack A. Threat Modeling: Designing for Security. Indianapolis: John Wiley & Sons, 2014

[31] Zhang J, Huang Y, Ye F, Yang Y. A novel proof-of-reputation consensus for storage allocation in edge blockchain systems. In: Proceedings of the 29th IEEE/ACM International Symposium on Quality of Service (IWQOS). 2021, 1−10

[32] Kamvar S D, Schlosser M T, Garcia-Molina H. The eigentrust algorithm for reputation management in P2P networks. In: Proceedings of the 12th International Conference on World Wide Web. 2003, 640−651

[33] Naehrig M, Niederhagen R, Schwabe P. New software speed records for cryptographic pairings. In: Proceedings of the 1st International Conference on Cryptology and Information Security in Latin America. 2010, 109−123

[34] Hossain M A, Islam M K, Das S K. Cryptanalyzing of message digest algorithms MD4 and MD5. International Journal on Cryptography and Information Security, 2012, 2 (1): 1-13.

[35] Luu L, Narayanan V, Zheng C, Baweja K, Gilbert S, Saxena P. A secure sharding protocol for open blockchains. In: Proceedings of 2016 ACM SIGSAC Conference on Computer and Communications Security. 2016, 17−30

[36] Yang J, Yue Y, Rashmi K V . A large-scale analysis of hundreds of in-memory key-value cache clusters at twitter. ACM Transactions on Storage, 2021, 17( 3): 17

[37] Egala B S, Pradhan A K, Badarla V, Mohanty S P . Fortified-chain: a blockchain-based framework for security and privacy-assured internet of medical things with effective access control. IEEE Internet of Things Journal, 2021, 8( 14): 11717–11731

[38] El Houda Z A, Hafid A, Khoukhi L. Co-IoT: a collaborative DDoS mitigation scheme in IoT environment based on blockchain using SDN. In: Proceedings of 2019 IEEE Global Communications Conference (GLOBECOM). 2019, 1−6

Acknowledgements

We thank anonymous reviewers for their insightful comments. This work was partially sponsored by the National Key R&D Program of China (No. 2020YFB1806700), and the National Natural Science Foundation of China (Grant Nos. 61972246, 61932014).

Competing interests

The authors declare that they have no competing interests or financial conflicts to disclose.

RIGHTS & PERMISSIONS

2025 Higher Education Press