Resource abstraction and data placement for distributed hybrid memory pool

Abstract

Emerging byte-addressable non-volatile memory (NVM) technologies offer higher density and lower cost than DRAM, at the expense of lower performance and limited write endurance. There have been many studies on hybrid NVM/DRAMmemory management in a single physical server. However, it is still an open problem on how to manage hybrid memories efficiently in a distributed environment. This paper proposes Alloy, a memory resource abstraction and data placement strategy for an RDMA-enabled distributed hybrid memory pool (DHMP). Alloy provides simple APIs for applications to utilize DRAM or NVM resource in the DHMP, without being aware of the hardware details of the DHMP. We propose a hotness-aware data placement scheme, which combines hot data migration, data replication and write merging together to improve application performance and reduce the cost of DRAM. We evaluate Alloy with several micro-benchmark workloads and public benchmark workloads. Experimental results show that Alloy can significantly reduce the DRAM usage in the DHMP by up to 95%, while reducing the total memory access time by up to 57% compared with the state-of-the-art approaches.

Keywords

load balance / distributed hybrid memory / clouds

References

[1] Kapoor R, Porter G, Tewari M, Voelker G M, Vahdat A. Chronos: predictable low latency for data center applications. In: Proceedings of the 3rd ACM Symposium on Cloud Computing. 2012, 1–14

[2] Ousterhout J K, Agrawal P, Erickson D, Kozyrakis C, Leverich J, Mazières D, Mitra S, Narayanan A, Parulkar G M, Rosenblum M, Rumble S M, Stratmann E, Stutsman R. The case for ramclouds: scalable high-performance storage entirely in DRAM. Operating Systems Review, 2009, 43(4): 92–105

[3] Xiao Y, Nazarian S, Bogdan P. Prometheus: processing-in-memory heterogeneous architecture design from a multi-layer network theoretic strategy. In: Proceedings of the 2018 Design, Automation & Test in Europe Conference & Exhibition. 2018, 1387–1392

[4] Wu C, Zhang G, Li K. Rethinking computer architectures and software systems for phase-change memory. ACM Journal on Emerging Technologies in Computing Systems, 2016, 12(4): 1–40

[5] Peng I B, Gioiosa R, Kestor G, Vetter J S, Cicotti P, Laure E, Markidis S. Characterizing the performance benefit of hybrid memory system for HPC applications. Parallel Computing, 2018, 76: 57–69

[6] Peng I B, Gioiosa R, Kestor G, Cicotti P, Laure E, Markidis S. RTHMS: a tool for data placement on hybrid memory system. In: Proceedings of the 2017 ACM SIGPLAN International Symposium onMemory Management. 2017, 82–91

[7] Wu K, Huang Y, Li D. Unimem: runtime data management on nonvolatile memory-based heterogeneous main memory. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. 2017, 1–14

[8] Lee SM, Yoon S K, Kim J G, Kim S D. Adaptive correlated prefetch with large-scale hybrid memory system for stream processing. The Journal of Supercomputing, 2018, 74(9): 4746–4770

[9] Li Y, Ghose S, Choi J, Sun J, Wang H, Mutlu O. Utility-based hybrid memory management. In: Proceedings of the 2017 IEEE International Conference on Cluster Computing. 2017, 152–165

[10] Xie Y, Feng D, Wang F, Zhang L. Non-sequential striping encoder from replication to erasure coding for distributed storage system. Frontiers of Computer Science, 2019, 13(6): 1356–1358

[11] Gutierrez-Garcia J O, Ramirez-Nafarrate A. Collaborative agents for distributed load management in cloud data centers using live migration of virtual machines. IEEE Transactions on Services Computing, 2015, 8(6): 916–929

[12] Desai M R, Patel H B. Efficient virtual machine migration in cloud computing. In: Proceedings of the 15th International Conference on Communication Systems and Network Technologies. 2015, 1015–1019

[13] Yin J, Wang J, Zhou J, Lukasiewicz T, Huang D, Zhang J. Opass: analysis and optimization of parallel data access on distributed file systems. In: Proceedings of the 2015 IEEE International Parallel and Distributed Processing Symposium. 2015, 623–632

[14] Kalia A, Kaminsky M, Andersen D G. FaSST: fast, scalable and simple distributed transactions with two-sided (RDMA) datagram RPCs. In: Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation. 2016, 185–201

[15] Duan Z, Liu H, Liao X, Jin H. HME: a lightweight emulator for hybrid memory. In: Proceedings of the 2018 Design, Automation & Test in Europe Conference & Exhibition. 2018, 1375–1380

[16] Dragojević A, Narayanan D, Castro M, Hodson O. FaRM: fast remote memory. In: Proceedings of the 11th USENIX Symposium on Networked Systems Design and Implementation. 2014, 401–414

[17] Lu Y, Shu J, Chen Y, Li T. Octopus: an RDMA-enabled distributed persistent memory file system. In: Proceedings of the 2017 USENIX Annual Technical Conference. 2017, 773–785

[18] Stuedi P, Trivedi A, Pfefferle J, Klimovic A, Schüpbach A, Metzler B. Unification of temporary storage in the nodekernel architecture. In: Proceedings of the 2019 USENIX Annual Technical Conference. 2019, 767–782

[19] Nelson J, Holt B, Myers B, Briggs P, Ceze L, Kahan S, Oskin M. Latencytolerant software distributed shared memory. In: Proceedings of the 2015 USENIX Annual Technical Conference. 2015, 291–305

[20] Shan Y, Tsai S Y, Zhang Y. Distributed shared persistent memory. In: Proceedings of the 2017 Symposium on Cloud Computing. 2017, 323–337

[21] Vavilapalli V K, Murthy A C, Douglas C, Agarwal S, Konar M, Evans R, Graves T, Lowe J, Shah H, Seth S, Saha B, Curino C, O’Malley O, Radia S, Reed B, Baldeschwieler E. Apache hadoop YARN: yet another resource negotiator. In: Proceedings of the 4th ACM Symposium on Cloud Computing. 2013, 1–16

[22] Ghemawat S, Gobioff H, Leung S T. The google file system. In: Proceedings of the 19th ACM Symposium on Operating Systems Principles. 2003, 29–43

[23] Munetomo M, Takai Y, Sato Y. A genetic approach to dynamic load balancing in a distributed computing system. In: Proceedings of the 1st IEEE Conference on Evolutionary Computation, IEEE World Congress on Computational Intelligence. 1994, 419–421

[24] Peng X, Liu X, Cao H, Zhang Z. An efficient energy aware virtual network migration based on genetic algorithm. Frontiers of Computer Science, 2019, 13(2): 440–442

[25] Chen X, Lin J, Ma Y, Lin B, Wang H, Huang G. Self-adaptive resource allocation for cloud-based software services based on progressive QoS prediction model. Science China Information Sciences, 2019, 62(11): 1–3

[26] Xia Y, Ren R, Cai H, Vasilakos A V, Lv Z. Daphne: a flexible and hybrid scheduling framework in multi-tenant clusters. IEEE Transactions on Network and Service Management, 2018, 15(1): 330–343

[27] Hindman B, Konwinski A, Zaharia M, Ghodsi A, Joseph A D, Katz R H, Shenker S, Stoica I. Mesos: a platform for fine-grained resource sharing in the data center. In: Proceedings of the 8th USENIX Symposium on Networked Systems Design and Implementation. 2011, 295–308

[28] Ghodsi A, Zaharia M, Hindman B, Konwinski A, Shenker S, Stoica I. Dominant resource fairness: fair allocation of multiple resource types. In: Proceedings of the 8th USENIX Symposium on Networked Systems Design and Implementation. 2011, 323–336

[29] Sanchez V A B, Kim W, Eom Y, Jin K, Nam M, Hwang D, Kim J, Nam B. EclipseMR: distributed and parallel task processing with consistent hashing. In: Proceedings of the 2017 IEEE International Conference on Cluster Computing. 2017, 322–332

[30] Shen L, Wu J, Wang Y, Liu L. Towards load balancing for LSH-based distributed similarity indexing in high-dimensional space. In: Proceedings of the 20th IEEE International Conference on High Performance Computing and Communications; the 16th IEEE International Conference on Smart City; the 4th IEEE International Conference on Data Science and Systems. 2018, 384–391

[31] Kulkarni C, Kesavan A, Zhang T, Ricci R, Stutsman R. Rocksteady: fast migration for low-latency in-memory storage. In: Proceedings of the 26th Symposium on Operating Systems Principles. 2017, 390–405

[32] Psychas K, Ghaderi J. Randomized algorithms for scheduling multiresource jobs in the cloud. IEEE/ACM Transactions on Networking, 2018, 26(5): 2202–2215

[33] Islam N S, Wasi-ur-Rahman M, Lu X, Panda D K. Efficient data access strategies for hadoop and spark on HPC cluster with heterogeneous storage. In: Proceedings of the 2016 IEEE International Conference on Big Data. 2016, 223–232

[34] Zhou P, Huang J, Qin X, Xie C. PaRS: a popularity-aware redundancy scheme for in-memory stores. IEEE Transactions on Computers, 2019, 68(4): 556–569

[35] Zhao N, Anwar A, Cheng Y, Salman M, Li D, Wan J, Xie C, He X, Wang F, Butt A R. Chameleon: an adaptive wear balancer for flash clusters. In: Proceedings of the 2018 IEEE International Parallel and Distributed Processing Symposium. 2018, 1163–1172

[36] Atikoglu B, Xu Y, Frachtenberg E, Jiang S, Paleczny M. Workload analysis of a large-scale key-value store. In: Proceedings of the 12th ACM SIGMETRICS/PERFORMANCE Joint International Conference on Measurement and Modeling of Computer Systems. 2012, 53–64

[37] Zhang W, Hwang J, Wood T, Ramakrishnan K K, Huang H H. Load balancing of heterogeneous workloads in memcached clusters. In: Proceedings of the 9th International Workshop on Feedback Computing. 2014, 1–8

[38] Salkhordeh R, Asadi H. An operating system level data migration scheme in hybrid DRAM-NVMmemory architecture. In: Proceedings of the 2016 Design, Automation & Test in Europe Conference & Exhibition. 2016, 936–941

[39] Zhou Y, Alagappan R, Memaripour A, Badam A, Wentzlaff D. HNVM: hybrid NVM enabled datacenter design and optimization. Microsoft Research Technical Report, 2017

[40] Calheiros R N, Ranjan R, Beloglazov A, Rose C A F D, Buyya R. Cloudsim: a toolkit for modeling and simulation of cloud computing environments and evaluation of resource provisioning algorithms. Software: Practice and Experience, 2011, 41(1): 23–50

[41] Izraelevitz J, Yang J, Zhang L, Kim J, Liu X, Memaripour A, Soh Y J, Wang Z, Xu Y, Dulloor S R, Zhao J, Swanson S. Basic performance measurements of the intel optane DC persistent memory module. 2019, arXiv preprint arXiv:1903.05714

[42] Hong Y J, Thottethodi M. Understanding and mitigating the impact of load imbalance in the memory caching tier. In: Proceedings of the ACM Symposium on Cloud Computing. 2013, 1–17

[43] Ou J, Shu J, Lu Y, Yi L, Wang W. EDM: an endurance-aware data migration scheme for load balancing in SSD storage clusters. In: Proceedings of the 28th IEEE International Parallel and Distributed Processing Symposium. 2014, 787–796

[44] Cooper B F, Silberstein A, Tam E, Ramakrishnan R, Sears R. Benchmarking cloud serving systems with YCSB. In: Proceedings of the 1st ACM Symposium on Cloud Computing. 2010, 143–154

RIGHTS & PERMISSIONS

2020 Higher Education Press