VBMq: pursuit baremetal performance by embracing block I/O parallelism in virtualization

Abstract

Barely acceptable block I/O performance prevents virtualization from being widely used in the High-Performance Computing field. Although the virtio paravirtual framework brings great I/O performance improvement, there is a sharp performance degradation when accessing high-performance NAND-flash-based devices in the virtual machine due to their data parallel design. The primary cause of this fact is the deficiency of block I/O parallelism in hypervisor, such as KVM and Xen. In this paper, we propose a novel design of block I/O layer for virtualization, named VBMq. VBMq is based on virtio paravirtual I/O model, aiming to solve the block I/O parallelism issue in virtualization. It uses multiple dedicated I/O threads to handle I/O requests in parallel. In the meanwhile, we use polling mechanism to alleviate overheads caused by the frequent context switches of the VM’s notification to and from its hypervisor. Each dedicated I/O thread is assigned to a non-overlapping core to improve performance by avoiding unnecessary scheduling. In addition, we configure CPU affinity to optimize I/O completion for each request. The CPU affinity setting is very helpful to reduce CPU cache miss rate and increase CPU efficiency. The prototype system is based on Linux 4.1 kernel and QEMU 2.3.1. Our measurements show that the proposed method scales graciously in the multi-core environment, and provides performance which is 39.6x better than the baseline at most, and approaches bare-metal performance.

Keywords

high-performance / parallelism / paravirtual I/O

References

[1] Walters J P, Chaudhary V, Cha M, Guercio Jr S, Gallo S. A comparison of virtualization technologies for HPC. In: Proceedings of the 22nd IEEE International Conference on Advanced Information Networking and Applications. 2008, 861–868

[2] Mergen M F, Uhlig V, Krieger O, Xenidis J. Virtualization for highperformance computing. Journal of ACM SIGOPS Operating Systems Review, 2006, 40(2): 8–11

[3] Huang W, Liu J, Abali B, Panda D K. A case for high performance computing with virtual machines. In: Proceedings of the 20th ACM Annual International Conference on Supercomputing. 2006, 125–134

[4] Bjorling M, Axboe J, Nellans D, Bonnet P. Linux block IO: introducing multi-queue SSD access on multi-core systems. In: Proceedings of the 6th ACM International Systems and Storage Conference. 2013, 22

[5] Bilas A. Scaling I/O in virtualized multicore servers: how much I/O in 10 years and how to get there. In: Proceedings of the 6th ACM International Workshop on Virtualization Technologies in Distributed Computing Date. 2012, 1–2

[6] Tezuka H, O’Carroll F, Hori A, Ishikawa Y. Pin-down cache: a virtual memory management technique for zero-copy communication. In: Proceedings of the 1st IEEE Merged International and Symposium on Parallel and Distributed Processing. 1998, 308–314

[7] Huffman A. NVM express, revision 1.0 c. Intel Corporation, 2012

[8] Gordon A, Har’El N, Landau A, Ben-Yehuda M, Traeger A. Towards exitless and efficient paravirtual I/O. In: Proceedings of the 5th ACM Annual International Systems and Storage Conference. 2012

[9] Adams K, Agesen O. A comparison of software and hardware techniques for x86 virtualization. Journal of ACM SIGOPS Operating Systems Review, 2006, 40(5): 2–13

[10] Har’El N, Gordon A, Landau A, Ben-Yehuda M, Traeger A, Ladelsky R. Efficient and scalable paravirtualI/O system. In: Proceedings of USENIX Annual Technical Conference. 2013, 231–242

[11] Maquelin O, Gao G R, Hum H H J, Theobald K B, Tian X M. Polling watchdog: combining polling and interrupts for efficient message handling. ACM SIGARCH Computer Architecture News, 1996, 24(2): 179–188

[12] Dovrolis C, Thayer B, Ramanathan P.HIP: hybrid interrupt-polling for the network interface. Journal of ACM SIGOPS Operating Systems Review, 2001, 35(4): 50–60

[13] Russell R. Virtio: towards a de-facto standard for virtual I/O devices. Journal of ACM SIGOPS Operating Systems Review, 2008, 42(5): 95–103

[14] Russell R, Tsirkin M S, Huck C, Moll P. Virtual I/O Device (VIRTIO) Version 1.0. OASIS Standard, OASIS Committee Specification, 2015, 2

[15] Martinez A, Chapple J, Sethi P, Bennett J. Circuitry to selectively produce MSI signals. U.S. Patent Application 10/881,076, 2004-6-29

[16] Sinharoy B, Van Norstrand J A, Eickemeyer R J, Le H Q, Leenstra J, Nguyen D Q, Konigsburg B, Ward K, Brown M D, Moreira J E, Levitan D, Tung S, Hrusecky D, Bishop J W, Gschwind M, Boersma M, Kroener M, Kaltenbach M, Karkhanis T, Fernsler K M. IBM POWER8 processor core micro architecture. IBMJournal of Research and Development, 2015, 59(1): 2

[17] Hung J J, Bu K, Sun Z L, Diao J T, Liu J B. PCI express-based NVMe solid state disk. In: Proceedings of Applied Mechanics and Materials. 2014, 365–368

[18] Dong Y, Dai J, Huang Z, Guan H, Tian K, Jiang Y. Towards highquality I/Ovirtualization. In: Proceedings of SYSTOR 2009: The Israeli Experimental Systems Conference. 2009

[19] Ben-Yehuda M, Mason J, Krieger O, Krieger O, Van Doorn L, Nakajima J, Wahlig E. Utilizing IOMMUs for virtualizationin Linux and Xen. In: Proceedings of the 2006 Ottawa Linux Symposium. 2006, 71–86

[20] AMD I, Virtualization O. Technology (IOMMU) Specification. 2007

[21] Ben-Yehuda M, Xenidis J, Ostrowski M, Rister K, Bruemmer A, Van Doorn L. The price of safety: evaluating IOMMU performance. In: Proceedings of the Ottawa Linux Symposium. 2007, 9–20

[22] Yassour B A, Ben-Yehuda M, Wasserman O. Direct device assignment for untrusted fully-virtualized virtual machines. IBM Research Report, 2008

[23] Zhai E, Cummings G D, Dong Y. Live migration with pass-through device for Linux VM. In: Proceedings of the 2008 Ottawa Linux Symposium. 2008, 261–268

[24] Ben-Yehuda M, Day M D, Dubitzky Z, Factor M, Har’El N, Gordon A, Liguori A, Wasserman O, Yassour , B. A. The turtles project: design and implementation of nested virtualization. In: Proceedings of the 9th USENIX Conference on Operating Systems Design and Implementation. 2010, 423–436

[25] SIG PCI. Single Root I/O Virtualization and Sharing Specification, Revision 1.0,2008

[26] Cully B, Wires J, Meyer D, Jamieson K, Fraser K, Deegan T, Stodden D, Lefebvre G, Ferstay D, Warfield A. Strata: high-performance scalable storage on virtualized non-volatile memory. In: Proceedings of the 12th USENIX conference on File and Storage Technologies. 2014, 17–31

[27] Clark C, Fraser K, Hand S,Hansen J G, Ju l E, Limpach C, Pratt I, Warfield A. Live migration of virtual machines. In: Proceedings of the 2nd Conference on Symposium on Networked Systems Design and Implementation-Volume 2. 2005, 273–286

[28] Yu W, Vetter J S. Xen-based HPC: a parallel I/O perspective. In: Proceedings of the 8th IEEE International Symposium on Cluster Computing and the Grid. 2008, 154–161

[29] Gordon A, Amit N, Har’El N, Ben-Yehuda M, Landau A, Schuster A, Tsafrir D. ELI: bare-metal performance forI/O virtualization. Journal of ACM SIGPLAN Notices, 2012, 47(4): 411–422

[30] Lei M. Virtio Blk multi-queue conversion. 2014

RIGHTS & PERMISSIONS

Higher Education Press and Springer-Verlag GmbH Germany, part of Springer Nature