Please wait a minute...

Frontiers of Computer Science

Front. Comput. Sci.    2018, Vol. 12 Issue (5) : 873-886
VBMq: pursuit baremetal performance by embracing block I/O parallelism in virtualization
Diming ZHANG1,2(), Fei XUE1, Hao HUANG1, Shaodi YOU3,4
1. Faculty of Computer Science and Technology, Nanjing University, Nanjing 210023, China
2. College of Computer Science and Engineering, Jiangsu University of Science and Technology, Zhenjiang 212003, China
3. College of Engineering and Computer Science, Australian National University, Canberra 2600, Australia
4. Data61-CSIRO, Australian National University, Canberra 2600, Australia
Download: PDF(881 KB)  
Export: BibTeX | EndNote | Reference Manager | ProCite | RefWorks

Barely acceptable block I/O performance prevents virtualization from being widely used in the High-Performance Computing field. Although the virtio paravirtual framework brings great I/O performance improvement, there is a sharp performance degradation when accessing high-performance NAND-flash-based devices in the virtual machine due to their data parallel design. The primary cause of this fact is the deficiency of block I/O parallelism in hypervisor, such as KVM and Xen. In this paper, we propose a novel design of block I/O layer for virtualization, named VBMq. VBMq is based on virtio paravirtual I/O model, aiming to solve the block I/O parallelism issue in virtualization. It uses multiple dedicated I/O threads to handle I/O requests in parallel. In the meanwhile, we use polling mechanism to alleviate overheads caused by the frequent context switches of the VM’s notification to and from its hypervisor. Each dedicated I/O thread is assigned to a non-overlapping core to improve performance by avoiding unnecessary scheduling. In addition, we configure CPU affinity to optimize I/O completion for each request. The CPU affinity setting is very helpful to reduce CPU cache miss rate and increase CPU efficiency. The prototype system is based on Linux 4.1 kernel and QEMU 2.3.1. Our measurements show that the proposed method scales graciously in the multi-core environment, and provides performance which is 39.6x better than the baseline at most, and approaches bare-metal performance.

Keywords high-performance      parallelism      paravirtual I/O     
Corresponding Authors: Diming ZHANG   
Just Accepted Date: 18 July 2017   Online First Date: 13 June 2018    Issue Date: 21 September 2018
 Cite this article:   
Diming ZHANG,Fei XUE,Hao HUANG, et al. VBMq: pursuit baremetal performance by embracing block I/O parallelism in virtualization[J]. Front. Comput. Sci., 2018, 12(5): 873-886.
E-mail this article
E-mail Alert
Articles by authors
Diming ZHANG
Shaodi YOU
1 Walters J P, Chaudhary V, Cha M, Guercio Jr S, Gallo S. A comparison of virtualization technologies for HPC. In: Proceedings of the 22nd IEEE International Conference on Advanced Information Networking and Applications. 2008, 861–868
2 Mergen M F, Uhlig V, Krieger O, Xenidis J. Virtualization for highperformance computing. Journal of ACM SIGOPS Operating Systems Review, 2006, 40(2): 8–11
3 Huang W, Liu J, Abali B, Panda D K. A case for high performance computing with virtual machines. In: Proceedings of the 20th ACM Annual International Conference on Supercomputing. 2006, 125–134
4 Bjorling M, Axboe J, Nellans D, Bonnet P. Linux block IO: introducing multi-queue SSD access on multi-core systems. In: Proceedings of the 6th ACM International Systems and Storage Conference. 2013, 22
5 Bilas A. Scaling I/O in virtualized multicore servers: how much I/O in 10 years and how to get there. In: Proceedings of the 6th ACM International Workshop on Virtualization Technologies in Distributed Computing Date. 2012, 1–2
6 Tezuka H, O’Carroll F, Hori A, Ishikawa Y. Pin-down cache: a virtual memory management technique for zero-copy communication. In: Proceedings of the 1st IEEE Merged International and Symposium on Parallel and Distributed Processing. 1998, 308–314
7 Huffman A. NVM express, revision 1.0 c. Intel Corporation, 2012
8 Gordon A, Har’El N, Landau A, Ben-Yehuda M, Traeger A. Towards exitless and efficient paravirtual I/O. In: Proceedings of the 5th ACM Annual International Systems and Storage Conference. 2012
9 Adams K, Agesen O. A comparison of software and hardware techniques for x86 virtualization. Journal of ACM SIGOPS Operating Systems Review, 2006, 40(5): 2–13
10 Har’El N, Gordon A, Landau A, Ben-Yehuda M, Traeger A, Ladelsky R. Efficient and scalable paravirtualI/O system. In: Proceedings of USENIX Annual Technical Conference. 2013, 231–242
11 Maquelin O, Gao G R, Hum H H J, Theobald K B, Tian X M. Polling watchdog: combining polling and interrupts for efficient message handling. ACM SIGARCH Computer Architecture News, 1996, 24(2): 179–188
12 Dovrolis C, Thayer B, Ramanathan P.HIP: hybrid interrupt-polling for the network interface. Journal of ACM SIGOPS Operating Systems Review, 2001, 35(4): 50–60
13 Russell R. Virtio: towards a de-facto standard for virtual I/O devices. Journal of ACM SIGOPS Operating Systems Review, 2008, 42(5): 95–103
14 Russell R, Tsirkin M S, Huck C, Moll P. Virtual I/O Device (VIRTIO) Version 1.0. OASIS Standard, OASIS Committee Specification, 2015, 2
15 Martinez A, Chapple J, Sethi P, Bennett J. Circuitry to selectively produce MSI signals. U.S. Patent Application 10/881,076, 2004-6-29
16 Sinharoy B, Van Norstrand J A, Eickemeyer R J, Le H Q, Leenstra J, Nguyen D Q, Konigsburg B, Ward K, Brown M D, Moreira J E, Levitan D, Tung S, Hrusecky D, Bishop J W, Gschwind M, Boersma M, Kroener M, Kaltenbach M, Karkhanis T, Fernsler K M. IBM POWER8 processor core micro architecture. IBMJournal of Research and Development, 2015, 59(1): 2
17 Hung J J, Bu K, Sun Z L, Diao J T, Liu J B. PCI express-based NVMe solid state disk. In: Proceedings of Applied Mechanics and Materials. 2014, 365–368
18 Dong Y, Dai J, Huang Z, Guan H, Tian K, Jiang Y. Towards highquality I/Ovirtualization. In: Proceedings of SYSTOR 2009: The Israeli Experimental Systems Conference. 2009
19 Ben-Yehuda M, Mason J, Krieger O, Krieger O, Van Doorn L, Nakajima J, Wahlig E. Utilizing IOMMUs for virtualizationin Linux and Xen. In: Proceedings of the 2006 Ottawa Linux Symposium. 2006, 71–86
20 AMD I, Virtualization O. Technology (IOMMU) Specification. 2007
21 Ben-Yehuda M, Xenidis J, Ostrowski M, Rister K, Bruemmer A, Van Doorn L. The price of safety: evaluating IOMMU performance. In: Proceedings of the Ottawa Linux Symposium. 2007, 9–20
22 Yassour B A, Ben-Yehuda M, Wasserman O. Direct device assignment for untrusted fully-virtualized virtual machines. IBM Research Report, 2008
23 Zhai E, Cummings G D, Dong Y. Live migration with pass-through device for Linux VM. In: Proceedings of the 2008 Ottawa Linux Symposium. 2008, 261–268
24 Ben-Yehuda M, Day M D, Dubitzky Z, Factor M, Har’El N, Gordon A, Liguori A, Wasserman O, Yassour , B. A. The turtles project: design and implementation of nested virtualization. In: Proceedings of the 9th USENIX Conference on Operating Systems Design and Implementation. 2010, 423–436
25 SIG PCI. Single Root I/O Virtualization and Sharing Specification, Revision 1.0,2008
26 Cully B, Wires J, Meyer D, Jamieson K, Fraser K, Deegan T, Stodden D, Lefebvre G, Ferstay D, Warfield A. Strata: high-performance scalable storage on virtualized non-volatile memory. In: Proceedings of the 12th USENIX conference on File and Storage Technologies. 2014, 17–31
27 Clark C, Fraser K, Hand S,Hansen J G, Ju l E, Limpach C, Pratt I, Warfield A. Live migration of virtual machines. In: Proceedings of the 2nd Conference on Symposium on Networked Systems Design and Implementation-Volume 2. 2005, 273–286
28 Yu W, Vetter J S. Xen-based HPC: a parallel I/O perspective. In: Proceedings of the 8th IEEE International Symposium on Cluster Computing and the Grid. 2008, 154–161
29 Gordon A, Amit N, Har’El N, Ben-Yehuda M, Landau A, Schuster A, Tsafrir D. ELI: bare-metal performance forI/O virtualization. Journal of ACM SIGPLAN Notices, 2012, 47(4): 411–422
30 Lei M. Virtio Blk multi-queue conversion. 2014
Related articles from Frontiers Journals
[1] Kenli LI, Zhao TONG, Dan LIU, Teklay TESFAZGHI, Xiangke LIAO. A PTS-PGATS based approach for data-intensive scheduling in data grids[J]. Front Comput Sci Chin, 2011, 5(4): 513-525.
[2] LONG Guilu, LIU Yang. Duality quantum computing[J]. Front. Comput. Sci., 2008, 2(2): 167-178.
[3] ZHANG Yunquan, CHEN Guoliang, SUN Guangzhong, MIAO Qiankun. Models of parallel computation: a survey and classification[J]. Front. Comput. Sci., 2007, 1(2): 156-165.
Full text