A survey of cloud resource management for complex engineering applications

Abstract

Traditionally, complex engineering applications (CEAs), which consist of numerous components (software) and require a large amount of computing resources, usually run in dedicated clusters or high performance computing (HPC) centers. Nowadays, Cloud computing system with the ability of providing massive computing resources and customizable execution environment is becoming an attractive option for CEAs. As a new type on Cloud applications, CEA also brings the challenges of dealing with Cloud resources. In this paper, we provide a comprehensive survey of Cloud resource management research for CEAs. The survey puts forward two important questions: 1) what are the main challenges for CEAs to run in Clouds? and 2) what are the prior research topics addressing these challenges? We summarize and highlight the main challenges and prior research topics. Our work can be probably helpful to those scientists and engineers who are interested in running CEAs in Cloud environment.

Keywords

Cloud computing / complex engineering application / resource management / virtualization

References

[1] Crago S, Dunn K, Eads P, Hochstein L, Kang D I, Kang M, Modium D, Singh K, Suh J, Walters J P. Heterogeneous cloud computing. In: Proceedings of 2011 IEEE International Conference on Cluster Computing (CLUSTER). 2011, 378–385

[2] Arian E. On the coupling of aerodynamic and structural design. Journal of Computational Physics, 1997, 135(1): 83–96

[3] Wang C, Xu L. Parameter mapping and data transformation for engineering application integration. Information Systems Frontiers, 2008, 10(5): 589–600

[4] Ong M, Thompson H. Challenges for wireless sensing in complex engineering applications. In: Proceedings of the 37th Annual Conference on IEEE Industrial Electronics Society (IECON). 2011, 2106–2111

[5] Bichon B, Eldred M, Swiler L, Mahadevan S, McFarland J. Multimodal reliability assessment for complex engineering applications using efficient global optimization. In: Proceedings of the 48th AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference, AIAA-2007-1946. 2007, 3029–3040

[6] Chacón Rebollo T, Gómez Mármol M, Restelli M. Numerical analysis of penalty stabilized finite element discretizations of evolution navier–stokes equations. Journal of Scientific Computing, 2015, 63(3): 885–912

[7] Campana E F, Liuzzi G, Lucidi S, Peri D, Piccialli V, Pinto A. New global optimization methods for ship design problems. Optimization and Engineering, 2009, 10(4): 533–555

[8] Jin C, Wang Y, Zhang W, Lin Y. Study on semi-finished ship structural components assembly sequence optimization. In: Proceedings of the 6th International Conference on Natural Computation (ICNC). 2010, 2706–2709

[9] Kwon S, Kim B C, Mun D, Han S. Simplification of feature-based 3D CAD assembly data of ship and offshore equipment using quantitative evaluation metrics. Computer-Aided Design, 2015, 59: 140–154

[10] Palankar M R, Iamnitchi A, Ripeanu M, Garfinkel S. Amazon S3 for science grids: a viable solution? In: Proceedings of the 2008 International Workshop on Data-aware Distributed Computing. 2008, 55–64

[11] Hazelhurst S. Scientific computing using virtual high-performance computing: a case study using the amazon elastic computing cloud. In: Proceedings of the 2008 Annual Research Conference of the South African Institute of Computer Scientists and Information Technologists on IT Research in Developing Countries: Riding the Wave of Technology. 2008, 94–103

[12] Vöckler J S, Juve G, Deelman E, Rynge M, Berriman B. Experiences using cloud computing for a scientific workflow application. In: Proceedings of the 2nd International Workshop on Scientific Cloud Computing. 2011, 15–24

[13] Juve G, Deelman E, Vahi K, Mehta G, Berriman B, Berman B P, Maechling P. Scientific workflow applications on amazon EC2. In: Proceedings of the 5th IEEE International Conference on E-ScienceWorkshops. 2009, 59–66

[14] Rehr J J, Vila F D, Gardner J P, Svec L, Prange M. Scientific computing in the cloud. Computing in Science & Engineering, 2010, 12(3): 34–43

[15] Lin G, Han B, Yin J, Gorton I. Exploring cloud computing for largescale scientific applications. In: Proceedings of the 9th IEEE World Congress on Services (SERVICES). 2013, 37–43

[16] Ramakrishnan L, Zbiegel P T, Campbell S, Bradshaw R, Canon R S, Coghlan S, Sakrejda I, Desai N, Declerck T, Liu A. Magellan: experiences from a science cloud. In: Proceedings of the 2nd International Workshop on Scientific Cloud Computing. 2011, 49–58

[17] Georgescu S, Chow P. GPU accelerated CAE using open solvers and the cloud. ACMSIGARCH Computer Architecture News, 2011, 39(4): 14–19

[18] Zhai Y, Liu M, Zhai J, Ma X, Chen W. Cloud versus in-house cluster: evaluating amazon cluster compute instances for running MPI applications. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC). 2011, 11

[19] Janapa Reddi V, Lee B C, Chilimbi T, Vaid K. Web search using mobile cores: quantifying and mitigating the price of efficiency. ACM SIGARCH Computer Architecture News, 2010, 38(3): 314–325

[20] Lim K, Ranganathan P, Chang J, Patel C, Mudge T, Reinhardt S. Understanding and designing new server architectures for emerging warehouse-computing environments. In: Proceedings of the 35th International Symposium on Computer Architecture (ISCA). 2008, 315–326

[21] Santos J R, Turner Y, Janakiraman G J, Pratt I. Bridging the gap between software and hardware techniques for I/O virtualization. In: Proceedings of the USENIX Annual Technical Conference (ATC). 2008, 29–42

[22] Ram K K, Santos J R, Turner Y, Cox A L, Rixner S. Achieving 10 Gb/s using safe and transparent network interface virtualization. In: Proceedings of the ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments (VEE). 2009, 61–70

[23] Chen H, Wu S, Shi X, Jin H, Fu Y. LCM: a lightweight communication mechanism in HPC cloud. In: Proceedings of the 6th International Conference on Pervasive Computing and Applications. 2011, 443–451

[24] Ram K K, Santos J R, Turner Y. Redesigning Xen’s memory sharing mechanism for safe and efficient I/O virtualization. In: Proceedings of the 2nd conference on I/O virtualization. 2010

[25] Jian Z, Xiaoyong L, Haibing G. The optimization of Xen network virtualization. In: Proceedings of the International Conference on Computer Science and Software Engineering. 2008, 431–436

[26] Guo D, Liao G, Bhuyan L N. Performance characterization and cacheaware core scheduling in a virtualized multi-core server under 10GbE. In: Proceedings of the IEEE International Symposium on Workload Characterization. 2009, 168–177

[27] Liao G, Guo D, Bhuyan L, King S R. Software techniques to improve virtualized I/O performance on multi-core systems. In: Proceedings of the 4th ACM/IEEE Symposium on Architectures for Networking and Communications Systems. 2008, 161–170

[28] Gordon A, Amit N, Har’El N, Ben-Yehuda M, Landau A, Schuster A, Tsafrir D. ELI: bare-metal performance for I/O virtualization. In: Proceedings of the 7th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS). 2012, 411–422

[29] Abramson D. Intel virtualization technology for directed I/O. Intel Technology Journal, 2006, 10(3): 179–192

[30] Rixner S. Network virtualization: breaking the performance barrier. Queue, 2008, 6(1): 37

[31] Willmann P, Shafer J, Carr D,Menon A, Rixner S, Cox A L, Zwaenepoel W. Concurrent direct network access for virtual machine monitors. In: Proccedings of the 13th IEEE International Symposium on High Performance Computer Architecture (HPCA). 2007, 306–317

[32] Liu J. Evaluating standard-based self-virtualizing devices: a performance study on 10 GbE NICs with SR-IOV support. In: Proceedings of the IEEE International Symposium on Parallel & Distributed Processing (IPDPS). 2010, 1–12

[33] Jones S T, Arpaci-Dusseau A C, Arpaci-Dusseau R H. Antfarm: tracking processes in a virtual machine environment. In: Proceedings of the USENIX Annual Technical Conference (ATC). 2006, 1–14

[34] Jin H, Ling X, Ibrahim S, Cao W, Wu S, Antoniu G. Flubber: two-level disk scheduling in virtualized environment. Future Generation Computer Systems, 2013, 29(8): 2222–2238

[35] Xu Y, Jiang S. A scheduling framework that makes any disk schedulers non-work-conserving solely based on request characteristics. In: Proceedings of the USENIX Conference on File and Storage Technologies. 2011, 119–132

[36] Zhang B B, Wang X L, Yang L, Lai R F, Wang Z L, Luo Y W, Li X M. Modifying guest OS to optimize I/O virtualization in KVM. Chinese Journal of Computers, 2010, 33(12): 2312–2319

[37] Ling X, Ibrahim S, Jin H, Wu S, Tao S. Exploiting spatial locality to improve disk efficiency in virtualized environments. In: Proceedings of the 21st IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems (MASCOTS). 2013

[38] Wang H, Varman P. A flexible approach to efficient resource sharing in virtualized environments. In: Proceedings of the 8th ACM International Conference on Computing Frontiers. 2011, 1–10

[39] Seelam S R, Teller P J. Virtual I/O scheduler: a scheduler of schedulers for performance virtualization. In: Proceedings of the 3rd International Conference on Virtual Execution Environments. 2007, 105–115

[40] Kesavan M, Gavrilovska A, Schwan K. Differential virtual time (DVT): rethinking I/O service differentiation for virtual machines. In: Proceedings of the 1st ACM Symposium on Cloud Computing. 2010, 27–38

[41] Gulati A, Ahmad I, Waldspurger C A. PARDA: proportional allocation of resources for distributed storage access. In: Proccedings of the 7th Conference on File and Storage Technologies (FAST). 2009, 85–98

[42] Gulati A, Merchant A, Varman P J. mClock: handling throughput variability for hypervisor I/O scheduling. In: Proceedings of the USENIX Symposium on Operating Systems Design and Implementation (OSDI). 2010, 1–7

[43] Arunagiri S, Kwok Y, Teller P J, Portillo R A, Seelam S R. FAIRIO: a throughput-oriented algorithm for differentiated I/O performance. International Journal of Parallel Programming, 2014, 42(1): 165–197

[44] Shue D, Freedman M J, Shaikh A. Performance isolation and fairness for multi-tenant cloud storage. In: Proceedings of the USENIX Symposium on Operating Systems Design and Implementation (OSDI). 2012, 349–362

[45] Lin C, Lu S. Scheduling scientific workflows elastically for cloud computing. In: Proceedings of the IEEE International Conference on Cloud Computing (CLOUD). 2011, 746–747

[46] Zhang F, Cao J, Hwang K, Wu C. Ordinal optimized scheduling of scientific workflows in elastic compute clouds. In: Proceedings of the 3rd IEEE International Conference on Cloud Computing Technology and Science (CloudCom). 2011, 9–17

[47] Rahman M, Hassan R, Ranjan R, Buyya R. Adaptive workflow scheduling for dynamic grid and cloud computing environment. Concurrency and Computation: Practice and Experience, 2013, 25(13): 1816–1842

[48] Liu B, Li J, Liu C. Cloud-based bioinformatics workflow platform for large-scale next-generation sequencing analyses. Journal of Biomedical Informatics, 2014, 49: 119–133

[49] Liu S W, Kong L M, Ren K J, Song J Q, Deng K F, Leng H Z. A twostep data placement and task scheduling strategy for optimizing scientific workflow performance on cloud computing platform. Chinese Journal of Computers, 2011, 34(11): 2121–2130

[50] Deelman E, Chervenak A. Data management challenges of dataintensive scientific workflows. In: Proceedings of the 8th IEEE International Symposium on Cluster Computing and the Grid (CCGRID). 2008, 687–692

[51] Yuan D, Yang Y, Liu X, Chen J. A cost-effective strategy for intermediate data storage in scientific cloud workflow systems. In: Proceedings of 2010 IEEE International Symposium on Parallel and Distributed Processing (IPDPS). 2010, 1–12

[52] Zheng P, Cui L Z, Wang H Y, Xu M. A data placement strategy for dataintensive applications in cloud. Chinese Journal of Computers, 2010, 33(8): 1472–1480

[53] He B, Fang W, Luo Q, Govindaraju N K, Wang T. Mars: a MapReduce framework on graphics processors. In: Proceedings of the 17th International Conference on Parallel Architectures and Compilation Techniques (PACT). 2008, 260–269

[54] Linderman M D, Collins J D, Wang H, Meng T H. Merge: a programming model for heterogeneous multi-core systems. In: Proceedings of the 13th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS). 2008, 287–296

[55] Boob S, Gonzalez-Velez H, Popescu A M. Automated instantiation of heterogeneous fast flow CPU/GPU parallel pattern applications in clouds. In: Proceedings of the 22nd Euromicro International Conference on Parallel, Distributed and Network-Based Processing (PDP). 2014, 162–169

[56] Campa S, Danelutto M, Goli M, González-Vélez H, Popescu A M, Torquati M. Parallel patterns for heterogeneous CPU/GPU architectures: structured parallelism from cluster to cloud. Future Generation Computer Systems, 2014, 37: 354–366

[57] Luk C K, Hong S, Kim H. Qilin: exploiting parallelism on heterogeneous multiprocessors with adaptive mapping. In: Proceedings of the 42nd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO). 2009, 45–55

[58] Ravi V T, Ma W, Chiu D, Agrawal G. Compiler and runtime support for enabling generalized reduction computations on heterogeneous parallel configurations. In: Proceedings of the 24th ACM International Conference on Supercomputing (ICS). 2010, 137–146

[59] Grewe D, O’Boyle M F. A static task partitioning approach for heterogeneous systems using OpenCL. Lecture Notes in Computer Science. 2011, 6601: 286–305

[60] Gupta V, Gavrilovska A, Schwan K, Kharche H, Tolia N, Talwar V, Ranganathan P. GViM: GPU-accelerated virtual machines. In: Proceedings of the 3rd ACM Workshop on System-level Virtualization for High Performance Computing. 2009, 17–24

[61] Shi L, Chen H, Sun J, Li K. vCUDA: GPU-accelerated high performance computing in virtual machines. IEEE Transactions on Computers, 2012, 61(6): 804–816

[62] Giunta G, Montella R, Agrillo G, Coviello G. A GPGPU transparent virtualization component for high performance computing clouds. In: Proceedings of Euro-Par 2010-Parallel Processing. 2010, 379–391

[63] Jo H, Jeong J, Lee M, Choi D H. Exploiting GPUs in virtual machine for BioCloud. BioMed Research International, 2013

[64] Shih C S, Wei J W, Hung S H, Chen J, Chang N. Fairness scheduler for virtual machines on heterogonous multi-core platforms. ACMSIGAPP Applied Computing Review, 2013, 13(1): 28–40

[65] Hu L, Che X, Xie Z. GPGPU cloud: a paradigm for general purpose computing. Tsinghua Science and Technology, 2013, 18(1): 22–23

[66] Chen H, Shi L, Sun J. VMRPC: a high efficiency and light weight RPC system for virtual machines. In: Proceedings of the 18th IEEE International Workshop on Quality of Service (IWQoS). 2010

[67] Montella R, Giunta G, Laccetti G. Virtualizing high-end GPGPUs on ARM clusters for the next generation of high performance cloud computing. Cluster Computing, 2014, 17(1): 139–152

[68] Cai Y, Li G, Wang H, Zheng G, Lin S. Development of parallel explicit finite element sheet forming simulation system based on GPU architecture. Advances in Engineering Software, 2012, 45(1): 370–379

[69] Ari I, Muhtaroglu N. Design and implementation of a cloud computing service for finite element analysis. Advances in Engineering Software, 2013, 60: 122–135

[70] Negrut D, Tasora A, Anitescu M, Mazhar H, Heyn T, Pazouki A. Solving large multi-body dynamics problems on the GPU. GPU Gems, 2011, 4: 269–280

[71] Hanniel I, Haller K. Direct rendering of solid CAD models on the GPU. In: Proceedings of the 12th International Conference on Computer-Aided Design and Computer Graphics (CAD/Graphics). 2011, 25–32

[72] Hsieh H T, Chu C H. Particle swarm optimisation (PSO)-based tool path planning for 5-axis flank milling accelerated by graphics processing unit (GPU). International Journal of Computer Integrated Manufacturing, 2011, 24(7): 676–687

[73] Hung Y, Wang W. Accelerating parallel particle swarm optimization via GPU. Optimization Methods and Software, 2012, 27(1): 33–51

[74] Jung H Y, Jun C W, Sohn J H. GPU-based collision analysis between a multi-body system and numerous particles. Journal of Mechanical Science and Technology, 2013, 27(4): 973–980

[75] Nguyen Van H, Dang Tran F, Menaud J M. Autonomic virtual resource management for service hosting platforms. In: Proceedings of the 2009 ICSE Workshop on Software Engineering Challenges of Cloud Computing. 2009, 1–8

[76] Mehta H K, Kanungo P, Chandwani M. Performance enhancement of scheduling algorithms in clusters and grids using improved dynamic load balancing techniques. In: Proceedings of the 20th International Conference Companion on World Wide Web. 2011, 385–390

[77] Chapman C, Emmerich W, Márquez F G, Clayman S, Galis A. Software architecture definition for on-demand cloud provisioning. Cluster Computing, 2012, 15(2): 79–100

[78] Ardagna D, Panicucci B, Passacantando M. A game theoretic formulation of the service provisioning problem in cloud systems. In: Proceedings of the 20th International Conference Companion on World Wide Web (WWW). 2011, 177–186

[79] Qiang L, Qin-Fen H, Li-Min X, Zhou-Jun L. Adaptive management and multi-objective optimization for virtual machine placement in cloud computing. Chinese Journal of Computers, 2011, 34(12): 2253–2264

[80] Kaur P D, Chana I. A resource elasticity framework for QoS-aware execution of cloud applications. Future Generation Computer Systems, 2014, 37: 14–25

[81] Son S, Jun S C. Negotiation-based flexible SLA establishment with SLA-driven resource allocation in cloud computing. In: Proceedings of the 13th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid). 2013, 168–171

[82] García A G, Espert I B, García V H. SLA-driven dynamic cloud resource management. Future Generation Computer Systems, 2014, 31: 1–11

[83] Wang L, Zhan J, Shi W, Liang Y, Yuan L. In cloud, do MTC or HTC service providers benefit from the economies of scale? In: Proceedings of the 2nd Workshop on Many-Task Computing on Grids and Supercomputers. 2009, 7

[84] Jin H, Qin H, Wu S, Guo X. CCAP: a cache contention-aware virtual machine placement approach for HPC cloud. International Journal of Parallel Programming, 2015, 43(3): 403–420

[85] Chen H, Wu S, Di S, Zhou B, Xie Z, Jin H, Shi X. Communicationdriven scheduling for virtual clusters in cloud. In: Proceedings of the 23rd International Symposium on High-performance Parallel and Distributed Computing (HPDC). 2014, 125–128

[86] Wu S, Chen H, Di S, Zhou B, Xie Z, Jin H, Shi X. Synchronizationaware scheduling for virtual clusters in cloud. IEEE Transactions on Parallel and Distributed Systems, 2015, 26(10): 2890–2901

[87] Eldred M S. Optimization strategies for complex engineering applications. Technical Report, Sandia National Labs., Albuquerque, NM (United States), 1998

[88] Keahey K. Cloud computing for science. In: Proceedings of the 21st International Conference on Scientific and Statistical Database Management. 2009, 478

[89] Deelman E, Singh G, Livny M, Berriman B, Good J. The cost of doing science on the cloud: the montage example. In: Proceedings of the 2008 ACM/IEEE Conference on Supercomputing (ICS). 2008, 50

[90] Wang L, Tao J, Kunze M, Castellanos A C, Kramer D, Karl W. Scientific cloud computing: Early definition and experience. In: Proceedings of the IEEE International Conference on High Performance Computing and Communications. 2008, 825–830

[91] Ludäscher B, Altintas I, Berkley C, Higgins D, Jaeger E, Jones M, Lee E A, Tao J, Zhao Y. Scientific workflow management and the Kepler system. Concurrency and Computation: Practice and Experience, 2006, 18(10): 1039–1065

[92] Oinn T, Addis M, Ferris J, Marvin D, Senger M, Greenwood M, Carver T, Glover K, Pocock M R, Wipat A, Li P. Taverna: a tool for the composition and enactment of bioinformatics workflows. Bioinformatics, 2004, 20(17): 3045–3054

RIGHTS & PERMISSIONS

2014 Higher Education Press and Springer-Verlag Berlin Heidelberg