User-level failure detection and auto-recovery of parallel programs in HPC systems

Abstract

As the mean-time-between-failures (MTBF) continues to decline with the increasing number of components on large-scale high performance computing (HPC) systems, program failures might occur during the execution period with high probability. Ensuring successful execution of the HPC programs has become an issue that the unprivileged users should be concerned. From the user perspective, if the program failure cannot be detected and handled in time, it would waste resources and delay the progress of program execution. Unfortunately, the unprivileged users are unable to perform program state checking due to execution control by the job management system as well as the limited privilege. Currently, automated tools for supporting user-level failure detection and autorecovery of parallel programs in HPC systems are missing. This paper proposes an innovative method for the unprivileged user to achieve failure detection of job execution and automatic resubmission of failed jobs. The state checker in our method is encapsulated as an independent job to reduce interference with the user jobs. In addition, we propose a dual-checker mechanism to improve the robustness of our approach.We implement the proposed method as a tool named automatic re-launcher (ARL) and evaluate it on the Tianhe-2 system. Experiment results show that ARL can detect the execution failures effectively on Tianhe-2 system. In addition, the communication and performance overhead caused by ARL is negligible. The good scalability of ARL makes it applicable for large-scale HPC systems.

Keywords

high performance computing / parallel program / failure detection / failure auto-recovery

References

[1] Egwutuoha I P, Levy D, Selic B, Chen S. A survey of fault tolerance mechanisms and checkpoint/restart implementations for high performance computing systems. The Journal of Supercomputing, 2013, 65: 1302–1326

[2] Lu C D. Failure data analysis of hpc systems. 2013, arXiv preprint arXiv:1302.4779

[3] Cappello F, Geist A, Gropp WD, Kale L V, Kramer WT, Snir M. Toward exascale resilience. International Journal of High Performance Computing Applications, 2009, 23: 374–385

[4] Bertier M, Marin O, Sens P. Performance analysis of a hierarchical failure detector. In: Proceedings of the 2003 International Conference on Dependable Systems and Networks. 2003, 635–644

[5] Luecke G R, Zou Y, Coyle J, Hoekstra J, Kraeva M. Deadlock detection in MPI programs. Concurrency and Computation: Practice and Experience, 2002, 14: 911–932

[6] Gao Q, Qin F, Panda D K. DMTracker: finding bugs in large-scale parallel programs by detecting anomaly in data movements. In: Proceedings of the 2007 ACM/IEEE Conference on Supercomputing. 2007, 1–12

[7] Arnold D C, Ahn D H, De Supinski B R, Lee G L, Miller B P, Schulz M. Stack trace analysis for large-scale debugging. In: Proceddings of the 2007 IEEE International Parallel and Distributed Processing Symposium. 2007, 1–10

[8] Laguna I, Gamblin T, De Supinski B R, Bagchi S, Bronevetsky G, Anh D H, Schulz M, Rountree B. Large scale debugging of parallel tasks with AutomaDeD. In: Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis. 2011, 1–10

[9] Wu X, Mueller F. Elastic and scalable tracing and accurate replay of nondeterministic events. In: Proceedings of the 27th International ACM Conference on International Conference on Supercomputing. 2013, 59–68

[10] Gupta R, Beckman P, Park B H, Lusk E, Hargrove P, Geist A, Panda D, Lumsdaine A, Dongarra J. CIFTS: a coordinated infrastructure for faulttolerant systems. In: Proceedings of the 2009 International Conference on Parallel Processing. 2009, 237–245

[11] Zhang G Z, Liu Y, Yang H L, Qian D P. A lightweight and flexible tool for distinguishing between hardware malfunctions and program bugs in debugging large-scale programs. IEEE Access, 2018, 6: 71892–71905

[12] Bosilca G, Bouteiller A, Guermouche A, Herault T, Robert Y, Sens P, Dongarra J. A failure detector for HPC platforms. The International Journal of High Performance Computing Applications, 2018, 32: 139–158

[13] Berrocal E, Bautista-Gomez L, Di S, Lan Z L, Cappello F. Lightweight silent data corruption detection based on runtime data analysis for HPC applications. In: Proceedings of the 24th International Symposium on High-Performance Parallel and Distributed Computing. 2015, 275–278

[14] Berrocal E, Bautista-Gomez L, Di S, Cappello F. Exploring partial replication to improve lightweight silent data corruption detection for HPC applications. In: Proceedings of Europe Conference on Parallel Processing. 2016, 419–430

[15] Berrocal E, Bautista-Gomez L, Di S, Lan Z L, Cappello F. Toward general software level silent data corruption detection for parallel applications. IEEE Transactions on Parallel and Distributed Systtems, 2017, 28(12): 3642–3655

[16] Guo L Z, Li D, Laguna I, Schulz M. FlipTracker: understanding natural error resilience in HPC applications. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis. 2018, 94–107

[17] Gomez L B, Cappello F. Detecting silent data corruption through data dynamic monitoring for scientific applications. In: Proceedings of the 19th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming. 2014, 381–382

[18] Subasi O, Di S, Gomez L B, Balaprakash P, Unsal O, Labarta J, Cristal A, Krishnamoorthy S, Cappello F. Exploring the capabilities of support vector machines in detecting silent data corruptions. Sustainable Computing: Informatics and Systems, 2018, 19: 277–290

[19] Liu J, Agrawal G. Soft error detection for iterative applications using offline training. In: Proceedings of the 23rd IEEE International Conference on High Performance Computing. 2016, 2–11

[20] Hassani A, Skjellum A, Brightwell R. Design and evaluation of FA-MPI, a transactional resilience scheme for non-blocking MPI. In: Proceedings of the 44th Annual IEEE/IFIP International Conference on Dependable Systems and Networks. 2014, 750–755

[21] Li Y, Lan Z, Gujrati P, Sun X H. Fault-aware runtime strategies for highperformance computing. IEEE Transactions on Parallel and Distributed Systems, 2009, 4(20): 460–473

[22] Hargrove P H, Duell J C. Berkeley lab checkpoint/restart (BLCR) for linux clusters. Journal of Physics: Conference Series, 2006, 46(1): 494–499

[23] Gomez L B, Tsuboi S, Komatitsch D, Cappello F, Maruyama N, Matsuoka S. FTI: high performance fault tolerance interface for hybrid systems. In: Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis. 2011, 1–12

[24] Buntinas D, Coti C, Herault T, Lemarinier P, Pilard L. Blocking vs. nonblocking coordinated checkpointing for large-scale fault tolerant MPI protocols. Future Generation Computer Systems, 2008, 24(1): 73–84

[25] Di S, Bouguerra M S, Gomez L B, Cappello F. Optimization of multilevel checkpoint model for large scale HPC applications. In: Proceedings of the 28th IEEE International Parallel and Distributed Processing Symposium. 2014, 1181–1190

[26] Ho J C Y,Wang C L, Lau F C M. Scalable group-based checkpoint/restart for large-scale message-passing systems. In: Proceedings of 2008 IEEE International Symposium on Parallel and Distributed Processing. 2008, 1–12

[27] Agarwal S, Garg R, GuptaMS,Moreira J E. Adaptive incremental checkpointing for massively parallel systems. In: Proceedings of the 18th Annual International Conference on Supercomputing. 2004, 277–286

[28] Nicolae B, Cappello F. AI-Ckpt: leveraging memory access patterns for adaptive asynchronous incremental checkpointing. In: Proceedings of the 22nd International Symposium on High-Performance Parallel and Distributed Computing. 2013, 155–166

[29] Ferreira K B, Riesen R, Bridges P, Arnold D, Brightwell R. Accelerating incremental checkpointing for extreme-scale computing. Future Generation Computer Systems, 2014, 30: 66–77

[30] Nicolae B, Moody A, Gonsiorowski E, Mohror K, Cappello F. VeloC: towards high performance adaptive asynchronous checkpointing at large scale. In: Proceedings of IEEE International Parallel and Distributed Processing Symposium. 2019, 911–920

[31] Losada N, Bosilca G, Bouteiller A, Gonzalez P, Martin M J. Local rollback for resilient MPI applications with application-level checkpointing and message logging. Future Generation Computer Systems, 2019, 91: 450–464

[32] Lan Z L, Li Y W. Adaptive fault management of parallel applications for high performance computing. IEEE Transactions on Computers, 2008, 57: 1647–1660

[33] Ibtesham D, Arnold D, Bridges P G, Ferreira K B, Brightwell R. On the viability of compression for reducing the overheads of checkpoint/restartbased fault tolerance. In: Proceedings of the 41st International Conference on Parallel Processing. 2012, 148–157

[34] Zhou P, Liu W, Fei L, Fei L, Lu S, Qin F, Zhou Y Y, Midkiff S, Torrellas G. Accmon: automatically detecting memory-related bugs via program counter-based invariants. In: Proceedings of the 37th Annual IEEE/ACM International Symposium on Microarchitecture. 2004, 269–280

[35] Zheng Z, Li Y, Lan Z L. Anomaly localization in large-scale clusters. In: Proceedings of 2007 IEEE International Conference on Cluster Computing. 2007, 322–330

[36] Yu L, Lan Z L. A scalable, non-parametric method for detecting perfor mance anomaly in large scale computing. IEEE Transactions on Parallel and Distributed Systems, 2016, 27(1): 1902–1914

[37] Zheng Z, Yu L, Tang W, Lan Z L, Gupta R, Desai N, Coghlan S, Buettner D. Co-analysis of RAS log and job log on Blue Gene/P. In: Proceddings of 2011 IEEE International Parallel & Distributed Processing Symposium. 2011, 840–851

[38] Berrocal E, Yu L, Wallace S, Papka M E, Lan Z L. Exploring void search for fault detection on extreme scale systems. In: Proceedings of IEEE International Conference on Cluster Computing. 2014, 1–9

[39] Gujrati P, Li Y, Lan Z L, Thakur R, White J. A meta-learning failure predictor for Blue Gene/L systems. In: Proceedings of 2007 International Conference on Parallel Processing. 2007

[40] Zheng Z, Lan Z L, Park B H, Geist A. System log pre-processing to improve failure prediction. In: Proceedings of 2009 IEEE/IFIP International Conference on Dependable Systems & Networks. 2009, 572–577

RIGHTS & PERMISSIONS

Higher Education Press