Towards HPC I/O Performance Prediction through Large-scale Log Analysis

Citations

SCOPUS

34

초록

Large-scale high performance computing (HPC) systems typically consist of many thousands of CPUs and storage units, while used by hundreds to thousands of users at the same time. Applications from these large numbers of users have diverse characteristics, such as varying compute, communication, memory, and I/O intensiveness. A good understanding of the performance characteristics of each user application is important for job scheduling and resource provisioning. Among these performance characteristics, the I/O performance is difficult to predict because the I/O system software is complex, the I/O system is shared among all users, and the I/O operations also heavily rely on networking systems. To improve the prediction of the I/O performance on HPC systems, we propose to integrate information from a number of different system logs and develop a regression-based approach that dynamically selects the most relevant features from the most recent log entries, and automatically select the best regression algorithm for the prediction task. Evaluation results show that our proposed scheme can predict the I/O performance with up to 84% prediction accuracy in the case of the I/O-intensive applications using the logs from CORI supercomputer at NERSC. © 2020 ACM.

키워드

distributed file system; high performance computing; I/O performance prediction; log analysis; Forecasting; Program processors; Supercomputers; Evaluation results; High performance computing systems; Networking systems; Performance characteristics; Performance prediction; Prediction accuracy; Regression algorithms; Relevant features; Large scale systems
제목
Towards HPC I/O Performance Prediction through Large-scale Log Analysis
저자
Kim, S.; Sim, A.; Wu, K.; Byna, S.; Son, Yong Seok; Eom, H.
DOI
10.1145/3369583.3392678
발행일
2020
유형
Conference Paper
저널명
HPDC 2020 - Proceedings of the 29th International Symposium on High-Performance Parallel and Distributed Computing
페이지
77 ~ 88