Design and implementation of dynamic I/O control scheme for large scale distributed file systems

  • Kim, Sunggon; 
  • Sim, Alex; 
  • Wu, Kesheng; 
  • Byna, Suren; 
  • Son, Yong Seok
Citations

WEB OF SCIENCE

0
Citations

SCOPUS

0

초록

In this work, we have analyzed the input/output (I/O) activities of Cori, which is a high-performance computing system at the National Energy Research Scientific Computing Center at Lawrence Berkeley National Laboratory. Our analysis results indicate that most users do not adjust storage configurations but rather use the default settings. In addition, owing to the interference from many applications running simultaneously, the performance varies based on the system status. To configure file systems autonomously in complex environments, we developed DCA-IO, a dynamic distributed file system configuration adjustment algorithm that utilizes the system log information to adjust storage configurations automatically. Our scheme aims to improve the application performance and avoid interference from other applications without user intervention. Moreover, DCA-IO uses the existing system logs and does not require code modifications, an additional library, or user intervention. To demonstrate the effectiveness of DCA-IO, we performed experiments using I/O kernels of real applications in both an isolated small-sized Lustre environment and Cori. Our experimental results shows that our scheme can improve the performance of HPC applications by up to 263% with the default Lustre configuration.

키워드

High-performance computing; Distributed dynamic resource management; Autonomous control; Parallel and distributed file system; Cloud system
제목
Design and implementation of dynamic I/O control scheme for large scale distributed file systems
저자
Kim, Sunggon; Sim, Alex; Wu, Kesheng; Byna, Suren; Son, Yong Seok
DOI
10.1007/s10586-022-03640-0
발행일
2022-12
유형
Article
저널명
Cluster Computing
권
25
호
6
페이지
4423 ~ 4438

파일 다운로드

Thumbnail