Design and implementation of a fast and predictable SSD liveness watchdog for storage systems

초록

Solid-state drives (SSDs) offer higher performance, greater reliability, and energy efficiency over hard disk drives (HDDs). Consequently, they have been widely adopted in various storage systems such as data centers, cloud infrastructure, and enterprise storage servers. Accordingly, the reliability of SSDs has become critical, and timely failure detection is essential to maintain data availability and system resilience. However, existing storage systems do not detect and notify of SSD failures promptly. This delay leads to prolonged service downtime and increased risk of data loss. By analyzing local and distributed storage systems, we identify three primary causes of delayed SSD failure detection: (1) loosely deterministic failure check, (2) fixed command timeout, and (3) delayed failure notification. First, the Linux kernel employs a passive, timeout-based failure check. This loosely deterministic mechanism often results in untimely or inaccurate detection. Second, the kernel uses a fixed timeout that fails to accommodate differences across SSD models and command types. Third, intermediate layers between the device driver and applications, such as RAID, file systems, and network layers, introduce further delays in failure notification due to their own failure-handling policies. To address these issues, in this article, we propose RL-Watchdog (RLW). It accelerates SSD failure detection in both local and distributed storage systems. RLW is organized into four key components: lightweight watchdog (LWW), reinforcement learning-based timeout predictor (RLTP), fast failure notification (FFN), and extended FFN (eFFN). First, LWW periodically checks SSD liveness using a lightweight special command to ensure deterministic failure detection. Second, RLTP predicts command latency by taking in-flight I/O count, write IOPS, and average write size as input features. It enables adaptation to different SSD models and command types. Third, FFN bypasses intermediate layers and directly notifies the application layer of SSD failure, ensuring rapid failure notification. Finally, for the distributed storage system, eFFN promptly delivers SSD failure notifications from storage nodes across the network, reducing failure detection time and minimizing data loss at the application level on client nodes. We implement RLW in the Linux kernel and evaluate it in both single-node and multi-node environments. Compared to the Linux kernel failure detection scheme with a fixed 1-second timeout, RLW reduces data loss by up to 96.7% and 44.6% in single-node and multi-node environments, respectively.

키워드