DeepSeek open sources and releases 3FS, a high-speed parallel file system optimized for AI data access

ChainCatcher news, according to DeepSeek's announcement, on the fifth day of Open Source Week, its Fire-Flyer file system (3FS) has officially been open-sourced. As a high-performance parallel file system, 3FS can fully utilize modern SSDs and RDMA networks to achieve high-speed data access, enhancing the efficiency of AI model training and inference.

Key performance indicators of 3FS:

• Achieved a total read throughput of 6.6 TiB/s in a 180-node cluster
• Reached a throughput of 3.66 TiB/min in a 25-node GraySort benchmark test
• Single-node KVCache query peak throughput exceeds 40+ GiB/s

3FS adopts a separated architecture, supporting data preprocessing, dataset loading, checkpoint storage/recovery, embedded vector search, and inference KVCache queries, with strong consistency semantics. DeepSeek also launched the Smallpond data processing framework to further optimize 3FS's data management capabilities.