3FS (Fire-Flyer File System)
A high-performance distributed file system designed specifically for AI training and inference workloads, offering disaggregated architecture, strong consistency, and familiar file interfaces.
Advanced Storage Technology
Key Features of 3FS
3FS combines modern storage technologies with innovative design principles to deliver exceptional performance for AI workloads.
- Disaggregated Architecture
Combines the throughput of thousands of SSDs with the network bandwidth of hundreds of storage nodes, allowing applications to access storage resources in a location-agnostic manner.
- Strong Consistency
Implements Chain Replication with Apportioned Queries (CRAQ) to provide strong consistency guarantees, making application code simple and easy to reason about.
- Familiar File Interface
Developed a stateless metadata service backed by a transactional key-value store (like FoundationDB), providing a familiar file interface without requiring learning new storage APIs.
- High Performance
Achieves peak throughput of approximately 6.6 TiB/s in read stress tests on a 180-node cluster, with efficient garbage collection operations.
- RDMA Networking
Leverages high-performance RDMA networking for efficient data transfer between storage and compute nodes, minimizing latency and maximizing throughput.
- Efficient KV Cache
Provides a cost-effective alternative to DRAM caching with high throughput and larger capacity, ideal for AI inference workloads.
System Design
3FS Architecture
3FS employs a modular design with separate components for storage, metadata management, client interfaces, and replication.
Storage Layer
The storage layer handles the physical storage and retrieval of data blocks. It leverages modern SSDs and RDMA networking to provide high-throughput, low-latency access to data. The storage nodes are organized in a disaggregated manner, allowing for independent scaling of storage and compute resources.
Metadata Service
The metadata service manages the file system structure and metadata. It is implemented as a stateless service backed by a transactional key-value store like FoundationDB. This design provides strong consistency guarantees while allowing the metadata service to scale horizontally to handle large file systems.
Benchmarks
3FS Performance
3FS delivers exceptional performance across various workloads, from high-throughput reads to complex data processing tasks.
Peak Read Throughput
6.6 TiB/s
Achieved in read stress tests on a 180-node cluster
GraySort Benchmark
3.66 TiB/min
Sorted 110.5 TiB of data in 30 minutes 14 seconds
KV Cache Performance
40 GiB/s
Peak read throughput for KV cache operations
Cluster Size
180 nodes
Scale tested with hundreds of storage nodes
Applications
3FS Use Cases
3FS is optimized for different stages of AI workflows, from data preparation to inference.
- Data Preparation
Organize outputs from data processing pipelines into hierarchical directory structures, efficiently managing large numbers of intermediate outputs.
- Data Loaders
Eliminate the need for prefetching or shuffling datasets by supporting random access to training samples across compute nodes.
- Checkpoint Saving
Support high-throughput parallel checkpoint saving for large-scale training, ensuring model progress is safely preserved.
- Inference KV Cache
Provide a cost-effective alternative to DRAM caching with high throughput and larger capacity for AI inference workloads.
Frequently Asked Questions
Can't find the answer you're looking for? Check out our GitHub repository or reach out to our team.
- What is 3FS?
- 3FS (Fire-Flyer File System) is a high-performance distributed file system designed specifically for AI training and inference workloads. It offers a disaggregated architecture, strong consistency guarantees, and familiar file interfaces.
- How does 3FS differ from other distributed file systems?
- 3FS is specifically optimized for AI workloads, with a focus on high throughput, strong consistency, and efficient handling of both small and large files. Its disaggregated architecture allows for independent scaling of storage and compute resources, and it leverages modern technologies like RDMA networking and transactional key-value stores for metadata management.
- What programming languages is 3FS implemented in?
- 3FS is primarily implemented in C++ (87.0%), with additional components in Rust (4.3%) and Python (2.1%). This combination provides both high performance and safety for critical system components.
- What kind of performance can I expect from 3FS?
- 3FS has demonstrated peak read throughput of approximately 6.6 TiB/s in stress tests on a 180-node cluster. In the GraySort benchmark, it sorted 110.5 TiB of data in 30 minutes 14 seconds, achieving an average throughput of 3.66 TiB/minute. For KV cache operations, it can reach peak read throughput of 40 GiB/s.
- What are the main components of 3FS?
- 3FS consists of several key components: a storage layer for handling data blocks, a metadata service for managing file system structure, client interfaces (including FUSE), and a replication system implementing the CRAQ protocol for strong consistency.
- Is 3FS open source?
- Yes, 3FS is available as an open-source project on GitHub at https://github.com/deepseek-ai/3FS. It is developed by DeepSeek AI to support high-performance AI infrastructure needs.