Amazon FSx for Lustre CSI Architecture

Amazon FSx for Lustre CSI Architecture An architecture diagram generated by Archify. ML training · data processing pods · Node 1 · Amazon EKS cluster ML training · data processing pods Node 1 FSx CSI driver · Node 1 DaemonSet · Amazon EKS cluster FSx CSI driver Node 1 DaemonSet ML inference · analysis pods · Node 2 · Amazon EKS cluster ML inference · analysis pods Node 2 FSx CSI driver · Node 2 DaemonSet · Amazon EKS cluster FSx CSI driver Node 2 DaemonSet FSx for Lustre · High-performance parallel FS · Architecture component FSx for Lustre High-performance parallel FS Amazon S3 · Data repository · Architecture component Amazon S3 Data repository Mounts volume Mounts volume Lustre client Data sync Amazon EKS cluster Legend Frontend Backend Database

Performance profile

  • • Choose throughput from 125 MB/s up to 1,000 MB/s per TiB of provisioned capacity
  • • Aggregate throughput scales close to linearly when thousands of clients read in parallel
  • • Metadata IOPS can be provisioned separately

Deployment types

  • • Scratch is for temporary high-performance work: no replication, lower cost
  • • Persistent replicates within the AZ, for longer retention and retryable workloads
  • • Scratch fits one-off training runs; persistent fits repeated training pipelines

S3 integration and use cases

  • • A data repository association lazy-loads S3 objects as files
  • • Export results back to S3 to keep training artifacts durably
  • • Use it for throughput-bound work: HPC simulation, large model training, genomics