Distributed Training Architecture

Distributed Training Architecture An architecture diagram generated by Archify. Launcher Pod · mpirun · MPI Job Launcher Pod mpirun Worker Pods ×4 · nvidia.com/gpu · MPI Job Worker Pods ×4 nvidia.com/gpu NVIDIA NCCL · Collective communication · Communication Layer NVIDIA NCCL Collective communication AWS OFI NCCL · libfabric integration · Communication Layer AWS OFI NCCL libfabric integration Elastic Fabric Adapter · EFA · Communication Layer Elastic Fabric Adapter EFA FSx for Lustre · high-performance parallel FS · Shared Storage FSx for Lustre high-performance parallel FS Amazon S3 · training data · model artifacts · Shared Storage Amazon S3 training data · model artifacts Checkpoint Storage · model checkpoints · Shared Storage Checkpoint Storage model checkpoints launch processes gradient exchange write checkpoints configured import/export MPI Job Communication Layer Shared Storage Legend Backend Database Cloud Message bus

Communication path

  • • NCCL can use EFA through AWS OFI NCCL and libfabric.
  • • Validate devices, drivers, plugins, networking and framework configuration.

Recovery

  • • FSx attachment alone does not guarantee checkpoint retention or S3 export.