Training Pipeline Overview

Training Pipeline Overview An architecture diagram generated by Archify. S3 Data Lake · Data Preparation S3 Data Lake Data Loader Pod · Data Preparation Data Loader Pod Preprocessing Job · Data Preparation Preprocessing Job Parameter Server · algorithm-specific option · Distributed Training Parameter Server algorithm-specific option AllReduce Communication · collectives such as DDP · Distributed Training AllReduce Communication collectives such as DDP Worker Pods · Distributed Training · Distributed Training Worker Pods Distributed Training Job Scheduler · Distributed Training Job Scheduler Checkpoint store · selected FS / object path · Checkpointing Checkpoint store selected FS / object path Save / restore checks · Checkpointing Save / restore checks Evaluation Job · Model Evaluation Evaluation Job Metrics Collection · Model Evaluation Metrics Collection Model Registry · Model Evaluation Model Registry load submit run when applicable gradient AllReduce save checkpoint evaluate register Data Preparation Distributed Training Checkpointing Model Evaluation Legend Backend Database Message bus

Recovery unit

  • • model + optimizer + scheduler + RNG + data cursor + shard state
  • • Atomic local replacement differs from remote durability.