Distributed Training Strategies

Distributed Training Strategies An architecture diagram generated by Archify. Data Shards 1-3 · per-shard input · Data Parallelism Data Shards 1-3 per-shard input Model Replicas 1-3 · basic DDP replica example · Data Parallelism Model Replicas 1-3 basic DDP replica example Gradient Sync · AllReduce · Data Parallelism Gradient Sync AllReduce Single Layer · attention / FFN · Tensor Parallelism Single Layer attention / FFN GPU 1 / GPU 2 · selected tensor partition · Tensor Parallelism GPU 1 / GPU 2 selected tensor partition Combine Results · intra-layer communication · Tensor Parallelism Combine Results intra-layer communication Micro-batches · fed in sequence · Pipeline Parallelism Micro-batches fed in sequence Stage 1 · layers 1-4 · GPU 1 · Pipeline Parallelism Stage 1 layers 1-4 · GPU 1 Stage 2 → 3 · layers 5-12 · GPU 2-3 · Pipeline Parallelism Stage 2 → 3 layers 5-12 · GPU 2-3 Router / Gating · picks experts per token · Expert Parallelism (MoE) Router / Gating picks experts per token Experts 1-4 · GPU 1-4 · Expert Parallelism (MoE) Experts 1-4 GPU 1-4 Combined Output · merges expert outputs · Expert Parallelism (MoE) Combined Output merges expert outputs gradients tensor split partials activations routing outputs Data Parallelism Tensor Parallelism Pipeline Parallelism Expert Parallelism (MoE) Legend Compute step Input data Communication / combine step

Global batch calculation

  • • microbatch × accumulation × DP replicas
  • • TP8 × PP4 × DP2 =64ranks; batch1 ×accum32 ×DP2 =64samples
  • • Account separately for FSDP/ZeRO and activation memory.