Kubeflow Trainer Reconciliation Flow

Kubeflow Trainer Reconciliation Flow An architecture diagram generated by Archify. TrainJob · script · args · worker count · Architecture component TrainJob script · args · worker count ClusterTrainingRuntime · image · launch mechanics · Architecture component ClusterTrainingRuntime image · launch mechanics Trainer Controller · reconciliation loop · Architecture component Trainer Controller reconciliation loop TrainJob.status · conditions / optional trainerStatus · Architecture component TrainJob.status conditions / optional trainerStatus JobSet · optional PodGroup policy · Architecture component JobSet optional PodGroup policy Training Pods / processes · Pod / process / EC2 counts differ · Architecture component Training Pods / processes Pod / process / EC2 counts differ Headless Service · peer DNS names · Architecture component Headless Service peer DNS names references watched by watched by creates discover peers opt-in status reports status Legend Input resources Controller / JobSet Status store Worker Pods Peer discovery

TrainJob and runtimes

  • • TrainJob: run settings; runtime: reusable template and ML policy
  • • Distinguish 2.2 RuntimePatches from 2.3 CRD/hook changes.

Conditional capabilities

  • • Default Torch runtime has no podGroupPolicy.
  • • TrainJobStatus alpha gate defaults false.
  • • Sequential node creation differs from synchronous process rendezvous.