How a Pipeline Run Flows Through the System

How a Pipeline Run Flows Through the System An architecture diagram generated by Archify. Python pipeline · @dsl.pipeline / @dsl.component · Local · KFP SDK Python pipeline @dsl.pipeline / @dsl.component KFP SDK Compiler · produces IR YAML · Local · KFP SDK KFP SDK Compiler produces IR YAML KFP API Server · stores pipeline, accepts Run · KFP backend (built on Argo Workflows) KFP API Server stores pipeline, accepts Run IR YAML → Argo Workflow · backend translation · KFP backend (built on Argo Workflows) IR YAML → Argo Workflow backend translation Argo Workflow Controller · sequence / create Pods · KFP backend (built on Argo Workflows) Argo Workflow Controller sequence / create Pods driver / launcher / work · K8s places Pods on nodes · KFP backend (built on Argo Workflows) driver / launcher / work K8s places Pods on nodes Object store · S3 / MinIO · artifacts · Architecture component Object store S3 / MinIO · artifacts ML Metadata (MLMD) · execution + artifact metadata · Architecture component ML Metadata (MLMD) execution + artifact metadata compile upload / create Run Workflow resource artifact read / write record metadata Local · KFP SDK KFP backend (built on Argo Workflows) Legend KFP SDK (local) KFP backend Storage Component execution

IR and execution

  • • Upload alone does not execute; backend compatibility still matters.
  • • Cache keys use artifact IDs and configuration, not file-byte hashes.

Storage and identity

  • • S3 is metered and needs root/provider/permission configuration.
  • • Inspect actual execution ServiceAccounts and API-server access.