How an Experiment Runs, End to End

How an Experiment Runs, End to End An architecture diagram generated by Archify. Experiment object created · objective · parameters · trialTemplate · Architecture component Experiment object created objective · parameters · trialTemplate Katib controller · reconciles the Experiment · Architecture component Katib controller reconciles the Experiment Suggestion service created · for the chosen algorithm · Architecture component Suggestion service created for the chosen algorithm Proposes hyperparameter set(s) · Trial concurrency, not GPU count · Architecture component Proposes hyperparameter set(s) Trial concurrency, not GPU count Trial 1 · training job · Architecture component Trial 1 training job Trial 2 · training job · Architecture component Trial 2 training job Trial N · training job · Architecture component Trial N training job Metrics through DB manager · pull collector / Push · Architecture component Metrics through DB manager pull collector / Push Controller terminal decision · Succeeded / Failed · Architecture component Controller terminal decision Succeeded / Failed Current best observation · currentOptimalTrial / conditions · Architecture component Current best observation currentOptimalTrial / conditions results feed back termination conditions Legend Experiment input Katib components Experiment status Trial training jobs Metrics reporting

Resource demand

  • • Concurrent Trials × Pods per Trial × resources per Pod + overhead
  • • maxTrialCount is neither successful-training count nor spending cap.

0.19.0 medianstop

  • • Unlike the documented median, code computes a mean of stored averages.
  • • Defaults: min_trials_required=3 / start_step=4