Kubernetes Cluster Backup and Recovery Process

Kubernetes Cluster Backup and Recovery Process An architecture diagram generated by Archify. Set Backup Schedule · CronJob · Backup Process Set Backup Schedule CronJob Create etcd Snapshot · etcdctl snapshot save · Backup Process Create etcd Snapshot etcdctl snapshot save Backup Resource YAMLs · kubectl get -o yaml · Backup Process Backup Resource YAMLs kubectl get -o yaml Backup Storage · /backup · Backup Process Backup Storage /backup Disaster Occurs · Recovery Process Disaster Occurs Restore etcd · etcdctl snapshot restore · Recovery Process Restore etcd etcdctl snapshot restore Restart Kubernetes Services · systemctl start · Recovery Process Restart Kubernetes Services systemctl start Validate Cluster · check kubectl get nodes / pods · Recovery Process Validate Cluster check kubectl get nodes / pods Restore Resources · Recovery Process Restore Resources etcd snapshot resource YAMLs Backup Process Recovery Process Legend Backend Database External

Backup

  • • etcd holds all cluster state, so regular backups are essential
  • • Create snapshots with etcdctl snapshot save; verify with snapshot status
  • • Save every namespace and cluster-scoped resource as YAML
  • • Automated daily at midnight with a CronJob

Recovery

  • • Stop kubelet, kube-apiserver and the other services first
  • • Back up the existing /var/lib/etcd, then run snapshot restore
  • • Restart etcd and the Kubernetes services, then verify cluster status