Alertmanager Clustering Architecture

Alertmanager Clustering Architecture An architecture diagram generated by Archify. Prometheus · Evaluates alert rules · Architecture component Prometheus Evaluates alert rules Alertmanager 1 · alertmanager-0 · sends after dedup · Alertmanager cluster (StatefulSet, replicas: 3) Alertmanager 1 alertmanager-0 · sends after dedup Alertmanager 2 · alertmanager-1 · Alertmanager cluster (StatefulSet, replicas: 3) Alertmanager 2 alertmanager-1 Alertmanager 3 · alertmanager-2 · Alertmanager cluster (StatefulSet, replicas: 3) Alertmanager 3 alertmanager-2 Receivers · Slack / PagerDuty / Email · Architecture component Receivers Slack / PagerDuty / Email alerts alerts alerts Gossip :9094 Gossip :9094 Gossip :9094 final notification Alertmanager cluster (StatefulSet, replicas: 3) Legend Backend External

Alert intake

  • • Register all three replicas under Prometheus alerting.alertmanagers
  • • Every replica receives the same alerts

Gossip cluster

  • • --cluster.listen-address=0.0.0.0:9094
  • • --cluster.peer links the three replicas to each other
  • • Syncs the notification log (nflog) and silences

Deduplication

  • • An alert another replica already sent is found in nflog and not resent
  • • If one replica fails, the others keep delivering