Monitoring and Logging Architecture

Monitoring and Logging Architecture An architecture diagram generated by Archify. NVIDIA DCGM Exporter · GPU metrics · Metrics Collection NVIDIA DCGM Exporter GPU metrics Node Exporter · Node metrics · Metrics Collection Node Exporter Node metrics Kube State Metrics · Cluster object state · Metrics Collection Kube State Metrics Cluster object state Prometheus · Metric store · rule evaluation · Monitoring Stack Prometheus Metric store · rule evaluation Alertmanager · Alert routing · Monitoring Stack Alertmanager Alert routing Grafana · Visualization · Monitoring Stack Grafana Visualization Alerts · GPU · Performance · Cost · Architecture component Alerts GPU · Performance · Cost Dashboards · GPU · Training · Inference · Cost · Architecture component Dashboards GPU · Training · Inference · Cost Fluent Bit · CRI / multiline / app logs · Logging Stack Fluent Bit CRI / multiline / app logs CloudWatch Logs · /eks/ml-cluster/pods · Logging Stack CloudWatch Logs /eks/ml-cluster/pods Elasticsearch · Log index · Logging Stack Elasticsearch Log index Kibana · Log search · Logging Stack Kibana Log search metrics alerting rules query Metrics Collection Monitoring Stack Logging Stack Legend Frontend Backend Database Cloud External

Metrics

  • • ServiceMonitor selects Service labels and named ports.
  • • GPU utilization differs from model quality and HPA Resource metrics.

Logs

  • • Distinguish CRI framing from application JSON.
  • • Validate output plugins, permissions, buffers and sensitive-data handling.