# Metrics Overview > **Last Updated**: September 13, 2026. Examples were checked locally with Prometheus 3.14.0 tooling; no cluster or cloud deployment was performed. ## Table of Contents - [Metrics Fundamentals](#metrics-fundamentals) - [Metric Types](#metric-types) - [Pull vs Push Model](#pull-vs-push-model) - [Cardinality and Metric Design](#cardinality-and-metric-design) - [Long-term Storage Requirements](#long-term-storage-requirements) - [Solution Comparison](#solution-comparison) - [Metrics Collection Architecture](#metrics-collection-architecture) ## Metrics Fundamentals Metrics describe system state and behavior numerically. A metric name and its complete label set identify a time series; each sample adds a value and a timestamp. Metrics support alerting, troubleshooting, capacity planning and performance analysis, but a sampled measurement does not preserve every individual event. For example, `http_requests_total` names a request counter; `method="GET"` and `status="200"` distinguish populations. The sample value is the accumulated count. Target labels such as `job` and `instance` are normally added during scraping. Timestamps depend on the format. An explicit timestamp in legacy Prometheus text exposition is Unix **milliseconds**, whereas OpenMetrics uses Unix **seconds**. Exporters normally omit explicit sample timestamps so Prometheus assigns the scrape time. Do not treat one unit as universal across telemetry protocols. ### Naming and units | Example | Interpretation | |---|---| | `http_requests_total` | Counter; `_total` indicates a cumulative count, not a physical unit | | `http_request_duration_seconds` | Duration in a base unit | | `node_memory_MemAvailable_bytes` | Existing node-exporter metric; retain its published spelling | | `requests` | Too little context for a new application metric | | `httpRequestDurationMs` | Does contain a unit, but uses camelCase and milliseconds rather than the usual Prometheus naming/base-unit convention | For new metrics, prefer descriptive prefixes, lowercase words separated by underscores, and units such as `_seconds` or `_bytes`. A naming convention does not authorize renaming an exporter's established API. The following `text` blocks are synthetic **Prometheus text exposition**, not YAML. Query expressions are separate `promql` blocks. Query selectors assume the shown scrape-job names; adapt them to your actual target labels. ## Metric Types Prometheus client libraries commonly expose Counter, Gauge, Histogram and Summary. Select the type from the meaning of the measurement, not merely from which query happens to accept its samples. ### 1. Counter A counter accumulates nonnegative increments, such as requests, errors or completed tasks. It can reset when the measured process/state is recreated; not every exporter restart necessarily resets the underlying counter. ```text # TYPE http_requests_total counter http_requests_total{method="GET",endpoint="/api/users",status="200"} 12345 http_requests_total{method="POST",endpoint="/api/users",status="500"} 23 ``` Per-series rate, service-wide rate and estimated increase: ```promql rate(http_requests_total{job="example-app"}[5m]) ``` ```promql sum(rate(http_requests_total{job="example-app"}[5m])) ``` ```promql increase(http_requests_total{job="example-app"}[1h]) ``` `rate()` accounts for observed counter resets and extrapolates over the requested window. It cannot recover increments lost between observations. `increase()` can therefore return a fractional estimate even for an integer counter. **Apply `rate()` before aggregating** so a reset in one instance is not hidden by growth in another. ### 2. Gauge A gauge represents current state and can rise or fall. These values illustrate real node-exporter and kube-state-metrics names alongside an application-defined temperature metric. ```text # TYPE node_memory_MemAvailable_bytes gauge node_memory_MemAvailable_bytes 8589934592 # TYPE node_memory_MemTotal_bytes gauge node_memory_MemTotal_bytes 17179869184 # TYPE kube_pod_status_ready gauge kube_pod_status_ready{namespace="example-app",pod="example-0",uid="00000000-0000-4000-8000-000000000001",condition="true"} 1 # TYPE temperature_celsius gauge temperature_celsius{location="datacenter-1"} 23.5 ``` ```promql 100 * (1 - node_memory_MemAvailable_bytes{job="node-exporter"} / node_memory_MemTotal_bytes{job="node-exporter"}) ``` ```promql max_over_time(temperature_celsius{job="example-app"}[1h]) ``` The memory expression is the share not reported as `MemAvailable`; it is not an application's resident-memory measurement. Pod readiness uses a particular `condition` series: a value of one on `condition="false"` has a different meaning from one on `condition="true"`. ### 3. Histogram A **classic histogram** counts observations into cumulative buckets in the instrumented application/exporter. Prometheus calculates quantiles later. `le` is an inclusive upper bound, and the `+Inf` bucket equals `_count`. ```text # TYPE http_request_duration_seconds histogram http_request_duration_seconds_bucket{le="0.005"} 24054 http_request_duration_seconds_bucket{le="0.01"} 33444 http_request_duration_seconds_bucket{le="0.025"} 100392 http_request_duration_seconds_bucket{le="0.05"} 129389 http_request_duration_seconds_bucket{le="0.1"} 133988 http_request_duration_seconds_bucket{le="0.25"} 144320 http_request_duration_seconds_bucket{le="+Inf"} 144320 http_request_duration_seconds_sum 4800.8625 http_request_duration_seconds_count 144320 ``` This is an illustrative distribution, not a benchmark. Its 144,320 observations total **4,800.8625 seconds**. The old sum of 53.42 seconds was inconsistent with the displayed bucket counts, which imply a lower bound above 2,704 seconds. The finite 0.25-second bucket also prevents the example's p95 from falling only in the unbounded bucket. Fleet-wide p95 and mean for matching bucket layouts: ```promql histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket{job="example-app"}[5m]))) ``` ```promql sum(rate(http_request_duration_seconds_sum{job="example-app"}[5m])) / sum(rate(http_request_duration_seconds_count{job="example-app"}[5m])) ``` Keep `le` when aggregating classic buckets. Quantiles interpolate within buckets; their accuracy depends on the distribution and bucket resolution. Changing classic bucket boundaries is an instrumentation change, not a query-only edit. Native histograms represent the distribution differently. Current Prometheus guidance prefers them when the client, scrape protocol and storage/query pipeline support them. Verify compatibility and configuration through the entire path; the classic examples here are not native-histogram wire output. ### 4. Summary A summary may calculate configured quantiles over a client-side window. Those values are generally **approximations with algorithm/window-dependent error**, not exact quantiles. Library support differs; a Summary implementation may expose only sum/count. ```text # TYPE rpc_request_duration_seconds summary rpc_request_duration_seconds{quantile="0.5"} 0.052 rpc_request_duration_seconds{quantile="0.9"} 0.089 rpc_request_duration_seconds{quantile="0.99"} 0.245 rpc_request_duration_seconds_sum 29969.50 rpc_request_duration_seconds_count 562887 ``` ```promql rpc_request_duration_seconds{job="example-app",quantile="0.99"} ``` ```promql sum(rate(rpc_request_duration_seconds_sum{job="example-app"}[5m])) / sum(rate(rpc_request_duration_seconds_count{job="example-app"}[5m])) ``` The first expression returns each matching instance's reported p99. Averaging or summing those p99 values does not produce a fleet p99. The second expression legitimately aggregates nonnegative-duration `_sum` and `_count` rates to calculate a fleet mean. | Question | Classic Histogram | Summary with quantiles | |---|---|---| | Where is the distribution processed? | Buckets at instrumentation; quantile at query time | Quantile at instrumentation | | Can instances be combined? | Compatible buckets can be aggregated | Quantiles cannot; sum/count can | | Error depends on | Bucket resolution and observations | Client algorithm, objective and time window | | Can a different percentile/window be queried later? | From retained bucket samples | Not from only the precomputed quantile | For zero traffic the mean can be `NaN`; absent series can produce an empty result. Neither should silently become evidence of healthy traffic. ## Pull vs Push Model ![Pull collection initiates requests from the collector; push collection initiates requests from the producer.](https://raw.githubusercontent.com/Atom-oh/kubernetes-docs/main/en/.gitbook/assets/en-observability-metrics-readme-0.png) [Interactive diagram](https://www.atomai.click/kubernetes-docs/archmaps/en-observability-metrics-readme-0.html) The figure illustrates connection direction. Real pipelines can mix both models: an agent can scrape endpoints and forward the results. Vendor names do not imply that every integration uses one model. ### Pull and Kubernetes discovery Pull collection centrally controls targets and intervals and makes endpoints easy to inspect. The collector needs outbound connectivity and the target needs permitted inbound access, with routing, TLS and authorization configured. NAT does not automatically make a target reachable. Prometheus `up` reports scrape success; it is not the application's availability SLO. This Prometheus configuration fragment selects **Running Pods in `example-app` with an opted-in, named TCP `metrics` container port**. It uses the discovered address rather than rewriting it with an IPv4-only regular expression. ```yaml # pod-scrape.yaml scrape_configs: - job_name: example-app kubernetes_sd_configs: - role: pod namespaces: names: - example-app relabel_configs: - source_labels: - __meta_kubernetes_pod_annotation_prometheus_io_scrape action: keep regex: 'true' - source_labels: - __meta_kubernetes_pod_phase action: keep regex: Running - source_labels: - __meta_kubernetes_pod_container_port_name action: keep regex: metrics - source_labels: - __meta_kubernetes_pod_container_port_protocol action: keep regex: TCP - source_labels: - __meta_kubernetes_pod_annotation_prometheus_io_path action: replace target_label: __metrics_path__ regex: (.+) - source_labels: - __meta_kubernetes_namespace target_label: namespace - source_labels: - __meta_kubernetes_pod_name target_label: pod ``` Prerequisites: the Pod annotation `prometheus.io/scrape: "true"`, a declared port named `metrics`, an optional `prometheus.io/path`, and Prometheus Kubernetes API credentials/RBAC to discover these Pods. The endpoint must actually serve metrics on that port/path. This is a configuration fragment, not a cluster installation or proof of reachability. ### Push and service-level batch jobs Push can fit producers with outbound access, including some short-lived workloads. It still requires receiver capacity, authentication, timeout/retry handling and a way to detect missing producers. A successful receiver health check is not proof that a batch job ran. Prometheus recommends Pushgateway for limited **service-level batch** use cases, not as the default for every short-lived Pod. Pushed groups do not expire automatically. Avoid per-Pod `HOSTNAME` grouping that leaves abandoned groups; define stable job ownership and an explicit retirement/cleanup process. The following is a **post-success integration fragment** for one logical batch, not a runnable Kubernetes Job. It assumes a reachable, authorized Pushgateway, `sh`, `awk` and `curl`. The batch supplies its measured duration, processed-record count and original completion timestamp. If delivery is retried, retain that original timestamp. Add environment-appropriate TLS/authentication without embedding secrets in examples. ```sh set -eu : "${PUSHGATEWAY_URL:?Set the reachable authorized Pushgateway base URL}" : "${DURATION_SECONDS:?Set the measured duration of the successful batch}" : "${RECORDS_PROCESSED:?Set the number of records processed by that batch}" : "${COMPLETED_AT_SECONDS:?Set its original Unix completion time in seconds}" # Reject nonnumeric metric values before sending anything. awk -v n="$DURATION_SECONDS" 'BEGIN { exit !(n ~ /^[0-9]+([.][0-9]+)?$/) }' case "$RECORDS_PROCESSED" in *[!0-9]*|'') exit 2;; esac case "$COMPLETED_AT_SECONDS" in *[!0-9]*|'') exit 2;; esac cat <