Observability Overview

Last reviewed: September 12, 2026.

Introduction

In modern distributed systems, especially Kubernetes-based microservices architectures, the ability to observe and understand the internal state of systems from external outputs is essential. This is called Observability.

Observability vs Monitoring

Monitoring is the activity of collecting and analyzing system behavior and responding to changes. Observability describes how well outputs let you understand internal state. They work together: monitoring can include logs, traces and diagnostic queries.

Aspect	Monitoring	Observability
Focus	Ongoing assessment of health and user impact	Visibility sufficient to explain system behavior
Practice	SLOs, dashboards, alerts and investigation	Instrumentation, context, correlation and exploration
Questions	What changed, and why is it failing?	Is there enough evidence to answer those questions?
Data	Metrics, logs, traces and other useful signals	Quality, coverage and connections between those signals
Complexity	Useful in simple and distributed systems	Designed around the system and operational goals

The Three Pillars of Observability

Logs, metrics and traces are three widely used signals. They are not an exhaustive definition of observability. Profiles can also describe code-level resource use; signal maturity differs in OpenTelemetry, where profile support remains under development.

1. Logs

Logs are records of individual events occurring in a system.

Characteristics:

• Records of discrete events; tamper resistance and retention guarantees depend on storage configuration
• Include timestamps and context information
• Structured (JSON) or unstructured format
• Useful for debugging and auditing, with deliberate event coverage, access controls and retention

Use Cases:

• Error and exception tracking
• Security auditing
• Compliance
• Detailed debugging

Tool roles: Loki, Elasticsearch/OpenSearch and CloudWatch Logs provide storage/query backends; Fluent Bit collects and forwards records.

2. Metrics

Metrics are numeric measurements over time.

Characteristics:

• Stored as time series data
• Support aggregation and mathematical operations
• Efficient storage when cardinality and collection volume are controlled
• Suitable for trend analysis

Common Prometheus Metric Types:

• Counter: Cumulative increasing values that can reset to zero, for example on restart (e.g., request count)
• Gauge: Current measurements that can increase or decrease (e.g., memory usage)
• Histogram: Distributions of observations (e.g., response time); classic and native histograms differ in representation and query syntax
• Summary: Observation count/sum and, depending on the implementation, precomputed quantiles; averaging instance quantiles does not produce a valid fleet-wide quantile

Tools: Prometheus, VictoriaMetrics, CloudWatch Metrics, Datadog

3. Traces

Traces describe the observed path of related work through spans. Instrumentation gaps, sampling, propagation failures and data loss can leave a partial trace.

Characteristics:

• Visualize request flow between services
• Measure latency at each step
• Identify bottlenecks
• Dependency analysis

Components:

• Trace: Related spans connected by a common TraceID
• Span: A single unit of work
• SpanContext: TraceID, SpanID, trace flags and tracestate used to propagate tracing context

Tools: Tempo, Jaeger, X-Ray, Zipkin, Datadog APM

Correlation Between the Three Pillars

Signals can be correlated when instrumentation, collection and backend links are configured. Their presence alone does not automatically connect every event.

An HTTP request fans out from an API gateway through user, order, and payment services; each service emits logs, metrics, and traces tagged with one shared trace ID, which links in both directions to a metric exemplar and to correlated log entries so you can pivot between the three pillars.Open full screen ↗

The figure abbreviates IDs for illustration. Use the valid full-width IDs shown below in real data. Metric correlation uses exemplar metadata; regular metric labels should remain bounded.

Trace-to-Log Correlation

Record TraceID and SpanID when an active trace context exists to link emitted log events to spans. Background or uninstrumented logs may have no such IDs. This is an application JSON example, not a complete OTLP request payload.

{
  "timestamp": "2025-02-15T10:30:00Z",
  "level": "ERROR",
  "message": "Payment processing failed",
  "traceId": "4bf92f3577b34da6a3ce929d0e0e4736",
  "spanId": "00f067aa0ba902b7",
  "service": "payment-service"
}

W3C-format TraceIDs contain 32 hexadecimal characters and SpanIDs contain 16; all-zero IDs are invalid. The JSON retains its illustrative historical event timestamp.

Metric-to-Trace Correlation (Exemplars)

An exemplar is separate metadata linking a particular observation to a trace. A per-request TraceID on ordinary metric series would create high cardinality, so use exemplars or log fields. The following is a complete classic-histogram OpenMetrics text example with illustrative measurements.

# HELP http_request_duration_seconds Observed HTTP request duration.
# TYPE http_request_duration_seconds histogram
# UNIT http_request_duration_seconds seconds
http_request_duration_seconds_bucket{le="0.5"} 1000 # {trace_id="4bf92f3577b34da6a3ce929d0e0e4736"} 0.42
http_request_duration_seconds_bucket{le="+Inf"} 1000
http_request_duration_seconds_sum 123.4
http_request_duration_seconds_count 1000
# EOF

An exemplar requires both a label set and an observation value (0.42); its timestamp is optional. Collection/storage must support and retain exemplars, and data-source links must map the trace_id label to the trace backend. The referenced trace must also have been sampled and retained.

OpenTelemetry and Standardization

OpenTelemetry (OTel) provides vendor-neutral APIs, SDKs, instrumentation and Collector tooling. Supported signals and automatic instrumentation vary by language, framework and component version.

Applications in any of four languages instrument via auto- or manual instrumentation, send data through the OpenTelemetry collector's receive, process, and export stages, and land in one of five observability backends.Open full screen ↗

This is one deployment pattern. Applications can also export directly to a supported backend. Collector receivers, processors and exporters must be configured for the chosen signals; component maturity varies.

Benefits of OpenTelemetry:

• Vendor-neutral standard
• Support for multiple language SDKs
• Auto-instrumentation capabilities
• Multi-backend support
• Active community

Observability Strategy for EKS Environments

Strategies for implementing effective observability in Amazon EKS:

1. Layer-based Observability

Infrastructure, Kubernetes, and application layers each feed their signals into a matching observability tool: infrastructure metrics go to CloudWatch, cluster and business metrics go to Prometheus/Grafana, and application traces and logs go to Tempo/X-Ray and Loki.Open full screen ↗

These are example signal paths, not exclusive tool-to-layer assignments. CloudWatch and other backends can cover multiple layers. Node-level collectors require the access and deployment model supported by the selected EKS compute mode.

2. Example Tool Choices

Function	Self-hosted Examples	AWS Managed Examples	Commercial Platforms
Metrics	Prometheus, VictoriaMetrics	CloudWatch, AMP	Datadog, New Relic
Logs	Loki, Elasticsearch/OpenSearch	CloudWatch Logs	Splunk, Datadog
Traces	Tempo, Jaeger	X-Ray	Datadog APM, Dynatrace
Visualization	Grafana	CloudWatch Dashboards	Datadog, Dynatrace

3. Cost Optimization Strategies

• Sampling: Balance retained trace volume against investigation coverage; tail sampling still receives and buffers data before deciding, so it does not remove every upstream cost
• Retention Policies: Optimize data retention periods
• Tiered Storage: Check the backend's supported storage tiers, query paths and required retention
• Aggregation and Cardinality: Control collection volume and label dimensions while assessing the diagnostic detail lost through aggregation

Observability Maturity Model

Use this practical planning model to review operational goals. It is not a product-purchase sequence or a universal certification scale. Assess signal coverage, investigation speed, actionable alerts and cost; introduce automated analysis where its behavior has been evaluated.

Review Area	Capability to Assess	Example Tools
Basic collection	Reliable coverage of needed service/platform signals	kubectl logs, CloudWatch
Centralization	Appropriate retention, access controls and querying	Loki, Prometheus, Grafana
Correlation	Links through request, service and deployment context	Tempo, exemplars, TraceID
Optional automation	Analysis assistance with tested false positives, gaps and response behavior	Datadog Watchdog, Dynatrace Intelligence

Section Guide

This observability section is organized as follows:

Logging

Tools and strategies for log collection, storage, and analysis:

• Loki: Log aggregation and query backend
• Fluent Bit: High-performance log collector
• CloudWatch Logs: AWS native logging

Metrics

Time series metric collection and analysis:

• Prometheus: Industry standard metrics system
• VictoriaMetrics: Metrics storage and query backend
• CloudWatch Metrics: AWS native metrics

Tracing

Distributed tracing and request flow analysis:

• Tempo: Grafana's distributed tracing backend
• X-Ray: AWS native distributed tracing
• OpenTelemetry: Standardized instrumentation
• Dynatrace: AI-powered APM

Grafana (Dashboards)

Unified visualization and dashboards:

• Data source integration
• Dashboard design patterns
• Alert configuration

Alerting

Actionable alerts, routing and response integrations.

Observability Optimization

Collection volume, cardinality, retention and operating cost.

Integrated Labs

Stepwise infrastructure, stack, application, load and tracing exercises.

Getting Started

Start with service goals, investigation questions and existing platform/collector capabilities. Adapt this example sequence to those requirements:

1. Set up metrics: Select Prometheus, VictoriaMetrics or an appropriate managed backend
2. Set up log collection: Deploy Loki and Fluent Bit
3. Set up tracing: Deploy Tempo or X-Ray
4. Visualization: Connect the needed data sources and dashboards in the chosen platform
5. Correlation: Configure TraceID-based linking

References

• OpenTelemetry signals
• OpenTelemetry Collector
• OpenTelemetry log data model
• W3C Trace Context
• Prometheus metric types
• OpenMetrics specification
• OpenTelemetry sampling
• Grafana OpenTelemetry documentation
• Amazon EKS monitoring and logging
• AWS Observability Best Practices
• SRE Workbook — Monitoring
• Datadog Watchdog
• Dynatrace Intelligence