可观测性分析：Logs/Metrics/Traces 关联

支持版本: Loki 3.0+, Tempo 2.4+, Prometheus 2.50+, Grafana 10.0+ 最后更新: February 23, 2026

1. 关联策略

有效的可观测性需要将 logs（日志）、metrics（指标）和 traces（追踪）关联起来，以理解系统行为。本节介绍 EKS 环境中跨信号关联的架构与实现。

Trace ID 传播标准

分布式 trace 上下文传播有两个主要标准：

W3C TraceContext（推荐）

traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
             |  |                                |                |
             |  |                                |                └─ flags (sampled)
             |  |                                └─ parent span ID (16 hex chars)
             |  └─ trace ID (32 hex chars)
             └─ version

tracestate: vendor1=value1,vendor2=value2

B3 Headers（旧版/Zipkin）

X-B3-TraceId: 463ac35c9f6413ad48485a3953bb6124
X-B3-SpanId: 0020000000000001
X-B3-ParentSpanId: 0010000000000000
X-B3-Sampled: 1
X-B3-Flags: 0

# Single header format
b3: 463ac35c9f6413ad48485a3953bb6124-0020000000000001-1-0010000000000000

OpenTelemetry SDK 插桩

配置 OTEL SDK 以实现自动 trace 传播：

yaml

# otel-config.yaml for Kubernetes deployment
apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-config
  namespace: observability
data:
  config.yaml: |
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
            cors:
              allowed_origins:
                - "*"

    processors:
      batch:
        timeout: 1s
        send_batch_size: 1024

      resource:
        attributes:
          - key: k8s.cluster.name
            value: "production"
            action: upsert
          - key: deployment.environment
            value: "production"
            action: upsert

      # Add Kubernetes metadata
      k8sattributes:
        auth_type: "serviceAccount"
        passthrough: false
        extract:
          metadata:
            - k8s.namespace.name
            - k8s.deployment.name
            - k8s.pod.name
            - k8s.node.name
          labels:
            - tag_name: app
              key: app.kubernetes.io/name
            - tag_name: version
              key: app.kubernetes.io/version

    exporters:
      otlp/tempo:
        endpoint: tempo-distributor.observability:4317
        tls:
          insecure: true

      prometheusremotewrite:
        endpoint: http://prometheus:9090/api/v1/write

      loki:
        endpoint: http://loki-gateway.observability:3100/loki/api/v1/push
        labels:
          resource:
            k8s.namespace.name: "namespace"
            k8s.pod.name: "pod"
            service.name: "service"

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [batch, resource, k8sattributes]
          exporters: [otlp/tempo]
        metrics:
          receivers: [otlp]
          processors: [batch, resource]
          exporters: [prometheusremotewrite]
        logs:
          receivers: [otlp]
          processors: [batch, resource]
          exporters: [loki]

应用程序插桩示例

使用 OTEL 的 Python Flask 应用程序：

python

# app.py
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from opentelemetry.propagate import set_global_textmap
from opentelemetry.propagators.composite import CompositePropagator
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator
from opentelemetry.propagators.b3 import B3MultiFormat
import logging
import json_log_formatter

# Configure trace propagation (both W3C and B3 for compatibility)
set_global_textmap(CompositePropagator([
    TraceContextTextMapPropagator(),
    B3MultiFormat()
]))

# Configure tracer
trace.set_tracer_provider(TracerProvider())
otlp_exporter = OTLPSpanExporter(endpoint="otel-collector:4317", insecure=True)
trace.get_tracer_provider().add_span_processor(BatchSpanProcessor(otlp_exporter))

# Configure logging with trace correlation
class TraceIdFilter(logging.Filter):
    def filter(self, record):
        span = trace.get_current_span()
        if span.is_recording():
            ctx = span.get_span_context()
            record.trace_id = format(ctx.trace_id, '032x')
            record.span_id = format(ctx.span_id, '016x')
        else:
            record.trace_id = '0' * 32
            record.span_id = '0' * 16
        return True

# JSON formatter for structured logging
formatter = json_log_formatter.JSONFormatter()
handler = logging.StreamHandler()
handler.setFormatter(formatter)
handler.addFilter(TraceIdFilter())

logger = logging.getLogger('app')
logger.addHandler(handler)
logger.setLevel(logging.INFO)

# Flask application
from flask import Flask, request
app = Flask(__name__)
FlaskInstrumentor().instrument_app(app)
RequestsInstrumentor().instrument()

@app.route('/api/orders/<order_id>')
def get_order(order_id):
    logger.info('Processing order request', extra={
        'order_id': order_id,
        'method': request.method,
        'path': request.path
    })
    # Business logic here
    return {'order_id': order_id, 'status': 'completed'}

Exemplars：从 Metrics 到 Traces

Exemplars 将高基数 metric 样本链接到特定 traces：

yaml

# Prometheus configuration to enable exemplars
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'app-metrics'
    scrape_interval: 15s
    static_configs:
      - targets: ['app:8080']
    # Enable exemplar storage
    enable_http2: true

用于发出 exemplars 的应用程序代码：

// Go application with exemplars
import (
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promauto"
    "go.opentelemetry.io/otel/trace"
)

var requestDuration = promauto.NewHistogramVec(
    prometheus.HistogramOpts{
        Name:    "http_request_duration_seconds",
        Help:    "HTTP request duration in seconds",
        Buckets: prometheus.DefBuckets,
    },
    []string{"method", "path", "status"},
)

func recordMetric(ctx context.Context, method, path string, status int, duration float64) {
    span := trace.SpanFromContext(ctx)
    if span.SpanContext().IsSampled() {
        requestDuration.WithLabelValues(method, path, strconv.Itoa(status)).(prometheus.ExemplarObserver).ObserveWithExemplar(
            duration,
            prometheus.Labels{
                "traceID": span.SpanContext().TraceID().String(),
                "spanID":  span.SpanContext().SpanID().String(),
            },
        )
    } else {
        requestDuration.WithLabelValues(method, path, strconv.Itoa(status)).Observe(duration)
    }
}

关联架构图

                    ┌─────────────────────────────────────────────────────────────┐
                    │                     Application                              │
                    │  ┌──────────────────────────────────────────────────────┐   │
                    │  │  Request with TraceContext Header                     │   │
                    │  │  traceparent: 00-abc123...-def456...-01               │   │
                    │  └──────────────────────────────────────────────────────┘   │
                    │         │              │                │                    │
                    │         ▼              ▼                ▼                    │
                    │    ┌────────┐    ┌─────────┐     ┌────────────┐             │
                    │    │ Logs   │    │ Metrics │     │   Traces   │             │
                    │    │traceID │    │exemplar │     │  spans     │             │
                    │    └────┬───┘    └────┬────┘     └─────┬──────┘             │
                    └─────────┼─────────────┼───────────────┼─────────────────────┘
                              │             │               │
              ┌───────────────┼─────────────┼───────────────┼─────────────────────┐
              │ OTEL Collector│             │               │                      │
              │               ▼             ▼               ▼                      │
              │         ┌─────────────────────────────────────────┐               │
              │         │  Enrich with K8s metadata               │               │
              │         │  namespace, pod, node, service          │               │
              │         └─────────────────────────────────────────┘               │
              └───────────────┬─────────────┬───────────────┬─────────────────────┘
                              │             │               │
                    ┌─────────┼─────────────┼───────────────┼─────────────────────┐
                    │ Storage │             │               │                      │
                    │         ▼             ▼               ▼                      │
                    │    ┌────────┐    ┌─────────┐     ┌────────────┐             │
                    │    │  Loki  │    │Prometheus│    │   Tempo    │             │
                    │    │        │    │         │     │            │             │
                    │    └────┬───┘    └────┬────┘     └─────┬──────┘             │
                    └─────────┼─────────────┼───────────────┼─────────────────────┘
                              │             │               │
                    ┌─────────┼─────────────┼───────────────┼─────────────────────┐
                    │ Grafana │             │               │                      │
                    │         ▼             ▼               ▼                      │
                    │  ┌─────────────────────────────────────────────────────┐    │
                    │  │              Unified Query Interface                 │    │
                    │  │  Logs ──(traceID)──▶ Traces                         │    │
                    │  │  Metrics ──(exemplar)──▶ Traces                     │    │
                    │  │  Traces ──(labels)──▶ Logs                          │    │
                    │  └─────────────────────────────────────────────────────┘    │
                    └─────────────────────────────────────────────────────────────┘

关联工作流

请求到达，带有或不带有 trace 上下文
应用程序创建/继续 trace，并记录带 traceID 的日志
记录 metrics，其中 exemplar 包含 traceID
OTEL Collector 增强所有信号，添加 K8s metadata
Grafana 查询可以使用共享标识符在信号之间导航

2. Loki LogQL 分析

Loki 提供强大的查询语言（LogQL）用于日志分析。本节介绍 EKS 运维分析中的实用模式。

错误率计算

从日志流计算错误率：

logql

# Error rate per service (last 5 minutes)
sum(rate({namespace="production"} |= "error" [5m])) by (app)
/ sum(rate({namespace="production"} [5m])) by (app)

# HTTP 5xx error rate from structured logs
sum(rate({namespace="production"} | json | status_code >= 500 [5m])) by (service)
/ sum(rate({namespace="production"} | json | status_code > 0 [5m])) by (service)

# Error rate with severity label
sum(rate({namespace="production", level="error"} [5m])) by (app)

从日志中提取延迟

从日志消息中提取延迟 metrics：

logql

# Extract duration from JSON logs
{namespace="production", app="api-gateway"}
| json
| duration_ms > 1000
| line_format "{{.method}} {{.path}} took {{.duration_ms}}ms"

# Calculate latency percentiles from logs
quantile_over_time(0.99,
  {namespace="production"}
  | json
  | unwrap duration_ms [5m]
) by (service)

# Average latency per endpoint
avg_over_time(
  {namespace="production", app="api"}
  | json
  | unwrap response_time_ms [5m]
) by (path)

基于日志的告警规则

从日志模式创建告警：

yaml

# loki-alert-rules.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: loki-alerts
  namespace: observability
spec:
  groups:
    - name: loki.alerts
      rules:
        # High error rate in logs
        - alert: HighLogErrorRate
          expr: |
            sum(rate({namespace="production"} |= "error" [5m])) by (app)
            / sum(rate({namespace="production"} [5m])) by (app)
            > 0.05
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "High error rate in logs for {{ $labels.app }}"
            description: "Error rate is {{ $value | printf \"%.2f\" }}%"

        # Out of memory errors
        - alert: OutOfMemoryErrors
          expr: |
            sum(count_over_time({namespace="production"}
              |~ "OutOfMemoryError|OOMKilled|memory allocation failed" [15m]
            )) by (pod) > 0
          for: 1m
          labels:
            severity: critical
          annotations:
            summary: "OOM errors detected in {{ $labels.pod }}"

        # Database connection errors
        - alert: DatabaseConnectionErrors
          expr: |
            sum(rate({namespace="production"}
              |~ "connection refused|connection timed out|too many connections" [5m]
            )) by (app) > 1
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "Database connection errors in {{ $labels.app }}"

        # Authentication failures
        - alert: HighAuthFailureRate
          expr: |
            sum(rate({namespace="production"}
              | json
              | event_type="authentication_failed" [5m]
            )) by (app) > 10
          for: 5m
          labels:
            severity: warning
            category: security
          annotations:
            summary: "High authentication failure rate in {{ $labels.app }}"

Label 策略与基数

管理 label 基数以防止性能问题：

yaml

# promtail-config.yaml - Label extraction strategy
scrape_configs:
  - job_name: kubernetes-pods
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      # Keep only essential labels
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod
      - source_labels: [__meta_kubernetes_pod_label_app]
        target_label: app
      # Drop high-cardinality labels
      - action: labeldrop
        regex: __meta_kubernetes_pod_label_(pod-template-hash|controller-revision-hash)
    pipeline_stages:
      - json:
          expressions:
            level: level
            # Don't extract high-cardinality fields as labels
      - labels:
          level:
      # Keep request_id in log line, not as label
      - output:
          source: message

LogQL 模式匹配与解析

高级解析模式：

logql

# Parse unstructured Nginx logs
{app="nginx"}
| pattern `<ip> - - [<timestamp>] "<method> <path> <_>" <status> <bytes>`
| status >= 500

# Parse with regex
{app="api"}
| regexp `(?P<timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}) \[(?P<level>\w+)\] (?P<message>.*)`
| level = "ERROR"

# Parse JSON and filter
{namespace="production"}
| json
| line_format `{{.timestamp}} [{{.level}}] {{.message}}`
| level = "error"
| message =~ ".*timeout.*"

# Unpack nested JSON
{app="api-gateway"}
| json
| json request_body="request.body"
| request_body != ""

聚合查询

聚合日志数据用于分析：

logql

# Count errors by service over time
sum by (service) (count_over_time({namespace="production", level="error"} [1h]))

# Top 10 error messages
topk(10, sum by (message) (count_over_time(
  {namespace="production"}
  | json
  | level = "error" [24h]
)))

# Log volume by namespace
sum by (namespace) (bytes_over_time({job="kubernetes-pods"} [1h]))

# Rate of specific events
sum(rate({namespace="production"} |= "payment_processed" [5m])) by (app)

多行日志处理

配置多行日志解析：

yaml

# promtail-config.yaml - Multi-line configuration
scrape_configs:
  - job_name: java-apps
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        target_label: app
    pipeline_stages:
      # Java stack trace multi-line
      - multiline:
          firstline: '^\d{4}-\d{2}-\d{2}'
          max_wait_time: 3s
          max_lines: 128
      - regex:
          expression: '^(?P<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}) (?P<level>\w+) .*'
      - labels:
          level:

  - job_name: python-apps
    kubernetes_sd_configs:
      - role: pod
    pipeline_stages:
      # Python traceback multi-line
      - multiline:
          firstline: '^\d{4}-\d{2}-\d{2}|^Traceback'
          max_wait_time: 3s

完整 Loki 告警规则 YAML

yaml

apiVersion: 1
groups:
  - name: loki-application-alerts
    rules:
      - alert: ApplicationErrorSpike
        expr: |
          sum(rate({namespace="production"} |= "error" [5m])) by (app)
          > 1.5 * sum(rate({namespace="production"} |= "error" [1h])) by (app)
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Error spike detected in {{ $labels.app }}"

      - alert: SlowRequestsDetected
        expr: |
          avg_over_time(
            {namespace="production"}
            | json
            | unwrap response_time_ms [5m]
          ) by (service) > 5000
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Slow requests in {{ $labels.service }}"

      - alert: UnusualLogVolume
        expr: |
          sum(rate({namespace="production"} [5m])) by (app)
          > 3 * avg_over_time(sum(rate({namespace="production"} [5m])) by (app) [1d])
        for: 15m
        labels:
          severity: info
        annotations:
          summary: "Unusual log volume from {{ $labels.app }}"

      - alert: CriticalPatternDetected
        expr: |
          count_over_time({namespace="production"}
            |~ "FATAL|panic|segfault|core dumped" [5m]) > 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Critical error pattern detected"

      - alert: PodCrashLoopDetected
        expr: |
          count_over_time({namespace="production"}
            |= "Back-off restarting failed container" [10m]) > 3
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Pod crash loop detected"

3. Prometheus PromQL 模式

PromQL 为 metrics 分析提供强大的查询能力。本节介绍 EKS 运维的基本模式。

RPS 计算

计算每秒请求数：

promql

# Total RPS across all services
sum(rate(http_requests_total[5m]))

# RPS by service
sum(rate(http_requests_total[5m])) by (service)

# RPS by endpoint (be careful with cardinality)
sum(rate(http_requests_total[5m])) by (service, path)

# RPS increase compared to yesterday
sum(rate(http_requests_total[5m]))
- sum(rate(http_requests_total[5m] offset 1d))

错误率（RED 方法）

Rate、Errors、Duration——RED 方法：

promql

# Error rate (errors / total requests)
sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
/ sum(rate(http_requests_total[5m])) by (service)

# Error rate with threshold
(
  sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
  / sum(rate(http_requests_total[5m])) by (service)
) > 0.01

# Client error rate (4xx)
sum(rate(http_requests_total{status=~"4.."}[5m])) by (service)
/ sum(rate(http_requests_total[5m])) by (service)

# Success rate (inverse of error rate)
1 - (
  sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
  / sum(rate(http_requests_total[5m])) by (service)
)

延迟百分位数

从直方图计算延迟百分位数：

promql

# P50 latency
histogram_quantile(0.50,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)
)

# P95 latency
histogram_quantile(0.95,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)
)

# P99 latency
histogram_quantile(0.99,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)
)

# Apdex score (target: 500ms, tolerated: 2s)
(
  sum(rate(http_request_duration_seconds_bucket{le="0.5"}[5m])) by (service)
  + sum(rate(http_request_duration_seconds_bucket{le="2"}[5m])) by (service)
) / 2
/ sum(rate(http_request_duration_seconds_count[5m])) by (service)

Istio Service Mesh Metrics

查询 Istio 的 telemetry：

promql

# Request rate by source and destination
sum(rate(istio_requests_total[5m])) by (source_workload, destination_workload)

# Service error rate
sum(rate(istio_requests_total{response_code=~"5.."}[5m])) by (destination_service)
/ sum(rate(istio_requests_total[5m])) by (destination_service)

# P99 latency by service
histogram_quantile(0.99,
  sum(rate(istio_request_duration_milliseconds_bucket[5m])) by (le, destination_service)
)

# TCP connections
sum(istio_tcp_connections_opened_total) by (source_workload, destination_workload)
- sum(istio_tcp_connections_closed_total) by (source_workload, destination_workload)

# Request size
histogram_quantile(0.99,
  sum(rate(istio_request_bytes_bucket[5m])) by (le, destination_service)
)

通过 CloudWatch 获取 ALB Metrics

查询从 CloudWatch 导出的 ALB metrics：

promql

# ALB request count
sum(rate(aws_applicationelb_request_count_sum[5m])) by (load_balancer)

# ALB target response time
aws_applicationelb_target_response_time_average

# ALB 5xx errors
sum(rate(aws_applicationelb_httpcode_elb_5xx_count_sum[5m])) by (load_balancer)

# ALB healthy host count
aws_applicationelb_healthy_host_count_average

# ALB active connection count
aws_applicationelb_active_connection_count_sum

Amazon Managed Prometheus（AMP）模式

针对 AMP 优化的查询：

promql

# Use recording rules to reduce query complexity
# AMP has query limits, so pre-aggregate where possible

# Efficient aggregation
sum by (namespace) (
  rate(container_cpu_usage_seconds_total[5m])
)

# Avoid high-cardinality queries
# Bad: sum(rate(http_requests_total[5m])) by (pod, path, method, status)
# Good: sum(rate(http_requests_total[5m])) by (service, status_class)

# Use label_replace to reduce cardinality
sum by (service, status_class) (
  label_replace(
    rate(http_requests_total[5m]),
    "status_class", "${1}xx", "status", "([0-9]).*"
  )
)

Recording Rules

预先计算昂贵查询：

yaml

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: recording-rules
  namespace: monitoring
spec:
  groups:
    - name: http.recording.rules
      interval: 30s
      rules:
        # Request rate by service
        - record: service:http_requests:rate5m
          expr: sum(rate(http_requests_total[5m])) by (service)

        # Error rate by service
        - record: service:http_errors:rate5m
          expr: sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)

        # Error ratio by service
        - record: service:http_error_ratio:rate5m
          expr: |
            service:http_errors:rate5m
            / service:http_requests:rate5m

        # P50 latency by service
        - record: service:http_latency_p50:rate5m
          expr: |
            histogram_quantile(0.50,
              sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)
            )

        # P95 latency by service
        - record: service:http_latency_p95:rate5m
          expr: |
            histogram_quantile(0.95,
              sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)
            )

        # P99 latency by service
        - record: service:http_latency_p99:rate5m
          expr: |
            histogram_quantile(0.99,
              sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)
            )

    - name: kubernetes.recording.rules
      interval: 30s
      rules:
        # Node CPU utilization
        - record: node:cpu_utilization:rate5m
          expr: |
            100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

        # Node memory utilization
        - record: node:memory_utilization:ratio
          expr: |
            1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)

        # Pod CPU usage by namespace
        - record: namespace:pod_cpu:rate5m
          expr: |
            sum(rate(container_cpu_usage_seconds_total{container!=""}[5m])) by (namespace)

        # Pod memory usage by namespace
        - record: namespace:pod_memory:bytes
          expr: |
            sum(container_memory_working_set_bytes{container!=""}) by (namespace)

    - name: istio.recording.rules
      interval: 30s
      rules:
        # Service request rate
        - record: service:istio_requests:rate5m
          expr: |
            sum(rate(istio_requests_total[5m])) by (destination_service)

        # Service error rate
        - record: service:istio_errors:rate5m
          expr: |
            sum(rate(istio_requests_total{response_code=~"5.."}[5m])) by (destination_service)

        # Service P99 latency
        - record: service:istio_latency_p99:rate5m
          expr: |
            histogram_quantile(0.99,
              sum(rate(istio_request_duration_milliseconds_bucket[5m])) by (le, destination_service)
            )

4. Tempo TraceQL 分析

Tempo 的 TraceQL 提供类似 SQL 的语法，用于搜索和分析分布式 traces。

基本 TraceQL 语法

traceql

# Find all traces for a service
{ resource.service.name = "api-gateway" }

# Filter by span name
{ name = "HTTP GET" }

# Filter by attribute
{ span.http.status_code >= 500 }

# Combine filters
{ resource.service.name = "order-service" && span.http.status_code = 500 }

# Duration filter
{ resource.service.name = "payment-service" && duration > 1s }

# Find traces with specific error
{ status = error && span.error.message =~ ".*timeout.*" }

延迟分析

查找和分析慢 traces：

traceql

# Traces slower than 5 seconds
{ duration > 5s }

# Slow database queries
{ span.db.system = "postgresql" && duration > 500ms }

# Slow HTTP calls
{ span.http.method = "POST" && duration > 2s }

# Find the slowest spans in a trace
{ duration > 1s } | select(duration, name, resource.service.name)

# P99 latency traces
{ resource.service.name = "checkout" && duration > 2s } | quantile_over_time(duration, 0.99)

错误 Trace 搜索

查找和分析错误 traces：

traceql

# All error traces
{ status = error }

# Errors by service
{ resource.service.name = "inventory-service" && status = error }

# Specific error types
{ span.exception.type = "java.lang.NullPointerException" }

# HTTP errors
{ span.http.status_code >= 500 }

# gRPC errors
{ span.rpc.grpc.status_code != 0 }

# Database errors
{ span.db.system = "mysql" && status = error }

Service 依赖关系映射

分析 service 依赖关系：

traceql

# Find all downstream calls from a service
{ resource.service.name = "api-gateway" && kind = client }

# Find all upstream callers of a service
{ resource.service.name = "user-service" && kind = server }

# Cross-service calls
{
  resource.service.name = "order-service"
  && span.peer.service = "payment-service"
}

# External dependency calls
{ span.http.url =~ ".*external-api.com.*" }

Span 属性过滤

按各种 span 属性过滤：

traceql

# Kubernetes metadata
{ resource.k8s.namespace.name = "production" }
{ resource.k8s.pod.name =~ "api-.*" }
{ resource.k8s.node.name = "ip-10-0-1-100.ec2.internal" }

# HTTP attributes
{ span.http.method = "POST" && span.http.route = "/api/orders" }
{ span.http.request_content_length > 1000000 }

# Database attributes
{ span.db.statement =~ ".*SELECT.*users.*" }
{ span.db.operation = "INSERT" }

# Custom attributes
{ span.user.id = "user-123" }
{ span.order.total > 1000 }

结构查询（父子关系）

查询 trace 结构：

traceql

# Find child spans of a specific parent
{ name = "HTTP POST /checkout" } >> { span.db.system = "postgresql" }

# Find parent of slow database queries
{ span.db.system = "postgresql" && duration > 1s } << { }

# Multi-level ancestry
{ name = "api-gateway" } >> { name = "order-service" } >> { span.db.system = "postgresql" }

# Sibling spans (same parent)
{ name = "inventory-check" } ~ { name = "payment-process" }

# Find traces where DB query is child of HTTP call
{ span.http.method = "GET" } >> { span.db.operation = "SELECT" && duration > 500ms }

Trace 比较

跨时间或版本比较 traces：

traceql

# Compare latency between deployments (using resource attributes)
{ resource.service.version = "v2.0.0" && duration > 1s }
{ resource.service.version = "v1.9.0" && duration > 1s }

# Find anomalous traces (compare to baseline)
{
  resource.service.name = "checkout"
  && duration > 2s
  && span.http.route = "/api/checkout"
}

# Traces by environment
{ resource.deployment.environment = "canary" && status = error }

Service Graph 查询

分析 service 拓扑：

traceql

# Service graph metrics (via Tempo metrics generator)
# These generate Prometheus metrics from trace data

# Request rate between services
traces_service_graph_request_total

# Error rate between services
traces_service_graph_request_failed_total

# Latency between services
traces_service_graph_request_server_seconds_bucket

Tempo Metrics Generator 配置：

yaml

# tempo-config.yaml
metrics_generator:
  registry:
    external_labels:
      source: tempo
      cluster: production
  storage:
    path: /var/tempo/generator/wal
    remote_write:
      - url: http://prometheus:9090/api/v1/write
        send_exemplars: true
  traces_storage:
    path: /var/tempo/generator/traces
  processor:
    service_graphs:
      dimensions:
        - k8s.namespace.name
        - k8s.deployment.name
      histogram_buckets: [0.01, 0.05, 0.1, 0.5, 1, 2, 5]
      max_items: 10000
      wait: 10s
      workers: 10
    span_metrics:
      dimensions:
        - service.name
        - span.name
        - http.method
        - http.status_code
      histogram_buckets: [0.002, 0.004, 0.008, 0.016, 0.032, 0.064, 0.128, 0.256, 0.512, 1.024]

5. Grafana Dashboards

Grafana dashboards 将 metrics、logs 和 traces 汇聚在一起，实现统一可观测性。

RED/USE 方法 Panels

RED 方法 Dashboard（面向请求）

json

{
  "panels": [
    {
      "title": "Request Rate",
      "type": "timeseries",
      "targets": [
        {
          "expr": "sum(rate(http_requests_total[5m])) by (service)",
          "legendFormat": "{{ service }}"
        }
      ]
    },
    {
      "title": "Error Rate",
      "type": "timeseries",
      "targets": [
        {
          "expr": "sum(rate(http_requests_total{status=~\"5..\"}[5m])) by (service) / sum(rate(http_requests_total[5m])) by (service)",
          "legendFormat": "{{ service }}"
        }
      ],
      "fieldConfig": {
        "defaults": {
          "unit": "percentunit",
          "thresholds": {
            "steps": [
              {"value": 0, "color": "green"},
              {"value": 0.01, "color": "yellow"},
              {"value": 0.05, "color": "red"}
            ]
          }
        }
      }
    },
    {
      "title": "Latency (P95)",
      "type": "timeseries",
      "targets": [
        {
          "expr": "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))",
          "legendFormat": "{{ service }}"
        }
      ],
      "fieldConfig": {
        "defaults": {
          "unit": "s"
        }
      }
    }
  ]
}

USE 方法 Dashboard（面向资源）

json

{
  "panels": [
    {
      "title": "CPU Utilization",
      "type": "gauge",
      "targets": [
        {
          "expr": "100 - (avg(rate(node_cpu_seconds_total{mode=\"idle\"}[5m])) * 100)",
          "legendFormat": "CPU %"
        }
      ],
      "fieldConfig": {
        "defaults": {
          "unit": "percent",
          "max": 100,
          "thresholds": {
            "steps": [
              {"value": 0, "color": "green"},
              {"value": 70, "color": "yellow"},
              {"value": 85, "color": "red"}
            ]
          }
        }
      }
    },
    {
      "title": "Memory Saturation",
      "type": "timeseries",
      "targets": [
        {
          "expr": "1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)",
          "legendFormat": "{{ instance }}"
        }
      ]
    },
    {
      "title": "Disk I/O Errors",
      "type": "stat",
      "targets": [
        {
          "expr": "rate(node_disk_io_time_seconds_total{device!~\"dm-.*\"}[5m])",
          "legendFormat": "{{ device }}"
        }
      ]
    }
  ]
}

跨 Datasource 链接

通过 Exemplars 从 Prometheus 到 Tempo

yaml

# Grafana datasource configuration
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    url: http://prometheus:9090
    jsonData:
      exemplarTraceIdDestinations:
        - name: traceID
          datasourceUid: tempo
          urlDisplayLabel: "View Trace"
      httpMethod: POST

通过 Derived Fields 从 Loki 到 Tempo

yaml

# Grafana datasource configuration
apiVersion: 1
datasources:
  - name: Loki
    type: loki
    url: http://loki:3100
    jsonData:
      derivedFields:
        - name: TraceID
          matcherRegex: '"traceId":"([a-f0-9]+)"'
          url: '$${__value.raw}'
          datasourceUid: tempo
          urlDisplayLabel: "View Trace"
        - name: TraceID-W3C
          matcherRegex: 'traceparent.*-([a-f0-9]{32})-'
          url: '$${__value.raw}'
          datasourceUid: tempo

Dashboard 预置

yaml

# grafana-dashboards-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-dashboards
  namespace: observability
  labels:
    grafana_dashboard: "1"
data:
  eks-overview.json: |
    {
      "dashboard": {
        "title": "EKS Cluster Overview",
        "uid": "eks-overview",
        "tags": ["eks", "kubernetes"],
        "timezone": "browser",
        "refresh": "30s",
        "templating": {
          "list": [
            {
              "name": "namespace",
              "type": "query",
              "datasource": "Prometheus",
              "query": "label_values(kube_namespace_labels, namespace)",
              "refresh": 2,
              "multi": true,
              "includeAll": true
            },
            {
              "name": "service",
              "type": "query",
              "datasource": "Prometheus",
              "query": "label_values(kube_service_info{namespace=~\"$namespace\"}, service)",
              "refresh": 2,
              "multi": true,
              "includeAll": true
            }
          ]
        },
        "panels": []
      }
    }

变量模板

json

{
  "templating": {
    "list": [
      {
        "name": "datasource",
        "type": "datasource",
        "query": "prometheus"
      },
      {
        "name": "cluster",
        "type": "query",
        "datasource": "${datasource}",
        "query": "label_values(up, cluster)",
        "refresh": 2
      },
      {
        "name": "namespace",
        "type": "query",
        "datasource": "${datasource}",
        "query": "label_values(kube_pod_info{cluster=\"$cluster\"}, namespace)",
        "refresh": 2,
        "multi": true,
        "includeAll": true
      },
      {
        "name": "workload",
        "type": "query",
        "datasource": "${datasource}",
        "query": "label_values(kube_deployment_labels{cluster=\"$cluster\", namespace=~\"$namespace\"}, deployment)",
        "refresh": 2,
        "multi": true
      },
      {
        "name": "interval",
        "type": "interval",
        "query": "1m,5m,15m,30m,1h,6h,12h,1d",
        "current": {
          "value": "5m"
        }
      }
    ]
  }
}

Dashboard JSON Model 示例

包含跨 datasource 链接的完整 panel：

json

{
  "title": "Service Latency with Traces",
  "type": "timeseries",
  "datasource": "Prometheus",
  "targets": [
    {
      "expr": "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{namespace=\"$namespace\", service=\"$service\"}[$interval])) by (le))",
      "legendFormat": "P99 Latency",
      "exemplar": true
    }
  ],
  "fieldConfig": {
    "defaults": {
      "unit": "s",
      "links": [
        {
          "title": "View slow traces",
          "url": "/explore?orgId=1&left=%7B%22datasource%22:%22Tempo%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22query%22:%22%7Bresource.service.name%3D%5C%22${service}%5C%22%20%26%26%20duration%20%3E%201s%7D%22%7D%5D%7D",
          "targetBlank": true
        },
        {
          "title": "View logs",
          "url": "/explore?orgId=1&left=%7B%22datasource%22:%22Loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22${namespace}%5C%22,%20app%3D%5C%22${service}%5C%22%7D%22%7D%5D%7D",
          "targetBlank": true
        }
      ]
    }
  },
  "options": {
    "tooltip": {
      "mode": "single"
    },
    "legend": {
      "displayMode": "list",
      "placement": "bottom"
    }
  }
}

端到端可观测性实验

ArgoCD 详解测验

Istio 详解测验

Linkerd 详解测验

Cilium 服务网格详解测验

Cilium 深入解析

Calico 深入解析

指标

日志

追踪

告警

可观测性实验

ArgoCD

流量管理

安全性

可观测性

弹性

高级

对比指南

指标

日志

追踪

告警

可观测性分析：Logs/Metrics/Traces 关联 ​

1. 关联策略 ​

Trace ID 传播标准 ​

OpenTelemetry SDK 插桩 ​

应用程序插桩示例 ​

Exemplars：从 Metrics 到 Traces ​

关联架构图 ​

关联工作流 ​

2. Loki LogQL 分析 ​

错误率计算 ​

从日志中提取延迟 ​

基于日志的告警规则 ​

Label 策略与基数 ​

LogQL 模式匹配与解析 ​

聚合查询 ​

多行日志处理 ​

完整 Loki 告警规则 YAML ​

3. Prometheus PromQL 模式 ​

RPS 计算 ​

错误率（RED 方法） ​

延迟百分位数 ​

Istio Service Mesh Metrics ​

通过 CloudWatch 获取 ALB Metrics ​

Amazon Managed Prometheus（AMP）模式 ​

Recording Rules ​

4. Tempo TraceQL 分析 ​

基本 TraceQL 语法 ​

延迟分析 ​

错误 Trace 搜索 ​

Service 依赖关系映射 ​

Span 属性过滤 ​

结构查询（父子关系） ​

Trace 比较 ​

Service Graph 查询 ​

5. Grafana Dashboards ​

RED/USE 方法 Panels ​

跨 Datasource 链接 ​

Dashboard 预置 ​

变量模板 ​

Dashboard JSON Model 示例 ​

相关资源 ​

可观测性分析：Logs/Metrics/Traces 关联

1. 关联策略

Trace ID 传播标准

OpenTelemetry SDK 插桩

应用程序插桩示例

Exemplars：从 Metrics 到 Traces

关联架构图

关联工作流

2. Loki LogQL 分析

错误率计算

从日志中提取延迟

基于日志的告警规则

Label 策略与基数

LogQL 模式匹配与解析

聚合查询

多行日志处理

完整 Loki 告警规则 YAML

3. Prometheus PromQL 模式

RPS 计算

错误率（RED 方法）

延迟百分位数

Istio Service Mesh Metrics

通过 CloudWatch 获取 ALB Metrics

Amazon Managed Prometheus（AMP）模式

Recording Rules

4. Tempo TraceQL 分析

基本 TraceQL 语法

延迟分析

错误 Trace 搜索

Service 依赖关系映射

Span 属性过滤

结构查询（父子关系）

Trace 比较

Service Graph 查询

5. Grafana Dashboards

RED/USE 方法 Panels

跨 Datasource 链接

Dashboard 预置

变量模板

Dashboard JSON Model 示例

相关资源