VictoriaMetrics

Supported Versions: VictoriaMetrics 1.x Last Updated: February 20, 2026

Introduction
Architecture Options
Single-Node Mode
Cluster Mode
vmagent
vmalert
MetricsQL
Helm Installation
Long-term Storage Configuration
Downsampling
Performance Optimization
Best Practices
Troubleshooting

Introduction

VictoriaMetrics is a high-performance, cost-effective time series database and monitoring solution. While fully compatible with Prometheus, it offers better compression rates, query performance, and scalability.

Key Features

Feature	Description
High Compression	Up to 7x more efficient data compression than Prometheus
Fast Query Performance	Up to 20x faster performance on complex queries
Horizontal Scaling	Unlimited scaling in cluster mode
Low Operational Overhead	Single binary deployment, minimal configuration
Prometheus Compatible	Full support for PromQL, Remote Write/Read API
Multi-tenancy	Isolated environments for multiple teams/projects
Long-term Storage	Efficient long-term metric storage and downsampling

VictoriaMetrics vs Prometheus

Item	Prometheus	VictoriaMetrics
Architecture	Single node	Single/Cluster
Horizontal scaling	Requires Thanos/Cortex	Native support
Disk usage	Baseline	~70% reduction
Query speed	Baseline	2-20x faster
Memory usage	High	Low
Cardinality limit	~10M time series	~100M+ time series
Query language	PromQL	MetricsQL (superset)

Architecture Options

VictoriaMetrics offers two deployment modes:

Selection Guide

Single-Node Mode

Single-node mode (vmsingle) is suitable for small to medium environments.

Features

Single binary provides all functionality
Simple setup and operation
Capable of processing millions of samples per second
Supports billions of active time series

StatefulSet Deployment

yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: vmsingle
  namespace: monitoring
spec:
  serviceName: "vmsingle"
  replicas: 1
  selector:
    matchLabels:
      app: vmsingle
  template:
    metadata:
      labels:
        app: vmsingle
    spec:
      containers:
      - name: vmsingle
        image: victoriametrics/victoria-metrics:v1.96.0
        args:
          - "--storageDataPath=/storage"
          - "--httpListenAddr=:8428"
          - "--retentionPeriod=1y"
          - "--search.latencyOffset=30s"
          - "--search.maxUniqueTimeseries=1000000"
          - "--search.maxSamplesPerQuery=1000000000"
          # Memory optimization
          - "--memory.allowedPercent=60"
          # Compression settings
          - "--dedup.minScrapeInterval=30s"
        ports:
        - containerPort: 8428
          name: http
        resources:
          requests:
            cpu: 500m
            memory: 2Gi
          limits:
            cpu: 2000m
            memory: 8Gi
        volumeMounts:
        - name: storage
          mountPath: /storage
        livenessProbe:
          httpGet:
            path: /health
            port: 8428
          initialDelaySeconds: 30
          periodSeconds: 30
        readinessProbe:
          httpGet:
            path: /health
            port: 8428
          initialDelaySeconds: 5
          periodSeconds: 15
      securityContext:
        fsGroup: 65534
        runAsNonRoot: true
        runAsUser: 65534
  volumeClaimTemplates:
  - metadata:
      name: storage
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: gp3
      resources:
        requests:
          storage: 100Gi
---
apiVersion: v1
kind: Service
metadata:
  name: vmsingle
  namespace: monitoring
spec:
  selector:
    app: vmsingle
  ports:
  - port: 8428
    targetPort: 8428
    name: http
  type: ClusterIP

Key Endpoints

Endpoint	Description
`/api/v1/write`	Prometheus Remote Write
`/api/v1/query`	Instant query
`/api/v1/query_range`	Range query
`/api/v1/series`	Series metadata
`/api/v1/labels`	Label list
`/api/v1/label/{name}/values`	Label value list
`/vmui`	Built-in UI
`/metrics`	Self metrics

Cluster Mode

Scalable cluster configuration for large-scale environments.

Architecture

Components

Component	Role	Scaling Method
vminsert	Write request routing	Horizontal scaling (Deployment)
vmstorage	Data storage	Horizontal scaling (StatefulSet)
vmselect	Query processing	Horizontal scaling (Deployment)

vmstorage Deployment

yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: vmstorage
  namespace: monitoring
spec:
  serviceName: "vmstorage"
  replicas: 3
  selector:
    matchLabels:
      app: vmstorage
  template:
    metadata:
      labels:
        app: vmstorage
    spec:
      containers:
      - name: vmstorage
        image: victoriametrics/vmstorage:v1.96.0-cluster
        args:
          - "--storageDataPath=/storage"
          - "--httpListenAddr=:8482"
          - "--vminsertAddr=:8400"
          - "--vmselectAddr=:8401"
          - "--retentionPeriod=1y"
          - "--dedup.minScrapeInterval=30s"
        ports:
        - containerPort: 8482
          name: http
        - containerPort: 8400
          name: vminsert
        - containerPort: 8401
          name: vmselect
        resources:
          requests:
            cpu: 500m
            memory: 2Gi
          limits:
            cpu: 2000m
            memory: 8Gi
        volumeMounts:
        - name: storage
          mountPath: /storage
        livenessProbe:
          httpGet:
            path: /health
            port: 8482
          initialDelaySeconds: 30
          periodSeconds: 30
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app: vmstorage
            topologyKey: kubernetes.io/hostname
  volumeClaimTemplates:
  - metadata:
      name: storage
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: gp3
      resources:
        requests:
          storage: 100Gi
---
apiVersion: v1
kind: Service
metadata:
  name: vmstorage
  namespace: monitoring
spec:
  selector:
    app: vmstorage
  clusterIP: None
  ports:
  - port: 8482
    name: http
  - port: 8400
    name: vminsert
  - port: 8401
    name: vmselect

vminsert Deployment

yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vminsert
  namespace: monitoring
spec:
  replicas: 3
  selector:
    matchLabels:
      app: vminsert
  template:
    metadata:
      labels:
        app: vminsert
    spec:
      containers:
      - name: vminsert
        image: victoriametrics/vminsert:v1.96.0-cluster
        args:
          - "--httpListenAddr=:8480"
          - "--storageNode=vmstorage-0.vmstorage:8400"
          - "--storageNode=vmstorage-1.vmstorage:8400"
          - "--storageNode=vmstorage-2.vmstorage:8400"
          - "--replicationFactor=2"
        ports:
        - containerPort: 8480
          name: http
        resources:
          requests:
            cpu: 200m
            memory: 256Mi
          limits:
            cpu: 1000m
            memory: 1Gi
        livenessProbe:
          httpGet:
            path: /health
            port: 8480
          initialDelaySeconds: 10
          periodSeconds: 30
---
apiVersion: v1
kind: Service
metadata:
  name: vminsert
  namespace: monitoring
spec:
  selector:
    app: vminsert
  ports:
  - port: 8480
    targetPort: 8480
    name: http
  type: ClusterIP

vmselect Deployment

yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vmselect
  namespace: monitoring
spec:
  replicas: 3
  selector:
    matchLabels:
      app: vmselect
  template:
    metadata:
      labels:
        app: vmselect
    spec:
      containers:
      - name: vmselect
        image: victoriametrics/vmselect:v1.96.0-cluster
        args:
          - "--httpListenAddr=:8481"
          - "--storageNode=vmstorage-0.vmstorage:8401"
          - "--storageNode=vmstorage-1.vmstorage:8401"
          - "--storageNode=vmstorage-2.vmstorage:8401"
          - "--search.maxUniqueTimeseries=1000000"
          - "--search.maxSamplesPerQuery=1000000000"
        ports:
        - containerPort: 8481
          name: http
        resources:
          requests:
            cpu: 200m
            memory: 512Mi
          limits:
            cpu: 1000m
            memory: 2Gi
        livenessProbe:
          httpGet:
            path: /health
            port: 8481
          initialDelaySeconds: 10
          periodSeconds: 30
---
apiVersion: v1
kind: Service
metadata:
  name: vmselect
  namespace: monitoring
spec:
  selector:
    app: vmselect
  ports:
  - port: 8481
    targetPort: 8481
    name: http
  type: ClusterIP

vmagent

vmagent is a lightweight agent for metric collection and forwarding.

Key Features

Compatible with Prometheus scrape configuration
Supports multiple Remote Write targets
Data buffering and retransmission
Low resource usage
Label rewriting and filtering

Deployment

yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vmagent
  namespace: monitoring
spec:
  replicas: 2
  selector:
    matchLabels:
      app: vmagent
  template:
    metadata:
      labels:
        app: vmagent
    spec:
      serviceAccountName: vmagent
      containers:
      - name: vmagent
        image: victoriametrics/vmagent:v1.96.0
        args:
          - "--promscrape.config=/etc/vmagent/prometheus.yml"
          - "--remoteWrite.url=http://vminsert:8480/insert/0/prometheus/api/v1/write"
          - "--remoteWrite.tmpDataPath=/tmp/vmagent-remotewrite-data"
          - "--remoteWrite.maxDiskUsagePerURL=1GB"
          - "--promscrape.cluster.membersCount=2"
          - "--promscrape.cluster.memberNum=$(POD_INDEX)"
        env:
        - name: POD_INDEX
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        ports:
        - containerPort: 8429
          name: http
        resources:
          requests:
            cpu: 100m
            memory: 256Mi
          limits:
            cpu: 500m
            memory: 1Gi
        volumeMounts:
        - name: config
          mountPath: /etc/vmagent
        - name: tmpdata
          mountPath: /tmp/vmagent-remotewrite-data
      volumes:
      - name: config
        configMap:
          name: vmagent-config
      - name: tmpdata
        emptyDir: {}
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: vmagent-config
  namespace: monitoring
data:
  prometheus.yml: |
    global:
      scrape_interval: 30s
      scrape_timeout: 10s

    scrape_configs:
      - job_name: 'kubernetes-pods'
        kubernetes_sd_configs:
          - role: pod
        relabel_configs:
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
            action: keep
            regex: true
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
            action: replace
            target_label: __metrics_path__
            regex: (.+)
          - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
            action: replace
            regex: ([^:]+)(?::\d+)?;(\d+)
            replacement: $1:$2
            target_label: __address__
          - source_labels: [__meta_kubernetes_namespace]
            target_label: namespace
          - source_labels: [__meta_kubernetes_pod_name]
            target_label: pod

      - job_name: 'kubernetes-nodes'
        kubernetes_sd_configs:
          - role: node
        scheme: https
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          insecure_skip_verify: true
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        relabel_configs:
          - action: labelmap
            regex: __meta_kubernetes_node_label_(.+)
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vmagent
  namespace: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: vmagent
rules:
- apiGroups: [""]
  resources: ["nodes", "nodes/proxy", "nodes/metrics", "services", "endpoints", "pods"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["networking.k8s.io"]
  resources: ["ingresses"]
  verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics", "/metrics/cadvisor"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: vmagent
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: vmagent
subjects:
- kind: ServiceAccount
  name: vmagent
  namespace: monitoring

vmagent Sharding

Distributes scrape targets across vmagent instances in large-scale environments:

yaml

args:
  - "--promscrape.cluster.membersCount=3"    # Total number of vmagents
  - "--promscrape.cluster.memberNum=0"       # Current instance number (0, 1, 2)
  - "--promscrape.cluster.replicationFactor=2"  # How many instances scrape each target

vmalert

vmalert is a component that evaluates alerting rules and generates alerts.

Deployment

yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vmalert
  namespace: monitoring
spec:
  replicas: 2
  selector:
    matchLabels:
      app: vmalert
  template:
    metadata:
      labels:
        app: vmalert
    spec:
      containers:
      - name: vmalert
        image: victoriametrics/vmalert:v1.96.0
        args:
          - "--datasource.url=http://vmselect:8481/select/0/prometheus"
          - "--remoteRead.url=http://vmselect:8481/select/0/prometheus"
          - "--remoteWrite.url=http://vminsert:8480/insert/0/prometheus"
          - "--notifier.url=http://alertmanager:9093"
          - "--rule=/etc/vmalert/rules/*.yaml"
          - "--evaluationInterval=30s"
          - "--external.url=http://vmalert:8880"
          - "--external.label=cluster=production"
        ports:
        - containerPort: 8880
          name: http
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 500m
            memory: 512Mi
        volumeMounts:
        - name: rules
          mountPath: /etc/vmalert/rules
        livenessProbe:
          httpGet:
            path: /health
            port: 8880
          initialDelaySeconds: 10
          periodSeconds: 30
      volumes:
      - name: rules
        configMap:
          name: vmalert-rules
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: vmalert-rules
  namespace: monitoring
data:
  kubernetes.yaml: |
    groups:
    - name: kubernetes
      interval: 30s
      rules:
      - alert: NodeMemoryHigh
        expr: |
          (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes)
          / node_memory_MemTotal_bytes * 100 > 90
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High memory usage on {{ $labels.instance }}"
          description: "Memory usage is {{ printf \"%.2f\" $value }}%"

      - alert: PodCrashLooping
        expr: increase(kube_pod_container_status_restarts_total[1h]) > 5
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} is crash looping"
          description: "Pod has restarted {{ $value }} times in the last hour"

    - name: recording-rules
      interval: 30s
      rules:
      - record: instance:node_cpu_utilization:rate5m
        expr: |
          100 - (avg by (instance)(irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

      - record: instance:node_memory_utilization:ratio
        expr: |
          1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes

MetricsQL

MetricsQL is VictoriaMetrics' query language and a superset of PromQL.

PromQL Compatibility

All PromQL queries work in MetricsQL:

promql

# Basic PromQL queries (also work in MetricsQL)
rate(http_requests_total[5m])
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

MetricsQL Extensions

promql

# Range function defaults
# PromQL: range specification required
rate(http_requests_total[5m])

# MetricsQL: range can be omitted (uses default)
rate(http_requests_total)

# keep_last_value: retain last value (gap filling)
keep_last_value(up)

# default: set default value
http_requests_total default 0

# label_set: set labels
label_set(up, "env", "production")

# label_del: delete labels
label_del(up, "instance")

# label_copy: copy labels
label_copy(up, "instance", "node")

# label_move: move labels
label_move(up, "instance", "node")

# label_join: join labels
label_join(up, "dst", "-", "job", "instance")

# label_transform: transform labels (regex)
label_transform(up, "instance", "([^:]+):.*", "$1")

# union: combine multiple series
union(up{job="api"}, up{job="web"})

# lag: time difference from previous value
lag(up)

# lifetime: series lifetime
lifetime(up)

# scrape_interval: estimate scrape interval
scrape_interval(up)

# range_* functions: usable without range vector
range_avg(http_requests_total)        # Average over entire range
range_max(http_requests_total)        # Maximum over entire range
range_min(http_requests_total)        # Minimum over entire range
range_sum(http_requests_total)        # Sum over entire range
range_first(http_requests_total)      # First value
range_last(http_requests_total)       # Last value

# rollup_* functions: advanced aggregation
rollup(http_requests_total[5m])       # Returns min, max, avg
rollup_rate(http_requests_total[5m])  # Rate min, max, avg
rollup_delta(gauge_metric[5m])        # Delta min, max, avg

# Anomaly detection functions
anomaly_score(http_requests_total[1h])  # Anomaly score (0-1)

# histogram_share: ratio below specific value in histogram
histogram_share(0.1, http_request_duration_seconds_bucket)  # Ratio below 100ms

Useful MetricsQL Queries

promql

# Time series count (cardinality)
count(up)

# Time series count by label
count by (job)(up)

# Fill gaps in data
keep_last_value(up)

# Multiple conditions OR combined
up{job="api"} or up{job="web"}

# MetricsQL: simpler syntax
union(up{job="api"}, up{job="web"})

# Rate calculation (safe division)
rate(http_requests_total{status=~"5.."}[5m])
/ (rate(http_requests_total[5m]) > 0)

# MetricsQL: simple with default
rate(http_requests_total{status=~"5.."}[5m])
/ rate(http_requests_total[5m]) default 0

# Histogram bucket ratio
histogram_share(0.5, http_request_duration_seconds_bucket)  # Ratio below 500ms

Helm Installation

victoria-metrics-k8s-stack

bash

# Add Helm repository
helm repo add vm https://victoriametrics.github.io/helm-charts/
helm repo update

# Install
helm install victoria-metrics vm/victoria-metrics-k8s-stack \
  --namespace monitoring \
  --create-namespace \
  -f values.yaml

values.yaml (Single Node)

yaml

# VictoriaMetrics Single
victoria-metrics-single:
  enabled: true
  server:
    retentionPeriod: 1y
    resources:
      requests:
        cpu: 500m
        memory: 2Gi
      limits:
        cpu: 2000m
        memory: 8Gi
    persistentVolume:
      enabled: true
      storageClassName: gp3
      size: 100Gi
    extraArgs:
      dedup.minScrapeInterval: 30s
      search.maxUniqueTimeseries: 1000000

# vmagent
vmagent:
  enabled: true
  spec:
    replicaCount: 2
    resources:
      requests:
        cpu: 100m
        memory: 256Mi
      limits:
        cpu: 500m
        memory: 1Gi
    extraArgs:
      promscrape.maxScrapeSize: 64MB
      promscrape.cluster.membersCount: "2"

# vmalert
vmalert:
  enabled: true
  spec:
    replicaCount: 2
    resources:
      requests:
        cpu: 100m
        memory: 128Mi
      limits:
        cpu: 500m
        memory: 512Mi

# Grafana
grafana:
  enabled: true
  persistence:
    enabled: true
    storageClassName: gp3
    size: 10Gi

# Alertmanager
alertmanager:
  enabled: true
  spec:
    replicaCount: 3
    storage:
      volumeClaimTemplate:
        spec:
          storageClassName: gp3
          resources:
            requests:
              storage: 10Gi

# kube-state-metrics
kube-state-metrics:
  enabled: true

# node-exporter
prometheus-node-exporter:
  enabled: true

values.yaml (Cluster)

yaml

# VictoriaMetrics Cluster
victoria-metrics-cluster:
  enabled: true

  vmselect:
    replicaCount: 3
    resources:
      requests:
        cpu: 200m
        memory: 512Mi
      limits:
        cpu: 1000m
        memory: 2Gi
    extraArgs:
      search.maxUniqueTimeseries: "1000000"

  vminsert:
    replicaCount: 3
    resources:
      requests:
        cpu: 200m
        memory: 256Mi
      limits:
        cpu: 1000m
        memory: 1Gi
    extraArgs:
      replicationFactor: "2"

  vmstorage:
    replicaCount: 3
    resources:
      requests:
        cpu: 500m
        memory: 2Gi
      limits:
        cpu: 2000m
        memory: 8Gi
    persistentVolume:
      enabled: true
      storageClassName: gp3
      size: 100Gi
    extraArgs:
      retentionPeriod: "1y"
      dedup.minScrapeInterval: "30s"

# Disable single node
victoria-metrics-single:
  enabled: false

Long-term Storage Configuration

Retention Period Settings

yaml

# vmsingle
args:
  - "--retentionPeriod=1y"         # 1 year retention
  - "--retentionPeriod=365d"       # 365 days retention
  - "--retentionPeriod=8760h"      # 8760 hours retention

# Size-based retention
args:
  - "--retentionPeriod=1y"
  - "--storage.maxDiskSpace=500GB"  # Maximum disk usage

Object Storage Backup

yaml

# S3 backup using vmbackup
apiVersion: batch/v1
kind: CronJob
metadata:
  name: vmbackup
  namespace: monitoring
spec:
  schedule: "0 2 * * *"  # Daily at 02:00
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: vmbackup
            image: victoriametrics/vmbackup:v1.96.0
            args:
              - "--storageDataPath=/storage"
              - "--snapshot.createURL=http://vmsingle:8428/snapshot/create"
              - "--dst=s3://my-bucket/vmbackup"
            env:
            - name: AWS_ACCESS_KEY_ID
              valueFrom:
                secretKeyRef:
                  name: aws-credentials
                  key: access-key-id
            - name: AWS_SECRET_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  name: aws-credentials
                  key: secret-access-key
            volumeMounts:
            - name: storage
              mountPath: /storage
              readOnly: true
          volumes:
          - name: storage
            persistentVolumeClaim:
              claimName: vmsingle-storage-vmsingle-0
          restartPolicy: OnFailure

Downsampling

VictoriaMetrics Enterprise supports downsampling. For open source, use recording rules.

Downsampling with Recording Rules

yaml

# vmalert rules
groups:
- name: downsampling
  interval: 5m
  rules:
  # 5-minute average CPU usage
  - record: cpu_usage:5m_avg
    expr: avg_over_time(node_cpu_seconds_total[5m])

  # 1-hour average (calculated every 5 minutes)
  - record: cpu_usage:1h_avg
    expr: avg_over_time(cpu_usage:5m_avg[1h])

  # Histogram downsampling
  - record: http_request_duration:5m
    expr: |
      histogram_quantile(0.50, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))
      or
      histogram_quantile(0.90, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))
      or
      histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))

Performance Optimization

Memory Optimization

yaml

args:
  # Memory usage limit (60% of available memory)
  - "--memory.allowedPercent=60"

  # Search cache size
  - "--search.maxMemoryPerQuery=512MB"

  # Index cache
  - "--storage.cacheSizeIndexDBDataBlocks=256MB"
  - "--storage.cacheSizeIndexDBIndexBlocks=128MB"

Query Optimization

yaml

args:
  # Maximum unique time series
  - "--search.maxUniqueTimeseries=1000000"

  # Maximum samples per query
  - "--search.maxSamplesPerQuery=1000000000"

  # Query timeout
  - "--search.maxQueryDuration=60s"

  # Maximum points per time series
  - "--search.maxPointsPerTimeseries=30000"

Write Optimization

yaml

args:
  # Deduplication (smaller than scrape interval)
  - "--dedup.minScrapeInterval=30s"

  # Maximum labels
  - "--maxLabelsPerTimeseries=30"

  # Maximum label value length
  - "--maxLabelValueLen=1024"

Best Practices

Production Checklist

Resource Allocation

yaml

resources:
  requests:
    cpu: 500m
    memory: 2Gi
  limits:
    cpu: 2000m
    memory: 8Gi

High Availability
- Set replicationFactor=2 in cluster mode
- Minimum 2 vmselect, vminsert replicas
- Minimum 3 vmstorage replicas

Monitoring

promql

# Monitor VictoriaMetrics self metrics
vm_app_version{job="vmsingle"}
vm_rows_inserted_total
vm_slow_queries_total
process_resident_memory_bytes{job=~"vm.*"}

Backup
- Regular backups with vmbackup
- Use snapshot feature

Migration Guide

Migrating from Prometheus to VictoriaMetrics:

yaml

# Step 1: Add remote_write to Prometheus
# prometheus.yml
remote_write:
  - url: http://victoriametrics:8428/api/v1/write

# Step 2: Add VictoriaMetrics data source to Grafana
# Operate both data sources in parallel for comparison

# Step 3: Migrate existing data (optional)
vmctl prometheus --prometheus.snapshot-path=/prometheus/snapshots/xxx \
  --vm-url=http://victoriametrics:8428

# Step 4: Complete transition
# Remove Prometheus remote_write, change Grafana default data source

Troubleshooting

Common Issues

1. High Memory Usage

bash

# Check memory usage
curl http://vmsingle:8428/api/v1/status/tsdb

# Solution: Set memory limit
args:
  - "--memory.allowedPercent=60"

2. Slow Queries

bash

# Check slow query log
curl http://vmsingle:8428/api/v1/status/top_queries

# Solution: Query optimization
# - Reduce time range
# - Add label filters
# - Increase step

3. Disk Space Full

bash

# Check disk usage
curl http://vmsingle:8428/api/v1/status/tsdb | jq .

# Solutions
# - Reduce retention period
# - Delete unnecessary metrics
curl -X POST "http://vmsingle:8428/api/v1/admin/tsdb/delete_series?match[]=go_.*"

Debugging Commands

bash

# Check status
curl http://vmsingle:8428/api/v1/status/tsdb
curl http://vmsingle:8428/api/v1/status/active_queries
curl http://vmsingle:8428/api/v1/status/top_queries

# Check metrics
curl http://vmsingle:8428/metrics

# Create snapshot
curl http://vmsingle:8428/snapshot/create

# Force merge (reclaim disk space)
curl http://vmsingle:8428/internal/force_merge

References

Quiz

To test your understanding of this chapter, try the VictoriaMetrics Quiz.

Traffic Management

Security

Observability

Resilience

Advanced

Comparison Guide

ArgoCD

Metrics

Logging

Tracing

Alerting

Observability End-to-End Labs

Cilium Deep Dive

Calico Deep Dive

Istio Detailed Quiz

Linkerd Detailed Quiz

Cilium Service Mesh Detailed Quiz

ArgoCD Detailed Quiz

Metrics

Logging

Tracing

Alerting

Observability Labs

VictoriaMetrics ​

Table of Contents ​

Introduction ​

Key Features ​

VictoriaMetrics vs Prometheus ​

Architecture Options ​

Selection Guide ​

Single-Node Mode ​

Features ​

StatefulSet Deployment ​

Key Endpoints ​

Cluster Mode ​

Architecture ​

Components ​

vmstorage Deployment ​

vminsert Deployment ​

vmselect Deployment ​

vmagent ​

Key Features ​

Deployment ​

vmagent Sharding ​

vmalert ​

Deployment ​

MetricsQL ​

PromQL Compatibility ​

MetricsQL Extensions ​

Useful MetricsQL Queries ​

Helm Installation ​

victoria-metrics-k8s-stack ​

values.yaml (Single Node) ​

values.yaml (Cluster) ​

Long-term Storage Configuration ​

Retention Period Settings ​

Object Storage Backup ​

Downsampling ​

Downsampling with Recording Rules ​

Performance Optimization ​

Memory Optimization ​

Query Optimization ​

Write Optimization ​

Best Practices ​

Production Checklist ​

Migration Guide ​

Troubleshooting ​

Common Issues ​

1. High Memory Usage ​

2. Slow Queries ​

3. Disk Space Full ​

Debugging Commands ​

References ​

Quiz ​

VictoriaMetrics

Table of Contents

Introduction

Key Features

VictoriaMetrics vs Prometheus

Architecture Options

Selection Guide

Single-Node Mode

Features

StatefulSet Deployment

Key Endpoints

Cluster Mode

Architecture

Components

vmstorage Deployment

vminsert Deployment

vmselect Deployment

vmagent

Key Features

Deployment

vmagent Sharding

vmalert

Deployment

MetricsQL

PromQL Compatibility

MetricsQL Extensions

Useful MetricsQL Queries

Helm Installation

victoria-metrics-k8s-stack

values.yaml (Single Node)

values.yaml (Cluster)

Long-term Storage Configuration

Retention Period Settings

Object Storage Backup

Downsampling

Downsampling with Recording Rules

Performance Optimization

Memory Optimization

Query Optimization

Write Optimization

Best Practices

Production Checklist

Migration Guide

Troubleshooting

Common Issues

1. High Memory Usage

2. Slow Queries

3. Disk Space Full

Debugging Commands

References

Quiz