FinOps コスト可視化プラットフォーム
対応バージョン: Kubernetes 1.28+, Kubecost 2.x, OpenCost 1.x 最終更新: April 25, 2026
< 前へ: イベント容量計画 | 目次 | 次へ: Tekton Pipelines >
概要
Kubernetes を大規模に運用すると、ワークロードが一時的であり、リソースが共有され、従来のサーバー単位のコスト配賦が適用できなくなるため、特有のコスト管理課題が生じます。意図的なコスト可視化がない場合、組織はクラウド請求額が想定を 2〜5 倍上回っていることに後から気づくことがよくあります。
FinOps (Financial Operations) は、クラウドコンピューティングの変動費モデルに財務上の説明責任を持ち込む実践です。FinOps ライフサイクルは、反復的な 3 つのフェーズで構成されます。
- Inform: どこで誰が費用を使っているかを可視化する
- Optimize: 無駄を削減し効率を高める機会を特定して実行する
- Operate: コスト効率を継続するためのガバナンス、自動化、文化的プラクティスを確立する
このガイドでは、OpenCost、Kubecost、Prometheus、Grafana を使用して、Kubernetes 上に完全な FinOps コスト可視化プラットフォームを構築します。
学習目標
- FinOps 運用モデルと、それが Kubernetes 環境にどのように適用されるかを理解する
- 正確なコスト配賦のために OpenCost と Kubecost をデプロイおよび設定する
- labels、namespaces、cost APIs を使用して showback と chargeback システムを実装する
- Slack への alerting pipelines によるコスト異常検知を構築する
- チーム向けのセルフサービス型コストダッシュボードと自動週次コストレポートを有効化する
- VPA recommendations と Goldilocks を使用したリソース rightsizing ワークフローを確立する
1. FinOps 運用モデル
1.1 Inform、Optimize、Operate サイクル
Inform フェーズ: コスト監視ツールのデプロイ、label 戦略の実装、showback ダッシュボードの構築により可視性を確立します。これは、すべての最適化作業の基盤です。
Optimize フェーズ: 可視化データを使用して無駄を特定します。これには、ワークロードの rightsizing、Spot instances と Savings Plans の活用、アイドルリソースのクリーンアップが含まれます。
Operate フェーズ: budget alerts、policy enforcement、定期的なコストレビュー会議を通じて、コスト効率を組織に定着させます。
1.2 組織上の役割
| 役割 | 責任 | 主なツール | 実施頻度 |
|---|---|---|---|
| FinOps Team | コスト配賦モデルの定義、ダッシュボードの維持、最適化の推進 | Kubecost, Grafana, AWS Cost Explorer | 日次監視、週次レポート |
| Engineering Teams | resource requests/limits の設定、コスト labels の適用、チームダッシュボードのレビュー | Team dashboards, VPA, Goldilocks | Sprint レベルのレビュー |
| Finance | 予算計画、予測の検証、chargeback 照合 | 月次コストレポート、showback データ | 月次照合 |
| Leadership | 予算承認、コスト目標設定、unit economics レビュー | Executive dashboards, trend reports | 月次/四半期レビュー |
| Platform Engineering | コストツールのデプロイと維持、セルフサービスダッシュボードの構築 | Kubecost, OpenCost, Kyverno, Prometheus | 継続的 |
1.3 成熟度レベル
| レベル | コスト配賦 | 最適化 | ガバナンス | タイムライン |
|---|---|---|---|---|
| Crawl | Namespace レベルの配賦、基本的な labels | 手動 rightsizing、アドホックなクリーンアップ | 正式な policies なし、リアクティブな alerts | 1〜3 か月 |
| Walk | 共有コスト分割を伴う label ベースの配賦、showback | VPA recommendations、Spot 採用 | Label enforcement、月次レビュー | 3〜6 か月 |
| Run | CUR 照合を伴うリアルタイム chargeback | 自動化された rightsizing pipelines | 自動化 policies、CI/CD の cost gates | 6〜12 か月 |
2. OpenCost/Kubecost の詳細設定
2.1 OpenCost のインストール (Open Source)
OpenCost は metrics 用に Prometheus を必要とし、独自のコスト配賦 API を公開します。
yaml
# opencost-values.yaml
# helm install opencost opencost/opencost -n opencost --create-namespace -f opencost-values.yaml
opencost:
exporter:
defaultClusterId: "production-eks-us-east-1"
image:
registry: ghcr.io
repository: opencost/opencost
tag: "1.112.0"
aws:
spot_data_region: "us-east-1"
spot_data_bucket: "my-company-spot-data-feed"
prometheus:
internal:
enabled: true
serviceName: prometheus-server
namespaceName: monitoring
port: 80
resources:
requests:
cpu: "100m"
memory: "256Mi"
limits:
cpu: "500m"
memory: "512Mi"
persistence:
enabled: true
storageClass: "gp3"
size: "32Gi"
cloudCost:
enabled: true
refreshRateHours: 6
ui:
enabled: true
ingress:
enabled: true
ingressClassName: "alb"
annotations:
alb.ingress.kubernetes.io/scheme: "internal"
alb.ingress.kubernetes.io/target-type: "ip"
alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS": 443}]'
hosts:
- host: "opencost.internal.mycompany.com"
paths:
- path: /
pathType: Prefix
metrics:
serviceMonitor:
enabled: true
namespace: monitoring
serviceAccount:
create: true
annotations:
eks.amazonaws.com/role-arn: "arn:aws:iam::123456789012:role/opencost-role"2.2 Kubecost Enterprise
Kubecost は、OpenCost の上に multi-cluster federation、S3 ETL storage、高度な配賦機能を追加します。
yaml
# kubecost-values.yaml
# helm install kubecost kubecost/cost-analyzer -n kubecost --create-namespace -f kubecost-values.yaml
global:
prometheus:
enabled: false
fqdn: "http://prometheus-server.monitoring.svc:80"
grafana:
enabled: false
domainName: "grafana.monitoring.svc"
kubecostProductConfigs:
clusterName: "production-eks-us-east-1"
currencyCode: "USD"
defaultModelPricing:
enabled: false
sharedNamespaces: "kube-system,kubecost,monitoring,cert-manager,ingress-nginx"
shareTenancyCosts: true
shareSplit: "weighted"
kubecostModel:
etl: true
etlBucketConfig:
enabled: true
federatedETL:
enabled: true
primaryCluster: true
resources:
requests:
cpu: "200m"
memory: "512Mi"
limits:
cpu: "1000m"
memory: "2Gi"
# S3 backend for ETL data
kubecostS3Config:
enabled: true
bucketName: "mycompany-kubecost-etl"
region: "us-east-1"
federatedETL:
federatedStore:
enabled: true
bucket: "mycompany-kubecost-federation"
region: "us-east-1"
kubecostAggregator:
enabled: true
replicas: 1
resources:
requests:
cpu: "500m"
memory: "1Gi"
limits:
cpu: "2000m"
memory: "4Gi"
ingress:
enabled: true
className: "alb"
annotations:
alb.ingress.kubernetes.io/scheme: "internal"
alb.ingress.kubernetes.io/target-type: "ip"
hosts:
- host: "kubecost.internal.mycompany.com"
paths:
- path: /
pathType: Prefix
serviceAccount:
create: true
annotations:
eks.amazonaws.com/role-arn: "arn:aws:iam::123456789012:role/kubecost-role"
podDisruptionBudget:
enabled: true
minAvailable: 12.3 AWS Cost and Usage Report (CUR) 統合
CUR は AWS 請求データの最も正確なソースを提供し、cluster 内の見積もりを実際の請求と照合できるようにします。
Terraform 設定
hcl
# cur-infrastructure.tf
terraform {
required_version = ">= 1.5.0"
required_providers { aws = { source = "hashicorp/aws"; version = "~> 5.0" } }
}
data "aws_caller_identity" "current" {}
resource "aws_s3_bucket" "cur_bucket" {
bucket = "mycompany-cur-reports"
tags = { Purpose = "cost-and-usage-reports", ManagedBy = "terraform" }
}
resource "aws_s3_bucket_versioning" "cur" {
bucket = aws_s3_bucket.cur_bucket.id
versioning_configuration { status = "Enabled" }
}
resource "aws_s3_bucket_lifecycle_configuration" "cur" {
bucket = aws_s3_bucket.cur_bucket.id
rule {
id = "transition-to-ia"
status = "Enabled"
transition { days = 90; storage_class = "STANDARD_IA" }
transition { days = 365; storage_class = "GLACIER" }
expiration { days = 730 }
}
}
resource "aws_s3_bucket_server_side_encryption_configuration" "cur" {
bucket = aws_s3_bucket.cur_bucket.id
rule { apply_server_side_encryption_by_default { sse_algorithm = "aws:kms" }; bucket_key_enabled = true }
}
resource "aws_s3_bucket_public_access_block" "cur" {
bucket = aws_s3_bucket.cur_bucket.id
block_public_acls = true; block_public_policy = true; ignore_public_acls = true; restrict_public_buckets = true
}
resource "aws_s3_bucket_policy" "cur" {
bucket = aws_s3_bucket.cur_bucket.id
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{ Sid = "AllowCURDelivery", Effect = "Allow", Principal = { Service = "billingreports.amazonaws.com" },
Action = ["s3:GetBucketAcl", "s3:GetBucketPolicy"], Resource = aws_s3_bucket.cur_bucket.arn },
{ Sid = "AllowCURWrite", Effect = "Allow", Principal = { Service = "billingreports.amazonaws.com" },
Action = "s3:PutObject", Resource = "${aws_s3_bucket.cur_bucket.arn}/*" }
]
})
}
resource "aws_cur_report_definition" "daily_cur" {
report_name = "mycompany-daily-cur"
time_unit = "DAILY"
format = "Parquet"
compression = "Parquet"
additional_schema_elements = ["RESOURCES"]
s3_bucket = aws_s3_bucket.cur_bucket.id
s3_region = "us-east-1"
s3_prefix = "cur-reports"
report_versioning = "OVERWRITE_REPORT"
refresh_closed_reports = true
additional_artifacts = ["ATHENA"]
}
# IAM role for Kubecost CUR access via IRSA
resource "aws_iam_role" "kubecost_cur" {
name = "kubecost-cur-reader"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Effect = "Allow"
Principal = { Federated = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:oidc-provider/${var.oidc_provider}" }
Action = "sts:AssumeRoleWithWebIdentity"
Condition = { StringEquals = {
"${var.oidc_provider}:sub" = "system:serviceaccount:kubecost:kubecost-cost-analyzer"
"${var.oidc_provider}:aud" = "sts.amazonaws.com"
}}
}]
})
}
resource "aws_iam_role_policy" "kubecost_cur" {
name = "kubecost-cur-read"
role = aws_iam_role.kubecost_cur.id
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{ Effect = "Allow", Action = ["s3:GetObject", "s3:ListBucket", "s3:GetBucketLocation"],
Resource = [aws_s3_bucket.cur_bucket.arn, "${aws_s3_bucket.cur_bucket.arn}/*"] },
{ Effect = "Allow", Action = ["athena:StartQueryExecution", "athena:GetQueryExecution", "athena:GetQueryResults"],
Resource = "arn:aws:athena:us-east-1:${data.aws_caller_identity.current.account_id}:workgroup/primary" },
{ Effect = "Allow", Action = ["glue:GetDatabase", "glue:GetTable", "glue:GetPartitions"],
Resource = ["arn:aws:glue:us-east-1:${data.aws_caller_identity.current.account_id}:catalog",
"arn:aws:glue:us-east-1:${data.aws_caller_identity.current.account_id}:database/athenacurcfn_*",
"arn:aws:glue:us-east-1:${data.aws_caller_identity.current.account_id}:table/athenacurcfn_*/*"] },
{ Effect = "Allow", Action = ["pricing:GetProducts", "ec2:DescribeInstances", "ec2:DescribeReservedInstances"], Resource = "*" }
]
})
}
variable "oidc_provider" { description = "OIDC provider URL (without https://)" ; type = string }
output "kubecost_role_arn" { value = aws_iam_role.kubecost_cur.arn }
output "cur_bucket_name" { value = aws_s3_bucket.cur_bucket.id }Kubecost Cloud Integration Values
yaml
# Add to kubecost-values.yaml for CUR reconciliation
kubecostProductConfigs:
cloudIntegrationJSON: |
{
"aws": [{
"athenaBucketName": "mycompany-cur-reports",
"athenaRegion": "us-east-1",
"athenaDatabase": "athenacurcfn_mycompany_daily_cur",
"athenaTable": "mycompany_daily_cur",
"athenaWorkgroup": "primary",
"projectID": "123456789012"
}]
}2.4 コスト精度のチューニング
カスタム料金設定
yaml
# custom-pricing-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: pricing-configs
namespace: kubecost
data:
default-pricing.json: |
{
"provider": "aws",
"description": "Custom pricing with negotiated EDP rates",
"CPU": "0.02835",
"RAM": "0.00356",
"GPU": "0.85",
"storage": "0.000054795",
"zoneNetworkEgress": "0.00",
"regionNetworkEgress": "0.01",
"internetNetworkEgress": "0.05",
"spotCPU": "0.0085",
"spotRAM": "0.00107",
"spotLabel": "karpenter.sh/capacity-type",
"spotLabelValue": "spot"
}共有コスト配賦ルール
yaml
# shared-cost-allocation-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: allocation-configs
namespace: kubecost
data:
shared-costs.json: |
{
"sharedCosts": [
{ "name": "Control Plane", "type": "weighted", "filter": { "namespace": "kube-system" }, "weight": "cpuCost" },
{ "name": "Monitoring Stack", "type": "weighted", "filter": { "namespace": "monitoring" }, "weight": "totalCost" },
{ "name": "Ingress Controllers", "type": "even", "filter": { "namespace": "ingress-nginx" } },
{ "name": "Service Mesh", "type": "weighted", "filter": { "namespace": "istio-system" }, "weight": "networkCost" },
{ "name": "Cert Manager", "type": "even", "filter": { "namespace": "cert-manager" } },
{ "name": "Platform Tools", "type": "even", "filter": { "namespace": "kubecost,argocd,kyverno" } }
],
"idleCostDistribution": "weighted"
}3. Showback/Chargeback の実装
Showback は認識のためにチームへコストを報告し、chargeback は実際に cost centers へ請求します。どちらも、組織単位に紐づいた正確なコスト配賦を必要とします。
3.1 Label 戦略
| Label | 目的 | 値の例 |
|---|---|---|
team | Engineering team へのコスト帰属 | platform, checkout, payments |
service | Service レベルのコスト追跡 | api-gateway, order-service |
environment | 環境の分離 | production, staging, development |
cost-center | Finance department へのマッピング | CC-1001, CC-2005 |
Kyverno Label 強制 Policy
yaml
# kyverno-cost-labels-policy.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: require-cost-labels
annotations:
policies.kyverno.io/title: Require Cost Attribution Labels
policies.kyverno.io/category: FinOps
policies.kyverno.io/severity: high
spec:
validationFailureAction: Enforce
background: true
rules:
- name: check-cost-labels-on-resource
match:
any:
- resources:
kinds:
- Deployment
- StatefulSet
- DaemonSet
exclude:
any:
- resources:
namespaces:
- kube-system
- kube-public
- kubecost
- monitoring
- ingress-nginx
- cert-manager
- argocd
- kyverno
validate:
message: >-
Resource {{request.object.kind}}/{{request.object.metadata.name}} is missing
required cost labels. All workloads must have: team, service, environment, cost-center.
pattern:
metadata:
labels:
team: "?*"
service: "?*"
environment: "?*"
cost-center: "?*"
- name: check-cost-labels-on-pod-template
match:
any:
- resources:
kinds:
- Deployment
- StatefulSet
- DaemonSet
exclude:
any:
- resources:
namespaces:
- kube-system
- kube-public
- kubecost
- monitoring
- ingress-nginx
- cert-manager
- argocd
- kyverno
validate:
message: "Pod template must also carry cost labels for accurate pod-level cost attribution."
pattern:
spec:
template:
metadata:
labels:
team: "?*"
service: "?*"
environment: "?*"
cost-center: "?*"
- name: validate-environment-values
match:
any:
- resources:
kinds:
- Deployment
- StatefulSet
- DaemonSet
exclude:
any:
- resources:
namespaces:
- kube-system
- kube-public
- kubecost
- monitoring
validate:
message: "Label 'environment' must be one of: production, staging, development, sandbox."
pattern:
metadata:
labels:
environment: "production | staging | development | sandbox"3.2 Namespace ベースのコスト配賦
Kubecost Allocation API の例
bash
# Cost allocation by namespace for the last 7 days
curl -s "http://kubecost.internal.mycompany.com/model/allocation\
?window=7d&aggregate=namespace&accumulate=true" \
| jq '.data[0] | to_entries[] | {namespace: .key, totalCost: .value.totalCost, cpuCost: .value.cpuCost}'
# Cost allocation by team label for the current month with shared costs
curl -s "http://kubecost.internal.mycompany.com/model/allocation\
?window=thismonth&aggregate=label:team&accumulate=true\
&shareIdle=weighted&shareNamespaces=kube-system,monitoring" \
| jq '.data[0] | to_entries | sort_by(-.value.totalCost) | .[] | {team: .key, totalCost: (.value.totalCost | round), cpuEfficiency: (.value.cpuEfficiency * 100 | round)}'
# Daily cost trend for a specific team over 30 days
curl -s "http://kubecost.internal.mycompany.com/model/allocation\
?window=30d&aggregate=label:team&step=1d&filterLabels=team:checkout" \
| jq '[.data[] | to_entries[] | {date: .key, cost: .value.totalCost}]'チーム Namespace ごとの ResourceQuota
yaml
# team-namespace-quota.yaml
apiVersion: v1
kind: Namespace
metadata:
name: team-checkout
labels:
team: checkout
cost-center: "CC-2005"
environment: production
---
apiVersion: v1
kind: ResourceQuota
metadata:
name: team-checkout-quota
namespace: team-checkout
spec:
hard:
requests.cpu: "40"
requests.memory: "80Gi"
limits.cpu: "80"
limits.memory: "160Gi"
persistentvolumeclaims: "20"
pods: "200"
---
apiVersion: v1
kind: LimitRange
metadata:
name: team-checkout-limits
namespace: team-checkout
spec:
limits:
- type: Container
default: { cpu: "500m", memory: "512Mi" }
defaultRequest: { cpu: "100m", memory: "128Mi" }
max: { cpu: "8", memory: "16Gi" }
min: { cpu: "10m", memory: "16Mi" }3.3 共有コスト配分
| 配分方法 | 使用する場面 | 長所 | 短所 |
|---|---|---|---|
| Weighted by CPU | Control plane コスト | 利用量に比例 | CPU-heavy なワークロードに不利 |
| Weighted by Total Cost | 一般的な共有 services | 全体として公平な配分 | 正確な基本配賦が必要 |
| Even Split | 小規模な共有 services | シンプルで透明 | チーム規模が異なる場合は不公平 |
| Weighted by Network | Ingress、service mesh | ネットワークコストに対して正確 | ネットワークコストは変動しやすい |
3.4 Grafana Showback ダッシュボード
次の Grafana dashboard JSON は、チーム変数セレクター付きで cost-per-team と cost-per-service の panels を提供します。Grafana UI からインポートするか、grafana_dashboard: "true" label を付けた ConfigMap として provision します。
json
{
"description": "FinOps Showback Dashboard",
"editable": true,
"panels": [
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": { "defaults": { "unit": "currencyUSD", "custom": { "drawStyle": "bars", "fillOpacity": 80, "stacking": { "mode": "normal" } } } },
"gridPos": { "h": 10, "w": 24, "x": 0, "y": 0 },
"id": 1, "title": "Daily Cost by Team", "type": "timeseries",
"targets": [{ "expr": "sum by (label_team) (sum by (namespace, label_team) (kubecost_container_cpu_allocation_cost{} * on(namespace) group_left(label_team) kube_namespace_labels{label_team!=\"\"}) + sum by (namespace, label_team) (kubecost_container_memory_allocation_cost{} * on(namespace) group_left(label_team) kube_namespace_labels{label_team!=\"\"}))", "legendFormat": "{{label_team}}" }]
},
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": { "defaults": { "unit": "currencyUSD" } },
"gridPos": { "h": 10, "w": 12, "x": 0, "y": 10 },
"id": 2, "title": "Monthly Cost by Service", "type": "bargauge",
"targets": [{ "expr": "sum by (label_service) ((kubecost_container_cpu_allocation_cost{} + kubecost_container_memory_allocation_cost{}) * on(pod) group_left(label_service) kube_pod_labels{label_service!=\"\"}) * 730", "legendFormat": "{{label_service}}" }]
},
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": { "defaults": { "unit": "percentunit", "min": 0, "max": 1 } },
"gridPos": { "h": 10, "w": 12, "x": 12, "y": 10 },
"id": 3, "title": "Resource Efficiency by Team", "type": "bargauge",
"targets": [{ "expr": "sum by (label_team) (rate(container_cpu_usage_seconds_total{namespace!~\"kube-system|monitoring\"}[1h]) * on(namespace) group_left(label_team) kube_namespace_labels{label_team!=\"\"}) / sum by (label_team) (kube_pod_container_resource_requests{resource=\"cpu\", namespace!~\"kube-system|monitoring\"} * on(namespace) group_left(label_team) kube_namespace_labels{label_team!=\"\"})", "legendFormat": "{{label_team}}" }]
},
{
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": { "defaults": { "unit": "currencyUSD" } },
"gridPos": { "h": 8, "w": 24, "x": 0, "y": 20 },
"id": 4, "title": "Team Cost Summary Table", "type": "table",
"targets": [
{ "expr": "sum by (label_team) (kubecost_container_cpu_allocation_cost{} + kubecost_container_memory_allocation_cost{}) * 730", "format": "table", "instant": true, "refId": "A" },
{ "expr": "sum by (label_team) (rate(container_cpu_usage_seconds_total{}[1h])) / sum by (label_team) (kube_pod_container_resource_requests{resource=\"cpu\"})", "format": "table", "instant": true, "refId": "B" }
]
}
],
"schemaVersion": 39, "tags": ["finops", "cost", "showback"],
"templating": { "list": [{ "name": "team", "type": "query", "query": "label_values(kube_namespace_labels{label_team!=\"\"}, label_team)", "includeAll": true, "multi": true }] },
"time": { "from": "now-30d", "to": "now" },
"title": "FinOps Showback Dashboard", "uid": "finops-showback-v1"
}4. コスト異常検知
コスト異常は、誤設定、トラフィック急増、または infrastructure 変更による予期しない支出変化を示します。早期に検知することで、請求ショックを防げます。
4.1 Kubecost Alert 設定
yaml
# kubecost-alerts-values.yaml (merge with main Kubecost Helm values)
kubecostProductConfigs:
alertConfigs:
enabled: true
frontendUrl: "https://kubecost.internal.mycompany.com"
alerts:
# Budget exceeded - any namespace over $5000/month
- type: budget
threshold: 5000
window: 30d
aggregation: namespace
slackWebhookUrl: "https://hooks.slack.com/services/T00/B00/XXX"
frequencyMinutes: 1440
# Budget warning at 80%
- type: budget
threshold: 4000
window: 30d
aggregation: namespace
slackWebhookUrl: "https://hooks.slack.com/services/T00/B00/XXX"
frequencyMinutes: 1440
# Cluster efficiency below 40%
- type: efficiency
threshold: 0.4
window: 48h
aggregation: cluster
slackWebhookUrl: "https://hooks.slack.com/services/T00/B00/XXX"
frequencyMinutes: 360
# 30% cost increase week over week per team
- type: recurringUpdate
threshold: 0.30
window: 7d
aggregation: "label:team"
slackWebhookUrl: "https://hooks.slack.com/services/T00/B00/XXX"
frequencyMinutes: 10080
# Daily spend exceeds 150% of 7-day average
- type: spendChange
threshold: 0.50
window: 1d
baselineWindow: 7d
aggregation: namespace
slackWebhookUrl: "https://hooks.slack.com/services/T00/B00/XXX"
frequencyMinutes: 3604.2 Prometheus ベースのコスト Alerting
yaml
# cost-anomaly-prometheus-rules.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: cost-anomaly-detection
namespace: monitoring
labels:
release: prometheus
spec:
groups:
- name: cost-anomaly-detection
interval: 30m
rules:
- alert: ClusterCostSpike
expr: |
(sum(kubecost_cluster_costs{}) / avg_over_time(sum(kubecost_cluster_costs{})[7d:1h])) > 1.5
for: 2h
labels:
severity: warning
category: finops
annotations:
summary: "Cluster cost spike detected"
description: "Current cost is {{ $value | humanizePercentage }} of 7-day average."
- alert: NamespaceCostDoubled
expr: |
(
sum by (namespace) (kubecost_container_cpu_allocation_cost{} + kubecost_container_memory_allocation_cost{})
/ sum by (namespace) (kubecost_container_cpu_allocation_cost{} offset 1d + kubecost_container_memory_allocation_cost{} offset 1d)
) > 2.0
for: 1h
labels:
severity: warning
category: finops
annotations:
summary: "Namespace {{ $labels.namespace }} cost doubled day-over-day"
- alert: LowClusterCPUEfficiency
expr: |
(
sum(rate(container_cpu_usage_seconds_total{namespace!~"kube-system|monitoring"}[1h]))
/ sum(kube_pod_container_resource_requests{resource="cpu", namespace!~"kube-system|monitoring"})
) < 0.30
for: 6h
labels:
severity: warning
category: finops
annotations:
summary: "Cluster CPU efficiency below 30%"
description: "Current efficiency: {{ $value | humanizePercentage }}. Review VPA recommendations."
- alert: HighIdleCost
expr: |
(sum(kubecost_cluster_costs{cost_type="idle"}) / sum(kubecost_cluster_costs{})) > 0.20
for: 24h
labels:
severity: info
category: finops
annotations:
summary: "Idle cost exceeds 20% of total cluster cost"
- alert: ProjectedMonthlyBudgetExceeded
expr: |
(sum(kubecost_cluster_costs{}) * 730) > 50000
for: 12h
labels:
severity: critical
category: finops
annotations:
summary: "Projected monthly cost exceeds $50,000 budget"
description: "Projected: ${{ $value | printf \"%.0f\" }}. Immediate review required."Alertmanager Route と Receiver
yaml
# alertmanager-finops-config.yaml
apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
name: finops-alerts
namespace: monitoring
spec:
route:
receiver: "finops-slack"
groupBy: ["alertname", "namespace"]
groupWait: 30s
groupInterval: 5m
repeatInterval: 4h
matchers:
- name: category
value: finops
routes:
- receiver: "finops-slack-critical"
matchers:
- name: severity
value: critical
repeatInterval: 1h
receivers:
- name: "finops-slack"
slackConfigs:
- apiURL:
name: finops-slack-webhook
key: webhook-url
channel: "#finops-alerts"
sendResolved: true
title: "[{{ .CommonLabels.severity | toUpper }}] {{ .CommonLabels.alertname }}"
text: |
{{ range .Alerts }}
*Description:* {{ .Annotations.description }}
{{ end }}
- name: "finops-slack-critical"
slackConfigs:
- apiURL:
name: finops-slack-webhook
key: webhook-url
channel: "#finops-critical"
sendResolved: true
title: "[CRITICAL] {{ .CommonLabels.alertname }}"
text: |
{{ range .Alerts }}
*Description:* {{ .Annotations.description }}
*Runbook:* {{ .Annotations.runbook_url }}
{{ end }}
color: "danger"
---
apiVersion: v1
kind: Secret
metadata:
name: finops-slack-webhook
namespace: monitoring
type: Opaque
stringData:
webhook-url: "https://hooks.slack.com/services/T00/B00/XXXXXXXXXXXXXXXXXXXXXXXX"4.3 AWS Cost Anomaly Detection 統合
AWS Cost Anomaly Detection は、Kubernetes レベルの監視を補完する ML ベースの異常検知を提供します。
hcl
# aws-cost-anomaly-detection.tf
resource "aws_ce_anomaly_monitor" "eks_monitor" {
name = "eks-cost-anomaly-monitor"
monitor_type = "DIMENSIONAL"
monitor_dimension = "SERVICE"
}
resource "aws_sns_topic" "finops_alerts" {
name = "finops-cost-anomaly-alerts"
}
resource "aws_sns_topic_subscription" "finops_email" {
topic_arn = aws_sns_topic.finops_alerts.arn
protocol = "email"
endpoint = "finops-team@mycompany.com"
}
resource "aws_ce_anomaly_subscription" "eks_alerts" {
name = "eks-anomaly-alerts"
frequency = "DAILY"
monitor_arn_list = [aws_ce_anomaly_monitor.eks_monitor.arn]
subscriber {
type = "SNS"
address = aws_sns_topic.finops_alerts.arn
}
threshold_expression {
dimension {
key = "ANOMALY_TOTAL_IMPACT_ABSOLUTE"
values = ["100"]
match_options = ["GREATER_THAN_OR_EQUAL"]
}
}
}5. チームセルフサービスのコスト管理
セルフサービス型のコスト管理は、FinOps を platform team の外へ拡張します。すべての engineering team が独立してコストを確認し、budget alerts に対応できるようになると、FinOps team は戦略に集中できます。
5.1 チームごとのコストダッシュボード
各チームが自分たちのコストだけを確認できる、変数駆動の Grafana dashboard です。主要な panels とその PromQL queries は次のとおりです。
yaml
# grafana-team-dashboard-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: grafana-team-cost-dashboard
namespace: monitoring
labels:
grafana_dashboard: "true"
data:
team-cost-dashboard.json: |
{
"panels": [
{ "id": 1, "title": "Projected Monthly Cost", "type": "stat", "gridPos": { "h": 4, "w": 8, "x": 0, "y": 0 },
"fieldConfig": { "defaults": { "unit": "currencyUSD" } },
"targets": [{ "expr": "sum(kubecost_container_cpu_allocation_cost{namespace=~\"team-$team.*\"} + kubecost_container_memory_allocation_cost{namespace=~\"team-$team.*\"}) * 730" }] },
{ "id": 2, "title": "CPU Efficiency", "type": "stat", "gridPos": { "h": 4, "w": 8, "x": 8, "y": 0 },
"fieldConfig": { "defaults": { "unit": "percentunit" } },
"targets": [{ "expr": "sum(rate(container_cpu_usage_seconds_total{namespace=~\"team-$team.*\"}[1h])) / sum(kube_pod_container_resource_requests{resource=\"cpu\", namespace=~\"team-$team.*\"})" }] },
{ "id": 3, "title": "Memory Efficiency", "type": "stat", "gridPos": { "h": 4, "w": 8, "x": 16, "y": 0 },
"fieldConfig": { "defaults": { "unit": "percentunit" } },
"targets": [{ "expr": "sum(container_memory_working_set_bytes{namespace=~\"team-$team.*\"}) / sum(kube_pod_container_resource_requests{resource=\"memory\", namespace=~\"team-$team.*\"})" }] },
{ "id": 4, "title": "Daily Cost Trend", "type": "timeseries", "gridPos": { "h": 8, "w": 24, "x": 0, "y": 4 },
"targets": [{ "expr": "sum(kubecost_container_cpu_allocation_cost{namespace=~\"team-$team.*\"} + kubecost_container_memory_allocation_cost{namespace=~\"team-$team.*\"}) * 24", "legendFormat": "Daily Cost" }] },
{ "id": 5, "title": "Cost by Service", "type": "piechart", "gridPos": { "h": 8, "w": 12, "x": 0, "y": 12 },
"targets": [{ "expr": "sum by (label_service) (kubecost_container_cpu_allocation_cost{namespace=~\"team-$team.*\"} + kubecost_container_memory_allocation_cost{namespace=~\"team-$team.*\"}) * 730", "legendFormat": "{{label_service}}" }] }
],
"templating": { "list": [{ "name": "team", "type": "query", "query": "label_values(kube_namespace_labels{label_team!=\"\"}, label_team)", "refresh": 2 }] },
"title": "Team Cost Self-Service", "uid": "finops-team-self-service-v1"
}5.2 Slack コストレポート Bot
Kubecost に問い合わせ、整形済みのコストレポートを Slack に投稿する週次 CronJob です。
yaml
# slack-cost-report-cronjob.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: cost-report-script
namespace: kubecost
data:
send-cost-report.sh: |
#!/bin/bash
set -euo pipefail
KUBECOST_URL="${KUBECOST_URL:-http://kubecost-cost-analyzer.kubecost.svc:9090}"
echo "Generating cost report for window: ${REPORT_WINDOW}"
ALLOCATION_DATA=$(curl -sf "${KUBECOST_URL}/model/allocation?window=${REPORT_WINDOW}&aggregate=label:team&accumulate=true&shareIdle=weighted&shareNamespaces=kube-system,monitoring")
TOTAL_COST=$(echo "${ALLOCATION_DATA}" | jq '[.data[0] | to_entries[].value.totalCost] | add | round')
TEAM_BREAKDOWN=$(echo "${ALLOCATION_DATA}" | jq -r '
.data[0] | to_entries | sort_by(-.value.totalCost) | .[]
| select(.key != "__idle__" and .key != "__unallocated__")
| "| \(.key) | $\(.value.totalCost | round) | \(.value.cpuEfficiency * 100 | round)% | \(.value.ramEfficiency * 100 | round)% |"
')
SLACK_PAYLOAD=$(cat <<PAYLOAD
{
"blocks": [
{ "type": "header", "text": { "type": "plain_text", "text": "Weekly Kubernetes Cost Report - ${CLUSTER_NAME}" } },
{ "type": "section", "text": { "type": "mrkdwn", "text": "*Report Period:* Last ${REPORT_WINDOW}\n*Total Cluster Cost:* \$${TOTAL_COST}" } },
{ "type": "divider" },
{ "type": "section", "text": { "type": "mrkdwn", "text": "*Cost by Team:*\n| Team | Cost | CPU Eff | Mem Eff |\n|------|------|---------|---------|${TEAM_BREAKDOWN}" } },
{ "type": "section", "text": { "type": "mrkdwn", "text": "<https://kubecost.internal.mycompany.com|View in Kubecost> | <https://grafana.internal.mycompany.com/d/finops-showback-v1|Dashboard>" } }
]
}
PAYLOAD
)
curl -sf -X POST "${SLACK_WEBHOOK_URL}" -H "Content-Type: application/json" -d "${SLACK_PAYLOAD}"
echo "Cost report sent successfully"
---
apiVersion: batch/v1
kind: CronJob
metadata:
name: weekly-cost-report
namespace: kubecost
spec:
schedule: "0 9 * * 1" # Every Monday 9:00 AM UTC
timeZone: "America/New_York"
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: 4
failedJobsHistoryLimit: 2
jobTemplate:
spec:
backoffLimit: 2
activeDeadlineSeconds: 300
template:
metadata:
labels: { app: cost-report-bot, team: platform }
spec:
serviceAccountName: cost-report-bot
restartPolicy: OnFailure
containers:
- name: cost-reporter
image: curlimages/curl:8.7.1
command: ["/bin/sh", "/scripts/send-cost-report.sh"]
env:
- name: KUBECOST_URL
value: "http://kubecost-cost-analyzer.kubecost.svc:9090"
- name: SLACK_WEBHOOK_URL
valueFrom:
secretKeyRef: { name: cost-report-slack-webhook, key: webhook-url }
- name: REPORT_WINDOW
value: "7d"
- name: CLUSTER_NAME
value: "production-eks-us-east-1"
resources:
requests: { cpu: "50m", memory: "64Mi" }
limits: { cpu: "200m", memory: "128Mi" }
volumeMounts:
- { name: scripts, mountPath: /scripts }
volumes:
- name: scripts
configMap: { name: cost-report-script, defaultMode: 0755 }
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: cost-report-bot
namespace: kubecost5.3 コスト予算設定と Alerts
yaml
# team-budgets-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: team-cost-budgets
namespace: kubecost
data:
budgets.json: |
{
"budgets": [
{ "team": "checkout", "monthlyBudget": 8000, "warningThreshold": 0.80, "criticalThreshold": 0.95, "slackChannel": "#checkout-alerts" },
{ "team": "payments", "monthlyBudget": 12000, "warningThreshold": 0.80, "criticalThreshold": 0.95, "slackChannel": "#payments-alerts" },
{ "team": "search", "monthlyBudget": 15000, "warningThreshold": 0.80, "criticalThreshold": 0.95, "slackChannel": "#search-alerts" },
{ "team": "platform", "monthlyBudget": 20000, "warningThreshold": 0.80, "criticalThreshold": 0.95, "slackChannel": "#platform-alerts" },
{ "team": "data-engineering", "monthlyBudget": 25000, "warningThreshold": 0.75, "criticalThreshold": 0.90, "slackChannel": "#data-eng-alerts" }
]
}
---
apiVersion: batch/v1
kind: CronJob
metadata:
name: budget-check
namespace: kubecost
spec:
schedule: "0 */6 * * *" # Every 6 hours
concurrencyPolicy: Forbid
jobTemplate:
spec:
backoffLimit: 1
activeDeadlineSeconds: 180
template:
spec:
serviceAccountName: cost-report-bot
restartPolicy: OnFailure
containers:
- name: budget-checker
image: curlimages/curl:8.7.1
command:
- /bin/sh
- -c
- |
set -euo pipefail
KUBECOST_URL="http://kubecost-cost-analyzer.kubecost.svc:9090"
ALLOCATION=$(curl -sf "${KUBECOST_URL}/model/allocation?window=thismonth&aggregate=label:team&accumulate=true&shareIdle=weighted")
DAY_OF_MONTH=$(date +%d)
DAYS_IN_MONTH=$(date -d "$(date +%Y-%m-01) +1 month -1 day" +%d)
TEAMS=$(cat /config/budgets.json | jq -r '.budgets[].team')
for TEAM in ${TEAMS}; do
BUDGET=$(jq -r ".budgets[] | select(.team == \"${TEAM}\") | .monthlyBudget" /config/budgets.json)
CRITICAL_PCT=$(jq -r ".budgets[] | select(.team == \"${TEAM}\") | .criticalThreshold" /config/budgets.json)
WARNING_PCT=$(jq -r ".budgets[] | select(.team == \"${TEAM}\") | .warningThreshold" /config/budgets.json)
CHANNEL=$(jq -r ".budgets[] | select(.team == \"${TEAM}\") | .slackChannel" /config/budgets.json)
ACTUAL=$(echo "${ALLOCATION}" | jq -r ".data[0][\"${TEAM}\"].totalCost // 0 | round")
PROJECTED=$(echo "scale=0; ${ACTUAL} * ${DAYS_IN_MONTH} / ${DAY_OF_MONTH}" | bc)
USAGE=$(echo "scale=4; ${PROJECTED} / ${BUDGET}" | bc)
if [ "$(echo "${USAGE} >= ${CRITICAL_PCT}" | bc)" = "1" ]; then
curl -sf -X POST "${SLACK_WEBHOOK_URL}" -H "Content-Type: application/json" \
-d "{\"channel\":\"${CHANNEL}\",\"text\":\":rotating_light: CRITICAL - Team *${TEAM}*: Projected \$${PROJECTED}/\$${BUDGET}\"}"
elif [ "$(echo "${USAGE} >= ${WARNING_PCT}" | bc)" = "1" ]; then
curl -sf -X POST "${SLACK_WEBHOOK_URL}" -H "Content-Type: application/json" \
-d "{\"channel\":\"${CHANNEL}\",\"text\":\":warning: Warning - Team *${TEAM}*: Projected \$${PROJECTED}/\$${BUDGET}\"}"
fi
done
env:
- name: SLACK_WEBHOOK_URL
valueFrom:
secretKeyRef: { name: cost-report-slack-webhook, key: webhook-url }
resources:
requests: { cpu: "50m", memory: "64Mi" }
limits: { cpu: "200m", memory: "128Mi" }
volumeMounts:
- { name: budget-config, mountPath: /config }
volumes:
- name: budget-config
configMap: { name: team-cost-budgets }6. リソース Rightsizing 自動化
Rightsizing は、resource requests と limits を実際のワークロード使用量に合わせます。過剰なプロビジョニングはコストを浪費し、過小なプロビジョニングは OOM kills や throttling を引き起こします。
6.1 VPA Recommendation ワークフロー
自動適用なしでリソース変更を提案するために、VPA を recommendation-only モード (updateMode: "Off") で実行します。
yaml
# vpa-recommendation-mode.yaml
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
name: order-service-vpa
namespace: team-checkout
labels:
team: checkout
finops-rightsizing: "true"
spec:
targetRef:
apiVersion: apps/v1
kind: Deployment
name: order-service
updatePolicy:
updateMode: "Off"
resourcePolicy:
containerPolicies:
- containerName: order-service
minAllowed: { cpu: "50m", memory: "64Mi" }
maxAllowed: { cpu: "4", memory: "8Gi" }
controlledResources: ["cpu", "memory"]
controlledValues: RequestsAndLimits
---
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
name: payment-processor-vpa
namespace: team-payments
labels:
team: payments
finops-rightsizing: "true"
spec:
targetRef:
apiVersion: apps/v1
kind: Deployment
name: payment-processor
updatePolicy:
updateMode: "Off"
resourcePolicy:
containerPolicies:
- containerName: payment-processor
minAllowed: { cpu: "100m", memory: "128Mi" }
maxAllowed: { cpu: "8", memory: "16Gi" }
controlledResources: ["cpu", "memory"]
controlledValues: RequestsAndLimitsRecommendations を確認します。
bash
# Get VPA recommendations across all namespaces
kubectl get vpa -A -o custom-columns=\
'NAMESPACE:.metadata.namespace,NAME:.metadata.name,TARGET_CPU:.status.recommendation.containerRecommendations[0].target.cpu,TARGET_MEM:.status.recommendation.containerRecommendations[0].target.memory'6.2 Goldilocks ダッシュボード
Goldilocks は、label 付けされた namespaces 内のすべての Deployment に対して VPA を実行し、現在のリソースと推奨リソースを比較する web dashboard を提供します。
yaml
# goldilocks-values.yaml
# helm install goldilocks fairwinds-stable/goldilocks -n goldilocks --create-namespace -f goldilocks-values.yaml
vpa:
enabled: true
updater:
enabled: false # Recommendations only
dashboard:
enabled: true
replicaCount: 2
resources:
requests: { cpu: "50m", memory: "64Mi" }
limits: { cpu: "200m", memory: "128Mi" }
ingress:
enabled: true
ingressClassName: "alb"
annotations:
alb.ingress.kubernetes.io/scheme: "internal"
hosts:
- host: "goldilocks.internal.mycompany.com"
paths:
- path: /
pathType: Prefix
controller:
enabled: true
resources:
requests: { cpu: "50m", memory: "64Mi" }
limits: { cpu: "200m", memory: "128Mi" }Namespaces で Goldilocks を有効化します。
bash
kubectl label namespace team-checkout goldilocks.fairwinds.com/enabled=true
kubectl label namespace team-payments goldilocks.fairwinds.com/enabled=true
kubectl label namespace team-search goldilocks.fairwinds.com/enabled=true
kubectl label namespace team-platform goldilocks.fairwinds.com/enabled=true
# Verify
kubectl get namespaces -l goldilocks.fairwinds.com/enabled=true6.3 自動リソース調整 Pipeline
成熟した組織では、VPA recommendations をレビュー用の pull requests を作成する自動 pipeline に流すことができます。
yaml
# rightsizing-pipeline-cronjob.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: rightsizing-script
namespace: kubecost
data:
collect-recommendations.sh: |
#!/bin/bash
set -euo pipefail
OUTPUT_DIR="/tmp/recommendations"
mkdir -p "${OUTPUT_DIR}"
VPAS=$(kubectl get vpa -A -l finops-rightsizing=true -o json)
echo "${VPAS}" | jq -c '.items[]' | while read -r VPA; do
NS=$(echo "${VPA}" | jq -r '.metadata.namespace')
TARGET_NAME=$(echo "${VPA}" | jq -r '.spec.targetRef.name')
REC_CPU=$(echo "${VPA}" | jq -r '.status.recommendation.containerRecommendations[0].target.cpu // empty')
REC_MEM=$(echo "${VPA}" | jq -r '.status.recommendation.containerRecommendations[0].target.memory // empty')
[ -z "${REC_CPU}" ] && continue
CURRENT=$(kubectl get deployment "${TARGET_NAME}" -n "${NS}" -o jsonpath='{.spec.template.spec.containers[0].resources.requests}')
CUR_CPU=$(echo "${CURRENT}" | jq -r '.cpu // "0"')
CUR_MEM=$(echo "${CURRENT}" | jq -r '.memory // "0"')
echo "${NS}/${TARGET_NAME}: CPU ${CUR_CPU} -> ${REC_CPU}, Memory ${CUR_MEM} -> ${REC_MEM}"
cat > "${OUTPUT_DIR}/${NS}-${TARGET_NAME}.json" <<EOF
{"namespace":"${NS}","name":"${TARGET_NAME}","current":{"cpu":"${CUR_CPU}","memory":"${CUR_MEM}"},"recommended":{"cpu":"${REC_CPU}","memory":"${REC_MEM}"}}
EOF
done
echo "Collected $(ls ${OUTPUT_DIR}/*.json 2>/dev/null | wc -l) recommendations"
---
apiVersion: batch/v1
kind: CronJob
metadata:
name: rightsizing-recommendations
namespace: kubecost
spec:
schedule: "0 6 * * 1" # Monday 6:00 AM UTC
concurrencyPolicy: Forbid
jobTemplate:
spec:
backoffLimit: 1
template:
spec:
serviceAccountName: rightsizing-bot
restartPolicy: OnFailure
containers:
- name: recommender
image: bitnami/kubectl:1.30
command: ["/bin/bash", "/scripts/collect-recommendations.sh"]
resources:
requests: { cpu: "100m", memory: "128Mi" }
limits: { cpu: "500m", memory: "256Mi" }
volumeMounts:
- { name: scripts, mountPath: /scripts }
volumes:
- name: scripts
configMap: { name: rightsizing-script, defaultMode: 0755 }
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: rightsizing-bot
namespace: kubecost
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: rightsizing-reader
rules:
- apiGroups: ["autoscaling.k8s.io"]
resources: ["verticalpodautoscalers"]
verbs: ["get", "list"]
- apiGroups: ["apps"]
resources: ["deployments", "statefulsets"]
verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: rightsizing-reader-binding
subjects:
- kind: ServiceAccount
name: rightsizing-bot
namespace: kubecost
roleRef:
kind: ClusterRole
name: rightsizing-reader
apiGroup: rbac.authorization.k8s.io7. コスト最適化ガバナンス
7.1 アイドルリソースの自動検出
意味のあるトラフィックなしにリソースを消費しているワークロードを特定するための PromQL queries です。
7 日間で CPU requests の 1% 未満しか使用していない Deployments:
promql
(
sum by (namespace, deployment) (
rate(container_cpu_usage_seconds_total{namespace!~"kube-system|monitoring|kubecost"}[7d])
)
/
sum by (namespace, deployment) (
kube_pod_container_resource_requests{resource="cpu", namespace!~"kube-system|monitoring|kubecost"}
* on(pod) group_left(deployment) kube_pod_owner{owner_kind="ReplicaSet"}
)
) < 0.017 日間で memory requests の 10% 未満しか使用していない Deployments:
promql
(
sum by (namespace, deployment) (
avg_over_time(container_memory_working_set_bytes{namespace!~"kube-system|monitoring"}[7d])
)
/
sum by (namespace, deployment) (
kube_pod_container_resource_requests{resource="memory", namespace!~"kube-system|monitoring"}
* on(pod) group_left(deployment) kube_pod_owner{owner_kind="ReplicaSet"}
)
) < 0.107 日間ネットワークトラフィックがゼロの Deployments:
promql
sum by (namespace, pod) (
increase(container_network_receive_bytes_total{namespace!~"kube-system|monitoring"}[7d])
) == 0bound されているがどの pod にも mounted されていない PVCs:
promql
kube_persistentvolumeclaim_status_phase{phase="Bound"}
unless on(persistentvolumeclaim, namespace) kube_pod_spec_volumes_persistentvolumeclaims_info7.2 コスト Policies (Kyverno)
Resource Limits のない Deployments をブロックする
yaml
# kyverno-require-resource-limits.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: require-resource-limits
annotations:
policies.kyverno.io/title: Require Resource Limits
policies.kyverno.io/category: FinOps
policies.kyverno.io/severity: high
spec:
validationFailureAction: Enforce
background: true
rules:
- name: validate-resource-limits
match:
any:
- resources:
kinds:
- Pod
exclude:
any:
- resources:
namespaces:
- kube-system
- kube-public
validate:
message: >-
All containers must define CPU and memory limits.
Add resources.limits.cpu and resources.limits.memory to your container spec.
foreach:
- list: "request.object.spec.containers"
deny:
conditions:
any:
- key: "{{ element.resources.limits.cpu || '' }}"
operator: Equals
value: ""
- key: "{{ element.resources.limits.memory || '' }}"
operator: Equals
value: ""過剰プロビジョニングされた Resources に警告する
yaml
# kyverno-warn-over-provisioned.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: warn-over-provisioned-resources
annotations:
policies.kyverno.io/title: Warn on Over-Provisioned Resources
policies.kyverno.io/category: FinOps
policies.kyverno.io/severity: medium
spec:
validationFailureAction: Audit
background: true
rules:
- name: warn-high-cpu-request
match:
any:
- resources:
kinds:
- Pod
exclude:
any:
- resources:
namespaces:
- kube-system
- monitoring
validate:
message: >-
Container '{{ element.name }}' requests {{ element.resources.requests.cpu }} CPU.
Requests above 4 CPU cores should be reviewed with VPA recommendations.
foreach:
- list: "request.object.spec.containers"
deny:
conditions:
all:
- key: "{{ element.resources.requests.cpu || '0' }}"
operator: GreaterThan
value: "4000m"
- name: warn-high-memory-request
match:
any:
- resources:
kinds:
- Pod
exclude:
any:
- resources:
namespaces:
- kube-system
- monitoring
validate:
message: >-
Container '{{ element.name }}' requests {{ element.resources.requests.memory }} memory.
Requests above 8Gi should be reviewed with VPA recommendations.
foreach:
- list: "request.object.spec.containers"
deny:
conditions:
all:
- key: "{{ element.resources.requests.memory || '0' }}"
operator: GreaterThan
value: "8Gi"7.3 定期的なコストレビュープロセス
レビューの実施頻度
| レビュー種別 | 頻度 | 参加者 | 所要時間 | 主なアジェンダ |
|---|---|---|---|---|
| Team Sprint Review | 2 週間ごと | Team lead、engineers | 15 分 | チームダッシュボードのレビュー、rightsizing recommendations への対応 |
| Weekly FinOps Standup | 週次 (月曜) | FinOps lead、platform eng | 30 分 | anomaly alerts のトリアージ、最適化アクションの優先順位付け |
| Monthly Cost Review | 月次 | Engineering leads、finance | 60 分 | 予算と実績、最適化 ROI、翌月予測 |
| Quarterly Business Review | 四半期ごと | Leadership、FinOps、finance | 90 分 | Unit economics、cost per customer、戦略的 savings |
月次レビューテンプレート
| セクション | 内容 | データソース |
|---|---|---|
| Executive Summary | 総支出、MoM 変化、予算状況 | Kubecost 月次レポート |
| Cost by Team | 効率スコア付きの内訳 | Kubecost Allocation API |
| Top 5 Cost Drivers | 支出または増加率が最も高い services | Kubecost trend analysis |
| Optimization Wins | rightsizing、クリーンアップによる削減 | Before/after comparisons |
| Anomalies | 調査済みの原因不明なコスト変化 | Anomaly alert history |
| Rightsizing Backlog | まだ適用されていない VPA recommendations | Goldilocks dashboard |
| Idle Resources | クリーンアップ対象として特定された resources | PromQL idle detection queries |
| Action Items | 割り当てられた owners と due dates | Previous review follow-up |
8. ベストプラクティス
最適化の前に可視化から始める。 最適化 recommendations を出す前に Kubecost または OpenCost をデプロイし、2〜4 週間分のデータを収集します。正確なコストデータがなければ、最適化は当て推量になります。
初日から labels を強制する。 最初から admission requirement としてコスト labels を強制するために Kyverno を使用します。数百のワークロードに後から labels を付けるのは大変であり、labels の欠落は信頼を損なう「unallocated」コストを生みます。
まず recommendation mode で VPA を使用する。 Recommendation mode で少なくとも 2 週間運用する前に、本番で VPA auto-update を有効にしてはいけません。Auto-updates は pod restarts を引き起こし、不正確な recommendations は障害を引き起こす可能性があります。
Showback と chargeback のタイムラインを分ける。 Chargeback を実装する前に、チームへ 2〜3 か月の showback 可視性を提供します。これによりデータへの信頼が構築され、チームに最適化する時間が与えられます。
共有コストを透明に扱う。 文書化された方法論を使用して共有 infrastructure コストを配分し、内訳をダッシュボードで明確に示します。隠れたコストは不信と争いを生みます。
15〜20% のバッファを持って予算を設定する。 厳しすぎる予算は alert fatigue を生み、実験を妨げます。チームがコスト管理への自信を高めるにつれて、徐々に引き締めます。
コストを個人ではなくチームレベルの metric にする。 個々の engineer レベルでのコスト説明責任は、ゆがんだインセンティブと非難文化を生みます。チームまたは service レベルに保ちます。
レビュープロセスを自動化する。 週次 Slack reports、budget alerts、rightsizing recommendation collection を自動化します。手動プロセスはスケールしません。
アンチパターン
| アンチパターン | 問題 | 解決策 |
|---|---|---|
| Cost data hoarding | Platform team だけがコストを見られ、engineers は盲目になる | チーム向けセルフサービスダッシュボードをデプロイし、週次 Slack reports を自動化する |
| Alert-only FinOps | Alerts は発火するが誰も対応しない | すべての alert に runbook と assigned owner を紐づけ、解決時間を追跡する |
| Over-optimizing non-production | Dev/staging (全体の小さな割合) に engineering time を費やす | まず本番に集中し、non-prod には単純な policies (夜間 scale-to-zero) を使用する |
| Ignoring data transfer costs | compute に集中する一方で network costs が静かに増える | network costs をダッシュボードに含め、CUR data を統合し、cross-AZ traffic をレビューする |
9. 参考資料
外部参考資料
- OpenCost ドキュメント - Open-source Kubernetes コスト監視
- Kubecost ドキュメント - Enterprise Kubernetes コスト管理
- AWS Cost and Usage Report - AWS 請求データエクスポート
- FinOps Foundation - FinOps のベストプラクティスとコミュニティ
- FinOps Framework - Inform、Optimize、Operate ライフサイクル
- Vertical Pod Autoscaler - Kubernetes VPA
- Fairwinds の Goldilocks - VPA recommendation dashboard
- Kyverno Policy Library - Kubernetes の Policy 例
内部参考資料
- EKS コスト最適化 - EKS 向け AWS 固有のコスト最適化戦略
- リソース最適化 - 詳細な resource requests/limits チューニングと framework 固有のガイド
- スケーリング戦略 - HPA、KEDA、VPA、Spot 利用戦略