VictoriaMetrics vs Thanos vs Grafana Cloud: Monthly Cost Comparison (1 Million Time Series)

Let's lock in the assumptions first

The calculations in this post are fixed to the conditions below. If your conditions differ, so will the conclusions—keep your own environment numbers next to you as you read.

Planning stages often assume "100 million samples/month," but 1 million active series × a 30-second scrape interval arithmetically yields ~86 billion samples/month. Get this order of magnitude wrong in storage estimates and costs will be off by two to three orders of magnitude. First sanity-check with active series × (2,592,000 seconds ÷ scrape interval).

At this scale, a single Prometheus typically hits three symptoms.

1. Memory: The head-block index and symbol table stay resident; 30–50 GB RSS is common at 1 million series. When WAL replay overlaps a reload or restart, memory spikes and you get OOMKilled.
2. Retention: Local-disk retention means you need TB-scale EBS to hold 13 months locally, and there is no backup/restore story.
3. HA: Even with two instances, each has a separate dataset, so query results drift slightly, and any window where one is down remains a permanent hole.

So the options effectively narrow to three.

• A) Self-hosted Prometheus + Thanos
• B) Grafana Cloud Metrics (managed)
• C) Self-hosted VictoriaMetrics (single node → cluster)

Note: this post covers the metrics axis only. Log-stack costs (Loki/ELK/CloudWatch Logs) are billed per GB ingested, so the cost structure is completely different—mixing them here would muddy the conclusion. For overall infrastructure TCO methodology, see Cloud Cost Optimization Guide: Serverless vs Kubernetes TCO Comparison.

30-second verdict table: 5-axis decision matrix

I recommend picking the answer first and checking the rationale afterward.

• If you landed on B, just look at the B formula in section 3, then check whether you fail the air-gap conditions in section 6.
• If you landed on C, skim section 3 and jump straight to the migration runbook in section 7.
• If you landed on A, you must read the compactor failure branches in section 5. A's cost comes from incidents, not instances.

Monthly costs of the three options, formula by formula

Unit-price disclaimer: The unit prices below are assumptions based on public price lists as of August 2026 and will vary by region, commitment, and contract terms. Re-validate with the AWS Pricing Calculator and vendor quotes before any real decision. Labor is assumed at 60,000 KRW/hour (fully loaded, including salary and overhead).

A) Self-hosted Prometheus + Thanos

Topology: 2× Prometheus (HA, 15-day local) + 2× Thanos sidecar + 1× store gateway + 1× query + 1× compactor; long-term storage on S3.

• Infrastructure: 887 × 1,380 = ~1,224,000 KRW
• Ops effort: 16 hours/month × 60,000 KRW = 960,000 KRW (compactor watch, overlapping-block response, version upgrades, rule management, capacity forecasting)
• A monthly TCO ≈ 2,184,000 KRW

B) Grafana Cloud Metrics

For managed metrics, the active-series unit price decides everything. You have to split into two scenarios for an honest comparison.

One practical conclusion falls out here. B is not won by "managed is convenient"—it is won or lost by the unit price you actually get. Dumping 1 million series onto list price is 5×+ vs A and C; securing a volume-contract unit price brings it down near A. Do not put B on the shortlist until you have a negotiated quote. Also, egress scales with sample count, not active series, so stretching the scrape interval from 30s → 60s halves it.

C) VictoriaMetrics single node

1 million active series at 33k samples/s is widely reported as a range a single node (vmsingle) can absorb comfortably. A cluster (vminsert/vmselect/vmstorage) is usually considered when you hit millions-to-tens-of-millions of series or need multi-tenancy.

Disk sizing rationale: post-compression, ~0.4–0.7 bytes per sample is the commonly reported range. 86 billion samples/month × 0.5B ≈ 43GB/month; 13 months ≈ 560GB. Doubling for cardinality growth and indexes, we sized 1TB.

• Infrastructure: 621 × 1,380 = ~857,000 KRW
• Ops effort: 8 hours/month × 60,000 KRW = 480,000 KRW
• C monthly TCO ≈ 1,337,000 KRW

Totals comparison

One reversal. Strip out ops effort and compare infrastructure only, and A (1.22M) beats B volume-contract (2.12M). Fold effort back in and the A–B gap shrinks to ~180k KRW, leaving only the question: "Do you actually have someone who will spend 16 hours/month on Thanos?" If you don't, A's real cost is billed in incident hours, not invoices. If you do, C is the cheapest of the three, and that gap (~1M KRW/month) is 12M KRW/year.

Why the same 1 million series consume such different resources

Prometheus TSDB: where the index eats memory

Prometheus keeps recent data in the head block, writes it to the WAL, then cuts disk blocks every 2 hours. The memory hog is not the samples themselves but the index and symbol table. Every active series keeps a label set, posting-list entries, and chunk references resident, so memory grows with series count, not sample count. Stretching the scrape interval barely moves memory; dropping one label drops it hard—that's the asymmetry.

That's also why memory spikes well above steady state during WAL replay on restart. If the node is already near its memory ceiling, you fall into a restart → OOM during replay → restart loop.

Thanos: cheap storage, paid for on the query path

Thanos has the sidecar upload 2-hour blocks to S3, and the store gateway queries them. The problem is the query path. The store gateway loads block index headers and range-GETs the needed chunk ranges from object storage. If the index cache is cold or the query window is wide, those network round-trips become query latency. That's why you get fast last-2-hours, noticeably slow last-quarter behavior. If dashboards frequently use a 3-month range, budget extra for store-gateway memory and cache (in-memory or memcached/Redis).

VictoriaMetrics: compression and background merge

VictoriaMetrics stores time-series data in a near-columnar layout and applies delta / delta-of-delta style encoding to timestamps and values. Incoming data is written as small parts and merged into larger parts in the background (LSM-tree-like), so the ingest path depends less on a resident index. As a result, lower memory and disk occupancy vs Prometheus at the same series count is reported repeatedly.

It's not free, though. If merge falls behind, disk usage balloons temporarily and merge I/O hits query latency. Put vm_rows_merged_total and part counts on a dashboard. Benchmark numbers vary widely by hardware, label shape, and query pattern, so treat the above as a tendency, not an absolute, and dual-run against your own data for two weeks.

When and how it blows up: failure branches

Thanos operational difficulty (option A)

• The compactor is effectively a single-instance constraint. Two compactors on the same bucket/stream tangle the blocks. So the compactor is close to an SPOF; if it's down, downsampling and retention quietly stop. About 80% of "S3 bills keep climbing and I don't know why" is a stopped compactor.
• Overlapping blocks: Wrong external labels on an HA pair, or a sidecar double-upload, produces overlapping blocks and the compactor halts. Recovery is manual: find the blocks with thanos tools bucket verify / bucket inspect, then mark and delete them.
• Downsampling lag: If 5m/1h downsample blocks aren't produced, long-range queries scrape original resolution and time out.

Cardinality explosion: symptoms by A/B/C and the first 24 hours

The trigger is always similar: pod names, request_id, user_id, or full URL paths leaking in as labels.

First-24-hour response order (common)

1. Identify top series counts per metric with topk(20, count by (__name__)({__name__=~".+"})).
2. Check unique values per label on the offending metric → pinpoint which label exploded.
3. Block immediately at collection: labeldrop the label or drop the metric via metric_relabel_configs in scrape config.
4. File a ticket with the application team to remove the label (root fix).
5. Clean already-stored series: A uses the admin API delete_series; C uses /api/v1/admin/tsdb/delete_series.

# 수집 단계에서 고카디널리티 라벨을 잘라내는 최소 방어
metric_relabel_configs:
  - regex: '(request_id|trace_id|session_id|pod_template_hash)'
    action: labeldrop
  - source_labels: [__name__]
    regex: 'app_http_request_duration_seconds_bucket'
    action: drop   # 히스토그램 버킷이 범인일 때 임시 차단

VictoriaMetrics has more cushion: defensive flags like -maxLabelsPerTimeseries, -maxLabelValueLen, -search.maxUniqueTimeseries, plus vmagent's -remoteWrite.maxHourlySeries and -remoteWrite.maxDailySeries can cap ingest itself. But series over the cap are dropped, so you must alert on that.

Difficulty of changing retention

Korean operational realities: KRW, air-gapping, and docs

FX risk. B's cost is quoted in USD. If FX moves from 1,380 to 1,500 KRW, B in the numbers above simply grows by ~180k KRW/month. A and C are also affected because cloud instances are USD-priced, but the large labor share is in KRW, so the swing is relatively smaller. Put FX scenarios (±10%) in the budget-approval doc.

Domestic region and egress. If the managed backend has no domestic region, you lose twice. First, remote-write RTT and retries back up the vmagent/Prometheus queue. Second, internet outbound transfer is billed as-is. Before contracting, measure "actual RTT from our cluster to that endpoint" and "monthly egress GB."

In an air-gapped network, B drops off the shortlist. In finance and public-sector closed-network requirements, sending metrics to an external SaaS is often simply not allowed (whether a specific regulation applies must be confirmed against your organization's security policy and the relevant supervisory notices). The verdict then simplifies.

• Strike the B column and compare A vs C only → from the table above, A 2.18M vs C 1.34M.
• Air-gapped environments have no internet mirrors, so image/binary import procedures apply. Component count equals import-ticket count. Thanos means sidecar/store/query/compactor plus object storage (MinIO, etc.) as import targets; VictoriaMetrics is two binaries: vmsingle + vmagent.
• Bottom line: in air-gapped networks, both cost and import friction favor C. The exception is an org that already runs in-house S3-compatible storage as standard and has accumulated Thanos ops experience.

Docs and response time zones. Official docs for all three projects are English-first, and community responses cluster in the Korean-time early morning. Without a commercial support contract, assume first-line incident response is entirely internal when you plan headcount.

Migration runbook: Prometheus → VictoriaMetrics

Step 1: Dual-write parallel run (2 weeks recommended)

Leave the existing Prometheus as-is and only add remote_write. The point is that rollback is deleting one config block.

# prometheus.yml
global:
  scrape_interval: 30s
  external_labels:
    cluster: prod-a
    replica: "0"

remote_write:
  - url: http://vmsingle.monitoring.svc:8428/api/v1/write
    remote_timeout: 30s
    queue_config:
      capacity: 10000          # 샤드당 버퍼
      max_shards: 50           # 백로그 시 확장 상한
      min_shards: 4
      max_samples_per_send: 2000
      batch_send_deadline: 5s
      min_backoff: 100ms
      max_backoff: 5s
    write_relabel_configs:
      - regex: '(request_id|trace_id)'
        action: labeldrop

Metrics to watch during the parallel period:

# 원격 쓰기 백로그 (0 근처를 유지해야 정상)
prometheus_remote_storage_samples_pending

# 재시도/실패율
rate(prometheus_remote_storage_samples_failed_total[5m])

# 큐 지연 (초 단위, 지속 상승하면 max_shards 상향)
prometheus_remote_storage_queue_highest_sent_timestamp_seconds
  - ignoring(url) prometheus_remote_storage_highest_timestamp_in_seconds

Healthy result: samples_pending returns near 0 within tens of seconds after a spike; samples_failed_total growth rate is 0. If pending accumulates continuously, it is usually not network bandwidth but insufficient max_shards or receiver CPU saturation. Watch vmsingle CPU utilization together with vm_rows_inserted_total.

Step 2: Historical data migration

# Prometheus 로컬 TSDB → VictoriaMetrics
# 이관 중 Prometheus는 읽기 전용 스냅샷을 사용
curl -XPOST http://prometheus:9090/api/v1/admin/tsdb/snapshot

vmctl prometheus \
  --prom-snapshot=/prometheus/snapshots/20260819T031500Z-abcdef \
  --vm-addr=http://vmsingle:8428 \
  --vm-concurrency=4 \
  --vm-batch-size=200000 \
  --prom-filter-time-start=2025-08-01T00:00:00Z

# Thanos 버킷(S3)에 이미 장기 데이터가 있다면
vmctl remote-read \
  --remote-read-src-addr=http://thanos-query:9090 \
  --remote-read-filter-time-start=2025-08-01T00:00:00Z \
  --remote-read-step-interval=day \
  --vm-addr=http://vmsingle:8428

Healthy result: vmctl prints processed series/sample counts and exits 0. If context deadline exceeded repeats, halve --vm-batch-size and lower --remote-read-step-interval to hour.

Step 3: Validation queries

Fire the same query at both sides and check the delta.

Q='sum(rate(container_cpu_usage_seconds_total[5m]))'
for U in http://prometheus:9090 http://vmsingle:8428; do
  curl -sG "$U/api/v1/query" --data-urlencode "query=$Q" | jq -r '.data.result[0].value[1]'
done

A reasonable baseline is within 1% for rate-style queries and within 0.1% for counter totals. Large deltas are usually (1) migration start times that don't overlap, or (2) different external_labels splitting the series.

Step 4: Dashboard and alert compatibility checklist

• Add a Grafana datasource of type Prometheus (VictoriaMetrics provides a PromQL-compatible API)
• Individually check panels using subquery, @ modifier, or histogram_quantile
• Migrate recording/alerting rules to vmalert; keep evaluation interval and for values identical
• Confirm the Alertmanager integration path still works — for alert-pipeline design see Grafana Dashboard Prometheus Integration and Alerting Setup
• Separately measure response time of long-range (3-month) panels
• Backup: snapshot via /snapshot/create API → cron upload to object storage

Step 5: Rollback decision criteria

During the parallel period, if any one of the following is violated continuously for 48 hours, remove the remote_write block and roll back.

Conclusion: for this case, just use this

1. No dedicated SRE and not air-gapped → B (Grafana Cloud). But never go list-price. If you cannot lock a volume-contract unit price in writing, B drops off the shortlist that instant.
2. Air-gapped, or budget is tight and you can attach even 0.3 of an engineer → C (VictoriaMetrics). Under the conditions above it has the lowest monthly TCO, and with only two binaries to import, closed-network process cost is also the smallest.
3. Thanos already runs stably and the org standard is the Prometheus ecosystem → keep A. Here the switching gain (~850k KRW/month) is smaller than switching and relearning cost. Do hang a compactor health-check alert today, though.

The cost of swapping a metrics backend is clearly lower than it used to be, thanks to OpenTelemetry and remote_write standardization. Put the other way: the reason to stay on an expensive option has shrunk by the same amount. If you are redrawing the whole monitoring-stack architecture, also see Prometheus vs Grafana vs Datadog Comparison Guide. Log-axis costs use a different billing unit and need a separate calculation.

FAQ

Q. Will VictoriaMetrics really hold 1 million active series on a single node? A. It is generally reported as a range one 8 vCPU / 64 GB-class instance can absorb. The real criterion is not series count alone but concurrent query count and query range. If several people hit 3-month-range dashboards at once, CPU saturates first—observe vm_concurrent_select_capacity and CPU utilization for two weeks, then consider clustering.

Q. Is there a way to stop Grafana Cloud bills from suddenly jumping? A. The root control is blocking at collection. Drop high-cardinality labels with metric_relabel_configs before remote write, and alert on active-series count itself (e.g., warn on +20% vs yesterday). Also negotiate the vendor's usage dashboard and cap/alert options at contract time.

Q. Can we run VictoriaMetrics alongside Thanos without tearing Thanos out? A. Yes. Adding VictoriaMetrics to Prometheus remote_write leaves the existing sidecar → S3 path running, so you can keep both backends during the parallel period. Infrastructure cost doubles during that window, so schedule a decision within 2–4 weeks.

Q. Can we put the unit prices in the cost table straight into a budget proposal? A. No. The unit prices in this post are assumptions based on public price lists as of August 2026 and will vary by region, commitment, and FX. Reuse only the formula structure (instances × hours + storage GB + transfer + ops effort); re-validate the numbers with the AWS Pricing Calculator and official vendor quotes.