Which Prometheus Label Is Multiplying Your Time Series?

A Prometheus server can run out of memory even when scrape traffic, retention and disk use look ordinary. The hidden multiplier is often a label whose values grow with users, request paths, containers, pods, sessions or IDs. Every unique metric name plus complete label set is a separate time series, so one new dimension can multiply—not merely add to—the active series held in the TSDB head.

Treat the incident as an ownership problem. First prove that head-series count or churn is rising; then identify the metric, label name and target responsible. Only after dashboards, alerts and recording rules are mapped should you drop a label, disable an exporter collector or redesign instrumentation. Adding RAM before that evidence can buy time while leaving the multiplier intact.

This runbook targets self-managed Prometheus. Managed services and compatible backends may expose different cardinality tools or impose their own limits, so use their documentation for product-specific controls.

Memory pressure is the symptom, not the owner

Prometheus keeps the current head block in memory and does not fully persist it until block compaction. Its storage documentation explains that boundary. A shorter historical retention period can reduce disk use, but it does not directly remove the active label sets currently occupying the head.

Begin with a synchronized sample of process memory, head series and series creation/removal rates:

process_resident_memory_bytes
prometheus_tsdb_head_series
rate(prometheus_tsdb_head_series_created_total[5m])
rate(prometheus_tsdb_head_series_removed_total[5m])

One high value is not enough. Record several samples across a representative interval and align them with a deployment, autoscaling event or target change. A stable head-series count with rising memory may point toward queries, compaction, remote write or another component; sustained series growth makes cardinality a stronger owner.

If Linux already killed the process, use kernel and cgroup OOM evidence to identify whether host exhaustion, a service limit or systemd-oomd acted. That evidence answers who terminated Prometheus; the TSDB investigation below answers why its working set grew.

One label value can multiply the whole metric

Prometheus metric naming guidance states that every unique combination of label values creates another series and warns against unbounded dimensions such as user IDs or email addresses. Labels are useful for small, meaningful groups such as method, status family or region. They are dangerous when a value behaves like an event identifier.

For a worked upper bound, imagine one histogram bucket family described by five HTTP methods, forty normalized routes, six status groups and 120 short-lived pods. If every combination occurs, one bucket can create 144,000 series before counting other histogram buckets or replicas.

Route templates such as /orders/{id} keep cardinality bounded; raw paths such as /orders/981273 create a new value for each order. Similar risk appears when Kubernetes pod UID, container hash or ephemeral hostname is retained even though queries need only workload, namespace or service.

The broader self-hosted application workload can include private metrics, automation and service checks on one server, but shared infrastructure does not make every label useful. Keep identity dimensions in logs or traces when operators need per-request detail; metrics should aggregate dimensions that support alerts and trends.

Ask the TSDB which metric owns the series

Prometheus exposes cardinality summaries at /api/v1/status/tsdb. Query it through a private administrative route or directly on the host; do not publish this endpoint merely for troubleshooting.

curl --fail --silent --show-error http://127.0.0.1:9090/api/v1/status/tsdb > /tmp/prometheus-tsdb-status.json
jq '.data.headStats, .data.seriesCountByMetricName[:20], .data.labelValueCountByLabelName[:20], .data.seriesCountByLabelValuePair[:20]' /tmp/prometheus-tsdb-status.json

The Prometheus HTTP API reference documents these fields. seriesCountByMetricName identifies large metric families; labelValueCountByLabelName exposes labels with many distinct values; seriesCountByLabelValuePair reveals combinations attached to many series. Memory-by-label-name is an estimate based on label-value string lengths, not a complete per-label share of process RSS.

Connect the metric to a scrape target

Once a metric name stands out, find which jobs and instances produce it:

count by (job, instance) ({__name__="http_request_duration_seconds_bucket"})

Replace the metric with the actual owner. If one target dominates, inspect its exporter or application instrumentation. If every target carries the same wasteful dimension, the change belongs in shared scrape configuration or a common library rather than one host.

Count the suspicious label directly

PromQL can compare how many series remain after removing a suspected dimension:

count({__name__="http_request_duration_seconds_bucket"})
count(count without (pod) ({__name__="http_request_duration_seconds_bucket"}))

The first expression counts current series. The second counts unique label sets after pod is ignored. Their difference estimates how much that dimension multiplies the selected metric, but it does not prove the label is safe to remove. Query owners may still depend on per-pod data.

Separate persistent cardinality from churn

High cardinality means many unique active label sets. Churn means series are created and removed rapidly, often as pods, containers or batch jobs change identity. Both consume resources, yet a point-in-time total can hide churn when old series disappear almost as quickly as new ones arrive.

Compare creation and removal rates with deployment events. For an offline copy of a TSDB, promtool tsdb analyze can report churn and label-pair cardinality. Never run an unplanned offline analysis against the live data directory; work from an approved copy or maintenance procedure so the active Prometheus process retains ownership.

Protect query contracts before changing ingestion

Dropping a label changes grouping, alert identity and dashboard drill-down. Search configuration repositories, Grafana dashboards, alert rules and recording rules for the metric and label before editing the scrape pipeline. Also note notification templates that display the label even when PromQL does not group by it.

Recording rules do not erase the source series by themselves. They create derived time series from queries while the original high-cardinality samples continue to ingest. A rule can give readers a stable aggregate, but memory falls only when instrumentation, exporter collection or ingestion changes.

Maintain outside-in visibility while the metrics contract changes. Private Uptime Kuma monitoring can verify a real endpoint independently of Prometheus, but it cannot replace internal saturation, queue or error signals. Use both views for different proof.

Remove the smallest unnecessary dimension

Choose the repair closest to the source of waste:

• Redesign application instrumentation when raw user, session, path or request identifiers were modeled as labels. Keep bounded categories in metrics and move detailed identity to logs or traces.
• Disable an exporter collector when its entire metric family has no alert, dashboard or capacity purpose.
• Drop one label during metric relabeling only when the metric remains useful without that dimension, every query owner approves the semantic change, and every resulting series remains uniquely labeled.
• Drop a metric family only when it has no required consumer and re-enabling it is documented.

Prometheus applies metric_relabel_configs after a target is scraped and before samples enter storage. A narrow example removes pod_uid from the dedicated app scrape job:

scrape_configs:
  - job_name: app
    static_configs:
      - targets: ["127.0.0.1:9101"]
    metric_relabel_configs:
      - regex: 'pod_uid'
        action: labeldrop

That rule drops pod_uid from every metric in the app scrape job, so use it only when the whole job treats that dimension as unnecessary and no two exposed series become identical after removal; labeldrop does not aggregate colliding series. When one metric alone is wrong, fixing instrumentation at source or isolating it into a separately reviewed scrape job is safer than a broad relabel rule.

Before reload, validate the complete file with the matching Prometheus release:

promtool check config /etc/prometheus/prometheus.yml

Stage the change on one non-critical Prometheus or one controlled target when topology permits. Save the previous configuration, expected series reduction, affected queries and a rollback owner. Reverting the configuration restores future ingestion, but samples dropped during the test were never stored.

Prove a stable series budget before resizing

After reload, watch head series, created/removed rates, scrape sample counts, rule evaluation failures and process memory through at least one normal workload cycle. Re-run the affected dashboards and alerts, then verify one representative service symptom through its real path.

Remote write adds its own memory cost because series IDs and labels are cached for WAL delivery. The remote-write tuning guide notes that users often report roughly 25% additional memory, with the actual amount depending on data shape and churn. A local series reduction should therefore be checked on both the Prometheus process and any remote-write queue.

Legitimate remaining demand may justify more memory, faster storage or isolation. Translate measured CPU, RAM, NVMe and network pressure through dedicated server hardware boundaries before choosing a platform. For sustained monitoring estates that need predictable resources, dedicated server capacity becomes defensible only after waste is removed and a normal series budget is recorded.

Set budget alerts on total head series, creation/removal rates, scrape samples per target and process memory. Budgets should reflect normal peaks plus deliberate headroom rather than a copied global threshold. Record the metric owner, expected label-value bounds and approval path for introducing a new dimension.

Close the incident with an owner and a bound

Prometheus cardinality is controlled when the team can name the metric, label and producer; explain why each retained dimension supports a query; and show a stable head-series range through a normal peak. Memory recovery alone is temporary if the label can resume unbounded growth tomorrow.

Save before/after TSDB summaries, affected queries, configuration diff, reload result, series budget and rollback record. Continue through Voxfor DevOps operations guides for adjacent monitoring and infrastructure workflows. The durable fix is a bounded metric contract, not merely a larger process limit.