calricon.
Systems & Architecture

Fix Prometheus Out-of-Order and Duplicate Sample Errors

Nov 19, 2026/5 min read/Verified Production Playbook

calricon / field note

Production Prometheus instances frequently drop metrics with

Nov 19, 2026/Systems & Architecture
370
Advertisement

Fix Prometheus Out-of-Order and Duplicate Sample Errors

When Prometheus logs Error on ingesting samples that are too old or are too new, Error on ingesting out-of-order samples, or Error on ingesting samples with different value but same timestamp, your monitoring pipeline is actively dropping telemetry. In production Kubernetes and distributed observability stacks, these rejections create silent gaps in alerting rules, break SLO burn-rate calculations, and trigger HTTP 400 spikes on /api/v1/write.

By default, the Prometheus Time Series Database (TSDB) enforces strict monotonic timestamp ordering per unique metric name and label set. Any incoming sample whose timestamp precedes the latest stored sample—or matches the exact timestamp with a different value—is rejected at ingestion. Resolving this reliably requires separating genuine timestamp ordering skew (such as OpenTelemetry batching and multi-replica collectors) from label-set collisions across misconfigured scrape targets.



Problem Diagnosis & Root Cause Architecture

Prometheus identifies every time series exclusively by its metric name combined with its exact set of key-value labels. When two samples arrive for the same series, the TSDB active block evaluates their Unix millisecond timestamps against the most recently appended data point:




  • Out-of-Order Sample Rejection: Occurs when an incoming sample carries a timestamp strictly older than the newest sample already committed to the active TSDB block for that series. Common triggers include multi-replica OpenTelemetry Collectors flushing overlapping batches, network jitter on remote write queues, and exporters caching stale client-side timestamps.

  • Duplicate Timestamp Collision: Occurs when two samples for the identical label set share the exact same millisecond timestamp but report different numerical values. This almost always signals that two distinct scrape targets or pods are emitting metrics with identical labels, or that metric relabeling stripped a differentiating label such as instance or pod.

  • Remote Write HTTP 400 Backpressure: When Prometheus acts as a remote write receiver or ingests OTLP payloads, rejected samples return HTTP 400 responses. Upstream collectors often retry failed batches indefinitely, amplifying CPU load and delaying fresh metrics behind stalled queues.



Never blindly raise the out-of-order window before verifying whether your errors stem from duplicate label sets. Expanding the ingestion window fixes delayed batch delivery, but it cannot resolve two targets fighting over the same series identity.


Why Standard Implementations Fail

Relying on Default Monotonic TSDB Ingestion

Standard Prometheus configurations leave storage.tsdb.out_of_order_time_window at 0s. While strict in-order appending works for single-instance pull scraping, modern push-based pipelines—including OpenTelemetry Collector batch processors and high-availability agent pairs—frequently deliver samples slightly out of chronological sequence.



Stripping Differentiating Labels During Relabeling

Engineers attempting to curb high cardinality often drop instance, pod, or container labels inside metric_relabel_configs without aggregating the values first. Once those labels disappear, every replica in a Deployment maps to the exact same series signature, causing immediate duplicate-timestamp and out-of-order collisions on every scrape interval.



Setting Unbounded Out-of-Order Windows

Configuring extreme values such as 75y on out_of_order_time_window forces Prometheus to keep older chunks open in memory for out-of-order compaction. This inflates RAM usage, slows down WAL replay on restarts, and masks underlying exporter bugs or clock drift across nodes.



Step-by-Step Resolution Playbook

Step 1: Quantify Rejection Rates Across Ingestion Paths

Query Prometheus internal counters to determine whether rejections happen during pull scrapes, TSDB ingestion, or remote write HTTP endpoints:



# Check total out-of-order samples rejected by TSDB
rate(prometheus_tsdb_out_of_order_samples_total[5m])

# Check pull-scrape out-of-order and duplicate timestamp failures
rate(prometheus_target_scrapes_sample_out_of_order_total[5m])
rate(prometheus_target_scrapes_sample_duplicate_timestamp_total[5m])

# Check HTTP 400 rejections on the remote write receiver
rate(prometheus_http_requests_total{code="400",handler="/api/v1/write"}[5m])


Step 2: Isolate the Offending Time Series and Labels

Prometheus omits the specific metric name and label set from default warning logs to prevent log flooding. Temporarily switch the log level to debug (or inspect a staging replica) to capture the exact series field:



# Start or restart Prometheus with debug logging to inspect rejected series
prometheus --config.file=/etc/prometheus/prometheus.yml --log.level=debug

# Filter journal or container logs for the exact offending series
journalctl -u prometheus -n 200 --no-pager | grep -E "out-of-order|duplicate|too old"

Inspect the returned series labels. If two pods or exporters share identical labels, restore the differentiating instance or pod label in your relabel_configs or metric_relabel_configs before proceeding.



Step 3: Configure a Bounded Out-of-Order Time Window

For legitimate timestamp skew caused by OpenTelemetry batching, multi-replica agents, or remote write retries, set out_of_order_time_window in prometheus.yml to a bounded duration such as 30m:



# /etc/prometheus/prometheus.yml
global:
scrape_interval: 15s
evaluation_interval: 15s

storage:
tsdb:
out_of_order_time_window: 30m

A 30m window absorbs standard collector batch delays and transient network partitions without keeping days of uncompacted blocks in active memory.



Step 4: Reload Configuration Without Dropping Scrapes

Apply the updated prometheus.yml at runtime via the lifecycle endpoint (when --web.enable-lifecycle is active) or by sending SIGHUP:



# Validate configuration syntax first
promtool check config /etc/prometheus/prometheus.yml

# Reload Prometheus gracefully
curl -X POST localhost:9090/-/reload


Step 5: Verify Production Recovery

Confirm that out-of-order rejections and remote write HTTP 400 errors drop to zero while TSDB memory remains stable:



# Confirm out-of-order and duplicate counters stop incrementing
curl -s "localhost:9090/api/v1/query?query=sum(rate(prometheus_tsdb_out_of_order_samples_total[5m]))"

# Verify remote write receiver returns 2xx/204 instead of 400
curl -s "localhost:9090/api/v1/query?query=sum(rate(prometheus_http_requests_total{handler='/api/v1/write'}[5m]))%20by%20(code)"

If prometheus_target_scrapes_sample_duplicate_timestamp_total continues climbing after enabling out_of_order_time_window, audit your service discovery targets for duplicate endpoints or broken relabeling rules that collapse distinct targets into a single label set.

Advertisement
Engineering Advisory 48-Hour Audit Turnaround

Experiencing this issue in your production systems?

Calricon conducts senior-level VoIP, SIP trunking, Asterisk, and cloud infrastructure audits. We review your live configs, identify packet drops or contention bottlenecks, and provide exact remediation runbooks.