The OpenTelemetry Collector OOM Crash Loop: Why Misordered Processors Blind SREs During Production Outages

An observability pipeline has one non-negotiable job: staying alive when the rest of your production infrastructure is on fire. Yet across thousands of cloud-native enterprises, the exact opposite happens during every major incident. A database connection pool saturates or an upstream payment gateway starts throwing HTTP 503 errors; microservice retry loops and verbose exception stack traces suddenly multiply telemetry volume by 15x; and within 45 seconds, every Grafana dashboard and distributed trace viewer goes completely blank. When the on-call SRE inspects the observability namespace, they find the OpenTelemetry Collector fleet trapped in a violent CrashLoopBackOff with exit code 137 (OOMKilled). The culprit is the OpenTelemetry Collector OOM Crash Loop: by placing stateful processors like batch, k8sattributes, or tail_sampling ahead of memory_limiter in the pipeline array, leaving Go's garbage collector unaware of Kubernetes cgroup boundaries (missing GOMEMLIMIT), and buffering exporter retries in volatile RAM, engineering teams build telemetry gateways that self-destruct under the exact diagnostic bursts they were deployed to capture.

An observability pipeline that crashes with OOMKilled during an outage is worse than having no observability at all. Every time the Linux kernel kills your OpenTelemetry Collector pod, it permanently vaporizes thousands of in-flight error spans buffered in RAM—destroying the crime scene evidence at the exact millisecond your SREs open their dashboards.

The Anatomy of an Outage Telemetry Tsunami

To understand why an OpenTelemetry Collector that runs peacefully at 35% memory utilization for three months suddenly explodes in 800 milliseconds during an incident, platform engineers must trace how failure cascades amplify telemetry memory pressure:

• Payload Size Inflation: A normal HTTP 200 OK span or log entry carries a tiny metadata footprint (~600 bytes). When a microservice throws an unhandled exception, the span or log event attaches a 12-kilobyte stack trace, serialized request parameters, and multiple retry child spans—inflating per-request memory allocation by 20x.
• Downstream Exporter Backpressure: During a regional incident or spike, your observability backend (whether Datadog, Honeycomb, Tempo, ClickHouse, or OpenSearch) inevitably experiences ingestion latency or enforces rate limiting. When the exporter slows down, the OpenTelemetry Collector cannot flush completed batches off the heap as fast as receivers ingest them.
• The Go Runtime GOGC Multiplier: The OpenTelemetry Collector is written in Go. By default, Go's garbage collector is governed by GOGC=100, meaning the runtime waits until the live heap doubles (+100% growth) before triggering a full garbage collection cycle. If your live buffered telemetry jumps to 600 MB inside a Kubernetes container with a 1,024 MB memory limit, Go happily waits for heap usage to reach 1,200 MB before collecting garbage—guaranteeing that the Linux kernel cgroup OOM-killer terminates the process first.

The Three Fatal Configuration Mistakes That Guarantee Collector OOM-Kills

When we audit enterprise OpenTelemetry deployments that fail during incidents, the same three architectural mistakes appear in nearly every collector ConfigMap:

1. Pipeline Processor Ordering Inversion (Putting batch Before memory_limiter)

In the OpenTelemetry Collector configuration YAML, the service.pipelines.<signal>.processors list is not an unordered set; it is a strict, sequential execution chain.

Consider this deceptively common configuration mistake:

# ANTI-PATTERN: Guaranteed OOMKilled under burst traffic
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [k8sattributes, tail_sampling, batch, memory_limiter]
      exporters: [otlphttp]

When a 15x burst of OTLP spans hits this pipeline:

1. Every span immediately enters k8sattributes, which allocates dozens of new string attributes (Pod labels, Namespace, Deployment name, Node UID) on the heap.
2. Next, the spans enter tail_sampling and batch, which hold tens of thousands of enriched spans in memory waiting for trace completion windows (e.g., 30 seconds) or batch timeouts (e.g., 5 seconds).
3. Because memory_limiter sits at the very end of the processor chain, it has zero visibility into the gigabytes of data accumulating upstream in tail_sampling and batch! By the time a batch finally flushes down to memory_limiter, the container has already breached its cgroup limit and crashed with OOMKilled.

2. Sluggish check_interval Polling (The 5-Second Blind Spot)

Even when teams place memory_limiter first in the processor list, they frequently copy-paste documentation examples that set check_interval: 5s.

The memory_limiter processor does not check memory allocation on every incoming span; doing so would cause severe CPU lock contention. Instead, a background goroutine wakes up once every check_interval to call runtime.ReadMemStats(). In a high-throughput Kubernetes gateway ingesting 150,000 spans per second during a retry storm, five seconds of unchecked ingestion allocates over 1.2 GB of heap memory. The collector breaches its container memory wall and gets killed by the Linux kernel 3.8 seconds after the burst begins—while memory_limiter is still asleep waiting for its 5-second timer to tick.

3. Unbounded In-Memory Exporter Queues (sending_queue Hoarding)

When a downstream observability backend slows down or returns HTTP 429 / 503 during an incident, the OpenTelemetry Collector's exporter helper stores failed batches in an internal sending_queue for retry.

By default, sending_queue lives entirely in volatile Go heap memory with a default queue_size of 1,000 batches. If your batch processor is configured with send_batch_size: 8192, a full in-memory sending_queue holds 8,192,000 spans in RAM! Even if memory_limiter eventually activates and begins rejecting new incoming spans at the receiver, the collector remains trapped at 99% memory saturation until the exporter queue drains or the pod OOMs.

Comparing OpenTelemetry Collector Resilience Architectures

Architecture Dimension	Default Unordered Pipeline (No GOMEMLIMIT)	Ordered memory_limiter (In-Memory Queue)	Calricon Hardened OTel Architecture
Survival During 15x Outage Surge	Crashes in <10s (OOMKilled exit 137)	Survives mild bursts; OOMs on slow backend	100% Crash-Proof (200ms check + disk WAL)
In-Flight Data Survival on Restart	0% (All buffered spans in RAM lost)	0% (RAM queue wiped on restart)	100% Preserved (file_storage disk WAL)
Go GC Efficiency Near Pod Limit	Fatal (Waits for 2x heap past cgroup)	Fragile (Spikes on forced runtime.GC())	Smooth (GOMEMLIMIT=80% proactive GC)

The Calricon Production Playbook: Engineering Crash-Proof OpenTelemetry Pipelines

Eliminating OpenTelemetry Collector OOM crashes requires coordinating three layers of defense: Go runtime cgroup awareness (GOMEMLIMIT), sub-second ingress circuit breaking (memory_limiter), and off-heap persistent disk queuing (file_storage).

Step 1: Set GOMEMLIMIT to 80% of the Kubernetes Pod Memory Limit

Before touching your collector YAML, fix the Go runtime. Setting the GOMEMLIMIT environment variable instructs the Go garbage collector to run aggressively whenever total memory approaches the soft limit—preventing lazy GOGC=100 heap doubling from breaching your Kubernetes container limit.

In your Kubernetes Deployment or DaemonSet manifest, pair resources.limits.memory with GOMEMLIMIT at 80% of the limit:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector-gateway
  namespace: observability
spec:
  replicas: 6
  template:
    spec:
      containers:
        - name: otelcol
          image: otel/opentelemetry-collector-contrib:0.114.0
          resources:
            requests:
              cpu: "2000m"
              memory: "4096Mi"
            limits:
              cpu: "2000m"
              memory: "4096Mi"
          env:
            # CRITICAL: Set GOMEMLIMIT to ~80% of container memory limit (3276MiB)
            - name: GOMEMLIMIT
              value: "3276MiB"
            - name: GOGC
              value: "80"

Step 2: Place memory_limiter Strictly First with a 200ms Check Interval

In every single pipeline (traces, metrics, logs), memory_limiter must be the very first processor in the array—immediately after the receiver and before any attribute enrichment, sampling, or batching.

Set check_interval to 200ms (never 5s), configure limit_percentage to 75, and spike_limit_percentage to 15 (giving you a soft backpressure threshold at 60% of container memory and a hard threshold at 75%, well below GOMEMLIMIT at 80% and the Linux OOM-killer at 100%).

Step 3: Offload Exporter Retry Buffers from RAM to Disk via file_storage

To prevent downstream backend slowdowns from filling your Go heap with millions of queued spans, attach the file_storage extension backed by a fast local SSD volume (/var/lib/otelcol/queue). When the exporter backs up, batches are serialized directly to disk rather than hoarding RAM, and survive pod restarts with zero data loss:

extensions:
  # Offloads exporter retry queues from Go RAM heap to persistent disk WAL
  file_storage/exporter_queue:
    directory: /var/lib/otelcol/queue
    timeout: 10s
    compaction:
      on_start: true
      on_rebound: true
      directory: /var/lib/otelcol/queue/compaction
      rebound_needed_threshold_mib: 100
      rebound_trigger_threshold_mib: 10

receivers:
  otlp:
    protocols:
      grpc:
        max_recv_msg_size_mib: 16
      http:

processors:
  # MUST BE FIRST: Polls every 200ms; triggers OTLP 429/ResourceExhausted backpressure at 60%
  memory_limiter:
    check_interval: 200ms
    limit_percentage: 75
    spike_limit_percentage: 15

  k8sattributes:
    extract:
      metadata: [k8s.namespace.name, k8s.deployment.name, k8s.pod.name]

  # MUST BE LAST: Batches only after memory safety is verified
  batch:
    send_batch_size: 2048
    send_batch_max_size: 4096
    timeout: 1s

exporters:
  otlphttp/backend:
    endpoint: "https://ingest.observability.internal:4318"
    retry_on_failure:
      enabled: true
      initial_interval: 2s
      max_interval: 30s
      max_elapsed_time: 300s
    sending_queue:
      enabled: true
      num_consumers: 10
      queue_size: 2500
      # CRITICAL: Binds retry queue to disk storage instead of Go heap RAM
      storage: file_storage/exporter_queue

service:
  extensions: [file_storage/exporter_queue]
  pipelines:
    traces:
      receivers: [otlp]
      # Strict execution order: memory_limiter FIRST, batch LAST
      processors: [memory_limiter, k8sattributes, batch]
      exporters: [otlphttp/backend]

Step 4: Alert on Refused Spans Before the Collector Hits Saturation

When memory_limiter engages its circuit breaker, it returns a retryable backpressure signal to upstream OpenTelemetry SDKs and increments otelcol_processor_refused_spans. Monitor this metric in Prometheus so you know the exact second your collector fleet begins shedding load:

- alert: OtelCollectorMemoryLimiterSheddingLoad
  expr: |
    rate(otelcol_processor_refused_spans[1m]) > 0
    or
    rate(otelcol_processor_refused_log_records[1m]) > 0
  for: 1m
  labels:
    severity: warning
  annotations:
    summary: "OpenTelemetry Collector memory_limiter is actively refusing telemetry due to memory pressure (>60% limit). Scale collector replicas."

Stop Letting Your Observability Pipeline Die First

An OpenTelemetry Collector is not a stateless pass-through router; it is a high-throughput stream-processing engine that ingests, mutates, batches, and queues gigabytes of structured data per minute. Treating its processor order casually or leaving its retry queues in volatile RAM guarantees that your observability stack will collapse the moment a real production outage tests your defenses.

Audit your OpenTelemetry Collector manifests today. Set GOMEMLIMIT to 80% of your container memory limit, move memory_limiter to index zero of every processor array with a 200ms check interval, move batch to the final position, and bind your exporter sending_queue to disk-backed file_storage. When production breaks at 3:00 AM, your telemetry pipeline should be the last component standing.