Fixing Redis OOM Errors and High Memory Fragmentation
calricon / field note
Redis instances fail writes with OOM command not allowed or get
Fixing Redis OOM Errors and High Memory Fragmentation
Your application suddenly fails writes with OOM command not allowed when used memory > 'maxmemory'—or your Redis pod exits with OOMKilled (exit code 137) even though used_memory is sitting comfortably at 60% of the container limit. In both cases, caching, rate-limiting, and session lookups break simultaneously. The root cause is usually one of two distinct mechanisms: a default noeviction or misapplied volatile-lru policy blocking writes at maxmemory, or jemalloc memory fragmentation and fork() copy-on-write spikes pushing physical RSS far beyond logical data size.
Problem Diagnosis & Root Cause Architecture
Redis reports both its allocator's logical usage (used_memory) and the operating system's resident physical footprint (used_memory_rss). Production outages happen when engineers monitor only one of those numbers or mix incompatible workloads on a single instance.
- noeviction & volatile-lru Traps: Upstream Redis defaults
maxmemory-policytonoeviction, rejecting all new writes onceused_memoryreachesmaxmemory. Worse, choosingvolatile-lruon a cache where keys lack a TTL behaves identically tonoevictionbecause zero keys are eligible for eviction. - jemalloc Fragmentation (mem_fragmentation_ratio > 1.5): High-churn workloads that rapidly create and expire variable-length keys leave scattered unused gaps in
jemallocpages. Physical RSS climbs well aboveused_memoryuntil the Linux OOM reaper terminates the process. - Fork Copy-on-Write (COW) Memory Storms: Background
BGSAVEsnapshots orBGREWRITEAOFcalls invokefork(). Under heavy write traffic, copy-on-write can nearly double physical RSS during the child process's lifetime, crashing containers sized above 50–60% of host RAM. - Shared Cache and Queue Eviction: Running BullMQ/Celery queues or distributed locks on the same Redis instance configured with
allkeys-lrucauses silent job loss whenever cache pressure triggers eviction.
Containers and the Linux OOM reaper do not enforce Redis's used_memory—they enforce physical RSS. If mem_fragmentation_ratio exceeds 1.5 or persistence forks under heavy writes, Redis can be OOMKilled well below maxmemory.
Why Standard Implementations Fail
1. Setting maxmemory Equal to Container Memory Limits
Setting maxmemory 4gb inside a 4 GiB Kubernetes container leaves zero headroom for jemalloc fragmentation, client output buffers, or RDB/AOF copy-on-write overhead, guaranteeing container restarts under load.
2. Enabling allkeys-lru on Multi-Purpose Redis Instances
Switching to allkeys-lru stops OOM command not allowed errors immediately, but if that same Redis server stores session state or background worker queues, Redis will evict active queue payloads and locks without warning.
3. Relying on MEMORY PURGE Without Active Defragmentation
Running MEMORY PURGE manually asks jemalloc to return dirty unused pages to the OS, which offers only temporary relief because it cannot relocate live fragmented objects inside active pages.
Step-by-Step Resolution Playbook
Step 1: Inspect Memory, Fragmentation Ratio, and Eviction Policy
Connect via redis-cli to compare used_memory against used_memory_rss, check mem_fragmentation_ratio, and audit the active maxmemory-policy and keyspace TTL distribution.
# Inspect logical memory, physical RSS, fragmentation ratio, and peak usage
redis-cli INFO memory | grep -E "used_memory_human|used_memory_rss_human|mem_fragmentation_ratio|maxmemory_human|maxmemory_policy|active_defrag_running"
# Audit keyspace to check how many keys actually have TTLs (expires)
redis-cli INFO keyspace
# Sample largest keys without blocking the main thread
redis-cli --bigkeys -i 0.1Step 2: Apply Workload-Specific Eviction and Right-Size maxmemory
Separate pure caches from stateful queues/locks onto dedicated instances. On pure cache instances, set maxmemory to ~70% of container RAM (or 50% if RDB/AOF persistence is enabled) and use allkeys-lru or allkeys-lfu. Keep queues on a separate instance with noeviction and TTLs on completed job metadata.
# Apply live on a dedicated cache instance (no restart required)
redis-cli CONFIG SET maxmemory 2800mb
redis-cli CONFIG SET maxmemory-policy allkeys-lru
redis-cli CONFIG SET maxmemory-samples 10
redis-cli CONFIG SET client-output-buffer-limit "normal 64mb 32mb 60"Step 3: Enable Active Defragmentation When Fragmentation Exceeds 1.5
When mem_fragmentation_ratio climbs above 1.5 due to high key churn, enable Redis activedefrag (Redis 4.0+) with bounded CPU cycle percentages so live keys are compacted in the background without blocking clients.
# Enable active defragmentation dynamically and persist to redis.conf
redis-cli CONFIG SET activedefrag yes
redis-cli CONFIG SET active-defrag-ignore-bytes 100mb
redis-cli CONFIG SET active-defrag-threshold-lower 10
redis-cli CONFIG SET active-defrag-cycle-min 1
redis-cli CONFIG SET active-defrag-cycle-max 25
redis-cli CONFIG REWRITEStep 4: Verify Fragmentation Recovery and Zero OOM Write Failures
Monitor mem_fragmentation_ratio as it converges back toward 1.05–1.20 and confirm rejected_connections and OOM write errors remain at zero:
# Watch fragmentation ratio and active defragmentation progress
watch -n 5 "redis-cli INFO memory | grep -E 'used_memory_human|used_memory_rss_human|mem_fragmentation_ratio|active_defrag_running'"
# Check evicted keys and command error stats
redis-cli INFO stats | grep -E "evicted_keys|rejected_connections"Experiencing this issue in your production systems?
Calricon conducts senior-level VoIP, SIP trunking, Asterisk, and cloud infrastructure audits. We review your live configs, identify packet drops or contention bottlenecks, and provide exact remediation runbooks.