Fixing Kubernetes CPU CFS Throttling and P99 Latency Spikes
calricon / field note
Title: Fixing Kubernetes CPU CFS Throttling and P99 Latency Spikes Slug: fixing-kubernetes-cpu-cfs-throttling-p99-latency-spikes Focus Keyword: kubernetes cpu cfs throttling nr_throttled Meta Title: Fix Kubernetes CPU CFS Throttling & P99 Latency Meta...
Fixing Kubernetes CPU CFS Throttling and P99 Latency Spikes
Production Kubernetes microservices—especially Go, JVM, and Python/Node.js APIs handling real-time Voice AI or SIP signaling—frequently suffer severe P99 latency spikes (jumping from 20ms to 400ms+) and readiness probe timeouts while Grafana CPU dashboards show average pod CPU utilization sitting below 30%.
The root cause is Linux Completely Fair Scheduler (CFS) bandwidth throttling enforced by resources.limits.cpu. Dashboards average CPU over 1-to-5-minute windows, whereas the Linux kernel enforces CFS quotas every 100ms (cpu.max in cgroups v2). A multi-threaded runtime with a 500m limit (50ms of CPU time per 100ms window) can burn its entire 50ms budget across 8 parallel threads in under 10ms, forcing the kernel to freeze all container threads for the remaining 90ms.
Problem Diagnosis & Root Cause Architecture
Resolving silent CFS throttling requires separating scheduler reservations (requests) from hard kernel quotas (limits) and matching runtime thread pools to container boundaries:
- 100ms CFS Quota Window Exhaustion: Every
1000m(1 vCPU) oflimits.cpugrants100,000usper100,000usperiod in/sys/fs/cgroup/cpu.max. Once spent within a 100ms window, every thread in the cgroup is parked until the next period starts, spiking tail latency while median (P50) latency stays flat. - Multi-Threaded Runtime Over-Allocation: Older Go runtimes (
GOMAXPROCSviaruntime.NumCPU()) and JVMs read the underlying 32-core host node instead of the cgroup quota, spawning dozens of OS threads and GC workers that burn the 100ms quota almost instantaneously. - Cgroups v2 Metric Visibility: On modern Linux nodes using cgroups v2, throttling counters live in
/sys/fs/cgroup/cpu.statasnr_throttledandthrottled_usecrather than legacy cgroups v1 paths. - Health Probe & Retry Amplification: When a container freezes for 80–90ms across consecutive CFS periods, liveness/readiness probes time out and upstream clients retry, compounding queue depth on the throttled pod.
CPU is a compressible resource: unlike memory exhaustion, high CPU usage does not trigger pod OOM eviction. Keeping accuraterequests.cpuwhile removinglimits.cpuon trusted, latency-sensitive services eliminates CFS stalls while preserving fair scheduling under node contention.
Why Standard Implementations Fail
1. Setting 100m or 200m CPU Limits on Active APIs
A 100m CPU limit grants only 10ms of total CPU time per 100ms period across all threads—guaranteeing severe throttling during TLS handshakes, JSON serialization, or garbage collection.
2. Checking Only Average CPU Utilization in kubectl top
kubectl top pods reports smoothed CPU averages that completely hide sub-second CFS quota pauses. A pod averaging 150m can still be throttled in 40%+ of its 100ms scheduling periods.
3. Leaving GOMAXPROCS or Thread Pools Unbounded
Raising limits.cpu without aligning runtime concurrency (GOMAXPROCS, -XX:ActiveProcessorCount, or worker thread counts) still lets bursty thread pools exhaust the cgroup quota early in each period.
Validated Step-by-Step Resolution Playbook
Step 1: Inspect Live Cgroup v2 Throttling Counters
Exec into an affected pod and inspect cpu.max and cpu.stat (for cgroups v2) or query Prometheus for throttled periods:
kubectl exec -it deploy/voice-gateway -- cat /sys/fs/cgroup/cpu.max
kubectl exec -it deploy/voice-gateway -- cat /sys/fs/cgroup/cpu.statWatch nr_throttled and throttled_usec over 30 seconds of load; a throttle ratio (nr_throttled / nr_periods) above 5–10% on latency-sensitive services confirms CFS throttling.
Step 2: Remove limits.cpu for Trusted Latency-Sensitive Services
Update your Deployment manifest to keep accurate requests.cpu (for scheduler bin-packing and fair share under contention) and limits.memory (to prevent node-wide OOM), while removing limits.cpu:
resources:
requests:
cpu: "500m"
memory: "512Mi"
limits:
memory: "512Mi"If corporate multi-tenancy policy requires a CPU ceiling, set limits.cpu to at least 2x–3x observed burst demand (minimum 1000m+ for multi-threaded APIs) or enable Kubernetes CPU Manager static policy for Guaranteed QoS pods with integer CPU requests.
Step 3: Align Runtime Parallelism with Cgroup Allocation
For Go services (prior to Go 1.25's cgroup-aware default), pass GOMAXPROCS via the Kubernetes Downward API or Uber's automaxprocs, and for JVM services set -XX:ActiveProcessorCount:
env:
- name: GOMAXPROCS
valueFrom:
resourceFieldRef:
containerName: voice-gateway
resource: requests.cpuStep 4: Production Verification
Roll out the updated Deployment, re-sample /sys/fs/cgroup/cpu.stat under peak traffic, and compare P99 latency before and after:
kubectl apply -f deployment.yaml
kubectl rollout status deployment/voice-gateway
kubectl exec -it deploy/voice-gateway -- cat /sys/fs/cgroup/cpu.statSeeing nr_throttled 0 (or near-zero throttled microseconds) alongside a sharp drop in P99 latency confirms that CFS bandwidth stalls are resolved.
Experiencing this issue in your production systems?
Calricon conducts senior-level VoIP, SIP trunking, Asterisk, and cloud infrastructure audits. We review your live configs, identify packet drops or contention bottlenecks, and provide exact remediation runbooks.