Fixing Kubernetes CPU Throttling and CFS Quota Latency Spikes
calricon / field note
Pods running at 35% average CPU can still suffer severe p99
Fixing Kubernetes CPU Throttling and CFS Quota Latency Spikes
Your Grafana cluster dashboard shows a calm 35% average CPU utilization across your Kubernetes nodes, yet your API gateway is firing p99 latency alerts and upstream gRPC deadlines are timing out. No pods are restarting, memory usage is flat, and network packet loss is zero. The root cause is Linux Completely Fair Scheduler (CFS) bandwidth throttling: when a container specifies resources.limits.cpu, the kernel enforces a strict microsecond budget per 100ms scheduling period rather than capping core count. A multi-threaded runtime bursting across multiple host cores exhausts that entire 100ms budget in a fraction of the window, prompting the kernel to freeze every runnable thread in the container until the next period begins.
Problem Diagnosis & Root Cause Architecture
In Linux cgroups v2, Kubernetes translates a container's CPU limit into the cpu.max interface, which pairs a quota in microseconds with a default 100000 microsecond (100ms) period. Because modern runtimes schedule work across multiple OS threads concurrently, CPU time is consumed in parallel across cores rather than sequentially on a single core.
- Parallel Budget Exhaustion: A pod with
limits.cpu: 500mreceives50000microseconds (50ms) of CPU time per 100ms period. If four worker threads execute simultaneously on a 32-core node, they burn through the entire 50ms budget in just 12.5ms of wall-clock time. - Mandatory Kernel Freeze Window: Once the cgroup hits
50000microseconds, the Linux kernel deschedules all tasks inside that cgroup for the remaining 87.5ms of the period—even if the underlying node has 80% idle CPU capacity. - Averaging Illusion in Metrics: Standard Prometheus and cAdvisor scrapes sample CPU usage over 15-to-60-second intervals, smoothing out hundreds of 80ms+ freeze events into an innocent-looking 35% average utilization line.
- Runtime Thread Over-Provisioning: Older Go binaries, JVM services without container awareness, and multi-worker Python or Node pools inspect host core counts via
sched_getaffinityand spawn 16 to 64 threads inside a container budgeted for 1 or 2 cores, guaranteeing rapid quota exhaustion on every request burst.
Setting a 500m CPU limit in Kubernetes does not give your pod half of one CPU core continuously; it gives your pod 50ms of total thread execution time per 100ms window, after which the kernel pauses the entire container until the next tick.
Why Standard Implementations Fail
1. Treating CPU Requests and Limits Like Memory Limits
Teams routinely apply the same mental model to CPU and memory: setting tight limits equal to requests to achieve Guaranteed QoS. Unlike memory—which is non-compressible and triggers an OOM event when exhausted—CPU is compressible. Enforcing a hard CFS quota prevents a latency-sensitive service from borrowing idle node cycles during brief traffic spikes or garbage collection pauses, trading predictable p99 latency for artificial throttling.
2. Silencing Prometheus Throttling Alerts Without Fixing Root Causes
Because default CPUThrottlingHigh alerts in kube-prometheus trigger noisily on init containers and low-priority batch jobs, platform teams often disable the alert cluster-wide. Without tracking nr_throttled and throttled_usec on latency-critical microservices, throttling silently degrades tail latency whenever node density or request concurrency increases.
3. Relying on Static Thread Counts Across Heterogeneous Nodes
Hardcoding worker counts or leaving default runtime parallelism unchanged fails when workloads migrate between 4-core and 64-core instance types. A service that runs cleanly on a small node can suddenly experience severe throttling when rescheduled onto a larger node because its thread pool expands to match the new host's logical CPU count.
Step-by-Step Resolution Playbook
Step 1: Verify CFS Throttling via cgroup v2 and Prometheus
Inspect the container's live cgroup v2 counters directly on the node or inside the pod to confirm whether the kernel is actively pausing runnable threads, and check Pressure Stall Information (PSI) to distinguish quota throttling from host-level CPU saturation.
# Run inside the target container to inspect cgroup v2 quota and throttling counters
cat /sys/fs/cgroup/cpu.max
cat /sys/fs/cgroup/cpu.stat
cat /sys/fs/cgroup/cpu.pressure
# Example cpu.stat output indicating severe CFS throttling:
# nr_periods 12480
# nr_throttled 3912
# throttled_usec 812340912In Prometheus, calculate the exact percentage of throttled 100ms periods and total throttled seconds per second for your namespace:
# Throttled period ratio (alert if > 0.05 on latency-sensitive services)
sum by (pod, container) (
rate(container_cpu_cfs_throttled_periods_total{namespace="production"}[5m])
)
/
sum by (pod, container) (
rate(container_cpu_cfs_periods_total{namespace="production"}[5m])
)
# Total CPU time lost to CFS throttling (seconds/sec)
sum by (pod, container) (
rate(container_cpu_cfs_throttled_seconds_total{namespace="production"}[5m])
)Step 2: Remove CPU Limits on Latency-Sensitive Services
For latency-sensitive APIs, Ingress controllers, and Voice AI pipelines, keep resources.requests.cpu accurate so the Kubernetes scheduler reserves capacity and assigns fair cpu.weight shares during contention, while removing resources.limits.cpu. Keep resources.limits.memory equal to resources.requests.memory to prevent memory-driven evictions.
apiVersion: apps/v1
kind: Deployment
metadata:
name: api-gateway
namespace: production
spec:
template:
spec:
containers:
- name: gateway
image: registry.internal/api-gateway:v2.4.1
resources:
requests:
cpu: "1000m"
memory: "1Gi"
limits:
memory: "1Gi"Step 3: Align Runtime Concurrency When CPU Limits Are Mandated
In strict multi-tenant or compliance clusters where limits.cpu cannot be removed, constrain runtime thread parallelism to match or stay below the integer CPU limit so threads do not race through the 100ms budget simultaneously.
# For Go services (Go 1.25+ is cgroup-aware by default; pin explicitly for older versions or sub-core limits)
env:
- name: GOMAXPROCS
value: "2"
# For JVM services (Java 17/21+): cap active processors to match integer CPU limit
env:
- name: JAVA_TOOL_OPTIONS
value: "-XX:ActiveProcessorCount=2 -XX:+UseG1GC"Step 4: Tune Kubelet CFS Period or Pin Dedicated CPUs for Ultra-Low Latency
If your cluster enforces CPU limits at the admission controller level and workloads still suffer 50ms+ tail spikes, shorten the kubelet CFS quota period from 100ms to 10ms on dedicated low-latency node pools, or enable the static CPU Manager policy for Guaranteed QoS pods with integer CPU requests.
# /var/lib/kubelet/config.yaml on dedicated low-latency worker nodes
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cpuCFSQuota: true
cpuCFSQuotaPeriod: 10ms
cpuManagerPolicy: static
reservedSystemCPUs: "0-1"Restart the kubelet on the target node pool (systemctl restart kubelet) after cordoning and draining, then verify the updated period and confirm throttling drops under load:
# Verify the new 10ms (10000us) period is active in cgroup v2
kubectl exec -n production deploy/api-gateway -- cat /sys/fs/cgroup/cpu.max
# Expected output for a 2000m limit with 10ms period: 20000 10000
# Watch nr_throttled delta over a 30-second load window
kubectl exec -n production deploy/api-gateway -- sh -c 'grep nr_throttled /sys/fs/cgroup/cpu.stat; sleep 30; grep nr_throttled /sys/fs/cgroup/cpu.stat'Experiencing this issue in your production systems?
Calricon conducts senior-level VoIP, SIP trunking, Asterisk, and cloud infrastructure audits. We review your live configs, identify packet drops or contention bottlenecks, and provide exact remediation runbooks.