The Kubernetes EndpointSlice Drain Race Trap: Why Rolling Updates
calricon / field note
Learn why Kubernetes EndpointSlice removal and SIGTERM race during
The Kubernetes EndpointSlice Drain Race Trap: Why Rolling Updates Trigger 502 Bad Gateway Spikes
In production Kubernetes clusters running high-throughput API gateways, microservice meshes, and payment backends, engineering teams frequently observe an infuriating deployment regression: even with maxUnavailable: 0 configured in the Deployment rolling update strategy and clean readiness probes, every CI/CD rollout or node cordon triggers a brief, sharp burst of HTTP 502 Bad Gateway and 503 Service Unavailable errors at the Ingress controller or cloud load balancer. Application logs show zero unhandled exceptions, and Kubernetes reports every replaced pod as cleanly terminated. The root cause is the Kubernetes EndpointSlice drain race—where kubelet delivers SIGTERM to the container in microseconds while asynchronous EndpointSlice removal takes seconds to propagate across kube-proxy, Ingress upstreams, and external load balancers.
Problem Diagnosis & Root Cause Architecture
When Kubernetes initiates pod termination (during a rolling deployment, HorizontalPodAutoscaler scale-in, or node drain), the control plane and node kubelet execute two parallel, uncoordinated workflows:
- Parallel Control-Plane vs. Kubelet Execution: The API server marks the pod as
Terminatingand instructs the EndpointSlice controller to remove the pod's IP address from eligible backend slices. Simultaneously—without waiting for any acknowledgment from network proxies—kubelet on the host node immediately initiates container shutdown by sendingSIGTERMto PID 1. - Eventually Consistent Proxy Propagation Lag: Removing a pod IP from an
EndpointSlicemust propagate to every node'skube-proxy(iptables or IPVS rules), every Ingress-NGINX or Envoy worker's in-memory upstream table, and any cloud load balancer (such as AWS ALB target group deregistration). On busy clusters, this propagation window spans 1 to 10 seconds, whereas localSIGTERMdelivery completes in microseconds. - Premature Listener Closure Under In-Flight Traffic: Because the application process receives
SIGTERMbefore upstream proxies stop routing to it, the HTTP server closes its listening socket or drops keep-alive connections while Ingress controllers are still forwarding new requests to that pod IP, producing immediate connection resets and502 Bad Gatewayresponses. - Single-Budget Grace Period Misalignment: Kubernetes enforces
terminationGracePeriodSeconds(defaulting to 30 seconds) as a single combined budget covering both thepreStoplifecycle hook and the application's graceful shutdown drain. If a hook consumes 20 seconds and the application needs 15 seconds to drain active transactions, kubelet sendsSIGKILLat second 30, severing live requests mid-flight.
"Kubernetes rolling updates are not atomic traffic switches. Because EndpointSlice propagation is eventually consistent while SIGTERM is instantaneous, every unbuffered pod termination creates a multi-second 502 window."
Why Standard Implementations Fail
Relying on maxUnavailable: 0 and Readiness Probes Alone
Setting maxUnavailable: 0 guarantees that new pods become ready before old pods are deleted, and readiness probes prevent unhealthy pods from receiving traffic. Neither setting delays SIGTERM on the terminating pod while upstream load balancers flush stale endpoint caches.
Using Shell-Form Entrypoints That Swallow SIGTERM
Dockerfiles written with shell-form commands (such as CMD python3 server.py or /bin/sh -c) run /bin/sh as PID 1. Because POSIX shells do not forward SIGTERM to child processes, the application ignores the shutdown signal completely until kubelet issues SIGKILL.
Ignoring Keep-Alive Socket Draining in Application Servers
Calling server.close() in Node.js or Go without closing idle keep-alive connections leaves persistent Ingress-to-pod TCP sockets open until the grace period expires, causing abrupt termination errors.
Validated Step-by-Step Resolution Playbook
Step 1: Audit Ingress Upstream Errors and Pod Termination Timing
Correlate Ingress controller 502 upstream connection-refused logs against pod termination timestamps during a rolling deployment:
# Inspect Ingress-NGINX logs for upstream premature closures during rollouts
kubectl logs -n ingress-nginx deploy/ingress-nginx-controller --since=10m | grep -E "502|upstream prematurely closed|Connection refused"
# Check current Deployment rollout strategy and terminationGracePeriodSeconds
kubectl get deploy api-service -o jsonpath='{.spec.strategy}{"\n"}{.spec.template.spec.terminationGracePeriodSeconds}{"\n"}'Step 2: Configure preStop Drain Buffers and Explicit Grace Budgets
Add a preStop sleep hook (using the native Kubernetes sleep action or /bin/sh) to hold off SIGTERM until EndpointSlice updates propagate across all proxies, and size terminationGracePeriodSeconds strictly as preStop + max_request_drain + buffer:
# /etc/k8s/api-service-deployment.yaml
# Zero-downtime Deployment spec with synchronized preStop and grace budget
apiVersion: apps/v1
kind: Deployment
metadata:
name: api-service
spec:
replicas: 4
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 0
maxSurge: 1
template:
spec:
# Total budget: 10s preStop + 30s active drain + 5s safety buffer = 45s
terminationGracePeriodSeconds: 45
containers:
- name: api
image: registry.internal/platform/api-service:v3.2.0
ports:
- containerPort: 8080
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "sleep 10"]Step 3: Enforce Exec-Form Entrypoints and Graceful SIGTERM Handlers
Ensure your container image uses JSON exec-form (CMD ["python3", "server.py"]) so PID 1 receives SIGTERM directly, and implement an application-level shutdown handler that stops accepting new connections only after preStop completes while draining in-flight requests:
# graceful_server.py
# Production ASGI/HTTP graceful shutdown handler for Kubernetes PID 1
import signal
import sys
import time
from http.server import ThreadingHTTPServer, BaseHTTPRequestHandler
shutdown_requested = False
def handle_sigterm(signum, frame):
global shutdown_requested
print("SIGTERM received after preStop drain window; draining in-flight requests...")
shutdown_requested = True
signal.signal(signal.SIGTERM, handle_sigterm)Step 4: Align Ingress-NGINX Keep-Alive and Upstream Retry Policies
Configure Ingress-NGINX ConfigMap settings to prevent stale keep-alive reuse against terminating pods and enable safe idempotent retries only for GET/OPTIONS or non-mutating calls:
# /etc/k8s/ingress-nginx-configmap.yaml
# Ingress-NGINX ConfigMap aligning upstream keepalive and safe retries
apiVersion: v1
kind: ConfigMap
metadata:
name: ingress-nginx-controller
namespace: ingress-nginx
data:
proxy-next-upstream: "error timeout"
proxy-next-upstream-tries: "2"
upstream-keepalive-timeout: "60"
keep-alive-requests: "1000"Step 5: Verify Zero 502 Errors Under Live Rolling Restart
Execute a sustained concurrent HTTP load test through the real Ingress endpoint while triggering a rolling restart, confirming 0% 502/503 responses across the entire rollout window:
# Trigger rolling restart in background and verify zero 5xx responses under load
kubectl rollout restart deploy/api-service &
python3 -c "
import http.client, time, concurrent.futures
def hit_ingress(_):
conn = http.client.HTTPConnection('127.0.0.1', 8080, timeout=5)
conn.request('GET', '/healthz')
resp = conn.getresponse()
conn.close()
return resp.status
errors = []
t_end = time.time() + 15
while time.time() < t_end:
with concurrent.futures.ThreadPoolExecutor(max_workers=16) as ex:
statuses = list(ex.map(hit_ingress, range(32)))
errors.extend([s for s in statuses if s >= 500])
time.sleep(0.2)
print('Total 5xx responses during rollout:', len(errors))
assert len(errors) == 0, 'Detected 502/503 errors during rolling update!'
print('Verification successful: Zero-downtime rollout confirmed with 0 upstream 5xx errors.')
"Experiencing this issue in your production systems?
Calricon conducts senior-level VoIP, SIP trunking, Asterisk, and cloud infrastructure audits. We review your live configs, identify packet drops or contention bottlenecks, and provide exact remediation runbooks.