Fixing Kubernetes Rolling Update 502 Bad Gateway Errors
calricon / field note
Zero-unavailable-replica rolling updates in Kubernetes still drop
Fixing Kubernetes Rolling Update 502 Bad Gateway Errors
Your Kubernetes Deployment rolls out with maxUnavailable: 0, every new pod passes its readiness probe before an old pod is deleted, and your CI/CD pipeline reports a clean deployment—yet your Ingress controller logs a burst of 502 Bad Gateway errors and upstream connection resets during every rollout. The root cause is an asynchronous race condition between pod termination and endpoint propagation: the kubelet delivers SIGTERM to your container in microseconds, while removing that pod's IP from the Service EndpointSlice, updating kube-proxy rules on every node, and reloading Ingress or load balancer upstream pools takes one to several seconds.
Problem Diagnosis & Root Cause Architecture
When a rolling update replaces an old pod, Kubernetes triggers two independent control loops concurrently rather than sequentially. Because neither loop waits for the other, traffic continues arriving at a process that has already closed its listening socket.
- Concurrent Control-Plane Loops: As soon as a pod transitions to
Terminating, the EndpointSlice controller removes the pod from ready endpoints in parallel with the node kubelet executing the container termination sequence. - Eventual Consistency Across Proxies: Every
kube-proxyinstance, NGINX or Envoy Ingress controller, and cloud load balancer (such as AWS ALB or GKE Ingress) watches the API server asynchronously, creating a 1-to-5-second window where stale upstream tables still route new HTTP requests to the terminating pod. - Immediate Socket Closure on SIGTERM: Without a
preStopdelay, the application receivesSIGTERMimmediately, stops accepting new TCP connections, and returnsECONNREFUSEDor premature connection resets to proxies that still consider the pod healthy. - Shell-Form Entrypoint Trap: Containers built with shell-form
ENTRYPOINTorCMDinstructions run/bin/sh -cas PID 1, which often swallowsSIGTERMinstead of forwarding it to the application process—leaving idle keep-alive sockets open until the grace period expires and the kernel forces termination.
Endpoint removal is eventually consistent across every proxy in your cluster, while SIGTERM delivery on the host takes microseconds. Any request routed during that propagation gap becomes a 502 Bad Gateway unless the pod stays alive long enough for proxies to drop it.
Why Standard Implementations Fail
1. Assuming maxUnavailable: 0 Prevents Dropped Requests
Setting maxUnavailable: 0 and maxSurge: 1 only ensures that replacement pods are Ready before old pods begin terminating. It does not pause traffic routing to the old pod before SIGTERM arrives, so every in-flight or newly routed request hitting the old pod's IP during endpoint propagation still fails.
2. Relying on Readiness Probes to Drain Rolling Updates
Teams often shorten readinessProbe intervals expecting faster draining. However, during pod deletion, Kubernetes marks the pod Terminating and removes it from EndpointSlice immediately regardless of probe state—the bottleneck is downstream propagation to Ingress controllers and cloud load balancers, not readiness probe detection.
3. Using server.close() Without Draining Keep-Alive Connections
In Node, Go, or Python servers, calling server.close() stops new connections but leaves persistent HTTP keep-alive connections open from Ingress proxies. If those idle sockets are not closed gracefully after in-flight requests finish, the container stalls until terminationGracePeriodSeconds expires and drops active work abruptly.
Step-by-Step Resolution Playbook
Step 1: Reproduce and Isolate the Rollout 502 Window
Confirm that 502 errors align with pod termination rather than application crashes by tailing Ingress controller logs alongside pod deletion timestamps during a controlled rollout restart.
# Terminal 1: Stream continuous requests and log non-200 status codes with timestamps
while true; do
code=$(curl -s -o /dev/null -w "%{http_code}" https://api.internal/healthz)
if [ "$code" != "200" ]; then
echo "$(date -u +%H:%M:%S.%3N) HTTP $code"
fi
sleep 0.1
done
# Terminal 2: Trigger a rolling restart and inspect Ingress upstream errors
kubectl rollout restart deployment/api-gateway -n production
kubectl logs -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx --since=2m | grep -E "502|upstream prematurely closed|Connection refused"Step 2: Add a Native preStop Sleep Hook and Size Grace Periods
Insert a preStop sleep hook so the container delays SIGTERM for 5 to 10 seconds—giving EndpointSlice, kube-proxy, and Ingress controllers time to drain the pod IP from upstream pools while the app continues serving late-arriving requests. On Kubernetes 1.30+, use the native sleep lifecycle action so distroless containers do not need a shell binary.
apiVersion: apps/v1
kind: Deployment
metadata:
name: api-gateway
namespace: production
spec:
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 0
maxSurge: 1
template:
spec:
terminationGracePeriodSeconds: 45
containers:
- name: gateway
image: registry.internal/api-gateway:v2.4.1
lifecycle:
preStop:
sleep:
seconds: 10Note that terminationGracePeriodSeconds covers both the preStop hook and the application shutdown phase combined. With a 10-second preStop sleep and terminationGracePeriodSeconds: 45, your application has 35 seconds after SIGTERM to drain in-flight requests.
Step 3: Use Exec-Form Entrypoints and Implement Graceful HTTP Draining
Ensure your container image runs the application binary directly as PID 1 using JSON exec-form syntax (ENTRYPOINT ["./server"]) so SIGTERM reaches your process immediately after the preStop hook finishes. In your application code, handle SIGTERM by draining active requests and closing idle keep-alive sockets:
// Example Go HTTP server graceful shutdown handler
srv := &http.Server{Addr: ":8080", Handler: router}
quit := make(chan os.Signal, 1)
signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
<-quit
ctx, cancel := context.WithTimeout(context.Background(), 25*time.Second)
defer cancel()
if err := srv.Shutdown(ctx); err != nil {
log.Fatalf("forced shutdown: %v", err)
}Step 4: Verify Zero 502s Under Active Load During Rollout
Deploy the updated manifest and run a concurrent load test across a full rolling restart to confirm zero non-200 responses and clean pod termination events:
# Apply updated deployment and watch rollout status
kubectl apply -f deployment.yaml
kubectl rollout status deployment/api-gateway -n production
# Verify no ForceDelete / abrupt termination events occurred
kubectl get events -n production --field-selector reason=Killing --sort-by='.lastTimestamp'Experiencing this issue in your production systems?
Calricon conducts senior-level VoIP, SIP trunking, Asterisk, and cloud infrastructure audits. We review your live configs, identify packet drops or contention bottlenecks, and provide exact remediation runbooks.