Fix Kubernetes Rolling Update 502 Drops With preStop Hooks
calricon / field note
Stop transient 502 Bad Gateway and connection-refused errors
Fix Kubernetes Rolling Update 502 Drops With preStop Hooks
During routine Kubernetes rolling deployments or Horizontal Pod Autoscaler scale-ins, clients experience brief bursts of 502 Bad Gateway or 503 connection errors for 2–5 seconds per replaced pod, even though readiness probes are configured and the new pods are already healthy.
The verified root cause is an asynchronous termination race condition: when a pod enters Terminating state, the kubelet immediately begins shutting down the container in parallel with the EndpointSlice controller, kube-proxy, and Ingress controllers removing that pod's IP from load-balancer upstream pools. Without a preStop delay, the application stops listening before iptables, IPVS, or Ingress rules finish converging across every cluster node.
Problem Diagnosis & Root Cause Architecture
- Parallel Endpoint Removal and Pod Shutdown: Marking a pod
Terminatingsets its EndpointSlicereadyflag tofalse, but propagating that update to every node'skube-proxyand Ingress controller takes 1–5 seconds asynchronously. - Immediate SIGTERM Without preStop Delay: Without a
preStoplifecycle hook, the kubelet sendsSIGTERMto PID 1 immediately, causing the server to close its listening port while routers still forward new requests to the pod IP. - Shell-Form ENTRYPOINT Swallowing SIGTERM: Containers launched with shell-form
ENTRYPOINTrun/bin/sh -cas PID 1, which often fails to forwardSIGTERMto the child application—forcing the container to sit idle until the 30-second grace period expires in forcible signal-9 termination. - Grace Period Countdown Includes preStop Time: Kubernetes starts the
terminationGracePeriodSecondstimer before runningpreStop. IfpreStopsleep plus in-flight request draining exceeds 30 seconds, the runtime terminates the pod mid-request.
Architectural Principle: DelaySIGTERMwith apreStopsleep long enough for EndpointSlice and Ingress convergence, deliverSIGTERMdirectly to the application as PID 1, and sizeterminationGracePeriodSecondsstrictly greater thanpreStopsleep plus maximum request-drain time.
Why Standard Implementations Fail
1. Relying Solely on readinessProbe Without preStop
A readinessProbe controls initial traffic admission and unhealthy pod removal, but during rolling deletion the pod transitions directly to Terminating without waiting for the next probe interval—making preStop mandatory to hold the socket open during endpoint propagation.
2. Setting preStop Sleep Longer Than terminationGracePeriodSeconds
If preStop sleeps for 15 seconds and request draining takes 20 seconds while terminationGracePeriodSeconds remains at the 30-second default, the kubelet forces immediate termination at second 30, dropping active connections.
3. Leaving Idle Keep-Alive Sockets Open on SIGTERM
In Node.js and Go HTTP servers, calling server.close() stops new connections but keeps idle keep-alive sockets open; handlers must close idle connections explicitly once active requests finish draining.
4. Testing Zero-Downtime Rollouts With Sequential curl Loops
A single-threaded curl loop misses connection-pool and concurrency races; realistic validation requires concurrent load generators such as hey or k6 running across the entire rollout.
Step-by-Step Resolution Playbook
Step 1: Audit Container Entrypoint for Exec-Form PID 1
Ensure your Dockerfile uses JSON exec-form ENTRYPOINT or CMD so the application runs directly as PID 1 and receives SIGTERM:
FROM node:20-alpine
WORKDIR /app
COPY . .
ENTRYPOINT ["node", "server.js"]Step 2: Implement Graceful SIGTERM Draining in Application Code
Close the listening server on SIGTERM while allowing active requests to complete and closing idle keep-alive sockets cleanly:
process.on("SIGTERM", () => {
server.close(() => process.exit(0));
if (typeof server.closeIdleConnections === "function") {
server.closeIdleConnections();
}
});Step 3: Add preStop Sleep and Extended terminationGracePeriodSeconds
Configure a 10-second preStop delay (or target-group deregistration delay plus 5 seconds on cloud ALBs) and raise terminationGracePeriodSeconds to 60 seconds in your Deployment manifest:
spec:
terminationGracePeriodSeconds: 60
containers:
- name: api-gateway
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "sleep 10"]
readinessProbe:
httpGet:
path: /healthz
port: 8080
periodSeconds: 5Step 4: Apply Manifest and Watch Endpoint Convergence
Apply the updated Deployment and monitor EndpointSlice transitions alongside pod termination events:
kubectl apply -f deployment.yaml
kubectl get endpointslices -l https://www.google.com/url?q=https://www.google.com/url?q%3Dhttp://kubernetes.io/service-name%253Dapi-gateway%26source%3Dgmail%26ust%3D1795352895232000%26sa%3DE&source=gmail&ust=1795353220117000&sa=E -wStep 5: Validate Zero 502s Under Concurrent Rollout Load
Trigger a rolling restart while driving concurrent HTTP traffic with hey and confirm 100% 200 status codes with zero 502 or 503 responses:
kubectl rollout restart deployment/api-gateway &
hey -z 60s -c 25 https://www.google.com/url?q=https://www.google.com/url?q%3Dhttps://api.example.internal/healthz%26source%3Dgmail%26ust%3D1795352895232000%26sa%3DE&source=gmail&ust=1795353220117000&sa=EExperiencing this issue in your production systems?
Calricon conducts senior-level VoIP, SIP trunking, Asterisk, and cloud infrastructure audits. We review your live configs, identify packet drops or contention bottlenecks, and provide exact remediation runbooks.