calricon.
DevOps & Cloud

Fix Kubernetes Rolling Update 502 Drops With preStop Hooks

Nov 21, 2026/4 min read/Verified Production Playbook

calricon / field note

Stop transient 502 Bad Gateway and connection-refused errors

Nov 21, 2026/DevOps & Cloud
373
Advertisement

Fix Kubernetes Rolling Update 502 Drops With preStop Hooks

During routine Kubernetes rolling deployments or Horizontal Pod Autoscaler scale-ins, clients experience brief bursts of 502 Bad Gateway or 503 connection errors for 2–5 seconds per replaced pod, even though readiness probes are configured and the new pods are already healthy.

The verified root cause is an asynchronous termination race condition: when a pod enters Terminating state, the kubelet immediately begins shutting down the container in parallel with the EndpointSlice controller, kube-proxy, and Ingress controllers removing that pod's IP from load-balancer upstream pools. Without a preStop delay, the application stops listening before iptables, IPVS, or Ingress rules finish converging across every cluster node.



Problem Diagnosis & Root Cause Architecture


  • Parallel Endpoint Removal and Pod Shutdown: Marking a pod Terminating sets its EndpointSlice ready flag to false, but propagating that update to every node's kube-proxy and Ingress controller takes 1–5 seconds asynchronously.

  • Immediate SIGTERM Without preStop Delay: Without a preStop lifecycle hook, the kubelet sends SIGTERM to PID 1 immediately, causing the server to close its listening port while routers still forward new requests to the pod IP.

  • Shell-Form ENTRYPOINT Swallowing SIGTERM: Containers launched with shell-form ENTRYPOINT run /bin/sh -c as PID 1, which often fails to forward SIGTERM to the child application—forcing the container to sit idle until the 30-second grace period expires in forcible signal-9 termination.

  • Grace Period Countdown Includes preStop Time: Kubernetes starts the terminationGracePeriodSeconds timer before running preStop. If preStop sleep plus in-flight request draining exceeds 30 seconds, the runtime terminates the pod mid-request.



Architectural Principle: Delay SIGTERM with a preStop sleep long enough for EndpointSlice and Ingress convergence, deliver SIGTERM directly to the application as PID 1, and size terminationGracePeriodSeconds strictly greater than preStop sleep plus maximum request-drain time.


Why Standard Implementations Fail

1. Relying Solely on readinessProbe Without preStop

A readinessProbe controls initial traffic admission and unhealthy pod removal, but during rolling deletion the pod transitions directly to Terminating without waiting for the next probe interval—making preStop mandatory to hold the socket open during endpoint propagation.



2. Setting preStop Sleep Longer Than terminationGracePeriodSeconds

If preStop sleeps for 15 seconds and request draining takes 20 seconds while terminationGracePeriodSeconds remains at the 30-second default, the kubelet forces immediate termination at second 30, dropping active connections.



3. Leaving Idle Keep-Alive Sockets Open on SIGTERM

In Node.js and Go HTTP servers, calling server.close() stops new connections but keeps idle keep-alive sockets open; handlers must close idle connections explicitly once active requests finish draining.



4. Testing Zero-Downtime Rollouts With Sequential curl Loops

A single-threaded curl loop misses connection-pool and concurrency races; realistic validation requires concurrent load generators such as hey or k6 running across the entire rollout.



Step-by-Step Resolution Playbook

Step 1: Audit Container Entrypoint for Exec-Form PID 1

Ensure your Dockerfile uses JSON exec-form ENTRYPOINT or CMD so the application runs directly as PID 1 and receives SIGTERM:


FROM node:20-alpine
WORKDIR /app
COPY . .
ENTRYPOINT ["node", "server.js"]


Step 2: Implement Graceful SIGTERM Draining in Application Code

Close the listening server on SIGTERM while allowing active requests to complete and closing idle keep-alive sockets cleanly:


process.on("SIGTERM", () => {
server.close(() => process.exit(0));
if (typeof server.closeIdleConnections === "function") {
server.closeIdleConnections();
}
});


Step 3: Add preStop Sleep and Extended terminationGracePeriodSeconds

Configure a 10-second preStop delay (or target-group deregistration delay plus 5 seconds on cloud ALBs) and raise terminationGracePeriodSeconds to 60 seconds in your Deployment manifest:


spec:
terminationGracePeriodSeconds: 60
containers:
- name: api-gateway
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "sleep 10"]
readinessProbe:
httpGet:
path: /healthz
port: 8080
periodSeconds: 5


Step 4: Apply Manifest and Watch Endpoint Convergence

Apply the updated Deployment and monitor EndpointSlice transitions alongside pod termination events:


kubectl apply -f deployment.yaml
kubectl get endpointslices -l https://www.google.com/url?q=https://www.google.com/url?q%3Dhttp://kubernetes.io/service-name%253Dapi-gateway%26source%3Dgmail%26ust%3D1795352895232000%26sa%3DE&source=gmail&ust=1795353220117000&sa=E -w


Step 5: Validate Zero 502s Under Concurrent Rollout Load

Trigger a rolling restart while driving concurrent HTTP traffic with hey and confirm 100% 200 status codes with zero 502 or 503 responses:


kubectl rollout restart deployment/api-gateway &
hey -z 60s -c 25 https://www.google.com/url?q=https://www.google.com/url?q%3Dhttps://api.example.internal/healthz%26source%3Dgmail%26ust%3D1795352895232000%26sa%3DE&source=gmail&ust=1795353220117000&sa=E
Advertisement
Engineering Advisory 48-Hour Audit Turnaround

Experiencing this issue in your production systems?

Calricon conducts senior-level VoIP, SIP trunking, Asterisk, and cloud infrastructure audits. We review your live configs, identify packet drops or contention bottlenecks, and provide exact remediation runbooks.