calricon.
VoIP & Telecom

The Kubernetes ExternalTrafficPolicy Local Trap: Why Node Health

Nov 11, 2026/8 min read/Verified Production Playbook

calricon / field note

Discover why externalTrafficPolicy Local causes carrier SIP

Nov 11, 2026/VoIP & Telecom
340
Advertisement

The Kubernetes ExternalTrafficPolicy Local Trap: Why Node Health Check Lag Drops Carrier SIP

In containerized telecommunications platforms running session border controllers (such as Kamailio) and real-time voice gateways on Kubernetes, preserving the client's original source IP address is an essential operational requirement. Source IP preservation is mandatory for telecom fraud detection, carrier IP access control lists (ACLs), regional least-cost routing, and regulatory emergency location tracking. To prevent kube-proxy from executing Source Network Address Translation (SNAT) on inbound carrier traffic, platform architects standardly configure public LoadBalancer and NodePort Services with externalTrafficPolicy: Local. However, during routine pod rescheduling, node drains, or horizontal pod autoscaling (HPA) events, a catastrophic signaling blackhole emerges: incoming SIP INVITE and registration requests intermittently vanish for ten to twenty seconds. Upstream carrier gateways receive no responses, triggering trunk failovers and dropping active call attempts. The failure is caused by an architectural timing asymmetry between externalTrafficPolicy: Local packet filtering and upstream cloud load balancer health check convergence.



Problem Diagnosis & Root Cause Architecture

The operational drop stems from a temporal disconnect between Kubernetes local node routing rules and asynchronous cloud load balancer health probes:




  • Local-Only Packet Dispatching: When a Service specifies externalTrafficPolicy: Local, kube-proxy programs node iptables or IPVS rules to forward incoming traffic *only* to local pod endpoints residing on that specific worker node. If traffic arrives on a node that lacks a healthy pod replica, kube-proxy deliberately drops the packet to prevent cross-node forwarding that would require SNAT.

  • The Node Health Check Endpoint: To prevent cloud network load balancers (such as AWS NLB, GCP Network Load Balancer, or Azure Standard Load Balancer) from sending traffic to nodes without pods, Kubernetes exposes an HTTP health check port (spec.healthCheckNodePort). This endpoint returns HTTP 200 OK only if the node hosts at least one active, healthy pod for that Service.

  • The Polling Latency Blackhole: When a SIP proxy pod terminates or scales down on Node A, kube-proxy immediately removes the local endpoint from the packet filter. However, the external cloud load balancer monitors Node A via periodic HTTP polling (standardly configured with a 5-second interval and a 2-consecutive-failure threshold). For 10 to 15 seconds after the pod begins shutting down, the cloud load balancer continues dispatching incoming carrier UDP packets to Node A.

  • Silent Kernel Drops: Because Node A no longer hosts an active pod endpoint, its local Netfilter table drops every incoming UDP packet. The carrier SBC receives zero responses to its SIP transactions, interprets the silence as an unrecoverable network failure, and cancels the call with SIP/2.0 504 Gateway Timeout.




"Preserving client source IPs via local traffic policies requires synchronized load balancer routing. When a pod terminates faster than upstream load balancer health checks can detect, the worker node turns into a packet blackhole, silently discarding live carrier traffic."


Why Standard Implementations Fail

Reverting to externalTrafficPolicy: Cluster

Setting externalTrafficPolicy: Cluster eliminates packet blackholes by enabling cross-node forwarding when a local pod is missing. However, cross-node routing requires kube-proxy to apply SNAT to the incoming packet, replacing the carrier's real IP with the node's internal overlay IP. The SIP proxy receives all traffic appearing to originate from internal cluster IPs, completely breaking carrier IP authorization, fraud mitigation, and location routing.



Relying on Default Kubernetes Graceful Termination

Setting terminationGracePeriodSeconds: 30 does not prevent the load balancer blackhole if the container application exits prematurely. When the container receives SIGTERM, standard SIP proxies begin immediate shutdown routines. If the application terminates its listening socket before the load balancer's polling window expires, incoming calls continue hitting the node with no application listening.



Configuring Kubernetes Readiness Probes Alone

Marking a pod as unready removes it from internal Kubernetes Service endpoints, but does not immediately alter the cloud load balancer's external health check state. As long as the load balancer's health probe window has not completed its failure threshold, external traffic continues to flow to the terminating node.



Production-Tested Step-by-Step Engineering Playbook

Step 1: Diagnose External Traffic Packet Blackholes and Health Check Skew

Verify whether inbound call drops coincide with pod termination events and load balancer routing to nodes lacking local endpoints.



# Monitor real-time inbound SIP packet arrivals and kernel drops on the worker node
tcpdump -i eth0 -n "udp port 5060" | grep "INVITE" &
journalctl -u kube-proxy -f | grep -E "Deleting endpoint|healthcheck" | sed 20q

# Sample telemetry log demonstrating packet drop during pod termination:
# I1111 03:22:01.110291 1942 proxier.go:842] "Removing local endpoint for service telecom/kamailio-ingress"
# [PACKET INGRESS 03:22:02.140Z] https://www.google.com/url?q=http://198.51.100.10:5060&source=gmail&ust=1794483688144000&sa=E (https://www.google.com/url?q=https://www.google.com/url?q%3Dhttp://198.51.100.10:5060%26amp;source%3Dgmail%26amp;ust%3D1794482534715000%26amp;sa%3DE&source=gmail&ust=1794483688144000&sa=E) -> https://www.google.com/url?q=http://10.244.2.11:5060&source=gmail&ust=1794483688144000&sa=E (https://www.google.com/url?q=https://www.google.com/url?q%3Dhttp://10.244.2.11:5060%26amp;source%3Dgmail%26amp;ust%3D1794482534715000%26amp;sa%3DE&source=gmail&ust=1794483688144000&sa=E) (SIP INVITE)
# [KERNEL DROP 03:22:02.140Z] iptables: KUBE-NODEPORTS: DROP (No local endpoints available)
# (Cloud load balancer continued sending traffic for 11.2 seconds after endpoint deletion)


Step 2: Implement PreStop Draining Delays on Ingress SIP Pods

Configure a preStop lifecycle hook in the pod specification that delays process termination, keeping the application socket active while the cloud load balancer completes its health probe draining cycle.



# /opt/k8s/sip-ingress-deployment.yaml (Deployment Specification)
apiVersion: apps/v1
kind: Deployment
metadata:
name: kamailio-ingress
namespace: telecom
spec:
replicas: 3
template:
spec:
terminationGracePeriodSeconds: 45
containers:
- name: kamailio
image: kamailio:production-v5.8
lifecycle:
preStop:
exec:
command:
- /bin/bash
- -c
- |
echo "PreStop hook initiated: Holding socket open for load balancer drain..."
# Wait 15 seconds to allow cloud load balancer health checks to mark node unhealthy
# while application continues servicing in-flight SIP transactions
sleep 15
echo "Load balancer drain window elapsed. Proceeding with graceful shutdown."


Step 3: Optimize Cloud Load Balancer Health Check Frequencies

Configure the Kubernetes Service manifest with explicit health check annotations and tuned threshold parameters to minimize the detection window during node transitions.



# /opt/k8s/sip-ingress-service.yaml (Service Manifest)
apiVersion: v1
kind: Service
metadata:
name: kamailio-lb-service
namespace: telecom
annotations:
# Tune cloud load balancer health check interval and failure thresholds (AWS NLB example)
service.beta.kubernetes.internal/aws-load-balancer-type: "external"
service.beta.kubernetes.internal/aws-load-balancer-nlb-target-type: "instance"
service.beta.kubernetes.internal/aws-load-balancer-healthcheck-interval: "2"
service.beta.kubernetes.internal/aws-load-balancer-healthcheck-timeout: "2"
service.beta.kubernetes.internal/aws-load-balancer-healthcheck-healthy-threshold: "2"
service.beta.kubernetes.internal/aws-load-balancer-healthcheck-unhealthy-threshold: "2"
spec:
type: LoadBalancer
# PRESERVE CLIENT SOURCE IP ADDRESS
externalTrafficPolicy: Local
selector:
app: kamailio-ingress
ports:
- name: sip-udp
port: 5060
targetPort: 5060
protocol: UDP

Apply the updated service and deployment manifests to the cluster:



kubectl apply -f /opt/k8s/sip-ingress-service.yaml
kubectl apply -f /opt/k8s/sip-ingress-deployment.yaml
kubectl -n telecom rollout status deployment/kamailio-ingress


Step 4: Configure Cilium eBPF Direct Server Return (DSR) as an Architectural Alternative

In modern eBPF-enabled clusters running Cilium CNI, configure Direct Server Return (DSR) mode. DSR preserves the original client IP without requiring externalTrafficPolicy: Local, allowing seamless cross-node packet forwarding without SNAT.



# values-cilium-dsr.yaml
bpf:
masquerade: true

# Enable Direct Server Return (DSR) mode with geneve/vxlan encapsulation
loadBalancer:
mode: dsr
algorithm: maglev
dsrDispatch: opt

# Apply updated Helm values to Cilium
helm upgrade cilium cilium/cilium \
--namespace kube-system \
--reuse-values \
-f values-cilium-dsr.yaml


Step 5: Production Verification and Node Drain Ingress Stress Testing

Validate that simulated pod terminations, scaling events, and node drains maintain 100% call setup success without a single dropped SIP packet.



# Execute continuous synthetic SIP call generation using sipp across the Service VIP
sipp -sn uac https://www.google.com/url?q=http://198.51.100.10:5060&source=gmail&ust=1794483688144000&sa=E (https://www.google.com/url?q=https://www.google.com/url?q%3Dhttp://198.51.100.10:5060%26amp;source%3Dgmail%26amp;ust%3D1794482534716000%26amp;sa%3DE&source=gmail&ust=1794483688144000&sa=E) -r 25 -s 1001 -m 2000 -trace_err &

# Trigger a rolling update restart of the SIP ingress pods during active traffic
kubectl -n telecom rollout restart deployment/kamailio-ingress

# Monitor SIP transaction results and response codes
# Verify that zero 504 Gateway Timeout or 408 Request Timeout errors occur
grep -c "SIP/2.0 [45][0-9][0-9]" sipp_uac_*.log
# Expected output: 0

# Confirm client IP preservation on the backend proxy
kamctl rpc core.psx | grep "si="
# Output confirms real carrier IP is preserved across all active transactions

By coupling externalTrafficPolicy: Local with an optimized 15-second preStop load balancer draining hook, tuning health probe intervals, or deploying eBPF Direct Server Return (DSR), Kubernetes telecommunications clusters eliminate packet blackholes. Inbound carrier SIP traffic flows continuously across pod rescheduling and node maintenance with zero dropped calls and complete source IP fidelity.

Advertisement
Engineering Advisory 48-Hour Audit Turnaround

Experiencing this issue in your production systems?

Calricon conducts senior-level VoIP, SIP trunking, Asterisk, and cloud infrastructure audits. We review your live configs, identify packet drops or contention bottlenecks, and provide exact remediation runbooks.