Fix Kubernetes 5-Second DNS Timeouts and conntrack Races
calricon / field note
Production Kubernetes workloads frequently hit intermittent
Fix Kubernetes 5-Second DNS Timeouts and conntrack Races
When microservices in Kubernetes log intermittent dial tcp: lookup ...: i/o timeout, EAI_AGAIN, or socket.gaierror stalls that cluster tightly around exactly 5.00 seconds while CoreDNS pods show zero restarts and sub-millisecond latency, the bottleneck is inside the Linux kernel rather than CoreDNS itself.
This failure occurs when Kubernetes' default ndots:5 resolver setting amplifies every external hostname lookup into eight UDP packets, colliding with a Linux netfilter conntrack race condition during kube-proxy ClusterIP DNAT translation. Fixing it permanently requires curbing search-domain amplification per workload and deploying NodeLocal DNSCache to bypass UDP DNAT connection tracking on the node.
Problem Diagnosis & Root Cause Architecture
In default ClusterFirst DNS mode, kubelet writes a /etc/resolv.conf inside every pod containing three search domains (<namespace>.svc.cluster.local, svc.cluster.local, cluster.local) and options ndots:5:
- ndots:5 Query Amplification: Any hostname containing fewer than five dots (such as
api.vendor.internal) is treated as relative. The pod resolver sequentially appends all three search suffixes and sends bothAandAAAAqueries for each—turning one external lookup into eight UDP DNS packets before ever querying the real FQDN. - Linux Netfilter conntrack DNAT Race: Standard
glibcresolvers dispatch theAandAAAAUDP queries in parallel from the same source port. Both packets hit thekube-dnsClusterIP DNAT rule simultaneously on different CPU cores, race to insert identical 5-tuple entries intonf_conntrack, and the losing packet is silently dropped by the kernel. The application blocks until the 5-second resolver timeout expires and retries on a new socket. - conntrack Table Saturation: Because UDP
conntrackentries linger for 30 seconds by default (nf_conntrack_udp_timeout), high-QPS pods runningndots:5can exhaustnf_conntrack_maxon busy worker nodes, causing widespread packet drops across unrelated services.
If your DNS failures cluster around exactly 5.00 seconds while CoreDNS logs show zero errors, the packet is being dropped silently inside kernel netfilter DNAT before it ever reaches CoreDNS.
Why Standard Implementations Fail
Scaling CoreDNS Replicas Without Fixing Node-Level DNAT
Adding more CoreDNS pods does not eliminate the conntrack race because the collision happens on the originating worker node during ClusterIP DNAT translation, long before packets reach a CoreDNS endpoint.
Using single-request-reopen on Alpine or musl Images
Adding single-request-reopen to pod dnsConfig forces glibc to open separate sockets for A and AAAA queries, mitigating the race for Debian or Ubuntu containers. However, Alpine Linux (musl libc) ignores single-request-reopen, leaving Alpine-based containers vulnerable unless NodeLocal DNSCache is deployed.
Lowering ndots Without Verifying Short Service Names
Setting ndots:1 globally breaks workloads that rely on two-part short names such as redis.cache without a .svc.cluster.local suffix. Setting ndots:2 preserves cross-namespace service.namespace resolution while eliminating search-list amplification for standard multi-dot external FQDNs.
Step-by-Step Resolution Playbook
Step 1: Confirm conntrack Drops and CoreDNS Amplification
Check node-level conntrack insert failures and table utilization on an affected worker node, and inspect CoreDNS NXDOMAIN amplification:
# Check conntrack insert_failed counters on the worker node
conntrack -S
# Compare active conntrack entries against the node maximum
sysctl net.netfilter.nf_conntrack_max
conntrack -CIf insert_failed or drop increments during 5-second application stalls, you have confirmed the kernel conntrack race.
Step 2: Tune Pod dnsConfig for External-Heavy Workloads
For workloads that frequently call external APIs, lower ndots to 2 and add single-request-reopen (for glibc-based images) in the Deployment's spec.template.spec.dnsConfig:
apiVersion: apps/v1
kind: Deployment
metadata:
name: payment-worker
spec:
template:
spec:
dnsPolicy: ClusterFirst
dnsConfig:
options:
- name: ndots
value: "2"
- name: single-request-reopen
- name: timeout
value: "2"
- name: attempts
value: "2"Step 3: Deploy NodeLocal DNSCache Across Worker Nodes
To eliminate ClusterIP DNAT and UDP conntrack entries cluster-wide—including for Alpine/musl containers—deploy NodeLocal DNSCache as a kube-system DaemonSet listening on a link-local address (such as 169.254.20.10):
# Inspect the existing kube-dns ClusterIP before templating nodelocaldns.yaml
kubectl get svc kube-dns -n kube-system -o jsonpath='{.spec.clusterIP}'
# Apply the NodeLocal DNSCache DaemonSet after substituting PILLAR variables
kubectl apply -f nodelocaldns.yamlNodeLocal DNSCache answers cached queries locally without touching iptables DNAT and upgrades upstream cache-miss connections to CoreDNS over TCP, where conntrack entries close immediately instead of lingering for 30 seconds.
Step 4: Update kubelet DNS Flag for IPVS Clusters
If your cluster runs kube-proxy in IPVS mode, point kubelet's --cluster-dns flag to the link-local NodeLocal DNSCache IP (169.254.20.10) and restart kubelet; in iptables mode, NodeLocal DNSCache binds both addresses automatically.
# Verify node-local-dns DaemonSet pods are Running on every node
kubectl get pods -n kube-system -l k8s-app=node-local-dns -o wideStep 5: Verify Latency and Zero conntrack Collisions in Production
Run a DNS resolution check from a pod and confirm sub-millisecond cache hits while conntrack -S stops incrementing insert_failed:
# Verify pod /etc/resolv.conf reflects updated options or link-local nameserver
kubectl exec -it deploy/payment-worker -- cat /etc/resolv.conf
# Confirm conntrack insert_failed stays flat under load
conntrack -SExperiencing this issue in your production systems?
Calricon conducts senior-level VoIP, SIP trunking, Asterisk, and cloud infrastructure audits. We review your live configs, identify packet drops or contention bottlenecks, and provide exact remediation runbooks.