calricon.
DevOps & Cloud

Fix Kubernetes 5-Second DNS Timeouts and conntrack Races

Nov 19, 2026/5 min read/Verified Production Playbook

calricon / field note

Production Kubernetes workloads frequently hit intermittent

Nov 19, 2026/DevOps & Cloud
371
Advertisement

Fix Kubernetes 5-Second DNS Timeouts and conntrack Races

When microservices in Kubernetes log intermittent dial tcp: lookup ...: i/o timeout, EAI_AGAIN, or socket.gaierror stalls that cluster tightly around exactly 5.00 seconds while CoreDNS pods show zero restarts and sub-millisecond latency, the bottleneck is inside the Linux kernel rather than CoreDNS itself.

This failure occurs when Kubernetes' default ndots:5 resolver setting amplifies every external hostname lookup into eight UDP packets, colliding with a Linux netfilter conntrack race condition during kube-proxy ClusterIP DNAT translation. Fixing it permanently requires curbing search-domain amplification per workload and deploying NodeLocal DNSCache to bypass UDP DNAT connection tracking on the node.



Problem Diagnosis & Root Cause Architecture

In default ClusterFirst DNS mode, kubelet writes a /etc/resolv.conf inside every pod containing three search domains (<namespace>.svc.cluster.local, svc.cluster.local, cluster.local) and options ndots:5:




  • ndots:5 Query Amplification: Any hostname containing fewer than five dots (such as api.vendor.internal) is treated as relative. The pod resolver sequentially appends all three search suffixes and sends both A and AAAA queries for each—turning one external lookup into eight UDP DNS packets before ever querying the real FQDN.

  • Linux Netfilter conntrack DNAT Race: Standard glibc resolvers dispatch the A and AAAA UDP queries in parallel from the same source port. Both packets hit the kube-dns ClusterIP DNAT rule simultaneously on different CPU cores, race to insert identical 5-tuple entries into nf_conntrack, and the losing packet is silently dropped by the kernel. The application blocks until the 5-second resolver timeout expires and retries on a new socket.

  • conntrack Table Saturation: Because UDP conntrack entries linger for 30 seconds by default (nf_conntrack_udp_timeout), high-QPS pods running ndots:5 can exhaust nf_conntrack_max on busy worker nodes, causing widespread packet drops across unrelated services.



If your DNS failures cluster around exactly 5.00 seconds while CoreDNS logs show zero errors, the packet is being dropped silently inside kernel netfilter DNAT before it ever reaches CoreDNS.


Why Standard Implementations Fail

Scaling CoreDNS Replicas Without Fixing Node-Level DNAT

Adding more CoreDNS pods does not eliminate the conntrack race because the collision happens on the originating worker node during ClusterIP DNAT translation, long before packets reach a CoreDNS endpoint.



Using single-request-reopen on Alpine or musl Images

Adding single-request-reopen to pod dnsConfig forces glibc to open separate sockets for A and AAAA queries, mitigating the race for Debian or Ubuntu containers. However, Alpine Linux (musl libc) ignores single-request-reopen, leaving Alpine-based containers vulnerable unless NodeLocal DNSCache is deployed.



Lowering ndots Without Verifying Short Service Names

Setting ndots:1 globally breaks workloads that rely on two-part short names such as redis.cache without a .svc.cluster.local suffix. Setting ndots:2 preserves cross-namespace service.namespace resolution while eliminating search-list amplification for standard multi-dot external FQDNs.



Step-by-Step Resolution Playbook

Step 1: Confirm conntrack Drops and CoreDNS Amplification

Check node-level conntrack insert failures and table utilization on an affected worker node, and inspect CoreDNS NXDOMAIN amplification:



# Check conntrack insert_failed counters on the worker node
conntrack -S

# Compare active conntrack entries against the node maximum
sysctl net.netfilter.nf_conntrack_max
conntrack -C

If insert_failed or drop increments during 5-second application stalls, you have confirmed the kernel conntrack race.



Step 2: Tune Pod dnsConfig for External-Heavy Workloads

For workloads that frequently call external APIs, lower ndots to 2 and add single-request-reopen (for glibc-based images) in the Deployment's spec.template.spec.dnsConfig:



apiVersion: apps/v1
kind: Deployment
metadata:
name: payment-worker
spec:
template:
spec:
dnsPolicy: ClusterFirst
dnsConfig:
options:
- name: ndots
value: "2"
- name: single-request-reopen
- name: timeout
value: "2"
- name: attempts
value: "2"


Step 3: Deploy NodeLocal DNSCache Across Worker Nodes

To eliminate ClusterIP DNAT and UDP conntrack entries cluster-wide—including for Alpine/musl containers—deploy NodeLocal DNSCache as a kube-system DaemonSet listening on a link-local address (such as 169.254.20.10):



# Inspect the existing kube-dns ClusterIP before templating nodelocaldns.yaml
kubectl get svc kube-dns -n kube-system -o jsonpath='{.spec.clusterIP}'

# Apply the NodeLocal DNSCache DaemonSet after substituting PILLAR variables
kubectl apply -f nodelocaldns.yaml

NodeLocal DNSCache answers cached queries locally without touching iptables DNAT and upgrades upstream cache-miss connections to CoreDNS over TCP, where conntrack entries close immediately instead of lingering for 30 seconds.



Step 4: Update kubelet DNS Flag for IPVS Clusters

If your cluster runs kube-proxy in IPVS mode, point kubelet's --cluster-dns flag to the link-local NodeLocal DNSCache IP (169.254.20.10) and restart kubelet; in iptables mode, NodeLocal DNSCache binds both addresses automatically.



# Verify node-local-dns DaemonSet pods are Running on every node
kubectl get pods -n kube-system -l k8s-app=node-local-dns -o wide


Step 5: Verify Latency and Zero conntrack Collisions in Production

Run a DNS resolution check from a pod and confirm sub-millisecond cache hits while conntrack -S stops incrementing insert_failed:



# Verify pod /etc/resolv.conf reflects updated options or link-local nameserver
kubectl exec -it deploy/payment-worker -- cat /etc/resolv.conf

# Confirm conntrack insert_failed stays flat under load
conntrack -S
Advertisement
Engineering Advisory 48-Hour Audit Turnaround

Experiencing this issue in your production systems?

Calricon conducts senior-level VoIP, SIP trunking, Asterisk, and cloud infrastructure audits. We review your live configs, identify packet drops or contention bottlenecks, and provide exact remediation runbooks.