Fix Kubernetes DNS 5-Second Timeouts and ndots:5 Amplification
calricon / field note
Production Kubernetes pods suffer intermittent 5-second API
Fix Kubernetes DNS 5-Second Timeouts and ndots:5 Amplification
In high-throughput Kubernetes clusters, microservices making outbound HTTPS or gRPC calls frequently experience intermittent, exact 5-second latency spikes and i/o timeout errors—even when CoreDNS pods show low CPU utilization and healthy readiness probes. Application traces reveal that getaddrinfo stalls for almost exactly 5,000 milliseconds before succeeding on retry, creating tail-latency cascades across upstream API gateways.
The root cause is a combination of two default behaviors: Kubernetes sets ndots:5 in every pod's /etc/resolv.conf, multiplying each external hostname lookup into up to eight sequential and parallel UDP packets, while glibc sends parallel IPv4 (A) and IPv6 (AAAA) queries from the same source socket. When those concurrent UDP packets hit kube-proxy DNAT rules simultaneously, they trigger a Linux kernel nf_conntrack insertion race that silently drops one packet, forcing the stub resolver to wait for its 5-second retry timer.
Problem Diagnosis & Root Cause Architecture
Every default ClusterFirst pod inherits a resolver configuration that amplifies DNS traffic and exposes a kernel connection-tracking race condition:
ndots:5Search Path Amplification: Any hostname containing fewer than five dots (such aspayments.vendor.examplewith two dots) is treated as a relative name. Before querying the external name directly, the pod resolver appends<namespace>.svc.cluster.local,svc.cluster.local, andcluster.local—turning one external lookup into four name queries, each issuing bothAandAAAArequests (eight UDP packets total).- Parallel Socket
nf_conntrackRace: On glibc-based containers,getaddrinfosendsAandAAAAUDP queries over the same source port. When two threads or packets race throughiptablesoripvsDNAT translation to the CoreDNS Service ClusterIP, the Linux kernel attempts to create two identical conntrack entries; one fails withinsert_failedand is silently dropped. - 5-Second Resolver Timeout Penalty: Because UDP provides no delivery acknowledgment, the pod's stub resolver waits for its default 5-second timeout before retransmitting the dropped query, producing clean 5,000 ms latency spikes in distributed traces.
Architectural Reality: Scaling CoreDNS replicas does not fix a 5-second UDP conntrack race. Ifconntrack -Sshows incrementinginsert_failedcounters on worker nodes, packets are being dropped inside the node's netfilter table before they ever leave the host or reach CoreDNS.
Why Standard Implementations Fail
1. Restarting or Over-Provisioning CoreDNS
Because CoreDNS pods are rarely CPU-saturated during a conntrack race, adding more CoreDNS replicas only distributes the same amplified UDP flood across more endpoints. The DNAT translation still happens on the originating worker node, so insert_failed drops continue unabated.
2. Lowering ndots Without Checking Internal Short Names
Changing ndots globally to 1 without auditing application configs breaks services that rely on short names like redis or postgres.payments instead of fully qualified domain names (postgres.payments.svc.cluster.local). Setting ndots:2 per workload or migrating internal callers to FQDNs avoids breaking intra-cluster discovery.
3. Relying on single-request-reopen on Alpine (musl) Images
Adding single-request-reopen to dnsConfig instructs glibc to open a new socket when parallel A and AAAA queries collide. However, Alpine-based containers use musl libc, which ignores single-request-reopen and continues firing concurrent UDP packets from a shared socket.
Step-by-Step Resolution Playbook
Step 1: Verify Conntrack Insert Failures and Pod Resolver Settings
Inspect the Linux kernel conntrack statistics on an affected worker node and check the active /etc/resolv.conf inside a running pod:
# On the worker node: check for netfilter insertion race drops
sudo conntrack -S
cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max
# Inside an affected pod: verify default ndots:5 and search domains
kubectl exec -it deploy/checkout-api -- cat /etc/resolv.confNon-zero insert_failed counters or nf_conntrack_count approaching nf_conntrack_max confirm that DNS packets are hitting netfilter drops.
Step 2: Tune Pod dnsConfig for External-Heavy Workloads
For deployments that predominantly call external APIs, override ndots to 2 (or 1 if all internal services use full FQDNs) and add single-request-reopen for glibc workloads:
apiVersion: apps/v1
kind: Deployment
metadata:
name: checkout-api
spec:
template:
spec:
dnsPolicy: ClusterFirst
dnsConfig:
options:
- name: ndots
value: "2"
- name: single-request-reopen
containers:
- name: checkout-api
image: registry.example/checkout-api:1.14.0Ensure any internal service references in environment variables use at least service.namespace.svc.cluster.local (or service.namespace when ndots is 2).
Step 3: Deploy NodeLocal DNSCache to Bypass Conntrack
For a cluster-wide structural fix that works for both glibc and Alpine/musl containers, deploy NodeLocal DNSCache as a DaemonSet. Pods send DNS queries to a link-local IP (169.254.20.10) on the same node, bypassing kube-proxy DNAT and conntrack entirely, while cache misses are forwarded to CoreDNS over persistent TCP connections:
# Retrieve your cluster DNS Service IP (typically 10.96.0.10)
CLUSTER_DNS=$(kubectl get svc kube-dns -n kube-system -o jsonpath='{.spec.clusterIP}')
echo "Cluster DNS IP: ${CLUSTER_DNS}"
# For managed clusters (e.g., GKE), enable the built-in NodeLocalDNS add-on
# or apply the upstream Kubernetes nodelocaldns.yaml manifest configured with:
# __PILLAR__DNS__SERVER__ -> ${CLUSTER_DNS}
# __PILLAR__LOCAL__DNS__ -> 169.254.20.10
kubectl apply -f nodelocaldns.yamlIn the node-local-dns Corefile, confirm that force_tcp is enabled on the upstream cluster forward block so cache misses never fall back to UDP conntrack.
Step 4: Point Kubelet ClusterDNS to the Local Cache IP
When using self-managed clusters in iptables mode, update the kubelet configuration on worker nodes (or via your node bootstrap template) so newly scheduled pods receive 169.254.20.10 in /etc/resolv.conf:
# /var/lib/kubelet/config.yaml excerpt
clusterDNS:
- 169.254.20.10Restart kubelet on rolling nodes and recycle application pods so they pick up the updated nameserver.
Step 5: Verify Cache Hits and Zero Conntrack Drops
Confirm that pods are resolving through 169.254.20.10, node-local-dns pods are healthy across every node, and CoreDNS upstream query volume has dropped:
kubectl get pods -n kube-system -l k8s-app=node-local-dns -o wide
kubectl exec -it deploy/checkout-api -- cat /etc/resolv.conf
kubectl exec -it deploy/checkout-api -- nslookup kubernetes.default.svc.cluster.local
# Verify local cache metrics on port 9253
kubectl logs -n kube-system -l k8s-app=node-local-dns --tail=50Monitor p99 DNS lookup duration and conntrack -S to confirm that 5-second timeout spikes and insert_failed increments have dropped to zero.
Experiencing this issue in your production systems?
Calricon conducts senior-level VoIP, SIP trunking, Asterisk, and cloud infrastructure audits. We review your live configs, identify packet drops or contention bottlenecks, and provide exact remediation runbooks.