Fixing Docker DNS Timeouts and systemd-resolved Failures on Linux
calricon / field note
Docker containers on Ubuntu and Debian hosts frequently suffer
Fixing Docker DNS Timeouts and systemd-resolved Failures on Linux
In Linux production environments running Docker or Docker Compose on Ubuntu and Debian hosts, containerized applications frequently fail with lookup api.internal on https://www.google.com/url?q=https://www.google.com/url?q%3Dhttp://127.0.0.11:53%26source%3Dgmail%26ust%3D1794917805779000%26sa%3DE&source=gmail&ust=1794918122599000&sa=E: i/o timeout, Temporary failure in name resolution, or mysterious 5-second connection delays before every outbound API call—even though dig works instantaneously on the Linux host itself.
The root cause is a namespace mismatch with systemd-resolved's loopback stub (127.0.0.53) combined with Docker's embedded DNS fallback behavior, host firewall (ufw/iptables) rule flushes, and glibc ndots search-domain amplification.
Problem Diagnosis & Root Cause Architecture
Tracing a DNS packet from inside a container reveals four common failure points:
- Loopback Stub Collision (
127.0.0.53): Modern Linux distributions symlink/etc/resolv.confto/run/systemd/resolve/stub-resolv.conf(nameserver 127.0.0.53). Containers cannot reach the host's loopback address from inside an isolated network namespace. When Docker sees127.0.0.53, it either falls back to/run/systemd/resolve/resolv.confor substitutes public8.8.8.8/8.8.4.4—breaking split-horizon internal DNS and timing out behind egress firewalls that block public port53. iptables/firewalldFlushing Embedded DNS DNAT: Docker's embedded DNS server (127.0.0.11on user-defined bridge networks) relies on netfilter DNAT rules in the container namespace andDOCKERchains on the host. Reloadingiptablesorufwwithout restarting Docker wipes those forwarding rules, causingconnection refusedor silent drops.- Default Bridge Network Blindness: Containers attached to Docker's legacy default
bridgenetwork receive no embedded127.0.0.11resolver and cannot resolve sibling containers by service name. - 5-Second Timeout & Search-Domain Amplification: With default glibc resolver timeouts (5 seconds per nameserver attempt) and multiple search domains, a single unreachable upstream DNS server multiplies lookup latency into 5-to-15-second stalls.
Production Rule: Point/etc/resolv.confat/run/systemd/resolve/resolv.conf(or configure explicit private/VPC DNS servers in/etc/docker/daemon.json), never put127.0.0.11indaemon.json, and recreate containers after any daemon DNS change.
Why Standard Implementations Fail
1. Adding 127.0.0.11 to daemon.json dns
Setting "dns": ["127.0.0.11"] in /etc/docker/daemon.json tells Docker's embedded DNS engine to forward unresolved external queries back to itself, creating an infinite forwarding loop and spiking CPU until queries time out.
2. Editing /etc/resolv.conf While It Remains a Symlink
Manually editing /etc/resolv.conf or disabling systemd-resolved without updating the symlink causes netplan or NetworkManager to overwrite your changes on the next DHCP renewal or reboot.
3. Restarting Docker Without Recreating Existing Containers
Updating /etc/docker/daemon.json and running systemctl restart docker only applies new dns and dns-opts settings to newly created containers—existing containers retain their old baked resolver configuration until recreated.
Validated Step-by-Step Resolution Playbook
Step 1: Isolate Embedded DNS vs. Host Egress Routing
Run a two-part diagnostic from inside the affected container to separate Docker's 127.0.0.11 forwarder from container-to-upstream network routing:
docker exec app-worker cat /etc/resolv.conf
docker exec app-worker nslookup api.internal 127.0.0.11
docker exec app-worker nslookup api.internal 1.1.1.1If querying 1.1.1.1 (or your internal VPC DNS IP) directly succeeds while 127.0.0.11 times out, the issue is Docker's upstream resolver inheritance or host systemd-resolved forwarding.
Step 2: Fix systemd-resolved Symlink for Real Upstream Inheritance
Switch /etc/resolv.conf to point to systemd-resolved's real upstream file instead of the 127.0.0.53 stub (preserving DHCP/Netplan updates without disabling systemd-resolved):
sudo ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf
cat /etc/resolv.confVerify that real upstream nameserver IPs (such as your VPC resolver or corporate DNS) now appear in /etc/resolv.conf.
Step 3: Pin Explicit DNS and Fast-Fail Options in /etc/docker/daemon.json
Configure explicit upstream DNS servers and tuned resolver options in /etc/docker/daemon.json so queries fail over in 2 seconds instead of stalling for 5+ seconds:
{
"dns": ["10.0.0.2", "1.1.1.1"],
"dns-opts": ["ndots:0", "timeout:2", "attempts:2"]
}Split-Horizon Note: Always list your internal VPC or private DNS server first (e.g., AWS VPC .2 resolver) so internal service names resolve without leaking to public DNS.
Step 4: Restart Docker and Recreate Containers
Apply the daemon configuration (and restore any missing Docker iptables chains), then recreate your Compose containers so they inherit the new resolver settings:
sudo systemctl restart docker
docker compose up -d --force-recreateStep 5: Verify Sub-50ms Resolution Inside Containers
Test both inter-container service discovery and external domain resolution from inside a recreated container:
docker exec app-worker cat /etc/resolv.conf
docker exec app-worker getent hosts postgres
docker exec app-worker time nslookup api.internalConfirm that options ndots:0 timeout:2 attempts:2 is present in container /etc/resolv.conf, internal service names resolve to bridge IPs, and external queries complete in milliseconds with zero https://www.google.com/url?q=https://www.google.com/url?q%3Dhttp://127.0.0.11:53%26source%3Dgmail%26ust%3D1794917805779000%26sa%3DE&source=gmail&ust=1794918122599000&sa=E timeouts.
Experiencing this issue in your production systems?
Calricon conducts senior-level VoIP, SIP trunking, Asterisk, and cloud infrastructure audits. We review your live configs, identify packet drops or contention bottlenecks, and provide exact remediation runbooks.