calricon.
DevOps & Cloud

Fixing Docker DNS Timeouts and systemd-resolved Failures on Linux

Nov 16, 2026/4 min read/Verified Production Playbook

calricon / field note

Docker containers on Ubuntu and Debian hosts frequently suffer

Nov 16, 2026/DevOps & Cloud
360
Advertisement

Fixing Docker DNS Timeouts and systemd-resolved Failures on Linux

In Linux production environments running Docker or Docker Compose on Ubuntu and Debian hosts, containerized applications frequently fail with lookup api.internal on https://www.google.com/url?q=https://www.google.com/url?q%3Dhttp://127.0.0.11:53%26source%3Dgmail%26ust%3D1794917805779000%26sa%3DE&source=gmail&ust=1794918122599000&sa=E: i/o timeout, Temporary failure in name resolution, or mysterious 5-second connection delays before every outbound API call—even though dig works instantaneously on the Linux host itself.

The root cause is a namespace mismatch with systemd-resolved's loopback stub (127.0.0.53) combined with Docker's embedded DNS fallback behavior, host firewall (ufw/iptables) rule flushes, and glibc ndots search-domain amplification.



Problem Diagnosis & Root Cause Architecture

Tracing a DNS packet from inside a container reveals four common failure points:



  • Loopback Stub Collision (127.0.0.53): Modern Linux distributions symlink /etc/resolv.conf to /run/systemd/resolve/stub-resolv.conf (nameserver 127.0.0.53). Containers cannot reach the host's loopback address from inside an isolated network namespace. When Docker sees 127.0.0.53, it either falls back to /run/systemd/resolve/resolv.conf or substitutes public 8.8.8.8/8.8.4.4—breaking split-horizon internal DNS and timing out behind egress firewalls that block public port 53.

  • iptables / firewalld Flushing Embedded DNS DNAT: Docker's embedded DNS server (127.0.0.11 on user-defined bridge networks) relies on netfilter DNAT rules in the container namespace and DOCKER chains on the host. Reloading iptables or ufw without restarting Docker wipes those forwarding rules, causing connection refused or silent drops.

  • Default Bridge Network Blindness: Containers attached to Docker's legacy default bridge network receive no embedded 127.0.0.11 resolver and cannot resolve sibling containers by service name.

  • 5-Second Timeout & Search-Domain Amplification: With default glibc resolver timeouts (5 seconds per nameserver attempt) and multiple search domains, a single unreachable upstream DNS server multiplies lookup latency into 5-to-15-second stalls.


Production Rule: Point /etc/resolv.conf at /run/systemd/resolve/resolv.conf (or configure explicit private/VPC DNS servers in /etc/docker/daemon.json), never put 127.0.0.11 in daemon.json, and recreate containers after any daemon DNS change.


Why Standard Implementations Fail

1. Adding 127.0.0.11 to daemon.json dns

Setting "dns": ["127.0.0.11"] in /etc/docker/daemon.json tells Docker's embedded DNS engine to forward unresolved external queries back to itself, creating an infinite forwarding loop and spiking CPU until queries time out.


2. Editing /etc/resolv.conf While It Remains a Symlink

Manually editing /etc/resolv.conf or disabling systemd-resolved without updating the symlink causes netplan or NetworkManager to overwrite your changes on the next DHCP renewal or reboot.


3. Restarting Docker Without Recreating Existing Containers

Updating /etc/docker/daemon.json and running systemctl restart docker only applies new dns and dns-opts settings to newly created containers—existing containers retain their old baked resolver configuration until recreated.



Validated Step-by-Step Resolution Playbook

Step 1: Isolate Embedded DNS vs. Host Egress Routing

Run a two-part diagnostic from inside the affected container to separate Docker's 127.0.0.11 forwarder from container-to-upstream network routing:


docker exec app-worker cat /etc/resolv.conf
docker exec app-worker nslookup api.internal 127.0.0.11
docker exec app-worker nslookup api.internal 1.1.1.1

If querying 1.1.1.1 (or your internal VPC DNS IP) directly succeeds while 127.0.0.11 times out, the issue is Docker's upstream resolver inheritance or host systemd-resolved forwarding.



Step 2: Fix systemd-resolved Symlink for Real Upstream Inheritance

Switch /etc/resolv.conf to point to systemd-resolved's real upstream file instead of the 127.0.0.53 stub (preserving DHCP/Netplan updates without disabling systemd-resolved):


sudo ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf
cat /etc/resolv.conf

Verify that real upstream nameserver IPs (such as your VPC resolver or corporate DNS) now appear in /etc/resolv.conf.



Step 3: Pin Explicit DNS and Fast-Fail Options in /etc/docker/daemon.json

Configure explicit upstream DNS servers and tuned resolver options in /etc/docker/daemon.json so queries fail over in 2 seconds instead of stalling for 5+ seconds:


{
"dns": ["10.0.0.2", "1.1.1.1"],
"dns-opts": ["ndots:0", "timeout:2", "attempts:2"]
}

Split-Horizon Note: Always list your internal VPC or private DNS server first (e.g., AWS VPC .2 resolver) so internal service names resolve without leaking to public DNS.



Step 4: Restart Docker and Recreate Containers

Apply the daemon configuration (and restore any missing Docker iptables chains), then recreate your Compose containers so they inherit the new resolver settings:


sudo systemctl restart docker
docker compose up -d --force-recreate


Step 5: Verify Sub-50ms Resolution Inside Containers

Test both inter-container service discovery and external domain resolution from inside a recreated container:


docker exec app-worker cat /etc/resolv.conf
docker exec app-worker getent hosts postgres
docker exec app-worker time nslookup api.internal

Confirm that options ndots:0 timeout:2 attempts:2 is present in container /etc/resolv.conf, internal service names resolve to bridge IPs, and external queries complete in milliseconds with zero https://www.google.com/url?q=https://www.google.com/url?q%3Dhttp://127.0.0.11:53%26source%3Dgmail%26ust%3D1794917805779000%26sa%3DE&source=gmail&ust=1794918122599000&sa=E timeouts.

Advertisement
Engineering Advisory 48-Hour Audit Turnaround

Experiencing this issue in your production systems?

Calricon conducts senior-level VoIP, SIP trunking, Asterisk, and cloud infrastructure audits. We review your live configs, identify packet drops or contention bottlenecks, and provide exact remediation runbooks.