Fix Temporary failure in name resolution: a host, Docker, and K8s DNS diagnostic runbook

Find the exact error message you just saw

If a server that was fine yesterday now dies on apt-get update with Temporary failure in name resolution, or restarting a container produces curl: (6) Could not resolve host, you are in the right place. This is not a DNS concepts guide — it is a runbook that goes error text → command → fix. We do not cover concepts (recursive queries, CoreDNS plugin architecture, and so on).

• If it fails only inside a Pod and you need to dig into CoreDNS itself → Kubernetes Pod DNS failure (CoreDNS) 5-minute diagnosis
• If the name resolves but the port connection is refused → connection refused / ECONNREFUSED 127.0.0.1 30-second diagnostic runbook

Scope: Ubuntu 22.04/24.04, Debian 12, and RHEL 9-family hosts; Docker Engine 24–27; Kubernetes 1.28–1.31. Start with the decision table below.

Error-text decision table: 5 messages, 5 layers

How to use this: once you have matched your message in the table, run the five-step diagnosis in the next section from top to bottom. The five steps are a layer separator that every message goes through.

30-second layer-separation diagnostic tree

① cat /etc/resolv.conf — is there even a resolver address?

cat /etc/resolv.conf
ls -l /etc/resolv.conf

Healthy output (systemd-resolved stack)

# This is /run/systemd/resolve/stub-resolv.conf
nameserver 127.0.0.53
options edns0 trust-ad
search ap-northeast-2.compute.internal

Unhealthy output examples

# Case A: completely empty (no output)
# Case B: search only, no nameserver line
search example.internal
# Case C: broken symlink
lrwxrwxrwx 1 root root 39 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf
cat: /etc/resolv.conf: No such file or directory

Cases A, B, and C produce Temporary failure in name resolution immediately. The one-liner below is a temporary recovery, but it will be overwritten on reboot or DHCP renew, so you must continue through the permanent host configuration below.

# temporary recovery (gone on reboot)
echo "nameserver 1.1.1.1" | sudo tee /etc/resolv.conf

② resolvectl status — does the stub resolver know its upstream?

resolvectl status

Healthy output (excerpt)

Link 2 (eth0)
    Current Scopes: DNS
         Protocols: +DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 10.0.0.2
       DNS Servers: 10.0.0.2
        DNS Domain: ap-northeast-2.compute.internal

Unhealthy output

Link 2 (eth0)
    Current Scopes: none          <- no upstream
       DNS Servers:               <- empty
         Protocols: ... DNSSEC=yes/supported   <- can cause SERVFAIL against internal DNS

Current Scopes: none means the resolver has nowhere to ask. If DNSSEC=yes while you are using internal/air-gapped DNS, that is a prime suspect for SERVFAIL.

③ dig @8.8.8.8 vs dig — the key branch point

dig +short @8.8.8.8 example.com
dig +short example.com
curl -sS -o /dev/null -w '%{http_code}\n' https://example.com

dig sends UDP queries directly and does not go through /etc/nsswitch.conf. If dig works but curl/ping do not, look at the NSS layer or proxy environment variables. Run the following as a control:

getent hosts example.com      # lookup that follows the NSS path as-is

④ ss -lunp | grep :53 — is the stub resolver actually listening?

sudo ss -lunp | grep :53

Healthy

UNCONN 0 0 127.0.0.53%lo:53 0.0.0.0:* users:(("systemd-resolve",pid=612,fd=13))

Unhealthy: if there is no output at all, resolv.conf points at 127.0.0.53 but nothing is listening there. That is the classic Temporary failure in name resolution combination.

systemctl status systemd-resolved
sudo systemctl enable --now systemd-resolved

⑤ nc -zvu + dig +tcp — is only UDP blocked, or TCP as well?

nc -zvu 10.0.0.2 53          # UDP 53 reachability
dig +tcp @10.0.0.2 example.com   # TCP 53 reachability

Healthy: Connection to 10.0.0.2 53 port [udp/domain] succeeded! Unhealthy: no response then timeout → outbound 53 is blocked by a security group, iptables, or NetworkPolicy. UDP is connectionless so nc -zvu can false-positive; always cross-check with dig +tcp.

Verdict → fix mapping

Layer-by-layer fixes ① Host

stub-resolv.conf vs resolv.conf: which should you point at?

To switch to the second form, replace the symlink as follows. The key is to change the link, not to edit the file directly.

sudo ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf
resolvectl status | head -20

Three ways to make it permanent — which to use where

netplan (Ubuntu 22.04/24.04 servers)

# /etc/netplan/01-netcfg.yaml
network:
  version: 2
  ethernets:
    eth0:
      dhcp4: true
      dhcp4-overrides:
        use-dns: false
      nameservers:
        addresses: [10.0.0.2, 1.1.1.1]
        search: [example.internal]

sudo chmod 600 /etc/netplan/01-netcfg.yaml
sudo netplan generate && sudo netplan apply
resolvectl status | grep -A2 'Link 2'

nmcli (RHEL 9 / NetworkManager)

nmcli con show
sudo nmcli con mod "System eth0" ipv4.dns "10.0.0.2 1.1.1.1"
sudo nmcli con mod "System eth0" ipv4.dns-search "example.internal"
sudo nmcli con mod "System eth0" ipv4.ignore-auto-dns yes
sudo nmcli con up "System eth0"

If you omit ipv4.ignore-auto-dns yes, DHCP-provided DNS stays in front and your settings are ignored.

systemd-resolved global config

# /etc/systemd/resolved.conf
[Resolve]
DNS=10.0.0.2 10.0.0.3
FallbackDNS=1.1.1.1
Domains=~.
DNSSEC=false
DNSStubListener=yes

sudo systemctl restart systemd-resolved
resolvectl status
resolvectl query example.com

Domains=~. is a routing directive meaning "send every domain to this link's DNS". It stops queries from leaking to the wrong link on VPNs or multi-interface hosts.

When to set DNSSEC=false: if public DNS works, querying internal DNS returns SERVFAIL, resolvectl status shows DNSSEC=yes, and dig +cd (checking disabled) succeeds — that three-part pattern strongly suggests the internal zone is not DNSSEC-signed. This setting gives up validation on the internet path, so apply it only on air-gapped/internal segments and document why you changed it.

dig +cd @10.0.0.2 internal.example.com   # if this succeeds, DNSSEC validation failure is the cause

Layer-by-layer fixes ② Docker

Why the host works but containers do not

If the host /etc/resolv.conf has only nameserver 127.0.0.53, that loopback address is meaningless inside the container namespace. Docker filters out these localhost-only addresses and substitutes a public DNS default (8.8.8.8 and similar). On an air-gapped network with no internet, container DNS dies at that moment. Confirm from inside a container.

docker run --rm alpine cat /etc/resolv.conf
# red flag: nameserver 8.8.8.8  (public DNS on an internal network)
docker run --rm alpine nslookup registry.example.internal

The real fix: daemon.json

{
  "dns": ["10.0.0.2", "10.0.0.3"],
  "dns-search": ["example.internal"],
  "dns-opts": ["ndots:1", "timeout:2", "attempts:2"]
}

sudo vi /etc/docker/daemon.json
sudo systemctl restart docker
docker run --rm alpine cat /etc/resolv.conf   # success if you see 10.0.0.2

Situation-specific alternatives

# docker-compose.yml
services:
  api:
    image: myorg/api:1.4.0
    dns:
      - 10.0.0.2
    dns_search:
      - example.internal

docker compose config | grep -A3 dns
docker compose up -d --force-recreate api

After editing daemon.json, already-running containers must be recreated for the new resolv.conf to take effect. A restart alone often does not change it.

Layer-by-layer fixes ③ Kubernetes

Latency and intermittent failures caused by ndots:5

A Pod's default /etc/resolv.conf looks like this.

search default.svc.cluster.local svc.cluster.local cluster.local ap-northeast-2.compute.internal
nameserver 10.96.0.10
options ndots:5

api.example.com has two dots, which is below the ndots:5 threshold, so the resolver first appends each search domain and fires 4–5 queries like api.example.com.default.svc.cluster.local. Only after every one returns NXDOMAIN does it try the absolute query. The result is hundreds of milliseconds to several seconds of latency, and intermittent failures under load.

Immediate fix 1 — trailing dot on the FQDN: change the hostname in code/config to api.example.com. (dot at the end) and the search step is skipped.

Immediate fix 2 — tune ndots

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  template:
    spec:
      dnsPolicy: ClusterFirst
      dnsConfig:
        options:
          - name: ndots
            value: "2"
          - name: timeout
            value: "2"
          - name: attempts
            value: "2"
      containers:
        - name: api
          image: myorg/api:1.4.0

Lowering to ndots:2 keeps search for in-cluster services in svc.ns form (one dot) while sending external domains straight to an absolute query. If any code uses dotless short names like service-name, verify there is no impact in staging first.

kubectl apply -f deploy.yaml
kubectl exec -it deploy/api -- cat /etc/resolv.conf   # confirm options ndots:2

If every Pod died at once: NetworkPolicy egress

If you apply an egress policy without opening kube-dns port 53, the entire namespace fails name resolution at once. Apply the following as-is.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-egress
  namespace: prod
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

If you omit TCP 53, only queries whose responses exceed 512 bytes and retry over TCP fail — a hard-to-reproduce outage. If you use an eBPF CNI such as Cilium, you can watch the drop point directly with hubble observe --protocol dns.

Turn on CoreDNS query logs

kubectl -n kube-system edit cm coredns

.:53 {
    log            # add this line
    errors
    health
    kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure }
    forward . /etc/resolv.conf
    cache 30
}

kubectl -n kube-system rollout restart deploy/coredns
kubectl logs -n kube-system -l k8s-app=kube-dns -f | grep example.com

forward . /etc/resolv.conf means follow the node's resolv.conf as-is. If one node's DNS config is polluted, cluster-wide queries can fail through the CoreDNS Pod scheduled on that node. If you suspect a node, pin the upstream explicitly.

forward . 10.0.0.2 10.0.0.3 {
    max_concurrent 1000
}

One-liner test Pod:

kubectl run -it --rm dnsutils --image=registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7 --restart=Never -- bash
# inside: dig +short kubernetes.default.svc.cluster.local ; dig +short api.example.com.

If you need to go deeper into CoreDNS itself, continue with Kubernetes Pod DNS failure (CoreDNS) 5-minute diagnosis. On large clusters where latency is a recurring problem, introducing NodeLocal DNSCache has become a standard response.

Preventing recurrence and three common misdiagnoses

Cache-flush command cheat sheet

resolvectl flush-caches                     # systemd-resolved
sudo systemctl restart dnsmasq              # dnsmasq
sudo nscd -i hosts                          # nscd environments
kubectl -n kube-system rollout restart deploy/coredns   # CoreDNS

Decision rule: if the symptom is unchanged after flushing the cache, the cache was not the cause. Stop repeating flushes and go back to diagnostic step ③ (the dig branch point).

Clearing up a TTL misconception

When a record change does not show up, it is usually waiting for TTL expiry, not an outage. Check remaining TTL as follows.

dig +noall +answer example.com
# example.com. 287 IN A 93.184.216.34   <- 287 seconds remaining
dig +noall +answer @8.8.8.8 example.com   # compare with the authoritative-path value

Three misdiagnoses and a one-line disproof each

Fix for the third case:

export no_proxy="localhost,127.0.0.1,.example.internal,10.0.0.0/8,.svc,.cluster.local"
export NO_PROXY="$no_proxy"

DNS health check on cron

#!/usr/bin/env bash
# /usr/local/bin/dns-healthcheck.sh
set -u
TARGETS=("internal.example.internal" "example.com")
FAILED=()
for d in "${TARGETS[@]}"; do
  dig +short +time=2 +tries=1 "$d" | grep -qE '^[0-9]' || FAILED+=("$d")
done
[ ${#FAILED[@]} -eq 0 ] && exit 0
logger -t dns-healthcheck "DNS resolution failed: ${FAILED[*]}"
exit 1

sudo chmod +x /usr/local/bin/dns-healthcheck.sh
echo '*/5 * * * * root /usr/local/bin/dns-healthcheck.sh' | sudo tee /etc/cron.d/dns-healthcheck

The key is to keep one internal domain and one external domain. If both fail, it is the resolver/network; if only internal fails, it is internal DNS or the forward config; if only external fails, it is the upstream/proxy path. You can swap logger for an internal webhook call on failure.

FAQ

Q. /etc/resolv.conf reverts after reboot even though I edited it directly. A. That file is generated by systemd-resolved or NetworkManager. Edit the actual source for your environment — netplan nameservers:, nmcli con mod ipv4.dns, or DNS= in /etc/systemd/resolved.conf — then run netplan apply or restart the service.

Q. dig works on the host but name resolution fails only inside containers. A. First run cat /etc/resolv.conf inside the container. If the host has only 127.0.0.53, Docker will not inherit it and substitutes public DNS, which fails on an air-gapped network. Put "dns": ["10.0.0.2"] in /etc/docker/daemon.json, restart the daemon, and recreate the container.

Q. External domain lookups from a Pod are slow and occasionally fail. Is the DNS server the problem? A. More often the cause is wasted search-domain queries from options ndots:5, not the server. Append a trailing dot to make the hostname an FQDN, or set ndots:2 via dnsConfig.options. Start by checking whether the option took effect with kubectl exec -- cat /etc/resolv.conf.