Kubernetes Pod DNS Failure (CoreDNS) 5-Minute Diagnosis: can't resolve & Temporary failure

Kubernetes Pod DNS Failure (CoreDNS) 5-Minute Diagnosis: can't resolve & Temporary failure

K8s_Troubleshooting_Guide, Part 8

"A Pod that was fine yesterday suddenly can't resolve service names or external domains." If you operate clusters, you will hit this at least once. Application logs only show connection refused or i/o timeout, so it looks like a network issue—but the real cause is often DNS resolution failure. If you cannot turn a name into an IP, every subsequent call fails.

This installment focuses on one layer only: DNS. The goal is to look at the original English error, immediately classify it as cluster-internal, external domain, or partial failure, and narrow the cause in five minutes with five copy-paste commands.

1. Classify from the original error first

With DNS failures, the error message alone can cut the suspect layer by more than half.

In short: if even internal services fail, it is CoreDNS itself or the DNS path; if only external fails, look at ndots/upstream; if only a specific domain fails, suspect NetworkPolicy or a search-domain combination.

2. Five copy-paste diagnostic commands

① Query directly from a temporary dnsutils Pod

kubectl run -it --rm dnsutils \
  --image=registry.k8s.io/e2e-test-images/agnhost:2.39 \
  -- nslookup kubernetes.default

Healthy output:

Server:    10.96.0.10
Address:   10.96.0.10:53
Name:      kubernetes.default.svc.cluster.local
Address:   10.96.0.1

Unhealthy output:

;; connection timed out; no servers could be reached
# 또는
nslookup: can't resolve 'kubernetes.default'

If Server: 10.96.0.10 (the kube-dns ClusterIP) answers, the DNS path is alive. timed out means port 53 traffic is blocked or CoreDNS is down.

② CoreDNS Pod status

kubectl get pods -n kube-system -o wide | grep coredns

Healthy is Running with READY 1/1, usually at least two replicas. CrashLoopBackOff, Pending, 0/1, or replica count 0 is the cause.

③ CoreDNS logs

kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50

If you see [ERROR] plugin/errors ... SERVFAIL or i/o timeout toward upstream, it is a forward config / external resolver problem.

④ Check the Pod's resolv.conf

kubectl exec -it dnsutils -- cat /etc/resolv.conf

Healthy example:

nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

If nameserver is empty or only the node's external DNS is set (e.g. 168.126.63.1), internal services will not resolve. If the file itself is missing, you get Temporary failure in name resolution.

⑤ Check NetworkPolicy

kubectl get networkpolicy -A

If an egress-restricting policy exists and does not allow 53/UDP to kube-system, DNS packets are dropped and you get connection timed out.

3. Why are external domains slow? — ndots:5 and the five-step search

The Pod default options ndots:5 means: if the name has fewer than five dots, append search domains and query those first. Looking up github.com (one dot) actually sends queries in this order.

github.com.default.svc.cluster.local   → NXDOMAIN
github.com.svc.cluster.local           → NXDOMAIN
github.com.cluster.local               → NXDOMAIN
github.com                             → 성공

So resolving one external domain costs four queries, which adds latency; combined with UDP packet loss it looks like intermittent failure. Call a ClusterIP service as the FQDN svc.namespace.svc.cluster.local (trailing dot included) to skip the search list and finish in one shot.

4. Fix recipes by cause

(a) CoreDNS down / replica 0

kubectl scale deployment coredns -n kube-system --replicas=2
kubectl describe pod -n kube-system -l k8s-app=kube-dns   # 리소스 부족/OOM 확인

(b) External lookups slow because of ndots:5 — add dnsConfig to the Pod:

spec:
  dnsConfig:
    options:
      - name: ndots
        value: "1"

Lowering ndots to 1 makes dotted external domains skip search domains and query directly.

(c) NetworkPolicy blocking port 53 — DNS egress allow policy:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: my-app
spec:
  podSelector: {}
  policyTypes: ["Egress"]
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

(d) conntrack/iptables UDP race — On the node, check conntrack -S for rising insert_failed. UDP race conditions often cause 5-second delays (use-vc / single-request loss). Force TCP (options use-vc) or use NodeLocal DNSCache.

(e) CoreDNS upstream misconfigured — check forward in the Corefile:

forward . /etc/resolv.conf {
    max_concurrent 1000
}

On an internal network, point at resolvers directly: forward . 10.0.0.2 8.8.8.8. After the change, kubectl rollout restart deployment/coredns -n kube-system.

5. A practitioner note + lasting mitigation

In practice, a large share of "DNS is sometimes slow" tickets is (b) ndots and (d) UDP race working together. Rather than patching one or two Pods, introduce NodeLocal DNSCache cluster-wide. Running a caching DNS agent on each node so Pods query the node-local cache first greatly cuts conntrack contention and extra external queries, and responses stabilize. If you use an eBPF CNI (Cilium), DNS-aware policies also make port-53 control cleaner—worth considering together.

One-page summary checklist

1. Classify the original error → internal / external / partial failure
2. From dnsutils, nslookup kubernetes.default → does Server: 10.96.0.10 answer?
3. CoreDNS Pods Running 1/1 and replica count
4. CoreDNS logs for SERVFAIL/timeout
5. resolv.conf nameserver · ndots · search
6. NetworkPolicy allows 53/UDP
7. Stabilize with NodeLocal DNSCache

References: official docs

The primary sources for the behavior, settings, and errors in this post are the official docs below. Check them for version-specific options and exact behavior.

• Kubernetes official documentation

FAQ

Q. nslookup kubernetes.default works, but external domains don't. A. Likely CoreDNS upstream forward config or ndots:5. Check CoreDNS logs for SERVFAIL and inspect the forward target resolvers in the Corefile. Querying the external domain as an FQDN (trailing .) or lowering ndots to 1 via dnsConfig often fixes it immediately.

Q. What does Temporary failure in name resolution mean? A. A socket-level error: packets never even reached a DNS server. Suspect a missing nameserver in the Pod's /etc/resolv.conf, a NetworkPolicy blocking 53/UDP to kube-dns, or node network / CoreDNS down.

Q. DNS intermittently delays by about 5 seconds. A. Classic UDP conntrack race. On the node, check insert_failed with conntrack -S. The lasting fix is NodeLocal DNSCache.

Next installment (Part 9) preview: "Endpoints don't attach to the Service — diagnosing Service/Endpoint connection failures and selector mismatches"