Kubernetes Service Discovery Principles: From How CoreDNS Works to Headless Services

A Complete Guide to Kubernetes Service Discovery: From How CoreDNS Works to Troubleshooting

Last time we covered how Kubernetes Services distribute traffic through load balancing, focusing on L4-level networking concepts. In real production environments, though, the problem you run into most often is finding the address.

Imagine a large office building where department A needs to find department B’s team lead—but that person’s desk was temporarily moved to another floor today. IP addresses always exist; knowing which IP currently points at a given service’s real port is the heart of service discovery.

In this third installment, we’ll take a deep look at how this “address lookup” magically works inside a Kubernetes cluster—from the inner workings of CoreDNS, the central piece, all the way to real-world troubleshooting.

Why Service Discovery Matters: Why DNS Is Essential

When we build applications, assume service A calls service B. If service B’s pods are constantly being created and destroyed in batches of ten, service A cannot hardcode a changing list of IP addresses.

That’s where DNS (Domain Name System) comes in. DNS turns a complex IP addressing scheme into human-readable names (FQDN: Fully Qualified Domain Name)—the world’s phone book. In Kubernetes, CoreDNS plays that DNS role.

Put simply: a service name (user-api.default.svc.cluster.local) is the building’s “department name,” and CoreDNS is the information desk that tells you, in real time, the current office location (IP address) of the available team lead for that department.

The Backbone of Kubernetes Networking: Understanding Services and Endpoints

To understand service discovery, you need a clear picture of how three Kubernetes objects relate.

1. Pod: An isolated group of containers where the actual application runs. Its IP address is ephemeral.
2. Service: Gives a pod group an abstract logical name. That name is a stable “attachment point” that does not change even if pods die and come back.
3. Endpoint: The list of actual pod IP addresses attached to a given Service. The Service uses this Endpoint list to distribute traffic.

🔍 Comparing Key Commands Hands-On

Let’s confirm how these three connect with kubectl.

Key point: The ClusterIP you see from kubectl get svc is not a real pod IP. It is only a virtual address assigned by Kubernetes’ internal load balancer; actual traffic is then spread from that IP across multiple Endpoints.

Deep Dive into How CoreDNS Works: From Query to IP Mapping

So when a request arrives by service name, what path does CoreDNS take to return a final IP? Think of it as a precise three-stage filter.

[CoreDNS flow (conceptual)]

Client request: http://user-api.default.svc.cluster.local:8080 $\downarrow$ 1. CoreDNS Query: Every nameserver inside the cluster intercepts this request. $\downarrow$ 2. Service Lookup: CoreDNS uses its internal mapping rules to find that Service Name and check the ClusterIP or A Record. $\downarrow$ 3. Endpoint Resolution: Finally, CoreDNS responds with the list of actual pod IPs (Endpoints) that Service points at.

💡 YAML comparison: ClusterIP vs. Headless Service

To make this concrete, compare the two most important modes in a Service definition YAML.

1. Regular Service (using ClusterIP):

apiVersion: v1
kind: Service
metadata:
  name: user-service
spec:
  selector:
    app: user
  ports:
    - port: 80
      targetPort: 8080
  type: ClusterIP # <--- this type is the key

• Behavior: CoreDNS returns this ClusterIP, and kube-proxy inside the cluster automatically load-balances traffic arriving at that virtual IP across multiple pod IPs. Clients only need to remember a single address.

2. Headless Service (clusterIP: None):

apiVersion: v1
kind: Service
metadata:
  name: user-headless-service
spec:
  selector:
    app: user
  ports:
    - port: 80
      targetPort: 8080
  clusterIP: None # <--- this setting is the key

• Behavior: With clusterIP: None, the Service skips the load-balancing layer and returns each pod’s real IP address list as the DNS query result. That is useful when you need direct communication between services (for example, direct API calls between microservices).

Real-World Scenarios and Troubleshooting: Strategies for DNS Resolution Failures

The most common production issue is “service name resolution failed.” It usually happens in one of three cases.

1. Typo: A developer called user-apis instead of user-api.
2. Service deleted/redeployed: The Service was temporarily deleted, or all pods died and the Endpoint list is empty.
3. TTL expiry: DNS records are caching information that is too old.

🛡️ Why TTL (Time To Live) settings matter

DNS uses caching to stay fast. If a service migrates from IP A to IP B, but a client or CoreDNS still holds the cached A address for a long time, traffic goes to the wrong place.

That’s why TTL (Time To Live) matters. Set it too long and changes propagate slowly; set it too short and every lookup hits DNS, adding overhead. In production you must choose a TTL that matches how often the service changes and how traffic actually looks.

✍️ Senior architect’s practical tip: When you design inter-service communication, don’t rely on Service names alone. As you scale, consider introducing a service mesh. CoreDNS is a solid foundation, but a service mesh (Istio, Linkerd, and so on) handles discovery, traffic splitting (canary releases), certificate management, and logging in a single layer—and that dramatically cuts operational complexity.

Conclusion: Next Steps for Reliable Service Communication

We’ve gone from the fundamentals of Kubernetes service discovery through CoreDNS’s mechanics. If you now understand the relationship between Service and Endpoint, and how Headless Services give you direct address access, your grasp of cluster networking has moved up a level.

The next step is to introduce a service mesh, abstract the networking layer, and experience policy-based traffic control. That is how you evolve from simply “finding an address” to “safe, controllable communication.”

References: Official docs

The primary source for the behavior, settings, and errors covered here is the official documentation. Check version-specific options and exact behavior there.

• Kubernetes official documentation

FAQ

Q1. How does Kubernetes service discovery work without CoreDNS? A1. CoreDNS is the default DNS solution for a Kubernetes cluster. Without it, the cluster loses standardized namespace lookup for service names, and inter-service communication becomes much more complex—or impossible.

Q2. Is ClusterIP always a fixed IP? A2. ClusterIP is a virtual IP address assigned to a Service inside the cluster. That IP itself is stable, but the backend pod IPs (Endpoints) that actually receive and handle traffic keep changing as pods restart.

Q3. Is calling by service name the safest approach? A3. Yes, it is the safest. Using the service name (service-name) lets the internal load balancer spread traffic across every currently live pod IP, so a failure of one pod does not take down the whole service.

(This post was written as a fundamentals check for deeper K8s networking study. In real production, we strongly recommend considering a service mesh.)