Endpoints <none> · no endpoints available: 7 Causes and a 30-Second Diagnosis

DNS Resolves, but Where Did the Traffic Go?

Part 1 covered CoreDNS name resolution, and Part 2 confirmed this is not a NetworkPolicy block. Even so, curl http://my-svc:8080 from an application pod still returns connection refused. kubectl describe svc my-svc puts the answer on a single line.

TEXT

Endpoints:         <none>

This series walks three layers: name resolution (Part 1) → policy blocks (Part 2) → binding Service to Pod (Part 3). This installment is the third: the name resolves, policy is open, and yet there is nowhere to send the packet.

Unrolled as text, the traffic path looks like this.

TEXT

Client Pod
  → DNS 조회 (my-svc.default.svc.cluster.local → 10.96.x.x)   [1편 영역]
  → ClusterIP 10.96.x.x:8080
  → kube-proxy가 설치한 DNAT 룰 (iptables / IPVS / nftables)
  → EndpointSlice에 등록된 Pod IP:targetPort 목록
  → Pod IP 10.244.x.x:8080                                   [4편 영역]

Endpoints: <none> means line four is an empty array. The control-plane endpointslice controller looks for pods that match the Service selector and are Ready, then fills EndpointSlice. If that result is zero, kube-proxy reflects “no backends” into the data plane.

The application log line no endpoints available for service "default/my-svc" is the same signal. It is what the API server proxy path or an Ingress controller emits when the backend list is empty, and the causal layer is identical to Endpoints: <none>.

That creates the first fork: connection refused vs timeout.

Symptom	kube-proxy behavior	Implication
Immediate `connection refused`	Zero backends, so a REJECT rule is installed (iptables default)	Endpoints are likely empty
`timeout` after several to tens of seconds	DNAT happened, but the packet vanished with no reply	Endpoints are populated; look at CNI, policy, or an unresponsive app

If connection refused comes back immediately, start with this post’s decision table. If you get a timeout, skip ahead to the second half of section 3 and to Part 4. Overlapping symptoms are covered in the baseline flow at kubectl get endpoints <none>·Service connection refused 5분 진단; this post focuses on decomposing seven causes, version-specific traps, and automation gates.

Scope: Kubernetes 1.21+ (after EndpointSlice became default), kubectl 1.25+, and all kube-proxy modes (iptables / IPVS / nftables).

30-Second First Pass: Narrow the Scope with Six Commands

In an incident, do not think—run these from the top. Copy them and change only the service name and namespace.

Bash

SVC=my-svc
NS=default

# 1) 레거시 Endpoints 객체 확인 (가장 빠른 신호)
kubectl -n "$NS" get endpoints "$SVC"

# 2) EndpointSlice 확인 (1.21+ 실제 소스 오브 트루스)
kubectl -n "$NS" get endpointslices -l "kubernetes.io/service-name=$SVC" -o wide

# 3) Service 정의와 이벤트
kubectl -n "$NS" describe svc "$SVC"

# 4) Service selector 원문
kubectl -n "$NS" get svc "$SVC" -o jsonpath='{.spec.selector}{"\n"}'

# 5) 파드 라벨 전량 확인
kubectl -n "$NS" get pods --show-labels

# 6) selector로 실제 매칭되는 파드 수
kubectl -n "$NS" get pods -l "$(kubectl -n "$NS" get svc "$SVC" -o jsonpath='{range .spec.selector.*}{"\n"}{end}' >/dev/null; kubectl -n "$NS" get svc "$SVC" -o jsonpath='{.spec.selector}' | tr -d '{}"' | tr ',' ',')"

Healthy output looks like this.

TEXT

NAME     ENDPOINTS                                   AGE
my-svc   10.244.1.7:8080,10.244.2.9:8080             12d

If the ENDPOINTS column shows <none>, you have confirmation. The same conclusion holds if command 2 shows no EndpointSlice at all, or an empty ENDPOINTS column.

Split the layers: curl the Pod IP directly

The critical judgment is “is the app dead, is the Service binding broken, or is node-to-node traffic blocked?” One temporary netshoot pod splits that in 30 seconds.

Bash

kubectl -n default run tmp-netshoot --rm -it --restart=Never \
  --image=nicolaka/netshoot -- /bin/bash

Inside the pod shell, hit the three layers in order.

Bash

# ① Pod IP 직접 (Service를 건너뜀)
curl -sS -m 3 -o /dev/null -w "podip:%{http_code}\n" http://10.244.1.7:8080/

# ② ClusterIP
curl -sS -m 3 -o /dev/null -w "clusterip:%{http_code}\n" http://10.96.30.11:8080/

# ③ NodePort (해당 서비스가 NodePort 타입일 때)
curl -sS -m 3 -o /dev/null -w "nodeport:%{http_code}\n" http://192.168.10.21:30080/

The combination of results cuts the scope immediately.

① Pod IP	② ClusterIP	③ NodePort	Verdict	Next action
Success	Fail	Fail	Service ↔ Pod binding problem	Walk decision table (a)–(g) in section 3
Fail	Fail	Fail	Application / container port problem	Check container logs and `ss -lntp` for the listen port
Success	Success	Fail	External node ingress / externalTrafficPolicy	Part 4, `externalTrafficPolicy: Local` section
Success	Intermittent fail	Intermittent fail	Some backends unhealthy, or node-to-node communication	Check kube-proxy rules, then Part 4 (CNI)

Run ss -lntp inside the target container like this.

Bash

kubectl -n default exec -it deploy/web -- sh -c "ss -lntp || netstat -lntp"

Healthy output includes a line such as LISTEN 0 128 0.0.0.0:8080. If you only see 127.0.0.1:8080, the app is bound to loopback only—and traffic fails even when Endpoints are populated.

One-screen combined diagnostic script

Keep a file around for repeat incidents.

Bash

#!/usr/bin/env bash
# svc-diag.sh — Service/Endpoint 결합 상태 일괄 점검
# usage: ./svc-diag.sh <service-name> [namespace]
set -euo pipefail

SVC="${1:?service name required}"
NS="${2:-default}"

line() { printf '\n=== %s ===\n' "$1"; }

line "Service spec"
kubectl -n "$NS" get svc "$SVC" -o yaml | grep -E 'clusterIP:|type:|externalTrafficPolicy:|publishNotReadyAddresses:' || true

line "Selector"
SELECTOR=$(kubectl -n "$NS" get svc "$SVC" -o jsonpath='{.spec.selector}')
echo "raw: ${SELECTOR:-<empty>}"

line "Ports (port -> targetPort)"
kubectl -n "$NS" get svc "$SVC" \
  -o jsonpath='{range .spec.ports[*]}{.name}{" "}{.port}{" -> "}{.targetPort}{"\n"}{end}'

line "Endpoints (legacy)"
kubectl -n "$NS" get endpoints "$SVC" -o wide || echo "no endpoints object"

line "EndpointSlices"
kubectl -n "$NS" get endpointslices -l "kubernetes.io/service-name=$SVC" -o wide || true

line "EndpointSlice ready conditions"
kubectl -n "$NS" get endpointslices -l "kubernetes.io/service-name=$SVC" \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{range .endpoints[*]}{.addresses[0]}{"=ready:"}{.conditions.ready}{" "}{end}{"\n"}{end}' || true

line "Matching pods"
SEL_KV=$(kubectl -n "$NS" get svc "$SVC" \
  -o jsonpath='{range .spec.selector}{@}{end}' | tr -d '{}"' )
if [ -n "$SEL_KV" ]; then
  kubectl -n "$NS" get pods -l "$SEL_KV" -o wide || true
else
  echo "selector is empty (selector-less Service: manual EndpointSlice required)"
fi

line "All pod labels in namespace"
kubectl -n "$NS" get pods --show-labels

line "Recent events"
kubectl -n "$NS" get events --sort-by=.lastTimestamp | tail -20

Bash

chmod +x svc-diag.sh
./svc-diag.sh my-svc default

If the Matching pods section prints No resources found, the cause is almost certainly (a) selector mismatch. If pods appear but EndpointSlice is empty, it is (c) Readiness-related.

Cause Decision Table: 7 Cases and a One-Line Fix

Start with the full map.

#	Symptom	Cause	Confirm command	One-line recovery
a	Endpoints `<none>`, 0 pods via selector	Selector label typo / mismatch	`kubectl get pods -l app=web`	`kubectl patch svc my-svc -p '{"spec":{"selector":{"app":"web-api"}}}'`
b	Endpoints exist but connect fails, or `<none>`	targetPort ↔ containerPort mismatch, named port undefined	`kubectl get svc my-svc -o jsonpath='{.spec.ports[*].targetPort}'`	`kubectl patch svc my-svc --type=json -p '[{"op":"replace","path":"/spec/ports/0/targetPort","value":8080}]'`
c	Pod is Running but EndpointSlice ready=false	Readiness probe failure → NotReady	READY column from `kubectl get pods -o wide` + `kubectl describe pod`	`kubectl patch deploy web --type=json -p '[{"op":"replace","path":"/spec/template/spec/containers/0/readinessProbe/httpGet/path","value":"/healthz"}]'`
d	Fails only from another namespace	Services do not cross namespaces	`kubectl get pods -A -l app=web`	Use FQDN: `curl http://my-svc.other-ns.svc.cluster.local:8080`
e	`Endpoints: <none>` but DNS returns many pod IPs	Headless Service (`clusterIP: None`) misconfigured	`kubectl get svc my-svc -o jsonpath='{.spec.clusterIP}'`	Delete the Service and re-apply a manifest without `clusterIP: None`
f	Selector empty and no EndpointSlice	Selector-less Service missing a manual EndpointSlice	`kubectl get svc my-svc -o jsonpath='{.spec.selector}'` is blank	Apply a manual EndpointSlice YAML (example below)
g	Only some pods registered; fails only on certain nodes	Port collision on hostNetwork pods	Duplicate node in `kubectl get pods -o wide`	`kubectl patch deploy web -p '{"spec":{"template":{"spec":{"affinity":{"podAntiAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":[{"labelSelector":{"matchLabels":{"app":"web"}},"topologyKey":"kubernetes.io/hostname"}]}}}}}}'`

(a) Selector label typo / mismatch

This is the most common case. The Deployment template label is app: web-api, but the Service selector says app: web.

YAML

# 잘못된 버전
apiVersion: v1
kind: Service
metadata:
  name: my-svc
  namespace: default
spec:
  selector:
    app: web          # 파드 라벨은 web-api
  ports:
    - port: 8080
      targetPort: 8080

YAML

# 수정 버전
apiVersion: v1
kind: Service
metadata:
  name: my-svc
  namespace: default
spec:
  selector:
    app: web-api
  ports:
    - name: http
      port: 8080
      targetPort: 8080
      protocol: TCP

Confirmation is one line.

Bash

kubectl -n default get pods -l app=web

No resources found in default namespace. means the selector matches nothing. Remember that a Service selector is an AND. If you set both app: web and tier: backend, only pods with both labels match. Missing tier on the pod yields zero results.

(b) targetPort ↔ containerPort mismatch, named port undefined

Named ports improve readability, but if the container does not define the name, the EndpointSlice port is empty or filled with the wrong value.

YAML

# 잘못된 버전: Service는 http라는 이름을 찾는데 컨테이너에 이름이 없음
apiVersion: v1
kind: Service
metadata:
  name: my-svc
spec:
  selector:
    app: web-api
  ports:
    - port: 8080
      targetPort: http
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web-api
  template:
    metadata:
      labels:
        app: web-api
    spec:
      containers:
        - name: app
          image: nginx:1.27
          ports:
            - containerPort: 8080   # name 지정 누락

YAML

# 수정 버전: 컨테이너 포트에 name: http 정의
apiVersion: v1
kind: Service
metadata:
  name: my-svc
spec:
  selector:
    app: web-api
  ports:
    - name: http
      port: 8080
      targetPort: http
      protocol: TCP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web-api
  template:
    metadata:
      labels:
        app: web-api
    spec:
      containers:
        - name: app
          image: nginx:1.27
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP

Check command and expected result:

Bash

kubectl -n default get endpointslices -l kubernetes.io/service-name=my-svc \
  -o jsonpath='{range .items[*]}{range .ports[*]}{.name}{":"}{.port}{"\n"}{end}{end}'

Healthy output looks like http:8080—an actual number. If you have a name but an empty port, (b) is confirmed.

(c) Readiness probe failure → NotReady

If the pod is Running but READY 0/1, the endpointslice controller marks conditions.ready: false, and kube-proxy excludes that address from the service backends.

Bash

kubectl -n default get pods -l app=web-api
kubectl -n default describe pod <pod-name> | grep -A5 "Readiness"

TEXT

NAME                   READY   STATUS    RESTARTS   AGE
web-6f9c8d5b4c-2xk7p   0/1     Running   0          3m

You will typically also see an event such as Warning Unhealthy ... Readiness probe failed: HTTP probe failed with statuscode: 404. Classic case: probe path or port does not match the app. If you need to drill into probe failures themselves, use the branch table in K8s Liveness/Readiness probe failed·connection refused 원인별 해결.

publishNotReadyAddresses: true is the option that also puts NotReady addresses into EndpointSlice.

YAML

apiVersion: v1
kind: Service
metadata:
  name: db-headless
spec:
  clusterIP: None
  publishNotReadyAddresses: true
  selector:
    app: postgres
  ports:
    - name: pg
      port: 5432
      targetPort: 5432

The legitimate use is clear: StatefulSet-based clustered software that must discover peers during boot. Flip this on a normal web Service and user requests flow to unready pods, and 5xx rates climb. Turning it on to hide an outage is irreversible debt—even as a temporary bypass, file a ticket and set a revert deadline.

(d) Pod lives in another namespace

A Service selector finds pods only in the same namespace. There is no cross-namespace selector.

Bash

kubectl get pods -A -l app=web-api -o wide

If the pod is in prod and the Service is in default, move the Service or have the client use the FQDN.

Bash

curl -sS http://my-svc.prod.svc.cluster.local:8080/healthz

To alias an external name, use ExternalName.

YAML

apiVersion: v1
kind: Service
metadata:
  name: my-svc
  namespace: default
spec:
  type: ExternalName
  externalName: my-svc.prod.svc.cluster.local

ExternalName returns only a CNAME, so empty Endpoints is expected. <none> in this case is not an incident.

(e) Headless Service misconfiguration

You copied a StatefulSet manifest and clusterIP: None came along for the ride.

YAML

# 잘못된 버전: 일반 API 서비스인데 headless
apiVersion: v1
kind: Service
metadata:
  name: my-svc
spec:
  clusterIP: None
  selector:
    app: web-api
  ports:
    - port: 8080
      targetPort: 8080

YAML

# 수정 버전: ClusterIP 할당
apiVersion: v1
kind: Service
metadata:
  name: my-svc
spec:
  type: ClusterIP
  selector:
    app: web-api
  ports:
    - name: http
      port: 8080
      targetPort: 8080

spec.clusterIP is immutable; you cannot patch it. Delete and recreate.

Bash

kubectl -n default delete svc my-svc
kubectl -n default apply -f my-svc-fixed.yaml

DNS response shape also distinguishes the two.

Bash

kubectl run dnsq --rm -it --restart=Never --image=nicolaka/netshoot -- \
  dig +short my-svc.default.svc.cluster.local

Setup	dig result	Client behavior
Regular ClusterIP	Single line `10.96.30.11`	kube-proxy load-balances
Headless	Multiple pod IPs such as `10.244.1.7`, `10.244.2.9`	Client picks directly; connection-pool skew risk

(f) Selector-less Service + manual EndpointSlice

This is the pattern for exposing an external DB or an out-of-cluster legacy API under an in-cluster name. With no selector, the controller fills nothing, so you must create the EndpointSlice yourself.

YAML

apiVersion: v1
kind: Service
metadata:
  name: legacy-db
  namespace: default
spec:
  ports:
    - name: pg
      port: 5432
      targetPort: 5432
      protocol: TCP
---
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
  name: legacy-db-1
  namespace: default
  labels:
    kubernetes.io/service-name: legacy-db
addressType: IPv4
ports:
  - name: pg
    port: 5432
    protocol: TCP
endpoints:
  - addresses:
      - "192.168.50.31"
    conditions:
      ready: true

Three things people routinely omit:

labels."kubernetes.io/service-name" — without this label the Service never binds and stays empty forever.
addressType — you must set one of IPv4, IPv6, or FQDN.
ports[].name — must match the Service port name exactly. A name on only one side breaks the match.

Bash

kubectl -n default apply -f legacy-db.yaml
kubectl -n default get endpointslices -l kubernetes.io/service-name=legacy-db -o wide

Healthy: the ENDPOINTS column shows 192.168.50.31.

(g) hostNetwork pods and port collisions

A hostNetwork: true pod uses the node’s network namespace as-is. If two pods using the same port land on the same node, the second fails to bind, stays NotReady, and you get a partial failure where only some backends register.

Bash

kubectl -n default get pods -l app=web-api -o wide

If the NODE column repeats the same node name and one of those pods is 0/1, this is the case.

YAML

# 수정 버전: 노드당 1개만 뜨도록 anti-affinity 부여
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web-api
  template:
    metadata:
      labels:
        app: web-api
    spec:
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchLabels:
                  app: web-api
              topologyKey: kubernetes.io/hostname
      containers:
        - name: app
          image: nginx:1.27
          ports:
            - name: http
              containerPort: 8080

If you omit dnsPolicy: ClusterFirstWithHostNet on a hostNetwork pod, it will not use cluster DNS and the name-resolution issues from Part 1 come back. Remember them as a pair.

Endpoints Are Populated and It Still Fails + Version Traps

Version differences change the diagnostic commands

1.21: EndpointSlice became the default data source. Since then kube-proxy watches EndpointSlice, and the legacy Endpoints object is mirrored by a controller for compatibility.
Recommended as of 1.33: Diagnose with kubectl get endpointslices. The Endpoints API is being cleaned up in a direction that does not reflect newer features (for example traffic-distribution fields, multiple addressTypes).
Split past 100: An EndpointSlice holds at most 100 endpoints by default. 250 backends become 3 slices. The mirrored legacy Endpoints object may truncate or fail to reflect the full set, which produces the false reading “there are only 100 endpoints.”

On large Services, get in the habit of summing every slice:

Bash

kubectl -n default get endpointslices -l kubernetes.io/service-name=my-svc \
  -o jsonpath='{range .items[*]}{range .endpoints[*]}{.addresses[0]}{"\n"}{end}{end}' \
  | sort -u | wc -l

Healthy: the number equals the actual Ready pod count. If that differs from kubectl get endpoints output length, trust the EndpointSlice number.

Failure branch tree: start with kube-proxy mode

If Endpoints look healthy but ClusterIP still fails, you are in the data plane. Check the mode first.

Bash

kubectl -n kube-system get cm kube-proxy -o yaml | grep -i "mode"

Mode-specific rule dumps. Run them on the node or from a privileged debug pod.

Bash

# iptables 모드
sudo iptables-save | grep my-svc

# IPVS 모드
sudo ipvsadm -Ln | grep -A3 10.96.30.11

# nftables 모드 (1.31+ 에서 사용 가능)
sudo nft list ruleset | grep my-svc

Expected healthy output:

Mode	Healthy signature	When unhealthy
iptables	`KUBE-SVC-XXXX` chain and one `KUBE-SEP-XXXX` jump per backend	Chain exists but 0 SEPs → Endpoints not reflected
IPVS	Real server list under `TCP 10.96.30.11:8080`	0 real-server lines → same
nftables	Service chain and verdict map in the `kube-proxy` table	Missing entries → check kube-proxy pod logs

If all three look healthy and it still fails, it is not kube-proxy itself but the node-to-node path—Part 4 territory.

One flow to flag: with eBPF CNIs such as Cilium, kube-proxy replacement makes every command above meaningless. Having no service chains in iptables-save is the healthy state; diagnosis moves to cilium service list and cilium endpoint list. Check the CNI config before concluding “no iptables rules = outage.”

The `externalTrafficPolicy: Local` single-node failure pattern

On NodePort/LoadBalancer, setting Local to preserve client source IP means a request that lands on a node with no backend pod is not forwarded—it is dropped. Classic cause of “fails only when I hit 1 of 3 nodes.”

Bash

kubectl -n default get svc my-svc -o jsonpath='{.spec.externalTrafficPolicy}{"\n"}'
kubectl -n default get pods -l app=web-api -o wide
kubectl get nodes -o name

Compare the nodes that have pods against the full node list, and check whether traffic is hitting nodes with no pods. Two fix directions:

Bash

# 1) 소스 IP 보존이 필수가 아니면 Cluster로 전환
kubectl -n default patch svc my-svc -p '{"spec":{"externalTrafficPolicy":"Cluster"}}'

YAML

# 2) Local을 유지해야 하면 모든 노드에 파드를 배치 (DaemonSet 또는 anti-affinity + 충분한 replicas)
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: web
spec:
  selector:
    matchLabels:
      app: web-api
  template:
    metadata:
      labels:
        app: web-api
    spec:
      containers:
        - name: app
          image: nginx:1.27
          ports:
            - name: http
              containerPort: 8080

If packets still vanish only for certain node combinations after this, you are looking at overlay tunnel / MTU / routing—the subject of Part 4.

Preventing Recurrence: CI Gates, Probe Design, Alerts

yq-based manifest consistency checks

Mechanically comparing labels and ports before deploy keeps cases (a) and (b) out of production.

Bash

#!/usr/bin/env bash
# validate-svc-match.sh — Service selector ↔ Deployment 라벨/포트 정합성 검증
# usage: ./validate-svc-match.sh deploy.yaml svc.yaml
set -euo pipefail

DEPLOY_FILE="${1:?deployment yaml required}"
SVC_FILE="${2:?service yaml required}"
FAIL=0

POD_LABELS=$(yq -o=json '.spec.template.metadata.labels' "$DEPLOY_FILE")
SELECTOR=$(yq -o=json '.spec.selector' "$SVC_FILE")

echo "pod labels : $POD_LABELS"
echo "selector   : $SELECTOR"

# selector의 모든 key/value가 pod labels에 포함되는지 검사
MISSING=$(echo "$SELECTOR" | jq -r --argjson labels "$POD_LABELS" \
  'to_entries[] | select(($labels[.key] // "") != .value) | .key')

if [ -n "$MISSING" ]; then
  echo "FAIL: selector keys not matched in pod labels -> $MISSING"
  FAIL=1
else
  echo "OK: selector matches pod labels"
fi

# targetPort가 숫자인 경우 containerPort 존재 확인
TARGET=$(yq '.spec.ports[0].targetPort' "$SVC_FILE")
if [[ "$TARGET" =~ ^[0-9]+$ ]]; then
  HIT=$(yq ".spec.template.spec.containers[].ports[] | select(.containerPort == $TARGET) | .containerPort" "$DEPLOY_FILE" || true)
  if [ -z "$HIT" ]; then
    echo "FAIL: targetPort $TARGET has no matching containerPort"
    FAIL=1
  else
    echo "OK: targetPort $TARGET matches containerPort"
  fi
else
  # named port인 경우 이름 정의 확인
  HIT=$(yq ".spec.template.spec.containers[].ports[] | select(.name == \"$TARGET\") | .name" "$DEPLOY_FILE" || true)
  if [ -z "$HIT" ]; then
    echo "FAIL: named targetPort '$TARGET' is not defined in containers[].ports[].name"
    FAIL=1
  else
    echo "OK: named port '$TARGET' defined"
  fi
fi

exit "$FAIL"

Pair it with kubeconform for schema validation.

Bash

kubeconform -strict -summary -kubernetes-version 1.33.0 deploy.yaml svc.yaml

GitHub Actions example:

YAML

name: k8s-manifest-gate
on: [pull_request]
jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install tools
        run: |
          sudo wget -qO /usr/local/bin/yq https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64
          sudo chmod +x /usr/local/bin/yq
          curl -sSL https://github.com/yannh/kubeconform/releases/latest/download/kubeconform-linux-amd64.tar.gz | tar xz
          sudo mv kubeconform /usr/local/bin/
      - name: Schema validation
        run: kubeconform -strict -summary -kubernetes-version 1.33.0 manifests/
      - name: Selector/port match
        run: ./validate-svc-match.sh manifests/deploy.yaml manifests/svc.yaml

Three principles for Readiness probe design

Do not put dependencies in the probe. A readiness check that also tests the DB connection will yank every pod out of backends during a brief DB blip and manufacture Endpoints: <none> yourself. Readiness should answer only “is this process ready to take requests?” Expose dependency health as a separate metric.
Prefer startupProbe over a long initialDelaySeconds. A large initialDelay on a slow-booting JVM app also delays failure detection. Isolate boot with startupProbe and keep the readiness period short.
Prevent a full NotReady during rollout. Set maxUnavailable together with a PodDisruptionBudget.

YAML

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 4
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  selector:
    matchLabels:
      app: web-api
  template:
    metadata:
      labels:
        app: web-api
    spec:
      containers:
        - name: app
          image: nginx:1.27
          ports:
            - name: http
              containerPort: 8080
          startupProbe:
            httpGet:
              path: /healthz
              port: http
            failureThreshold: 30
            periodSeconds: 5
          readinessProbe:
            httpGet:
              path: /healthz
              port: http
            periodSeconds: 5
            timeoutSeconds: 2
            failureThreshold: 3
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: web-api

Alerting: catch zero endpoints within 5 minutes

A PrometheusRule based on kube-state-metrics.

YAML

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: service-endpoint-rules
  namespace: monitoring
  labels:
    release: kube-prometheus-stack
spec:
  groups:
    - name: service-endpoints
      rules:
        - alert: ServiceHasNoEndpoints
          expr: |
            kube_endpoint_address_available == 0
            and on (namespace, service)
            label_replace(
              kube_service_spec_type{type!="ExternalName"},
              "service", "$1", "service", "(.*)"
            )
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "Service {{ $labels.namespace }}/{{ $labels.service }} has zero endpoints"
            description: "5분 이상 사용 가능한 엔드포인트가 0개입니다. selector 라벨, targetPort, readiness probe 순서로 확인하세요."
        - alert: ServiceEndpointsDropped
          expr: |
            delta(kube_endpoint_address_available[10m]) < 0
            and kube_endpoint_address_available < 2
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "Endpoints decreasing for {{ $labels.namespace }}/{{ $labels.service }}"
            description: "엔드포인트 수가 감소해 2개 미만입니다. 롤아웃 또는 readiness 실패 여부를 확인하세요."

ExternalName Services normally have empty Endpoints, so the exclusion is mandatory or you will page on noise. Metric names may also appear as the kube_endpointslice_* family depending on kube-state-metrics version—check the metrics list on the version you actually run before applying.

This Installment’s Checklist and a Preview of the Next

In an incident, just run from the top.

Confirm emptiness with kubectl get endpoints <svc> and kubectl get endpointslices -l kubernetes.io/service-name=<svc>.
From netshoot, curl Pod IP → ClusterIP → NodePort to split the layers.
If kubectl get pods -l <selector> is 0, suspect (a) selector mismatch first.
If pods match but Endpoints are empty, check the READY column and readiness events for (c).
If targetPort is a named port, confirm the same name is defined on the container.
If the selector is empty, inspect the manual EndpointSlice kubernetes.io/service-name label and addressType.
If Endpoints are healthy and it still fails, inspect kube-proxy rules by mode and externalTrafficPolicy: Local.

Recovery is sometimes not instant: kube-proxy can take a few seconds to notice EndpointSlice changes and sync rules, and leftover conntrack sessions keep the old path. Do not pile on extra changes—recheck after about 30 seconds.

Next is Part 4, “Endpoints Are Healthy but Crossing Nodes Breaks — Debugging CNI Overlay, MTU, and the kube-proxy Data Plane.” Same-node works, cross-node dies; large responses vanish from MTU mismatch; tracing the VXLAN encapsulation path.

Official references worth reading alongside this: the Kubernetes docs on Service, EndpointSlice, and Virtual IPs and Service Proxies.

FAQ

Q1. Should I look at Endpoints or EndpointSlice? A. Since 1.21 the real data source is EndpointSlice. kubectl get endpoints is fine for a quick check, but once backends exceed 100 and slices split, the list can look truncated. For an accurate call, treat kubectl get endpointslices -l kubernetes.io/service-name=<svc> as the source of truth.

Q2. I want to send traffic to NotReady pods too. A. Set spec.publishNotReadyAddresses: true. The legitimate use is peer discovery during boot, as with StatefulSets. On a user-facing Service it sends requests to unready pods and produces 5xx. If it is a temporary bypass, set a revert deadline.

Q3. On a Headless Service, is Endpoints: <none> always an incident? A. No. A Headless Service with clusterIP: None can look different in kubectl output, and ExternalName has no endpoint concept at all—empty is normal. Judge by whether dig returns multiple pod IPs and whether addresses actually exist on the EndpointSlice.

Q4. I fixed the manifest. Why didn’t it recover immediately? A. The endpointslice controller has to apply the change, and kube-proxy has to sync rules on every node. On top of that, existing conntrack sessions keep the old destination, so recovery feels delayed. Force clients to open a new connection (or recycle the connection pool) to tell the difference.

Q5. What do connection refused vs timeout imply? A. Immediate connection refused usually means zero backends and a REJECT rule answering—start with the seven causes in this post. A timeout means the packet vanished with no reply, so suspect a NetworkPolicy block (Part 2), a CNI path/MTU issue (Part 4), or an app that is not responding.

확인 정보

#쿠버네티스 트러블슈팅 #kubernetes Service #EndpointSlice #kube-proxy

✦ ✦ ✦

편집 검토 · Editorial Review

Nodelog는 모든 콘텐츠의 내용과 출처를 공개 전에 검토합니다. 환경(OS·버전)에 따라 결과가 달라질 수 있는 기술 정보는 공식 문서와 함께 확인하며, 검토 기준과 정정 원칙은 편집 정책에서 안내합니다. 오류를 발견하시면 이메일로 제보해 주세요 — 확인 후 신속히 정정합니다.

편집 책임 · Nodelog 기술 편집팀·발행 · Aug 5, 2026·업데이트 · Sep 17, 2026

Endpoints <none> · no endpoints available: 7 Causes and a 30-Second Diagnosis

DNS Resolves, but Where Did the Traffic Go?

30-Second First Pass: Narrow the Scope with Six Commands

Split the layers: curl the Pod IP directly

One-screen combined diagnostic script

Cause Decision Table: 7 Cases and a One-Line Fix

(a) Selector label typo / mismatch

(b) targetPort ↔ containerPort mismatch, named port undefined

(c) Readiness probe failure → NotReady

(d) Pod lives in another namespace

(e) Headless Service misconfiguration

(f) Selector-less Service + manual EndpointSlice

(g) hostNetwork pods and port collisions

Endpoints Are Populated and It Still Fails + Version Traps

Version differences change the diagnostic commands

Failure branch tree: start with kube-proxy mode

The externalTrafficPolicy: Local single-node failure pattern

Preventing Recurrence: CI Gates, Probe Design, Alerts

yq-based manifest consistency checks

Three principles for Readiness probe design

Alerting: catch zero endpoints within 5 minutes

This Installment’s Checklist and a Preview of the Next

FAQ

Comments

The `externalTrafficPolicy: Local` single-node failure pattern