## DNS Resolves, but Where Did the Traffic Go? Part 1 covered CoreDNS name resolution, and Part 2 confirmed this is not a NetworkPolicy block. Even so, `curl http://my-svc:8080` from an application pod still returns `connection refused`. `kubectl describe svc my-svc` puts the answer on a single line. ```text Endpoints: ``` This series walks three layers: **name resolution (Part 1) → policy blocks (Part 2) → binding Service to Pod (Part 3)**. This installment is the third: the name resolves, policy is open, and yet **there is nowhere to send the packet**. Unrolled as text, the traffic path looks like this. ```text Client Pod → DNS 조회 (my-svc.default.svc.cluster.local → 10.96.x.x) [1편 영역] → ClusterIP 10.96.x.x:8080 → kube-proxy가 설치한 DNAT 룰 (iptables / IPVS / nftables) → EndpointSlice에 등록된 Pod IP:targetPort 목록 → Pod IP 10.244.x.x:8080 [4편 영역] ``` `Endpoints: ` means **line four is an empty array**. The control-plane endpointslice controller looks for pods that match the Service `selector` and are Ready, then fills EndpointSlice. If that result is zero, kube-proxy reflects “no backends” into the data plane. The application log line `no endpoints available for service "default/my-svc"` is the same signal. It is what the API server proxy path or an Ingress controller emits when the backend list is empty, and the causal layer is identical to `Endpoints: `. That creates the first fork: **`connection refused` vs `timeout`**. | Symptom | kube-proxy behavior | Implication | |---|---|---| | Immediate `connection refused` | Zero backends, so a REJECT rule is installed (iptables default) | Endpoints are likely empty | | `timeout` after several to tens of seconds | DNAT happened, but the packet vanished with no reply | Endpoints are populated; look at CNI, policy, or an unresponsive app | If `connection refused` comes back immediately, start with this post’s decision table. If you get a `timeout`, skip ahead to the second half of section 3 and to Part 4. This post focuses on decomposing seven causes, version-specific traps, and automation gates. Scope: Kubernetes 1.21+ (after EndpointSlice became default), kubectl 1.25+, and all kube-proxy modes (iptables / IPVS / nftables). ## 30-Second First Pass: Narrow the Scope with Six Commands In an incident, do not think—run these from the top. Copy them and change only the service name and namespace. ```bash SVC=my-svc NS=default # 1) 레거시 Endpoints 객체 확인 (가장 빠른 신호) kubectl -n "$NS" get endpoints "$SVC" # 2) EndpointSlice 확인 (1.21+ 실제 소스 오브 트루스) kubectl -n "$NS" get endpointslices -l "kubernetes.io/service-name=$SVC" -o wide # 3) Service 정의와 이벤트 kubectl -n "$NS" describe svc "$SVC" # 4) Service selector 원문 kubectl -n "$NS" get svc "$SVC" -o jsonpath='{.spec.selector}{"\n"}' # 5) 파드 라벨 전량 확인 kubectl -n "$NS" get pods --show-labels # 6) selector로 실제 매칭되는 파드 수 # jsonpath '{.spec.selector}' prints JSON such as {"app":"web"}, which -l does not accept. # Convert it to key=value,key=value with jq. SEL_KV=$(kubectl -n "$NS" get svc "$SVC" -o json | jq -r '.spec.selector // {} | to_entries | map("\(.key)=\(.value)") | join(",")') [ -n "$SEL_KV" ] && kubectl -n "$NS" get pods -l "$SEL_KV" -o wide ``` Healthy output looks like this. ```text NAME ENDPOINTS AGE my-svc 10.244.1.7:8080,10.244.2.9:8080 12d ``` If the `ENDPOINTS` column shows ``, you have confirmation. The same conclusion holds if command 2 shows no EndpointSlice at all, or an empty `ENDPOINTS` column. ### Split the layers: curl the Pod IP directly The critical judgment is “is the app dead, is the Service binding broken, or is node-to-node traffic blocked?” One temporary netshoot pod splits that in 30 seconds. ```bash kubectl -n default run tmp-netshoot --rm -it --restart=Never \ --image=nicolaka/netshoot -- /bin/bash ``` Inside the pod shell, hit the three layers in order. ```bash # ① Pod IP 직접 (Service를 건너뜀) curl -sS -m 3 -o /dev/null -w "podip:%{http_code}\n" http://10.244.1.7:8080/ # ② ClusterIP curl -sS -m 3 -o /dev/null -w "clusterip:%{http_code}\n" http://10.96.30.11:8080/ # ③ NodePort (해당 서비스가 NodePort 타입일 때) curl -sS -m 3 -o /dev/null -w "nodeport:%{http_code}\n" http://192.168.10.21:30080/ ``` The combination of results cuts the scope immediately. | ① Pod IP | ② ClusterIP | ③ NodePort | Verdict | Next action | |---|---|---|---|---| | Success | Fail | Fail | Service ↔ Pod binding problem | Walk decision table (a)–(g) in section 3 | | Fail | Fail | Fail | Application / container port problem | Check container logs and `ss -lntp` for the listen port | | Success | Success | Fail | External node ingress / externalTrafficPolicy | Part 4, `externalTrafficPolicy: Local` section | | Success | Intermittent fail | Intermittent fail | Some backends unhealthy, or node-to-node communication | Check kube-proxy rules, then Part 4 (CNI) | Run `ss -lntp` inside the target container like this. ```bash kubectl -n default exec -it deploy/web -- sh -c "ss -lntp || netstat -lntp" ``` Healthy output includes a line such as `LISTEN 0 128 0.0.0.0:8080`. If you only see `127.0.0.1:8080`, the app is bound to loopback only—and traffic fails even when Endpoints are populated. ### One-screen combined diagnostic script Keep a file around for repeat incidents. ```bash #!/usr/bin/env bash # svc-diag.sh — Service/Endpoint 결합 상태 일괄 점검 # usage: ./svc-diag.sh [namespace] set -euo pipefail SVC="${1:?service name required}" NS="${2:-default}" line() { printf '\n=== %s ===\n' "$1"; } line "Service spec" kubectl -n "$NS" get svc "$SVC" -o yaml | grep -E 'clusterIP:|type:|externalTrafficPolicy:|publishNotReadyAddresses:' || true line "Selector" SELECTOR=$(kubectl -n "$NS" get svc "$SVC" -o jsonpath='{.spec.selector}') echo "raw: ${SELECTOR:-}" line "Ports (port -> targetPort)" kubectl -n "$NS" get svc "$SVC" \ -o jsonpath='{range .spec.ports[*]}{.name}{" "}{.port}{" -> "}{.targetPort}{"\n"}{end}' line "Endpoints (legacy)" kubectl -n "$NS" get endpoints "$SVC" -o wide || echo "no endpoints object" line "EndpointSlices" kubectl -n "$NS" get endpointslices -l "kubernetes.io/service-name=$SVC" -o wide || true line "EndpointSlice ready conditions" kubectl -n "$NS" get endpointslices -l "kubernetes.io/service-name=$SVC" \ -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{range .endpoints[*]}{.addresses[0]}{"=ready:"}{.conditions.ready}{" "}{end}{"\n"}{end}' || true line "Matching pods" SEL_KV=$(kubectl -n "$NS" get svc "$SVC" -o json \ | jq -r '.spec.selector // {} | to_entries | map("\(.key)=\(.value)") | join(",")') if [ -n "$SEL_KV" ]; then kubectl -n "$NS" get pods -l "$SEL_KV" -o wide || true else echo "selector is empty (selector-less Service: manual EndpointSlice required)" fi line "All pod labels in namespace" kubectl -n "$NS" get pods --show-labels line "Recent events" kubectl -n "$NS" get events --sort-by=.lastTimestamp | tail -20 ``` ```bash chmod +x svc-diag.sh ./svc-diag.sh my-svc default ``` If the `Matching pods` section prints `No resources found`, the cause is almost certainly (a) selector mismatch. If pods appear but EndpointSlice is empty, it is (c) Readiness-related. ## Cause Decision Table: 7 Cases and a One-Line Fix Start with the full map. | # | Symptom | Cause | Confirm command | One-line recovery | |---|---|---|---|---| | a | Endpoints ``, 0 pods via selector | Selector label typo / mismatch | `kubectl get pods -l app=web` | `kubectl patch svc my-svc -p '{"spec":{"selector":{"app":"web-api"}}}'` | | b | Endpoints exist but connect fails, or `` | targetPort ↔ containerPort mismatch, named port undefined | `kubectl get svc my-svc -o jsonpath='{.spec.ports[*].targetPort}'` | `kubectl patch svc my-svc --type=json -p '[{"op":"replace","path":"/spec/ports/0/targetPort","value":8080}]'` | | c | Pod is Running but EndpointSlice ready=false | Readiness probe failure → NotReady | READY column from `kubectl get pods -o wide` + `kubectl describe pod` | `kubectl patch deploy web --type=json -p '[{"op":"replace","path":"/spec/template/spec/containers/0/readinessProbe/httpGet/path","value":"/healthz"}]'` | | d | Fails only from another namespace | Services do not cross namespaces | `kubectl get pods -A -l app=web` | Use FQDN: `curl http://my-svc.other-ns.svc.cluster.local:8080` | | e | `Endpoints: ` but DNS returns many pod IPs | Headless Service (`clusterIP: None`) misconfigured | `kubectl get svc my-svc -o jsonpath='{.spec.clusterIP}'` | Delete the Service and re-apply a manifest without `clusterIP: None` | | f | Selector empty and no EndpointSlice | Selector-less Service missing a manual EndpointSlice | `kubectl get svc my-svc -o jsonpath='{.spec.selector}'` is blank | Apply a manual EndpointSlice YAML (example below) | | g | Only some pods registered; fails only on certain nodes | Port collision on hostNetwork pods | Duplicate node in `kubectl get pods -o wide` | `kubectl patch deploy web -p '{"spec":{"template":{"spec":{"affinity":{"podAntiAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":[{"labelSelector":{"matchLabels":{"app":"web"}},"topologyKey":"kubernetes.io/hostname"}]}}}}}}'` | ### (a) Selector label typo / mismatch This is the most common case. The Deployment template label is `app: web-api`, but the Service selector says `app: web`. ```yaml # 잘못된 버전 apiVersion: v1 kind: Service metadata: name: my-svc namespace: default spec: selector: app: web # 파드 라벨은 web-api ports: - port: 8080 targetPort: 8080 ``` ```yaml # 수정 버전 apiVersion: v1 kind: Service metadata: name: my-svc namespace: default spec: selector: app: web-api ports: - name: http port: 8080 targetPort: 8080 protocol: TCP ``` Confirmation is one line. ```bash kubectl -n default get pods -l app=web ``` `No resources found in default namespace.` means the selector matches nothing. Remember that a Service selector is an **AND**. If you set both `app: web` and `tier: backend`, only pods with both labels match. Missing `tier` on the pod yields zero results. ### (b) targetPort ↔ containerPort mismatch, named port undefined Named ports improve readability, but if the container does not define the name, the EndpointSlice port is empty or filled with the wrong value. ```yaml # 잘못된 버전: Service는 http라는 이름을 찾는데 컨테이너에 이름이 없음 apiVersion: v1 kind: Service metadata: name: my-svc spec: selector: app: web-api ports: - port: 8080 targetPort: http --- apiVersion: apps/v1 kind: Deployment metadata: name: web spec: replicas: 2 selector: matchLabels: app: web-api template: metadata: labels: app: web-api spec: containers: - name: app image: nginx:1.27 ports: - containerPort: 8080 # name 지정 누락 ``` ```yaml # 수정 버전: 컨테이너 포트에 name: http 정의 apiVersion: v1 kind: Service metadata: name: my-svc spec: selector: app: web-api ports: - name: http port: 8080 targetPort: http protocol: TCP --- apiVersion: apps/v1 kind: Deployment metadata: name: web spec: replicas: 2 selector: matchLabels: app: web-api template: metadata: labels: app: web-api spec: containers: - name: app image: nginx:1.27 ports: - name: http containerPort: 8080 protocol: TCP ``` Check command and expected result: ```bash kubectl -n default get endpointslices -l kubernetes.io/service-name=my-svc \ -o jsonpath='{range .items[*]}{range .ports[*]}{.name}{":"}{.port}{"\n"}{end}{end}' ``` Healthy output looks like `http:8080`—an actual number. If you have a name but an empty port, (b) is confirmed. ### (c) Readiness probe failure → NotReady If the pod is `Running` but `READY 0/1`, the endpointslice controller marks `conditions.ready: false`, and kube-proxy excludes that address from the service backends. ```bash kubectl -n default get pods -l app=web-api kubectl -n default describe pod | grep -A5 "Readiness" ``` ```text NAME READY STATUS RESTARTS AGE web-6f9c8d5b4c-2xk7p 0/1 Running 0 3m ``` You will typically also see an event such as `Warning Unhealthy ... Readiness probe failed: HTTP probe failed with statuscode: 404`. Classic case: probe path or port does not match the app. If you need to drill into probe failures themselves, use the branch table in [K8s Liveness/Readiness probe failed·connection refused 원인별 해결](/blog/k8s-livenessreadiness-probe-failedconnection-refused-원인별-해결). `publishNotReadyAddresses: true` is the option that also puts NotReady addresses into EndpointSlice. ```yaml apiVersion: v1 kind: Service metadata: name: db-headless spec: clusterIP: None publishNotReadyAddresses: true selector: app: postgres ports: - name: pg port: 5432 targetPort: 5432 ``` The legitimate use is clear: StatefulSet-based clustered software that must discover peers during boot. Flip this on a normal web Service and user requests flow to unready pods, and 5xx rates climb. **Turning it on to hide an outage is irreversible debt**—even as a temporary bypass, file a ticket and set a revert deadline. ### (d) Pod lives in another namespace A Service selector finds pods **only in the same namespace**. There is no cross-namespace selector. ```bash kubectl get pods -A -l app=web-api -o wide ``` If the pod is in `prod` and the Service is in `default`, move the Service or have the client use the FQDN. ```bash curl -sS http://my-svc.prod.svc.cluster.local:8080/healthz ``` To alias an external name, use ExternalName. ```yaml apiVersion: v1 kind: Service metadata: name: my-svc namespace: default spec: type: ExternalName externalName: my-svc.prod.svc.cluster.local ``` ExternalName returns only a CNAME, so empty `Endpoints` is expected. `` in this case is not an incident. ### (e) Headless Service misconfiguration You copied a StatefulSet manifest and `clusterIP: None` came along for the ride. ```yaml # 잘못된 버전: 일반 API 서비스인데 headless apiVersion: v1 kind: Service metadata: name: my-svc spec: clusterIP: None selector: app: web-api ports: - port: 8080 targetPort: 8080 ``` ```yaml # 수정 버전: ClusterIP 할당 apiVersion: v1 kind: Service metadata: name: my-svc spec: type: ClusterIP selector: app: web-api ports: - name: http port: 8080 targetPort: 8080 ``` `spec.clusterIP` is immutable; you cannot patch it. Delete and recreate. ```bash kubectl -n default delete svc my-svc kubectl -n default apply -f my-svc-fixed.yaml ``` DNS response shape also distinguishes the two. ```bash kubectl run dnsq --rm -it --restart=Never --image=nicolaka/netshoot -- \ dig +short my-svc.default.svc.cluster.local ``` | Setup | dig result | Client behavior | |---|---|---| | Regular ClusterIP | Single line `10.96.30.11` | kube-proxy load-balances | | Headless | Multiple pod IPs such as `10.244.1.7`, `10.244.2.9` | Client picks directly; connection-pool skew risk | ### (f) Selector-less Service + manual EndpointSlice This is the pattern for exposing an external DB or an out-of-cluster legacy API under an in-cluster name. With no selector, the controller fills nothing, so you must create the EndpointSlice yourself. ```yaml apiVersion: v1 kind: Service metadata: name: legacy-db namespace: default spec: ports: - name: pg port: 5432 targetPort: 5432 protocol: TCP --- apiVersion: discovery.k8s.io/v1 kind: EndpointSlice metadata: name: legacy-db-1 namespace: default labels: kubernetes.io/service-name: legacy-db addressType: IPv4 ports: - name: pg port: 5432 protocol: TCP endpoints: - addresses: - "192.168.50.31" conditions: ready: true ``` Three things people routinely omit: 1. `labels."kubernetes.io/service-name"` — without this label the Service never binds and stays empty forever. 2. `addressType` — you must set one of `IPv4`, `IPv6`, or `FQDN`. 3. `ports[].name` — must match the Service port name exactly. A name on only one side breaks the match. ```bash kubectl -n default apply -f legacy-db.yaml kubectl -n default get endpointslices -l kubernetes.io/service-name=legacy-db -o wide ``` Healthy: the `ENDPOINTS` column shows `192.168.50.31`. ### (g) hostNetwork pods and port collisions A `hostNetwork: true` pod uses the node’s network namespace as-is. If two pods using the same port land on the same node, the second fails to bind, stays NotReady, and you get a **partial failure where only some backends register**. ```bash kubectl -n default get pods -l app=web-api -o wide ``` If the `NODE` column repeats the same node name and one of those pods is `0/1`, this is the case. ```yaml # 수정 버전: 노드당 1개만 뜨도록 anti-affinity 부여 apiVersion: apps/v1 kind: Deployment metadata: name: web spec: replicas: 3 selector: matchLabels: app: web-api template: metadata: labels: app: web-api spec: hostNetwork: true dnsPolicy: ClusterFirstWithHostNet affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchLabels: app: web-api topologyKey: kubernetes.io/hostname containers: - name: app image: nginx:1.27 ports: - name: http containerPort: 8080 ``` If you omit `dnsPolicy: ClusterFirstWithHostNet` on a `hostNetwork` pod, it will not use cluster DNS and the name-resolution issues from Part 1 come back. Remember them as a pair. ## Endpoints Are Populated and It Still Fails + Version Traps ### Version differences change the diagnostic commands - **1.21**: EndpointSlice became the default data source. Since then kube-proxy watches EndpointSlice, and the legacy `Endpoints` object is mirrored by a controller for compatibility. - **Recommended as of 1.33**: Diagnose with `kubectl get endpointslices`. The Endpoints API is being cleaned up in a direction that does not reflect newer features (for example traffic-distribution fields, multiple addressTypes). - **Split past 100**: An EndpointSlice holds at most 100 endpoints by default. 250 backends become 3 slices. The mirrored legacy Endpoints object may truncate or fail to reflect the full set, which produces the false reading “there are only 100 endpoints.” On large Services, get in the habit of summing every slice: ```bash kubectl -n default get endpointslices -l kubernetes.io/service-name=my-svc \ -o jsonpath='{range .items[*]}{range .endpoints[*]}{.addresses[0]}{"\n"}{end}{end}' \ | sort -u | wc -l ``` Healthy: the number equals the actual Ready pod count. If that differs from `kubectl get endpoints` output length, trust the EndpointSlice number. ### Failure branch tree: start with kube-proxy mode If Endpoints look healthy but ClusterIP still fails, you are in the data plane. Check the mode first. ```bash kubectl -n kube-system get cm kube-proxy -o yaml | grep -i "mode" ``` Mode-specific rule dumps. Run them on the node or from a privileged debug pod. ```bash # iptables 모드 sudo iptables-save | grep my-svc # IPVS 모드 sudo ipvsadm -Ln | grep -A3 10.96.30.11 # nftables 모드 (1.31+ 에서 사용 가능) sudo nft list ruleset | grep my-svc ``` Expected healthy output: | Mode | Healthy signature | When unhealthy | |---|---|---| | iptables | `KUBE-SVC-XXXX` chain and one `KUBE-SEP-XXXX` jump per backend | Chain exists but 0 SEPs → Endpoints not reflected | | IPVS | Real server list under `TCP 10.96.30.11:8080` | 0 real-server lines → same | | nftables | Service chain and verdict map in the `kube-proxy` table | Missing entries → check kube-proxy pod logs | If all three look healthy and it still fails, it is not kube-proxy itself but the node-to-node path—Part 4 territory. One flow to flag: with eBPF CNIs such as Cilium, **kube-proxy replacement** makes every command above meaningless. Having no service chains in `iptables-save` is the healthy state; diagnosis moves to checking the service-to-backend mapping inside the Cilium agent pod with `cilium-dbg service list` and `cilium-dbg endpoint list` (older CLIs: `cilium service list`) ([Cilium kube-proxy replacement docs](https://docs.cilium.io/en/stable/network/kubernetes/kubeproxy-free/)). Check the CNI config before concluding “no iptables rules = outage.” ### The `externalTrafficPolicy: Local` single-node failure pattern On NodePort/LoadBalancer, setting `Local` to preserve client source IP means **a request that lands on a node with no backend pod is not forwarded—it is dropped.** Classic cause of “fails only when I hit 1 of 3 nodes.” ```bash kubectl -n default get svc my-svc -o jsonpath='{.spec.externalTrafficPolicy}{"\n"}' kubectl -n default get pods -l app=web-api -o wide kubectl get nodes -o name ``` Compare the nodes that have pods against the full node list, and check whether traffic is hitting nodes with no pods. Two fix directions: ```bash # 1) 소스 IP 보존이 필수가 아니면 Cluster로 전환 kubectl -n default patch svc my-svc -p '{"spec":{"externalTrafficPolicy":"Cluster"}}' ``` ```yaml # 2) Local을 유지해야 하면 모든 노드에 파드를 배치 (DaemonSet 또는 anti-affinity + 충분한 replicas) apiVersion: apps/v1 kind: DaemonSet metadata: name: web spec: selector: matchLabels: app: web-api template: metadata: labels: app: web-api spec: containers: - name: app image: nginx:1.27 ports: - name: http containerPort: 8080 ``` If packets still vanish only for certain node combinations after this, you are looking at overlay tunnel / MTU / routing—the subject of Part 4. ### In a service mesh (Istio), the symptom string is different Where an Envoy sidecar receives traffic, as in Istio, zero backends often show up not as `connection refused` but as the sidecar answering HTTP **503 `no healthy upstream`**. The Envoy access log records the response flag `UH` (no healthy hosts in the upstream cluster) ([Envoy access log response flags](https://www.envoyproxy.io/docs/envoy/latest/configuration/observability/access_log/usage)). ```bash # endpoints and health (HEALTHY/UNHEALTHY) as seen by the client pod's sidecar istioctl proxy-config endpoint . --cluster "outbound|80||my-svc.default.svc.cluster.local" ``` If the list is empty, check the Kubernetes EndpointSlice with this post's decision table first; if entries exist but are all UNHEALTHY, look at outlier detection or health-check settings. ## Preventing Recurrence: CI Gates, Probe Design, Alerts ### yq-based manifest consistency checks Mechanically comparing labels and ports before deploy keeps cases (a) and (b) out of production. ```bash #!/usr/bin/env bash # validate-svc-match.sh — Service selector ↔ Deployment 라벨/포트 정합성 검증 # usage: ./validate-svc-match.sh deploy.yaml svc.yaml set -euo pipefail DEPLOY_FILE="${1:?deployment yaml required}" SVC_FILE="${2:?service yaml required}" FAIL=0 POD_LABELS=$(yq -o=json '.spec.template.metadata.labels' "$DEPLOY_FILE") SELECTOR=$(yq -o=json '.spec.selector' "$SVC_FILE") echo "pod labels : $POD_LABELS" echo "selector : $SELECTOR" # selector의 모든 key/value가 pod labels에 포함되는지 검사 MISSING=$(echo "$SELECTOR" | jq -r --argjson labels "$POD_LABELS" \ 'to_entries[] | select(($labels[.key] // "") != .value) | .key') if [ -n "$MISSING" ]; then echo "FAIL: selector keys not matched in pod labels -> $MISSING" FAIL=1 else echo "OK: selector matches pod labels" fi # targetPort가 숫자인 경우 containerPort 존재 확인 TARGET=$(yq '.spec.ports[0].targetPort' "$SVC_FILE") if [[ "$TARGET" =~ ^[0-9]+$ ]]; then HIT=$(yq ".spec.template.spec.containers[].ports[] | select(.containerPort == $TARGET) | .containerPort" "$DEPLOY_FILE" || true) if [ -z "$HIT" ]; then echo "FAIL: targetPort $TARGET has no matching containerPort" FAIL=1 else echo "OK: targetPort $TARGET matches containerPort" fi else # named port인 경우 이름 정의 확인 HIT=$(yq ".spec.template.spec.containers[].ports[] | select(.name == \"$TARGET\") | .name" "$DEPLOY_FILE" || true) if [ -z "$HIT" ]; then echo "FAIL: named targetPort '$TARGET' is not defined in containers[].ports[].name" FAIL=1 else echo "OK: named port '$TARGET' defined" fi fi exit "$FAIL" ``` Pair it with kubeconform for schema validation. ```bash kubeconform -strict -summary -kubernetes-version 1.33.0 deploy.yaml svc.yaml ``` GitHub Actions example: ```yaml name: k8s-manifest-gate on: [pull_request] jobs: validate: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Install tools run: | sudo wget -qO /usr/local/bin/yq https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 sudo chmod +x /usr/local/bin/yq curl -sSL https://github.com/yannh/kubeconform/releases/latest/download/kubeconform-linux-amd64.tar.gz | tar xz sudo mv kubeconform /usr/local/bin/ - name: Schema validation run: kubeconform -strict -summary -kubernetes-version 1.33.0 manifests/ - name: Selector/port match run: ./validate-svc-match.sh manifests/deploy.yaml manifests/svc.yaml ``` ### Three principles for Readiness probe design 1. **Do not put dependencies in the probe.** A readiness check that also tests the DB connection will yank every pod out of backends during a brief DB blip and manufacture `Endpoints: ` yourself. Readiness should answer only “is this process ready to take requests?” Expose dependency health as a separate metric. 2. **Prefer `startupProbe` over a long `initialDelaySeconds`.** A large initialDelay on a slow-booting JVM app also delays failure detection. Isolate boot with startupProbe and keep the readiness period short. 3. **Prevent a full NotReady during rollout.** Set `maxUnavailable` together with a PodDisruptionBudget. ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: web spec: replicas: 4 strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 maxSurge: 1 selector: matchLabels: app: web-api template: metadata: labels: app: web-api spec: containers: - name: app image: nginx:1.27 ports: - name: http containerPort: 8080 startupProbe: httpGet: path: /healthz port: http failureThreshold: 30 periodSeconds: 5 readinessProbe: httpGet: path: /healthz port: http periodSeconds: 5 timeoutSeconds: 2 failureThreshold: 3 --- apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: web-pdb spec: minAvailable: 2 selector: matchLabels: app: web-api ``` ### Alerting: catch zero endpoints within 5 minutes A PrometheusRule based on kube-state-metrics. ```yaml apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: service-endpoint-rules namespace: monitoring labels: release: kube-prometheus-stack spec: groups: - name: service-endpoints rules: - alert: ServiceHasNoEndpoints expr: | kube_endpoint_address_available == 0 and on (namespace, service) label_replace( kube_service_spec_type{type!="ExternalName"}, "service", "$1", "service", "(.*)" ) for: 5m labels: severity: critical annotations: summary: "Service {{ $labels.namespace }}/{{ $labels.service }} has zero endpoints" description: "5분 이상 사용 가능한 엔드포인트가 0개입니다. selector 라벨, targetPort, readiness probe 순서로 확인하세요." - alert: ServiceEndpointsDropped expr: | delta(kube_endpoint_address_available[10m]) < 0 and kube_endpoint_address_available < 2 for: 10m labels: severity: warning annotations: summary: "Endpoints decreasing for {{ $labels.namespace }}/{{ $labels.service }}" description: "엔드포인트 수가 감소해 2개 미만입니다. 롤아웃 또는 readiness 실패 여부를 확인하세요." ``` ExternalName Services normally have empty Endpoints, so the exclusion is mandatory or you will page on noise. Metric names may also appear as the `kube_endpointslice_*` family depending on kube-state-metrics version—check the metrics list on the version you actually run before applying. ## This Installment’s Checklist and a Preview of the Next In an incident, just run from the top. 1. Confirm emptiness with `kubectl get endpoints ` and `kubectl get endpointslices -l kubernetes.io/service-name=`. 2. From netshoot, curl Pod IP → ClusterIP → NodePort to split the layers. 3. If `kubectl get pods -l ` is 0, suspect (a) selector mismatch first. 4. If pods match but Endpoints are empty, check the READY column and readiness events for (c). 5. If `targetPort` is a named port, confirm the same name is defined on the container. 6. If the selector is empty, inspect the manual EndpointSlice `kubernetes.io/service-name` label and `addressType`. 7. If Endpoints are healthy and it still fails, inspect kube-proxy rules by mode and `externalTrafficPolicy: Local`. Recovery is sometimes not instant: kube-proxy can take a few seconds to notice EndpointSlice changes and sync rules, and leftover conntrack sessions keep the old path. Do not pile on extra changes—recheck after about 30 seconds. Next is **Part 4, “Endpoints Are Healthy but Crossing Nodes Breaks — Debugging CNI Overlay, MTU, and the kube-proxy Data Plane.”** Same-node works, cross-node dies; large responses vanish from MTU mismatch; tracing the VXLAN encapsulation path. Official docs to read alongside this: - [Service](https://kubernetes.io/docs/concepts/services-networking/service/) - [EndpointSlices](https://kubernetes.io/docs/concepts/services-networking/endpoint-slices/) - [Debug Services](https://kubernetes.io/docs/tasks/debug/debug-application/debug-service/) - [Virtual IPs and Service Proxies](https://kubernetes.io/docs/reference/networking/virtual-ips/) ## FAQ **Q1. Should I look at Endpoints or EndpointSlice?** A. Since 1.21 the real data source is EndpointSlice. `kubectl get endpoints` is fine for a quick check, but once backends exceed 100 and slices split, the list can look truncated. For an accurate call, treat `kubectl get endpointslices -l kubernetes.io/service-name=` as the source of truth. **Q2. I want to send traffic to NotReady pods too.** A. Set `spec.publishNotReadyAddresses: true`. The legitimate use is peer discovery during boot, as with StatefulSets. On a user-facing Service it sends requests to unready pods and produces 5xx. If it is a temporary bypass, set a revert deadline. **Q3. On a Headless Service, is `Endpoints: ` always an incident?** A. No. A Headless Service with `clusterIP: None` can look different in kubectl output, and ExternalName has no endpoint concept at all—empty is normal. Judge by whether `dig` returns multiple pod IPs and whether addresses actually exist on the EndpointSlice. **Q4. I fixed the manifest. Why didn’t it recover immediately?** A. The endpointslice controller has to apply the change, and kube-proxy has to sync rules on every node. On top of that, existing conntrack sessions keep the old destination, so recovery feels delayed. Force clients to open a new connection (or recycle the connection pool) to tell the difference. **Q5. What do `connection refused` vs `timeout` imply?** A. Immediate `connection refused` usually means zero backends and a REJECT rule answering—start with the seven causes in this post. A `timeout` means the packet vanished with no reply, so suspect a NetworkPolicy block (Part 2), a CNI path/MTU issue (Part 4), or an app that is not responding. **Q6. Endpoints have IPs, but I still get `connection refused`.** A. The backends are registered, but nothing is listening on the forwarded port. Check that `targetPort` matches the port the container actually listens on, and that the app isn't bound only to `127.0.0.1` (loopback). Requesting the Pod IP directly from inside the cluster with `kubectl run tmp --rm -it --image=busybox --restart=Never -- wget -qO- http://:` tells you right away whether it's a Service problem or an app problem.

#kube-proxy

Endpoints <none> · no endpoints available: 7 Causes and a 30-Second Diagnosis