## When endpoints are healthy but the browser still returns 404 This is **part 4 of the K8s networking deep-dive series**. We start from the assumption that, in part 3, you already confirmed with `kubectl get endpointslice` that backend Pods are correctly registered on the Service. A typical situation looks like this. ```bash kubectl port-forward svc/myapp 8080:80 curl -s -o /dev/null -w '%{http_code}\n' http://localhost:8080/ # 200 curl -s -o /dev/null -w '%{http_code}\n' https://example.com/ # 404 ``` The moment port-forward returns 200, the **Service → Pod hop is already cleared**. The problem sits in front of that — the L7 ingress path. ``` [Client] │ ① DNS / firewall / security groups ▼ [Cloud LB (ELB/GLB/Azure LB)] │ ② NodePort or LB target registration / health check ▼ [Ingress Controller Pod (ingress-nginx, etc.)] │ ③ Ingress resource matching: host / path / ingressClassName ▼ [Service (ClusterIP)] │ ④ selector → EndpointSlice ← part 3 scope ▼ [Pod] ``` **Explicit branch note:** If `kubectl get endpointslice -l kubernetes.io/service-name=myapp` shows empty `ADDRESSES` or ``, this is not the article you want — go to part 3 (Endpoints `` · no endpoints available). This article covers **only the ①–③ path, where endpoints are populated but external access still fails**. If the Ingress Controller's basic setup and routing concepts are still unfamiliar, skim [A complete Kubernetes Ingress guide — routing external traffic with Nginx Ingress Controller](/engineer/kubernetes-ingress-nginx-controller) first; the diagnostic steps in this article will read much faster. ## Response-body decision table — jump straight to the causal layer from a single string You can narrow the causal layer from the string printed by the browser or `curl -v` alone. Find your situation in the table first. | Response body / log | Causal layer | Next action | |---|---|---| | `HTTP/1.1 404 Not Found` + header `Server: nginx` + body `404 Not Found

404 Not Found

nginx` | ③ Ingress resource match failure — one of host / path / `ingressClassName` does not match, so traffic falls through to the **default backend** | Confirmation steps ①–③ | | `HTTP/1.1 503 Service Temporarily Unavailable` + controller log `no healthy upstream` or `upstream connect error` | ③→④ boundary — Ingress matched, but backend Service name/port mismatch, or upstream is empty | Confirmation steps ④–⑤ | | `HTTP/1.1 502 Bad Gateway` + log `upstream sent too big header while reading response header from upstream` | ③ proxy buffer too small (large Set-Cookie / JWT headers) | Fix-table `proxy-buffer-size` row | | `curl: (28) Operation timed out` / browser `ERR_CONNECTION_TIMED_OUT` — **no response headers at all** | ①–② the path **before** Ingress: LB, security groups, NodePort, etc. | Confirmation step ⑥ + the failure branch at the end | | TLS warning + certificate CN is `Kubernetes Ingress Controller Fake Certificate` | ③ TLS Secret not mounted, or `tls.hosts` does not match the request host | Fix-table TLS row | The key to the diagnosis is **whether response headers exist at all**. If even one header came back, the packet reached the Ingress Controller; if there are none, it was cut off in front of it. ## 30-second confirmation: 6 steps Copy-paste and run them in order. Compare the healthy vs. unhealthy output of each step side by side. ### ① Has the Ingress been picked up by the controller? ```bash kubectl get ingress -o wide ``` Healthy output: ``` NAME CLASS HOSTS ADDRESS PORTS AGE myapp nginx example.com 203.0.113.10 80, 443 12m ``` Unhealthy output: ``` NAME CLASS HOSTS ADDRESS PORTS AGE myapp * 80 12m ``` If `CLASS` is `` or `ADDRESS` stays blank, the controller has **not picked up this Ingress at all**. Skip to ③. ### ② Check Events and default backend with describe ```bash kubectl describe ingress myapp ``` Healthy: ``` Rules: Host Path Backends ---- ---- -------- example.com / myapp:80 (10.244.1.23:8080,10.244.2.11:8080) Events: Type Reason Age From Message Normal Sync 2m nginx-ingress-controller Scheduled for sync ``` Unhealthy (two representative patterns): ``` Default backend: () ``` ``` / myapp:8080 () ``` The first means no matching rule is alive; the second is the 503 fast-path where the Service was found but **upstream is empty**. If `Events` has no `Sync` at all, the controller is not watching this resource. ### ③ Check ingressClassName (mandatory on v1.22+) ```bash kubectl get ingress myapp -o jsonpath='{.spec.ingressClassName}{"\n"}' kubectl get ingress myapp -o jsonpath='{.metadata.annotations}{"\n"}' kubectl get ingressclass ``` Healthy: ``` nginx {"kubernetes.io/ingress.class":"nginx"} NAME CONTROLLER PARAMETERS AGE nginx k8s.io/ingress-nginx 30d ``` Unhealthy: the first command prints only a blank line, and `kubernetes.io/ingress.class` exists only as an annotation. The legacy annotation was targeted for removal in Kubernetes 1.22 and can be silently ignored depending on controller version and startup flags. The `NAME` from `kubectl get ingressclass` and the `spec.ingressClassName` value must be **identical character-for-character**. ### ④ Cross-check backend Service name and port ```bash kubectl get ingress myapp -o jsonpath='{range .spec.rules[*].http.paths[*]}{.backend.service.name}{" -> "}{.backend.service.port}{"\n"}{end}' kubectl get svc myapp -o jsonpath='{range .spec.ports[*]}{.name}{" "}{.port}{" -> "}{.targetPort}{"\n"}{end}' ``` Healthy: ``` myapp -> {"number":80} http 80 -> 8080 ``` Unhealthy: ``` myapp -> {"name":"https"} http 80 -> 8080 ``` If the port **name** referenced by the Ingress is not in the Service's port-name list, no upstream is created and you get 503. When referencing by number, use the Service's `port` value, not `targetPort`. ### ⑤ Tail the controller logs live ```bash kubectl logs -n ingress-nginx deploy/ingress-nginx-controller --tail=50 ``` Healthy (request actually arrives and returns 200): ``` 10.0.1.5 - - [10/Aug/2026:04:11:02 +0000] "GET / HTTP/1.1" 200 1256 "-" "curl/8.4.0" 84 0.004 [default-myapp-80] [] 10.244.1.23:8080 1256 0.004 200 ``` Unhealthy: ``` 2026/08/10 04:12:31 [error] 31#31: *117 no live upstreams while connecting to upstream, client: 10.0.1.5, server: example.com, request: "GET / HTTP/1.1", upstream: "http://upstream-default-backend/" ``` ``` 2026/08/10 04:13:02 [error] 31#31: *120 upstream sent too big header while reading response header from upstream, client: 10.0.1.5, ... ``` If the bracketed `[default-myapp-80]` is empty (`[]`) or shows `upstream-default-backend`, matching failure is confirmed. ### ⑥ Rule out DNS ```bash curl -H 'Host: example.com' http:/// -v ``` Get `` with `kubectl get svc -n ingress-nginx ingress-nginx-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}'`. Healthy: ``` < HTTP/1.1 200 OK < Server: nginx ``` Unhealthy A — 200 with a Host header, but the domain fails → **DNS/CNAME problem**. The cluster is fine. Unhealthy B — `curl: (28)` timeout here as well → LB / security group / NodePort path. Jump to the failure branch below. ## Fix table by cause | Symptom | Check command | Fix | Verify | |---|---|---|---| | 404 default backend, `CLASS ` | `kubectl get ingress -o wide` | Add `ingressClassName` | CLASS and ADDRESS appear in `kubectl get ingress -o wide` | | `/api` works, only `/api/users` is 404 | `kubectl get ing myapp -o yaml \| grep pathType` | Change to `pathType: Prefix` | `curl -o /dev/null -w '%{http_code}' https://example.com/api/users` | | 503, describe shows `()` | Confirmation ④ above | Correct Service port name/number | IP list appears in `kubectl describe ing myapp` | | 502 + `too big header` | Controller logs | `proxy-buffer-size` annotation | Error gone from logs, 200 response | | Backend does not know the `/api` prefix | Check app routing | `rewrite-target` + capture groups | Confirm upstream path in logs | | Fake Certificate warning | `kubectl get secret myapp-tls` | Attach `spec.tls` | Certificate CN from `curl -vI https://example.com` | ### 1) Add ingressClassName ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: myapp namespace: default spec: ingressClassName: nginx rules: - host: example.com http: paths: - path: / pathType: Prefix backend: service: name: myapp port: number: 80 ``` ### 2) Fix pathType — the hidden 404 regular `Exact` matches only when the string is identical. If you set `Exact` on `/api`, `/api/users` will not match and falls through to the default backend. ```yaml paths: - path: /api pathType: Prefix # Exact → Prefix backend: service: name: myapp port: number: 80 ``` | pathType | `/api` request | `/api/users` | `/apiv2` | |---|---|---|---| | `Exact` | match | no match | no match | | `Prefix` | match | match | no match (compared per path element) | | `ImplementationSpecific` | delegated to the controller implementation | differs by implementation | differs by implementation | `ImplementationSpecific` can change routing the moment you swap controllers, so avoid it if you need portability. ### 3) Correct the backend Service port ```yaml # Service apiVersion: v1 kind: Service metadata: name: myapp spec: selector: app: myapp ports: - name: http port: 80 targetPort: 8080 --- # Ingress backend — when referencing by name, it must match Service ports[].name service: name: myapp port: name: http ``` ### 4) proxy-buffer-size (502 too big header) ```yaml metadata: annotations: nginx.ingress.kubernetes.io/proxy-buffer-size: "16k" nginx.ingress.kubernetes.io/proxy-buffers-number: "4" ``` The default is too small and blows up on response headers that carry large `Set-Cookie` values or long JWTs. Raise it 8k → 16k → 32k and recheck. ### 5) rewrite-target + regex capture groups This is the case where you want an `/api/users` request rewritten to `/users` on the backend. ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: myapp annotations: nginx.ingress.kubernetes.io/use-regex: "true" nginx.ingress.kubernetes.io/rewrite-target: /$2 spec: ingressClassName: nginx rules: - host: example.com http: paths: - path: /api(/|$)(.*) pathType: ImplementationSpecific backend: service: name: myapp port: number: 80 ``` The common mistake: if you use capture group `$1`, only `/` or an empty string is forwarded. `(/|$)` is the first group, so the actual path is `$2`. For regex paths you must use `pathType: ImplementationSpecific`, not `Prefix`, or it will not behave as intended. ### 6) Attach the TLS Secret ```bash kubectl create secret tls myapp-tls --cert=tls.crt --key=tls.key -n default ``` ```yaml spec: ingressClassName: nginx tls: - hosts: - example.com secretName: myapp-tls rules: - host: example.com ... ``` The `tls.hosts` value, `rules.host`, and the certificate SAN must **all three match** for the Fake Certificate to go away. The Secret must be in the same namespace as the Ingress. ### Three traps that fail silently - **Namespace boundary**: An Ingress can only reference Services in the same namespace. If you write a Service name from another ns, there is no error — upstream is simply empty and you get 503. A workaround is to place an `ExternalName` Service in the same ns. - **Duplicate host declarations**: If multiple Ingresses declare the same host, paths are merged, but when the same path overlaps, the resource created earlier usually wins. Check for duplicates first with `kubectl get ingress -A | grep example.com`. - **IngressClass name mismatch**: If the controller startup flag `--ingress-class=nginx` and the IngressClass resource name differ, nothing matches. Cross-check with `kubectl -n ingress-nginx get deploy ingress-nginx-controller -o yaml | grep ingress-class`. ## Controller differences and version branches Even for the same symptom, the message and defaults differ by controller. | Item | ingress-nginx | Traefik | AWS ALB Controller | |---|---|---|---| | Match-failure message | nginx default 404 page (`nginx`) | `404 page not found` plain text | ALB default 404 (JSON/empty body, `Server: awselb/2.0`) | | No upstream | `503` + log `no live upstreams` | `503 Service Unavailable` | `502` / target unhealthy | | Annotation prefix | `nginx.ingress.kubernetes.io/` | `traefik.ingress.kubernetes.io/` | `alb.ingress.kubernetes.io/` | | Default proxy timeout | 60s family (read/send) | effectively unlimited; set explicitly | idle timeout 60s | | Default request body size | 1m (change with `proxy-body-size`) | effectively unlimited | separate ALB-layer limits | Exact defaults depend on the chart version you deployed, so confirm actual values in each project's official docs and with `kubectl -n ingress-nginx get cm ingress-nginx-controller -o yaml`. ### Version branch table | Change | Version | Symptom | Response | |---|---|---|---| | `networking.k8s.io/v1beta1` removed | Kubernetes 1.22 | `error: unable to recognize ... no matches for kind "Ingress" in version "networking.k8s.io/v1beta1"` | Convert to `v1`. Also requires the structural change `serviceName/servicePort` → `service.name/service.port` | | `kubernetes.io/ingress.class` annotation deprecated | deprecated in 1.18, effectively cleaned up after 1.22 | resource is created but CLASS is blank / unmatched | Use `spec.ingressClassName` | | snippet annotation restrictions | ingress-nginx 1.x | admission webhook rejects `configuration-snippet` | Review the ConfigMap `allow-snippet-annotations` policy; prefer replacing with standard annotations where possible | | annotation value blocklist / regex validation tightened | ingress-nginx 1.x | `admission webhook "validate.nginx.ingress.kubernetes.io" denied the request` | Read the denial reason as-is and clean up the offending annotation value | When apply is blocked by a webhook, reading the denial message verbatim is faster than hunting for a bypass. Which annotation and which token tripped it is usually written out as-is. For context, the Ingress API is effectively feature-frozen, and new routing features are moving toward Gateway API. This installment's diagnostic scope is still limited to Ingress. To look at 5xx more broadly including Gateway API, see [K8s 5xx error root-cause analysis: a 7-step Ingress/Gateway API debugging guide](/blog/k8s-5xx-에러-원인-분석-ingressgateway-api-7단계-디버깅-가이드). ## Failure branch when it still doesn't work If you ran all 6 steps and it still isn't fixed, check these three in order. **1) NetworkPolicy blocking** — the application namespace has a default deny and does not allow traffic from the ingress-nginx namespace. Ingress matches, but the upstream connection times out. ```bash kubectl get networkpolicy -A ``` ```yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-from-ingress-nginx namespace: default spec: podSelector: matchLabels: app: myapp policyTypes: ["Ingress"] ingress: - from: - namespaceSelector: matchLabels: kubernetes.io/metadata.name: ingress-nginx ``` **2) externalTrafficPolicy: Local** — requests that land on a node with no controller Pod are dropped, showing up as intermittent "sometimes works, sometimes doesn't" failures. ```bash kubectl -n ingress-nginx get svc ingress-nginx-controller -o jsonpath='{.spec.externalTrafficPolicy}{"\n"}' ``` **3) LB health-check path mismatch** — the cloud LB health-checks `/` but the application only returns 200 on `/healthz`, so every target goes unhealthy. The symptom is a timeout with no headers. You have to check the target-group status in the cloud console directly. If Pods themselves never come up and upstream is empty, continue with [Pod Pending FailedScheduling 0/3 nodes: a 30-second diagnose-and-recover runbook](/blog/pod-pending-failedscheduling-03-nodes-30초-진단복구-런북). If routing is fine but latency is the problem, the next document is [Kubernetes network latency: a complete guide to diagnosing root cause with eBPF and optimizing performance](/blog/kubernetes-네트워크-지연-ebpf로-근본-원인-진단하고-성능-최적화하는-완벽-가이드). Part 5 covers the path after L7 ingress: **TLS termination and cert-manager certificate issuance failures**. ## FAQ **Q. I get 404 even though I definitely created the Ingress resource. What should I look at first?** A. The `CLASS` and `ADDRESS` columns from `kubectl get ingress -o wide`. If `CLASS` is ``, `spec.ingressClassName` is missing or does not match the IngressClass name. If `ADDRESS` stays blank, the controller has not recognized the resource. The resource existing and the controller having applied it are two different things. **Q. How do I distinguish a `no healthy upstream` 503 from a `502 Bad Gateway`?** A. 503 means there is no upstream to send to at all (Service name/port mismatch, empty endpoints). 502 means a connection to upstream was made but response handling failed. 502 leaves a concrete reason in the controller logs — `upstream sent too big header`, `connection reset`, and the like — so read the log verbatim first. **Q. `/api` works but `/api/users` returns 404.** A. `pathType: Exact` is the most likely cause. Switching to `Prefix` fixes it. If you also need to strip the prefix before sending to the backend, use the combination `use-regex: "true"` + `path: /api(/|$)(.*)` + `rewrite-target: /$2`. The frequently missed detail is that it is `$2`, not `$1`.

#ingress-nginx

Ingress 404 default backend · 503 no healthy upstream: a 30-second diagnostic chart