Ingress 404 default backend · 503 no healthy upstream: a 30-second diagnostic chart

When endpoints are healthy but the browser still returns 404

This is part 4 of the K8s networking deep-dive series. We start from the assumption that, in part 3, you already confirmed with kubectl get endpointslice that backend Pods are correctly registered on the Service.

A typical situation looks like this.

kubectl port-forward svc/myapp 8080:80
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:8080/
# 200

curl -s -o /dev/null -w '%{http_code}\n' https://example.com/
# 404

The moment port-forward returns 200, the Service → Pod hop is already cleared. The problem sits in front of that — the L7 ingress path.

[Client]
    │  ① DNS / firewall / security groups
    ▼
[Cloud LB (ELB/GLB/Azure LB)]
    │  ② NodePort or LB target registration / health check
    ▼
[Ingress Controller Pod (ingress-nginx, etc.)]
    │  ③ Ingress resource matching: host / path / ingressClassName
    ▼
[Service (ClusterIP)]
    │  ④ selector → EndpointSlice  ← part 3 scope
    ▼
[Pod]

Explicit branch note: If kubectl get endpointslice -l kubernetes.io/service-name=myapp shows empty ADDRESSES or <none>, this is not the article you want — go to part 3 (Endpoints <none> · no endpoints available). This article covers only the ①–③ path, where endpoints are populated but external access still fails.

If the Ingress Controller's basic setup and routing concepts are still unfamiliar, skim A complete Kubernetes Ingress guide — routing external traffic with Nginx Ingress Controller first; the diagnostic steps in this article will read much faster.

Response-body decision table — jump straight to the causal layer from a single string

You can narrow the causal layer from the string printed by the browser or curl -v alone. Find your situation in the table first.

The key to the diagnosis is whether response headers exist at all. If even one header came back, the packet reached the Ingress Controller; if there are none, it was cut off in front of it.

30-second confirmation: 6 steps

Copy-paste and run them in order. Compare the healthy vs. unhealthy output of each step side by side.

① Has the Ingress been picked up by the controller?

kubectl get ingress -o wide

Healthy output:

NAME    CLASS   HOSTS         ADDRESS         PORTS     AGE
myapp   nginx   example.com   203.0.113.10    80, 443   12m

Unhealthy output:

NAME    CLASS    HOSTS         ADDRESS   PORTS   AGE
myapp   <none>              *           80      12m

If CLASS is <none> or ADDRESS stays blank, the controller has not picked up this Ingress at all. Skip to ③.

② Check Events and default backend with describe

kubectl describe ingress myapp

Healthy:

Rules:
  Host          Path  Backends
  ----          ----  --------
  example.com
                /     myapp:80 (10.244.1.23:8080,10.244.2.11:8080)
Events:
  Type    Reason  Age   From                      Message
  Normal  Sync    2m    nginx-ingress-controller  Scheduled for sync

Unhealthy (two representative patterns):

Default backend:  <default> (<error: endpoints "default-http-backend" not found>)

                /     myapp:8080 (<none>)

The first means no matching rule is alive; the second is the 503 fast-path where the Service was found but upstream is empty. If Events has no Sync at all, the controller is not watching this resource.

③ Check ingressClassName (mandatory on v1.22+)

kubectl get ingress myapp -o jsonpath='{.spec.ingressClassName}{"\n"}'
kubectl get ingress myapp -o jsonpath='{.metadata.annotations}{"\n"}'
kubectl get ingressclass

Healthy:

nginx
{"kubernetes.io/ingress.class":"nginx"}
NAME    CONTROLLER                      PARAMETERS   AGE
nginx   k8s.io/ingress-nginx            <none>       30d

Unhealthy: the first command prints only a blank line, and kubernetes.io/ingress.class exists only as an annotation. The legacy annotation was targeted for removal in Kubernetes 1.22 and can be silently ignored depending on controller version and startup flags. The NAME from kubectl get ingressclass and the spec.ingressClassName value must be identical character-for-character.

④ Cross-check backend Service name and port

kubectl get ingress myapp -o jsonpath='{range .spec.rules[*].http.paths[*]}{.backend.service.name}{" -> "}{.backend.service.port}{"\n"}{end}'
kubectl get svc myapp -o jsonpath='{range .spec.ports[*]}{.name}{" "}{.port}{" -> "}{.targetPort}{"\n"}{end}'

Healthy:

myapp -> {"number":80}
http 80 -> 8080

Unhealthy:

myapp -> {"name":"https"}
http 80 -> 8080

If the port name referenced by the Ingress is not in the Service's port-name list, no upstream is created and you get 503. When referencing by number, use the Service's port value, not targetPort.

⑤ Tail the controller logs live

kubectl logs -n ingress-nginx deploy/ingress-nginx-controller --tail=50

Healthy (request actually arrives and returns 200):

10.0.1.5 - - [10/Aug/2026:04:11:02 +0000] "GET / HTTP/1.1" 200 1256 "-" "curl/8.4.0" 84 0.004 [default-myapp-80] [] 10.244.1.23:8080 1256 0.004 200

Unhealthy:

2026/08/10 04:12:31 [error] 31#31: *117 no live upstreams while connecting to upstream, client: 10.0.1.5, server: example.com, request: "GET / HTTP/1.1", upstream: "http://upstream-default-backend/"

2026/08/10 04:13:02 [error] 31#31: *120 upstream sent too big header while reading response header from upstream, client: 10.0.1.5, ...

If the bracketed [default-myapp-80] is empty ([]) or shows upstream-default-backend, matching failure is confirmed.

⑥ Rule out DNS

curl -H 'Host: example.com' http://<LB-IP>/ -v

Get <LB-IP> with kubectl get svc -n ingress-nginx ingress-nginx-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}'.

Healthy:

< HTTP/1.1 200 OK
< Server: nginx

Unhealthy A — 200 with a Host header, but the domain fails → DNS/CNAME problem. The cluster is fine. Unhealthy B — curl: (28) timeout here as well → LB / security group / NodePort path. Jump to the failure branch below.

Fix table by cause

1) Add ingressClassName

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myapp
  namespace: default
spec:
  ingressClassName: nginx
  rules:
  - host: example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: myapp
            port:
              number: 80

2) Fix pathType — the hidden 404 regular

Exact matches only when the string is identical. If you set Exact on /api, /api/users will not match and falls through to the default backend.

      paths:
      - path: /api
        pathType: Prefix      # Exact → Prefix
        backend:
          service:
            name: myapp
            port:
              number: 80

ImplementationSpecific can change routing the moment you swap controllers, so avoid it if you need portability.

3) Correct the backend Service port

# Service
apiVersion: v1
kind: Service
metadata:
  name: myapp
spec:
  selector:
    app: myapp
  ports:
  - name: http
    port: 80
    targetPort: 8080
---
# Ingress backend — when referencing by name, it must match Service ports[].name
          service:
            name: myapp
            port:
              name: http

4) proxy-buffer-size (502 too big header)

metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-buffer-size: "16k"
    nginx.ingress.kubernetes.io/proxy-buffers-number: "4"

The default is too small and blows up on response headers that carry large Set-Cookie values or long JWTs. Raise it 8k → 16k → 32k and recheck.

5) rewrite-target + regex capture groups

This is the case where you want an /api/users request rewritten to /users on the backend.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myapp
  annotations:
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  ingressClassName: nginx
  rules:
  - host: example.com
    http:
      paths:
      - path: /api(/|$)(.*)
        pathType: ImplementationSpecific
        backend:
          service:
            name: myapp
            port:
              number: 80

The common mistake: if you use capture group $1, only / or an empty string is forwarded. (/|$) is the first group, so the actual path is $2. For regex paths you must use pathType: ImplementationSpecific, not Prefix, or it will not behave as intended.

6) Attach the TLS Secret

kubectl create secret tls myapp-tls --cert=tls.crt --key=tls.key -n default

spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - example.com
    secretName: myapp-tls
  rules:
  - host: example.com
    ...

The tls.hosts value, rules.host, and the certificate SAN must all three match for the Fake Certificate to go away. The Secret must be in the same namespace as the Ingress.

Three traps that fail silently

• Namespace boundary: An Ingress can only reference Services in the same namespace. If you write a Service name from another ns, there is no error — upstream is simply empty and you get 503. A workaround is to place an ExternalName Service in the same ns.
• Duplicate host declarations: If multiple Ingresses declare the same host, paths are merged, but when the same path overlaps, the resource created earlier usually wins. Check for duplicates first with kubectl get ingress -A | grep example.com.
• IngressClass name mismatch: If the controller startup flag --ingress-class=nginx and the IngressClass resource name differ, nothing matches. Cross-check with kubectl -n ingress-nginx get deploy ingress-nginx-controller -o yaml | grep ingress-class.

Controller differences and version branches

Even for the same symptom, the message and defaults differ by controller.

Exact defaults depend on the chart version you deployed, so confirm actual values in each project's official docs and with kubectl -n ingress-nginx get cm ingress-nginx-controller -o yaml.

Version branch table

When apply is blocked by a webhook, reading the denial message verbatim is faster than hunting for a bypass. Which annotation and which token tripped it is usually written out as-is.

For context, the Ingress API is effectively feature-frozen, and new routing features are moving toward Gateway API. This installment's diagnostic scope is still limited to Ingress. To look at 5xx more broadly including Gateway API, see K8s 5xx error root-cause analysis: a 7-step Ingress/Gateway API debugging guide.

Failure branch when it still doesn't work

If you ran all 6 steps and it still isn't fixed, check these three in order.

1) NetworkPolicy blocking — the application namespace has a default deny and does not allow traffic from the ingress-nginx namespace. Ingress matches, but the upstream connection times out.

kubectl get networkpolicy -A

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-from-ingress-nginx
  namespace: default
spec:
  podSelector:
    matchLabels:
      app: myapp
  policyTypes: ["Ingress"]
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: ingress-nginx

2) externalTrafficPolicy: Local — requests that land on a node with no controller Pod are dropped, showing up as intermittent "sometimes works, sometimes doesn't" failures.

kubectl -n ingress-nginx get svc ingress-nginx-controller -o jsonpath='{.spec.externalTrafficPolicy}{"\n"}'

3) LB health-check path mismatch — the cloud LB health-checks / but the application only returns 200 on /healthz, so every target goes unhealthy. The symptom is a timeout with no headers. You have to check the target-group status in the cloud console directly.

If Pods themselves never come up and upstream is empty, continue with Pod Pending FailedScheduling 0/3 nodes: a 30-second diagnose-and-recover runbook. If routing is fine but latency is the problem, the next document is Kubernetes network latency: a complete guide to diagnosing root cause with eBPF and optimizing performance.

Part 5 covers the path after L7 ingress: TLS termination and cert-manager certificate issuance failures.

FAQ

Q. I get 404 even though I definitely created the Ingress resource. What should I look at first? A. The CLASS and ADDRESS columns from kubectl get ingress -o wide. If CLASS is <none>, spec.ingressClassName is missing or does not match the IngressClass name. If ADDRESS stays blank, the controller has not recognized the resource. The resource existing and the controller having applied it are two different things.

Q. How do I distinguish a no healthy upstream 503 from a 502 Bad Gateway? A. 503 means there is no upstream to send to at all (Service name/port mismatch, empty endpoints). 502 means a connection to upstream was made but response handling failed. 502 leaves a concrete reason in the controller logs — upstream sent too big header, connection reset, and the like — so read the log verbatim first.

Q. /api works but /api/users returns 404. A. pathType: Exact is the most likely cause. Switching to Prefix fixes it. If you also need to strip the prefix before sending to the backend, use the combination use-regex: "true" + path: /api(/|$)(.*) + rewrite-target: /$2. The frequently missed detail is that it is $2, not $1.