K8s Liveness/Readiness Probe Failed · Connection Refused: Fixes by Cause

Liveness/Readiness Probe Failed, Connection Refused · Context Deadline: A 30-Second Decision Tree

K8s_Troubleshooting_Guide, Part 21

The Pod is clearly Running, but the RESTARTS count keeps climbing, or traffic never reaches the Service. Run kubectl describe pod and you’ll see a single probe failed line at the bottom of Events. This runbook is built so you can Ctrl+F that exact error string and jump straight to the fix. Minimal theory — just follow: console string → root cause → copy-paste YAML.

Step 1: Which probe is the culprit? (symptom decision table)

Identify the probe type first. The symptom almost always tells you which one it is.

Key rule: Liveness failure = restart, Readiness failure = removed from Endpoints (traffic blocked). Restart loop → suspect Liveness first. 502 / connection failures → suspect Readiness first.

Step 2: Where to read the exact error string

# — describe Events
kubectl describe pod <pod-name> | tail -30

# when
kubectl get events --sort-by=.lastTimestamp -n <ns>

The probe message is printed verbatim on the Warning Unhealthy line in the Events section. Find that one line in the table below.

Step 3: Exact-match table for error strings

Step 4: Decision tree

probe failed found
├─ connection refused / dial tcp refused
│   ├─ Port numbers differ → fix so containerPort == probe.port
│   └─ Ports match (still booting) → raise initialDelaySeconds or add startupProbe
├─ context deadline exceeded → raise timeoutSeconds (1→3) + check app latency
├─ statuscode: 500 → check health-endpoint dependencies (DB/cache) (NOT a probe-config issue!)
└─ Restart loop but the app is healthy → Liveness is too aggressive; loosen thresholds

Step 5: Four copy-paste YAML snippets

① Slow-booting Spring/JVM apps — startupProbe + relaxed Liveness

For apps with a long cold start, protect them with startupProbe so Liveness only runs after startup completes.

startupProbe:
  httpGet:
    path: /actuator/health
    port: 8080
  failureThreshold: 30   # 30 × 10s = up to 5 min for startup
  periodSeconds: 10
livenessProbe:
  httpGet:
    path: /actuator/health/liveness
    port: 8080
  periodSeconds: 10
  failureThreshold: 3
  timeoutSeconds: 3
readinessProbe:
  httpGet:
    path: /actuator/health/readiness
    port: 8080
  periodSeconds: 5
  failureThreshold: 3

Liveness/Readiness wait until startupProbe succeeds. The app will no longer be killed by Liveness during boot.

② connection refused — fix httpGet path/port

containerPort and the probe port must be identical.

ports:
  - containerPort: 8080   # port the app actually listens on
readinessProbe:
  httpGet:
    path: /healthz
    port: 8080            # must match containerPort above!

③ context deadline exceeded — adjust the timeout

The default timeoutSeconds: 1 is too tight for JVM apps that pause briefly for GC.

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  timeoutSeconds: 3     # 1 → 3
  periodSeconds: 10
  failureThreshold: 3

④ Choosing tcpSocket vs httpGet

For DBs, message queues, and anything without an HTTP endpoint, tcpSocket (just check that the port is open) is the right probe.

readinessProbe:
  tcpSocket:
    port: 5432           # check that the port is open
  periodSeconds: 5
# grpc field is GA in 1.29+ for gRPC apps
livenessProbe:
  grpc:
    port: 50051
  periodSeconds: 10

Practitioner note: Don’t set Liveness too aggressively

The most common self-inflicted outage in production is setting Liveness so tight that it kills a healthy app. A brief slowdown under a traffic spike makes Liveness fail → restart → traffic piles onto a cold new Pod → slowdown again → another restart. You amplify the incident yourself. Keep Liveness loose so it only catches true unrecoverable deadlocks; leave transient slowness to Readiness. Also, in service-mesh environments like Istio, readiness false positives are common because of sidecar startup order (the app is up but Envoy isn’t yet). Guarantee order with startupProbe + sidecar holdApplicationUntilProxyStarts.

Recommended thresholds to prevent recurrence

References: official docs

The primary source for the behavior, settings, and errors covered here is the official documentation below. Check it for version-specific options and exact semantics.

• Kubernetes official documentation

FAQ

Q. The Pod is Running but traffic never arrives. Why? A. Most likely a Readiness probe failure. Check whether the Pod IP is missing with kubectl get endpoints <svc>. A Readiness failure removes the Pod from Endpoints without restarting it.

Q. RESTARTS keep climbing. There are no errors in the app logs. A. Liveness is too aggressive. Raise timeoutSeconds from 1 to 3. For slow-booting apps, add a startupProbe (failureThreshold: 30) so Liveness does not run until startup completes.

Q. Can I fix statuscode: 500 by changing the probe config? A. No. 500 means the health endpoint actually returned 500 — it is not a probe-config problem. Check connectivity to dependencies such as DB and cache.

Part 22 covers a runbook that matches CrashLoopBackOff to causes by exit code.