6 Major Causes of Liveness/Readiness Probe Failures and How to Fix Them: Stopping Infinite Pod Restarts

"The App Is Fine, So Why Does the Pod Keep Restarting?" — When Probe Configuration Becomes the Problem

If you're a DevOps engineer, you've probably hit this most confusing situation at least once. Application logs look healthy, and everything works perfectly in the test environment—but the Pod you deployed to the actual Kubernetes cluster is uniquely unstable. kubectl get pods shows CrashLoopBackOff or Error, and Ready is 0/1, so the Pod receives no traffic at all.

Most of the time, the cause is not a bug in the application code. It comes from a misunderstanding of Liveness Probe or Readiness Probe. These probes exist to protect us, but misconfigured they become "false watchdogs" that kill the application instead.

This article goes beyond parsing error messages. We fully break down how the three probe types work, cover the six failure causes that show up most often in production, and give you YAML prescriptions you can apply immediately—a practical, hands-on guide.

The Probe Trio — Liveness, Readiness, and Startup: How They Actually Work

These three probes have similar names, but the roles they play inside Kubernetes and the impact of failure are completely different. Understanding that difference is 80% of the solution.

The core model:

• Liveness failure $\rightarrow$ Restart
• Readiness failure $\rightarrow$ Traffic blocked (removed from Service Endpoints)
• Startup failure $\rightarrow$ Wait, then start checks (delay Liveness/Readiness checks themselves)

🚨 First Diagnostic Step: How to Read Error Messages with kubectl describe

When something goes wrong, look for evidence—not gut feel. If kubectl get pod <pod-name> shows READY as 0/1, or RESTARTS is abnormally high, immediately run kubectl describe pod <pod-name>.

The sections to watch are Events and Conditions.

Example error messages and how to interpret them:

1. Liveness failure example:

   Liveness probe failed: HTTP probe failed with statuscode: 503

   • Interpretation: The container is alive (it may not have been restarted), but the /healthz endpoint is returning 503, explicitly saying it cannot accept traffic right now. (→ Likely a Readiness Probe issue)
2. Readiness failure example:

   Readiness probe failed: Get "http://10.x.x.x:8080/healthz": dial tcp connection refused

   • Interpretation: The container refused the connection on that port (8080) entirely. The port is not open, or the app has not bound to it yet. (→ Suspect insufficient initialDelaySeconds or a port typo)
3. Startup failure example:

   Startup probe failed: ...

   • Interpretation: Boot is so slow that startup itself is failing before Liveness/Readiness checks even begin. (→ Adding a startupProbe is urgent.)

🛠️ The 6 Most Common Probe Failure Causes in Production, with YAML Prescriptions

The six scenarios you hit most often in production, organized as symptom $\rightarrow$ cause $\rightarrow$ fix.

1. Insufficient Initial Delay (initialDelaySeconds Too Low)

• Symptom: Right after deploy, the Pod fails immediately and enters a restart loop.
• Cause: The app needs time for JVM loading or DB connection initialization, but the probe starts checking too soon.
• Fix: Increase initialDelaySeconds enough. (e.g., 30 seconds)

2. Port or Path Typo / Mismatch

• Symptom: connection refused or unknown endpoint errors.
• Cause: The port in the YAML (e.g., 8080) differs from the port the app actually listens on.
• Fix: Use kubectl exec to run curl http://localhost:8080/healthz yourself, confirm the exact port and path, then fix the YAML.

3. The Status Code Trap (When It's Not 200 OK)

• Symptom: The probe can communicate successfully, but it keeps failing and the Pod is unstable.
• Cause: Even if the health-check endpoint responds, a 3xx redirect or 401 (auth required) is treated as failure by the probe.
• Fix: The health-check path must return 200 OK. Fix this at the application level.

4. Waiting on External Dependencies (DB/Redis, etc.)

• Symptom: The Pod does not restart, but it receives no traffic and you get 503 errors.
• Cause: The app tries to connect to the DB as soon as it starts, but the DB is not ready yet so the connection fails.
• Fix: Use a Readiness Probe, and split the logic so this probe checks whether the DB connection succeeded. (The most ideal pattern)

5. Timeouts Under Load (timeoutSeconds / periodSeconds)

• Symptom: Fine under normal load, then sudden failures when traffic spikes.
• Cause: If timeoutSeconds is too short, every time load makes the response take more than 1 second, the probe is marked failed.
• Fix: Increase timeoutSeconds enough, or increase periodSeconds to check less frequently.

6. Death of a Slow-Booting App (No Startup Probe)

• Symptom: Intermittent restarts only on JVM-based apps or apps with complex migration logic.
• Cause: Liveness/Readiness probes run too early, before boot completes, and force-kill the app.
• Fix: Always use a startupProbe to give the app a grace period to finish booting.

💡 Practical YAML Prescription: Example Combining All Three Probes

The following is a recommended combination (HTTP Get).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app-deployment
spec:
  template:
    spec:
      containers:
      - name: my-container
        image: your-registry/my-app:latest
        ports:
        - containerPort: 8080
        
        # 1. Startup Probe: 부팅 완료까지 최대 120초 허용 (12초 * 10회)
        startupProbe:
          httpGet:
            path: /bootstrap-ready # 부팅 전용 경로
            port: 8080
          failureThreshold: 10 # 최대 10번 실패 허용
          periodSeconds: 12 # 12초마다 검사
          
        # 2. Readiness Probe: 트래픽 수신 준비 완료 시점 체크
        readinessProbe:
          httpGet:
            path: /readyz # 트래픽 수신 준비 경로
            port: 8080
          initialDelaySeconds: 30 # 30초 대기 후 검사 시작
          periodSeconds: 10
          failureThreshold: 3
          
        # 3. Liveness Probe: 치명적 오류 감지용 (최후의 보루)
        livenessProbe:
          httpGet:
            path: /livez # 생존 여부만 확인하는 경량 경로
            port: 8080
          initialDelaySeconds: 60 # 가장 늦게 검사 시작
          periodSeconds: 20
          failureThreshold: 5

📌 Tip: Calculating the maximum allowed time for a Startup Probe: Max allowed time $\approx$ failureThreshold $\times$ periodSeconds (Example above: $10 \times 12 = 120$ seconds)

🚀 Advanced Pattern for Zero-Downtime Deploys: Using a PreStop Hook

It's not enough for the Pod to become ready. During a rolling update you also need to prevent clients from dropping connections. That's where the preStop hook comes in.

The preStop hook is a script Kubernetes runs immediately before terminating the Pod. At that moment you can signal the application: "I'm about to shut down, so stop accepting new connections and drain existing ones."

        lifecycle:
          preStop:
            exec:
              command: ["/bin/sleep", "10"] # 10초 동안 대기하며 연결 종료 유도

With preStop you buy time, and if you also cause the readinessProbe to fail, the Service immediately stops sending traffic to this Pod, minimizing 502/503 errors.

⏱️ 5-Minute Diagnostic Checklist: Probe Troubleshooting Order

When a problem occurs, checking in this order solves 90%+ of cases.

1. Check kubectl describe pod: Get the exact error message from Events (503, connection refused, etc.).
2. Check initialDelaySeconds: Confirm you allowed enough time given the app's actual boot time.
3. Verify directly with kubectl exec: Run kubectl exec -it <pod-name> -- curl http://localhost:8080/path and confirm the port/path in the YAML actually works.
4. Split and inspect probes:
   • Slow app? $\rightarrow$ Add a startupProbe (do this first).
   • Traffic blocked? $\rightarrow$ Isolate and verify readinessProbe logic (DB connections, etc.).
   • Fatal error? $\rightarrow$ Slim down and re-evaluate the livenessProbe.
5. Separate dependencies: If there are external dependencies (DB), put that check only in the readinessProbe, and keep the livenessProbe as a simple memory/liveness check.

A practitioner's note: Early on I often made the mistake of pointing both Liveness and Readiness probes at the same /health endpoint. The result: a brief DB disconnect made the Liveness Probe fail, the Pod restarted, and that restart itself looked like a service outage to clients—a vicious cycle. The most important principle is to separate the logic for "alive" (Liveness) from "ready to serve" (Readiness).

Frequently Asked Questions (FAQ)

Q. What happens if I set both a Liveness Probe and a Readiness Probe? A. If both fail, and the Liveness Probe fails first, the Pod restarts and the service is interrupted. If the Readiness Probe fails, only traffic is blocked. You must clearly separate the failure causes and blast radius of the two probes.

Q. Is setting failureThreshold high always a good idea? A. No. failureThreshold means "how many failures are acceptable." Set it too high and recovery takes longer when a real outage happens. Choose a reasonable value (usually 3–5) based on the app's characteristics and acceptable downtime.

Q. Does a failed readinessProbe restart the Pod? A. No. When a Readiness Probe fails, Kubernetes only removes (deregisters) the Pod from the Service's Endpoint list. It does not restart the container. Restarting is the job of the Liveness Probe.