Fixing CrashLoopBackOff: Diagnose an Infinite Pod Restart Loop with kubectl in 5 Minutes

CrashLoopBackOff Fully Solved: Diagnose an Infinite Pod Restart Loop with kubectl in 5 Minutes

K8s Troubleshooting Guide, Part 18

"Back-off restarting failed container" — If This Is What You're Looking At

The deploy finished, but the Pod keeps coming up and dying, and the RESTARTS count just keeps climbing. Odds are you're staring at something like this.

$ kubectl get pods
NAME                      READY   STATUS             RESTARTS      AGE
api-server-7d9f8c-x2kpq   0/1     CrashLoopBackOff   6 (90s ago)   8m

When STATUS is CrashLoopBackOff and RESTARTS keeps increasing, Kubernetes is looping on "this container died again; I'll wait a bit and start it over." The restart interval grows exponentially: 10s → 20s → 40s… up to a 5-minute cap (that's the back-off).

Let's clear up the look-alike errors in one line each before we go further.

• ImagePullBackOff: The container never even starts. The image couldn't be pulled (registry / tag / auth problem).
• OOMKilled: The kernel killed the process for exceeding the memory limit (Exit 137). Can also be a cause of CrashLoop.
• CrashLoopBackOff: The container starts, then dies, over and over. ← This post is aimed squarely at this case.

In other words, the core issue is: "the image pulled fine, but the process exits right after start — or shortly after."

First-pass diagnosis trio: describe · logs --previous · Exit Code

Don't guess at the cause — narrow it down with commands. Run these three in order.

① Check Events and Last State with describe

kubectl describe pod api-server-7d9f8c-x2kpq

The key sections to look at in the output:

    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Wed, 25 Jun 2026 10:11:02 +0900
      Finished:     Wed, 25 Jun 2026 10:11:03 +0900
...
Events:
  Type     Reason     Age                 From     Message
  ----     ------     ----                ----     -------
  Warning  BackOff    20s (x6 over 7m)    kubelet  Back-off restarting failed container

The Exit Code and Reason under Last State: Terminated are your first clues. The gap between Started and Finished also tells you whether it died in a second or ran for a while first.

② Look at the logs from just before it died (most important)

The current container has often already died and been replaced, so kubectl logs comes back empty. You need the logs from the previous (dead) container.

kubectl logs api-server-7d9f8c-x2kpq --previous

--previous (or -p) is the heart of CrashLoop diagnosis. Most of the real causes (stack traces, "config not found", "connection refused") are printed here.

③ Extract the Exit Code precisely

kubectl get pod api-server-7d9f8c-x2kpq \
  -o jsonpath='{.status.containerStatuses[0].lastState.terminated.exitCode}'

# 이벤트만 시간순으로
kubectl get events --field-selector involvedObject.name=api-server-7d9f8c-x2kpq \
  --sort-by=.lastTimestamp

Exit Code diagnostic table

Five copy-paste prescriptions by cause

1) Bad command/args (Exit 127/126)

Symptom: logs --previous shows exec: "start.sh": not found or permission denied.

Check:

kubectl get pod <pod> -o jsonpath='{.spec.containers[0].command}'

Fix YAML (correct the path and permissions):

spec:
  containers:
    - name: api
      image: myregistry/api:1.4.0
      command: ["/app/bin/server"]   # 절대경로, 실제 존재하는 바이너리
      args: ["--port=8080"]

If it's a script, don't forget RUN chmod +x /app/bin/server when you build the image (prevents 126).

2) Dependency not ready yet (wait with an initContainer)

Symptom: The app tries to connect to DB/Redis, gets connection refused, Exit 1. If the dependency comes up late, you get an infinite restart loop.

Check:

kubectl logs <pod> --previous | grep -i "refused\|timeout\|unreachable"

Fix YAML (wait-for pattern):

spec:
  initContainers:
    - name: wait-for-db
      image: busybox:1.36
      command:
        - sh
        - -c
        - |
          until nc -z postgres 5432; do
            echo "waiting for postgres..."; sleep 2;
          done
  containers:
    - name: api
      image: myregistry/api:1.4.0

3) Missing env vars / ConfigMap / Secret (Exit 1)

A classic in GitOps: the Deployment syncs first while the ConfigMap is still missing.

Symptom: Error: configmap "app-config" not found, or the app log shows KeyError: DATABASE_URL.

Check:

kubectl get configmap app-config
kubectl describe pod <pod> | grep -A5 "Environment"

Fix YAML:

      envFrom:
        - configMapRef:
            name: app-config
        - secretRef:
            name: app-secret

Double-check that the mounted key names match the env var names the app actually reads.

4) Over-aggressive liveness probe

Symptom: The app is fine, but boot is slow so liveness kills it first → infinite restarts. (The proper fix is in the next section.)

5) OOM-related (Exit 137)

Symptom: describe shows Reason: OOMKilled, Exit 137. Check: kubectl describe pod <pod> → Last State Reason. Raise the memory limits or tune the app heap. See the OOMKilled post for details.

Break the loop by tuning probes: startupProbe in practice

The most frustrating CrashLoop is the "app came up fine, then liveness slaughtered it" pattern. Slow-boot apps (JVM, large model loading) can't survive on initialDelaySeconds alone. The right answer is to split the boot window out with a startupProbe.

    livenessProbe:
      httpGet: { path: /healthz, port: 8080 }
      periodSeconds: 10
      failureThreshold: 3        # 부팅과 무관하게 짧게 유지
    readinessProbe:
      httpGet: { path: /ready, port: 8080 }
      periodSeconds: 5
    startupProbe:
      httpGet: { path: /healthz, port: 8080 }
      periodSeconds: 10
      failureThreshold: 30       # 10s * 30 = 최대 300초까지 부팅 허용

Until the startupProbe succeeds, liveness and readiness stay quiet. Once boot finishes, liveness kicks in — so you cleanly separate "slow boot" from "failure in production."

Recommended probe values

A note from the field: Since Kubernetes 1.30+, sidecars became first-class as native initContainers (restartPolicy: Always), which cut down a lot of CrashLoops caused by log-shipper / proxy sidecars coming up after the main container. That said, CrashLoops from a missing ConfigMap sync in GitOps (Argo CD, etc.) are still #1. My habit: after every deploy, look at logs --previous first, then the Exit Code. Output beats guessing.

Wrap-up: recurrence-prevention checklist

Once you've broken the loop, nail it down so it doesn't happen again.

• Verify the container runs standalone locally: bring it up outside K8s first with docker run --rm <image>
• Diff ConfigMap/Secret mount keys against the keys the app actually reads — they must match 1:1
• Split probes by role: boot = startupProbe, runtime health = liveness, traffic admission = readiness
• Dependencies: protect DB/cache with an initContainer wait-for or a readinessProbe
• Monitor/alert on RESTARTS: alert on the kube_pod_container_status_restarts_total metric

Related Errors (internal series links)

• ImagePullBackOff post — when the image can't be pulled at all
• OOMKilled post — deep-dive on Exit 137 memory exhaustion
• PVC Pending post — Pod won't start because volume binding failed

Coming up next

Part 19 tracks Pod Pending & scheduling failures (node resource shortage, taint/toleration, nodeSelector mismatch) starting from the FailedScheduling event in kubectl describe.

References: official docs

The primary source for the behavior, settings, and errors covered here is the official documentation below. Check it for version-specific options and exact semantics.

• Kubernetes 공식 문서

FAQ

Q. kubectl logs is empty — how do I see why it died? A. The current container may have just been replaced, so there are no logs yet. Use kubectl logs <pod> --previous (-p) to see the logs from the previously dead container. Most root causes show up there.

Q. Does Exit Code 137 always mean out of memory? A. 137 means it received SIGKILL (128+9). It's usually OOMKilled, but node pressure or a forced kill can also produce it. Confirm it's a memory problem by checking Last State → Reason: OOMKilled in kubectl describe pod.

Q. The app is healthy but it keeps restarting. What should I look at first? A. An over-aggressive liveness probe is the likely culprit. Split boot time out with a startupProbe and keep liveness failureThreshold short. If liveness runs before boot finishes, even a healthy app will restart forever.