Solving CrashLoopBackOff: Diagnosing 7 Causes of Infinite Pod Restarts

Completely Fixing CrashLoopBackOff: Diagnosing 7 Causes of Infinite Pod Restarts

The deploy succeeded, but the Pod keeps restarting

You ran kubectl get pod and the STATUS column shows CrashLoopBackOff, with the RESTARTS count climbing every five minutes. This article is for that exact moment.

The first thing to understand is that CrashLoopBackOff is a symptom, not a root cause. It simply means the container starts → crashes → Kubernetes restarts it → it crashes again, so kubelet retries with a backoff (delayed retry) interval. The real cause is why the process died inside the container.

Part 1 of this series covered Pending, a scheduling-stage problem (resource shortage, node selectors). Part 2 covered ImagePullBackOff, where the image never gets pulled at all. CrashLoopBackOff is one layer deeper: the image was pulled and the container started, then died immediately. So the debugging focus is not the scheduler, but the container process's exit code and logs.

The 1-minute diagnostic routine: describe → logs --previous → Exit Code

Whatever the cause, always start with these three steps. Follow them in order and you'll narrow the scope within a minute about 90% of the time.

Step 1: Check Events with describe

kubectl describe pod my-app-7d9f8-abcde

The Events section at the bottom is what matters.

Events:
  Type     Reason     Age                From     Message
  ----     ------     ----               ----     -------
  Normal   Pulled     2m                 kubelet  Successfully pulled image
  Normal   Created    2m (x4 over 3m)    kubelet  Created container app
  Normal   Started    2m (x4 over 3m)    kubelet  Started container app
  Warning  BackOff    30s (x8 over 3m)   kubelet  Back-off restarting failed container

Back-off restarting failed container is the common signal for CrashLoopBackOff. When you see it, you know the container started and then died. Now you need to find out why.

Step 2: Capture the dead container's logs with logs --previous

The currently running container may be about to die again, so its logs can be empty. You need the logs from the previously terminated container.

kubectl logs my-app-7d9f8-abcde --previous
# 컨테이너가 여러 개면
kubectl logs my-app-7d9f8-abcde -c app --previous

A lot of people forget --previous (short form -p) and waste time wondering why there are no logs. Half of crash debugging hangs on this one flag.

Step 3: Check the Exit Code

Read the exit code from the Last State block in the describe output.

    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137

Also watch the RESTARTS count from kubectl get pod. If it spikes quickly, the process is crashing immediately. If the container stays up for a while then dies, you're more likely looking at a memory leak or a probe issue.

Error message → cause mapping table

Diagnosing and fixing the 7 causes

Cause 1. Application exception / entrypoint error (Exit 1)

The most common case. The app throws an exception and dies as soon as it starts.

kubectl logs <pod> --previous   # 스택트레이스 확인

The logs will show messages like NullPointerException, Cannot find module, or panic: verbatim. This is an application bug, not an infra problem, so the fix is the code or config the logs point to. If you're on the infra side, capture the logs and hand them to the development team as-is.

Cause 2. OOMKilled (Exit 137)

kubectl describe pod <pod> | grep -A2 "Last State"   # Reason: OOMKilled 확인
kubectl top pod <pod>                                 # 실사용 메모리 확인

137 = 128 + 9, i.e. SIGKILL. The kernel forcibly killed the process because it exceeded its memory limit.

Before (limit too tight)

resources:
  limits:
    memory: "128Mi"
  requests:
    memory: "128Mi"

After (raised based on actual usage)

resources:
  requests:
    memory: "256Mi"   # 평상시 사용량
  limits:
    memory: "512Mi"   # 피크 여유분

The real fix is to measure actual usage with kubectl top pod and set the limit accordingly. For workloads that OOM often, auto-tuning requests/limits with VPA (Vertical Pod Autoscaler) is a common approach these days.

Cause 3. Missing environment variables or secrets

kubectl logs <pod> --previous           # "secret not found" / "env XXX is undefined"
kubectl get secret db-credentials        # 시크릿 존재 여부
kubectl get secret db-credentials -o yaml # 키 이름 확인

Before (referenced secret missing or name typo)

envFrom:
  - secretRef:
      name: db-credential   # 's' 누락

After

envFrom:
  - secretRef:
      name: db-credentials

The real fix: deploy Secrets/ConfigMaps before the Pod, and cross-check names and keys with kubectl get secret. Also confirm they live in the same namespace.

Cause 4. Liveness probe failure

The container itself is fine, but the probe fails before the app is fully up, so kubelet kills it.

Warning  Unhealthy  kubelet  Liveness probe failed: HTTP probe failed with statuscode: 500
Normal   Killing    kubelet  Container failed liveness probe, will be restarted

Before (no startup grace period)

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 1
  failureThreshold: 1

After (startup grace + relaxed thresholds)

startupProbe:          # 1.29+ 권장: 기동 전용 프로브 분리
  httpGet:
    path: /healthz
    port: 8080
  failureThreshold: 30
  periodSeconds: 5
livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 10
  failureThreshold: 3
  periodSeconds: 10

The real fix: split slow-starting apps onto a startupProbe, and configure liveness conservatively so it only fires when the process is actually dead. Separating readiness/liveness/startup by role is the recommended pattern as of 1.29+.

Cause 5. Dependent service (DB) connection failure

kubectl logs <pod> --previous   # "connection refused" / "dial tcp 10.x:5432"

The app comes up before DB or Redis is ready and dies on a connection failure.

Fix: wait for the dependency with an initContainer

initContainers:
  - name: wait-for-db
    image: busybox:1.36
    command: ['sh', '-c',
      'until nc -z postgres 5432; do echo waiting; sleep 2; done']

Fundamentally, putting backoff retry logic in the app is the most robust approach. An initContainer guarantees startup order, but resilience when the DB blips during operation is the app-level retry's job.

Cause 6. ConfigMap mount path error

kubectl logs <pod> --previous   # "no such file or directory: /config/app.yaml"
kubectl describe pod <pod>      # Volumes / Mounts 확인

Before (no subPath — mounting a directory overwrites existing files)

volumeMounts:
  - name: config
    mountPath: /app/config/app.yaml   # 파일을 디렉터리로 마운트

After (inject a single file with subPath)

volumeMounts:
  - name: config
    mountPath: /app/config/app.yaml
    subPath: app.yaml
volumes:
  - name: config
    configMap:
      name: app-config

The real fix: use subPath when injecting a single file, and confirm mountPath matches the path the app actually reads.

Cause 7. Wrong command / args

If you override the image entrypoint incorrectly, you'll see exec: "xxx": executable file not found.

Before

command: ["python3"]
args: ["app.py"]   # 작업 디렉터리에 app.py가 없음 → 즉시 종료

After

command: ["python3"]
args: ["/app/main.py"]

The real fix: check the image's default ENTRYPOINT/CMD, and override only when you actually need to.

Exit code cheat sheet

If you get Exit 0 but it still restarts, it's almost always a "run once and exit" script deployed as a Deployment. Switch one-shot work to a Job.

Debugging instant exits that leave no logs

The most frustrating case. Even --previous returns empty logs, and the container dies too fast to exec in. In that situation, keep the container alive on purpose and inspect it from the inside.

# 임시로 엔트리포인트를 sleep으로 덮어 컨테이너를 살려둠
command: ["sleep", "3600"]

Deploy it that way, then exec in and run the process by hand.

kubectl exec -it <pod> -- sh
# 안에서 직접 실행해 진짜 에러 메시지 확인
/app/entrypoint.sh

If you don't want to touch the original image, attaching an ephemeral container with kubectl debug is also a good option.

kubectl debug -it <pod> --image=busybox --target=app -- sh

A note from the field: In overnight incident response, more than 70% of CrashLoopBackOff cases turned out to be either a "secret name typo" or "OOMKilled". So when I get paged, I always start with the Exit Code from describe. 137 means memory, a secret message means config, otherwise I go straight to logs --previous. That one branch cut average response time in half.

Conclusion: diagnostic checklist

1. kubectl get pod → check RESTARTS count and STATUS
2. kubectl describe pod → check Events for Back-off restarting... and the Exit Code
3. kubectl logs <pod> --previous → capture the dead container's logs
4. Branch on Exit Code: 137 → memory, 1 → app exception, secret/env → config, probe → probes
5. If there are no logs, override with command: sleep or exec in via kubectl debug

For ImagePullBackOff (image never pulled), see Part 2. For Pending (stuck at scheduling), see Part 1. If the container is up but you can't reach it from outside, that continues in Part 4: troubleshooting Service/Endpoint connection failures.

References: official docs

The primary source for the behavior, settings, and errors covered in this article is the official documentation below. Check it for version-specific options and exact behavior.

• Kubernetes official documentation

FAQ

Q. I ran kubectl logs --previous and got "previous terminated container not found". A. Either the container hasn't restarted yet (right after the first crash), or the node already cleaned up the previous container. Wait until RESTARTS increments and try again, or check the termination reason in the Last State block of kubectl describe.

Q. Exit Code is 137, but kubectl top pod shows memory below the limit. Why did it OOM? A. It may have spiked over the limit and died, and you're looking at the post-death measurement. top only shows the current value, so check the peak of container_memory_working_set_bytes with Prometheus or similar and raise the limit above that. Because 137 is SIGKILL, also consider an external kill (e.g. node resource pressure).

Q. In CrashLoopBackOff, the restart interval keeps getting longer. Is that normal? A. Yes. Kubernetes exponentially backs off from 10 seconds up to a maximum of 5 minutes. Once you fix the cause and the container starts cleanly, the counter resets. A long interval does not mean a worse outage — focus on the exit code and logs, not the gap.