Solving CrashLoopBackOff: Diagnosing 7 Causes of Infinite Pod Restarts

Completely Fixing CrashLoopBackOff: Diagnosing 7 Causes of Infinite Pod Restarts

What this covers: kubectl get pod shows STATUS CrashLoopBackOff, and kubectl describe pod Events show Back-off restarting failed container. The container starts and then dies, so pick the cause from the exit code and the previous container's logs.

If a referenced Secret or ConfigMap does not exist at all, the STATUS is CreateContainerConfigError, not CrashLoopBackOff (see Cause 3).

The deploy succeeded, but the Pod keeps restarting

You ran kubectl get pod and the STATUS column shows CrashLoopBackOff, with the RESTARTS count climbing. This article is for that exact moment.

The first thing to understand is that CrashLoopBackOff is a symptom, not a root cause. It simply means the container starts → crashes → Kubernetes restarts it → it crashes again, so kubelet retries with a backoff (delayed retry) interval. The real cause is why the process died inside the container.

Part 1 of this series covered Pending, a scheduling-stage problem (resource shortage, node selectors). Part 2 covered ImagePullBackOff, where the image never gets pulled at all. CrashLoopBackOff is one layer deeper: the image was pulled and the container started, then died immediately. So the debugging focus is not the scheduler, but the container process's exit code and logs.

The 1-minute diagnostic routine: describe → logs --previous → Exit Code

Whatever the cause, always start with these three steps. Follow them in order and the scope narrows quickly.

Step 1: Check Events with describe

kubectl describe pod my-app-7d9f8-abcde

The Events section at the bottom is what matters.

Events:
  Type     Reason     Age                From     Message
  ----     ------     ----               ----     -------
  Normal   Pulled     2m                 kubelet  Successfully pulled image
  Normal   Created    2m (x4 over 3m)    kubelet  Created container app
  Normal   Started    2m (x4 over 3m)    kubelet  Started container app
  Warning  BackOff    30s (x8 over 3m)   kubelet  Back-off restarting failed container

Back-off restarting failed container is the common signal for CrashLoopBackOff. When you see it, you know the container started and then died. Now you need to find out why.

Step 2: Capture the dead container's logs with logs --previous

The currently running container may be about to die again, so its logs can be empty. You need the logs from the previously terminated container.

kubectl logs my-app-7d9f8-abcde --previous
# if the Pod has multiple containers
kubectl logs my-app-7d9f8-abcde -c app --previous

A lot of people forget --previous (short form -p) and waste time wondering why there are no logs. It is the first flag to reach for in crash debugging.

Step 3: Check the Exit Code

Read the exit code from the Last State block in the describe output.

    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137

Also watch the RESTARTS count from kubectl get pod. If it spikes quickly, the process is crashing immediately. If the container stays up for a while then dies, you're more likely looking at a memory leak or a probe issue.

Error message → cause mapping table

Diagnosing and fixing the 7 causes

Cause 1. Application exception / entrypoint error (Exit 1)

The most common case. The app throws an exception and dies as soon as it starts.

kubectl logs <pod> --previous   # check the stack trace

The logs will show messages like NullPointerException, Cannot find module, or panic: verbatim. This is an application bug, not an infra problem, so the fix is the code or config the logs point to. If you're on the infra side, capture the logs and hand them to the development team as-is.

Cause 2. OOMKilled (Exit 137)

kubectl describe pod <pod> | grep -A2 "Last State"   # look for Reason: OOMKilled
kubectl top pod <pod>                                 # actual memory use (needs metrics-server)

137 = 128 + 9, i.e. SIGKILL. The kernel OOM killer terminated the process because it exceeded its memory limit, and describe records it as Reason: OOMKilled, Exit Code: 137 (resource management docs).

Before (limit too tight)

resources:
  limits:
    memory: "128Mi"
  requests:
    memory: "128Mi"

After (raised based on actual usage)

resources:
  requests:
    memory: "256Mi"   # typical usage
  limits:
    memory: "512Mi"   # headroom for peaks

The real fix is to measure actual usage with kubectl top pod (requires metrics-server) and set the limit accordingly. For workloads that OOM often, auto-tuning requests/limits with VPA (Vertical Pod Autoscaler) is a common approach these days.

Cause 3. Missing environment variables or secrets

kubectl logs <pod> --previous           # "secret not found" / "env XXX is undefined"
kubectl get secret db-credentials        # does the Secret exist?
kubectl describe secret db-credentials  # key names and sizes only (values are not printed)

First, tell two cases apart. If the referenced Secret does not exist at all, the container never starts, so the STATUS is CreateContainerConfigError rather than CrashLoopBackOff, and Events show Error: secret "db-credential" not found. The kubelet keeps retrying until the Secret appears (Secret docs). You get CrashLoopBackOff when the Secret exists but a key name differs, so the env var you expected from envFrom is empty and the app dies during its startup config check. -o yaml prints the Secret values (base64) to your terminal, so describe is the safer way to check keys.

Before (Secret name typo → CreateContainerConfigError)

envFrom:
  - secretRef:
      name: db-credential   # missing 's'

After

envFrom:
  - secretRef:
      name: db-credentials

The real fix: deploy Secrets/ConfigMaps before the Pod, and cross-check names and keys with kubectl get secret. Also confirm they live in the same namespace.

Cause 4. Liveness probe failure

The container itself is fine, but the probe fails before the app is fully up, so kubelet kills it.

Warning  Unhealthy  kubelet  Liveness probe failed: HTTP probe failed with statuscode: 500
Normal   Killing    kubelet  Container failed liveness probe, will be restarted

Before (no startup grace period)

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 1
  failureThreshold: 1

After (startup grace + relaxed thresholds)

startupProbe:          # startup-only probe (GA in 1.20): holds off liveness until it passes
  httpGet:
    path: /healthz
    port: 8080
  failureThreshold: 30
  periodSeconds: 5
livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 10
  failureThreshold: 3
  periodSeconds: 10

The real fix: split slow-starting apps onto a startupProbe, and configure liveness conservatively so it only fires when the process is actually dead. startupProbe went GA in 1.20 (feature gate list), and it defers liveness and readiness checks until it succeeds (probe docs). The example above waits up to 30×5 s = 150 s for startup.

Probe field defaults

Defaults are from the Probe API reference. Rather than copying the numbers as-is, measure your real startup time and health-endpoint latency and set the fields from those.

Cause 5. Dependent service (DB) connection failure

kubectl logs <pod> --previous   # "connection refused" / "dial tcp 10.x:5432"

The app comes up before DB or Redis is ready and dies on a connection failure.

Fix: wait for the dependency with an initContainer

initContainers:
  - name: wait-for-db
    image: busybox:1.36
    command: ['sh', '-c',
      'until nc -z postgres 5432; do echo waiting; sleep 2; done']

Fundamentally, putting backoff retry logic in the app is the most robust approach. An initContainer guarantees startup order, but resilience when the DB blips during operation is the app-level retry's job.

Cause 6. ConfigMap mount path error

kubectl logs <pod> --previous   # "no such file or directory: /config/app.yaml"
kubectl describe pod <pod>      # check Volumes / Mounts

Before (no subPath — mounting a directory overwrites existing files)

volumeMounts:
  - name: config
    mountPath: /app/config/app.yaml   # mounts a directory where a file is expected

After (inject a single file with subPath)

volumeMounts:
  - name: config
    mountPath: /app/config/app.yaml
    subPath: app.yaml
volumes:
  - name: config
    configMap:
      name: app-config

The real fix: use subPath when injecting a single file, and confirm mountPath matches the path the app actually reads. Note that a file mounted with subPath does not receive ConfigMap updates, so restart the Pod after changing the ConfigMap (ConfigMap docs).

Cause 7. Wrong command / args

If you override the image entrypoint incorrectly, you'll see exec: "xxx": executable file not found.

Before

command: ["python3"]
args: ["app.py"]   # app.py is not in the working directory → exits immediately

After

command: ["python3"]
args: ["/app/main.py"]

The real fix: check the image's default ENTRYPOINT/CMD, and override only when you actually need to.

Exit code cheat sheet

If you get Exit 0 but it still restarts, it's usually a "run once and exit" script deployed as a Deployment. Switch one-shot work to a Job.

126 and 127 are returned by shells that follow the Bash exit-status convention (127 if a command is not found, 126 if it is found but not executable). You mostly see them when the entrypoint is a shell script and a command inside it fails. If the entrypoint binary itself is missing from the image, no shell is involved: the container runtime fails to start the process, Reason shows StartError, and the message contains executable file not found in $PATH (the exit code in that case depends on the runtime).

Exit 0 (Completed) but it keeps restarting

In a Deployment with restartPolicy: Always, even a clean exit (0) gets restarted. If it isn't a one-off job and still exits 0, the main process (PID 1) usually isn't staying in the foreground.

• A daemon that forks into the background: run nginx in the foreground, e.g. nginx -g 'daemon off;'. The official nginx image's default CMD already does this.
• An entrypoint script that starts the server with & and exits: change the last line to exec <server command> so the server becomes PID 1. With exec, SIGTERM also reaches the server process directly.

#!/bin/sh
# entrypoint.sh — do setup, then run the server as PID 1
set -e
/app/migrate.sh
exec /app/server --port=8080

exec format error: image and node CPU architecture mismatch

If logs --previous shows only exec format error and the container dies immediately, the image was most likely built for a different architecture than the node — typically an image built on an arm64 Mac running on amd64 nodes.

# node architecture
kubectl get nodes -L kubernetes.io/arch
# platforms included in the image
docker buildx imagetools inspect <registry>/<image>:<tag>

If the node architecture is missing from the image's list, rebuild for that platform. If your cluster mixes architectures, build and push a multi-platform image (Docker multi-platform build docs).

docker buildx build --platform linux/amd64,linux/arm64 -t <registry>/<image>:<tag> --push .

Debugging instant exits that leave no logs

The most frustrating case. Even --previous returns empty logs, and the container dies too fast to exec in. In that situation, keep the container alive on purpose and inspect it from the inside.

# temporarily override the entrypoint with sleep to keep the container alive
command: ["sleep", "3600"]

Deploy it that way, then exec in and run the process by hand.

kubectl exec -it <pod> -- sh
# run it by hand inside to see the real error message
/app/entrypoint.sh

If you don't want to touch the original image, attaching an ephemeral container with kubectl debug is also a good option.

kubectl debug -it <pod> --image=busybox:1.36 --target=app -- sh

If the container dies too fast to attach with --target, make a copy of the Pod with the command replaced by a shell. The original Pod stays as it is, and you can run the entrypoint by hand inside the copy (official debugging docs).

kubectl debug <pod> -it --copy-to=<pod>-debug --container=app -- sh
# delete the copy when you're done
kubectl delete pod <pod>-debug

A note from the field: In overnight incident response, the CrashLoopBackOff causes I ran into most often were "wrong env var or secret key" and "OOMKilled". So when I get paged, I always start with the Exit Code from describe. 137 means memory, a config-related message means Secrets/ConfigMaps, otherwise I go straight to logs --previous.

Conclusion: diagnostic checklist

1. kubectl get pod → check RESTARTS count and STATUS
2. kubectl describe pod → check Events for Back-off restarting... and the Exit Code
3. kubectl logs <pod> --previous → capture the dead container's logs
4. Branch on Exit Code: 137 → memory, 1 → app exception, secret/env → config, probe → probes
5. If there are no logs, override with command: sleep or exec in via kubectl debug

For ImagePullBackOff (image never pulled), see Part 2. For Pending (stuck at scheduling), see Part 1. If the container is up but you can't reach it from outside, that continues in Part 4: troubleshooting Service/Endpoint connection failures.

References: official docs

The primary source for the behavior, settings, and errors covered in this article is the official documentation below. Check it for version-specific options and exact behavior.

• Pod lifecycle: container restarts and backoff
• Debug running Pods (kubectl debug)
• Configure liveness, readiness, and startup probes
• Resource management for containers (OOMKilled)
• Secrets · ConfigMaps
• Probe API reference (field defaults)
• Bash exit status (126, 127) · Docker multi-platform builds

FAQ

Q. I ran kubectl logs --previous and got "previous terminated container not found". A. Either the container hasn't restarted yet (right after the first crash), or the node already cleaned up the previous container. Wait until RESTARTS increments and try again, or check the termination reason in the Last State block of kubectl describe.

Q. Exit Code is 137, but kubectl top pod shows memory below the limit. Why did it OOM? A. It may have spiked over the limit and died, and you're looking at the post-death measurement. top only shows the current value, so check the peak of container_memory_working_set_bytes with Prometheus or similar and raise the limit above that. Because 137 is SIGKILL, also consider an external kill (e.g. node resource pressure).

Q. In CrashLoopBackOff, the restart interval keeps getting longer. Is that normal? A. Yes. The kubelet doubles the restart delay (10 s, 20 s, 40 s, …) and caps it at 300 s (5 minutes). Once a container runs for 10 minutes without problems, the backoff timer resets; the RESTARTS number itself stays (Pod lifecycle docs). From 1.35, the KubeletCrashLoopBackOffMax feature (beta, enabled by default) lets you tune the maximum delay per node (feature gate list). A long interval does not mean a worse outage — focus on the exit code and logs, not the gap.