# Completely Fixing CrashLoopBackOff: Diagnosing 7 Causes of Infinite Pod Restarts **What this covers:** `kubectl get pod` shows STATUS `CrashLoopBackOff`, and `kubectl describe pod` Events show `Back-off restarting failed container`. The container starts and then dies, so pick the cause from the exit code and the previous container's logs. | Cause | Check | Fix | |---|---|---| | App exception or bad config (Exit 1) | `kubectl logs --previous` | Fix the code or env var the log points to | | Memory limit exceeded (`OOMKilled`, Exit 137) | `Last State` in `kubectl describe pod ` | Raise the memory limit or cut usage | | Liveness probe failing | `Liveness probe failed` in Events | Add a `startupProbe`; relax liveness thresholds | | Dependency not ready (`connection refused`) | `kubectl logs --previous` | Retry logic in the app; an initContainer if needed | | Wrong command or path (`no such file or directory`) | Command and Mounts in `kubectl describe pod ` | Check the image ENTRYPOINT; fix the path or `subPath` | If a referenced Secret or ConfigMap does not exist at all, the STATUS is `CreateContainerConfigError`, not CrashLoopBackOff (see Cause 3). ## The deploy succeeded, but the Pod keeps restarting You ran `kubectl get pod` and the STATUS column shows `CrashLoopBackOff`, with the RESTARTS count climbing. This article is for that exact moment. The first thing to understand is that **CrashLoopBackOff is a symptom, not a root cause**. It simply means the container starts → crashes → Kubernetes restarts it → it crashes again, so kubelet retries with a backoff (delayed retry) interval. The real cause is **why the process died inside the container**. Part 1 of this series covered [Pending](/blog/kubernetes-pod-pending-원인-7가지-kubectl-describe로-완벽-진단하는-실전-가이드), a scheduling-stage problem (resource shortage, node selectors). Part 2 covered [ImagePullBackOff](/blog/imagepullbackofferrimagepull-해결-events로-원인-진단하고-복붙으로-끝내기), where the image never gets pulled at all. CrashLoopBackOff is one layer deeper: **the image was pulled and the container started, then died immediately**. So the debugging focus is not the scheduler, but the container process's exit code and logs. ## The 1-minute diagnostic routine: describe → logs --previous → Exit Code Whatever the cause, always start with these three steps. Follow them in order and the scope narrows quickly. ### Step 1: Check Events with describe ```bash kubectl describe pod my-app-7d9f8-abcde ``` The Events section at the bottom is what matters. ``` Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Pulled 2m kubelet Successfully pulled image Normal Created 2m (x4 over 3m) kubelet Created container app Normal Started 2m (x4 over 3m) kubelet Started container app Warning BackOff 30s (x8 over 3m) kubelet Back-off restarting failed container ``` `Back-off restarting failed container` is the **common signal** for CrashLoopBackOff. When you see it, you know the container started and then died. Now you need to find out *why*. ### Step 2: Capture the dead container's logs with logs --previous The currently running container may be about to die again, so its logs can be empty. You need the logs from the **previously terminated container**. ```bash kubectl logs my-app-7d9f8-abcde --previous # if the Pod has multiple containers kubectl logs my-app-7d9f8-abcde -c app --previous ``` A lot of people forget `--previous` (short form `-p`) and waste time wondering why there are no logs. It is the first flag to reach for in crash debugging. ### Step 3: Check the Exit Code Read the exit code from the `Last State` block in the describe output. ``` Last State: Terminated Reason: OOMKilled Exit Code: 137 ``` Also watch the RESTARTS count from `kubectl get pod`. If it spikes quickly, the process is crashing immediately. If the container stays up for a while then dies, you're more likely looking at a memory leak or a probe issue. ### Error message → cause mapping table | Log / Events message | Likely cause | Jump to | |---|---|---| | `Back-off restarting failed container` | Common signal (container crash) | All | | `OOMKilled` / Exit 137 | Memory limit exceeded | Cause 2 | | `missing env` / `env XXX is undefined` (app log) | Missing env var or secret key | Cause 3 | | `Error: secret "xxx" not found` (STATUS `CreateContainerConfigError`) | Referenced Secret does not exist (not CrashLoopBackOff) | Cause 3 | | `Liveness probe failed` | Probe misconfiguration | Cause 4 | | `connection refused` / `dial tcp ...` | Dependent service (DB) not ready | Cause 5 | | `no such file or directory` | Mount path / command error | Causes 6 and 7 | | Exit 1 + stack trace | Generic application exception | Cause 1 | ## Diagnosing and fixing the 7 causes ### Cause 1. Application exception / entrypoint error (Exit 1) The most common case. The app throws an exception and dies as soon as it starts. ```bash kubectl logs --previous # check the stack trace ``` The logs will show messages like `NullPointerException`, `Cannot find module`, or `panic:` verbatim. This is an **application bug**, not an infra problem, so the fix is the code or config the logs point to. If you're on the infra side, capture the logs and hand them to the development team as-is. ### Cause 2. OOMKilled (Exit 137) ```bash kubectl describe pod | grep -A2 "Last State" # look for Reason: OOMKilled kubectl top pod # actual memory use (needs metrics-server) ``` 137 = 128 + 9, i.e. [SIGKILL](/blog/npm-err-code-elifecycle-해결법-errno-1134sigkill-원인별-진단). The kernel OOM killer terminated the process because it exceeded its memory limit, and describe records it as `Reason: OOMKilled`, `Exit Code: 137` ([resource management docs](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/)). **Before (limit too tight)** ```yaml resources: limits: memory: "128Mi" requests: memory: "128Mi" ``` **After (raised based on actual usage)** ```yaml resources: requests: memory: "256Mi" # typical usage limits: memory: "512Mi" # headroom for peaks ``` The real fix is to measure actual usage with `kubectl top pod` (requires [metrics-server](https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/)) and set the limit accordingly. For workloads that OOM often, auto-tuning requests/limits with VPA (Vertical Pod Autoscaler) is a common approach these days. ### Cause 3. Missing environment variables or secrets ```bash kubectl logs --previous # "secret not found" / "env XXX is undefined" kubectl get secret db-credentials # does the Secret exist? kubectl describe secret db-credentials # key names and sizes only (values are not printed) ``` First, tell two cases apart. If the referenced Secret does not exist at all, the container never starts, so the STATUS is `CreateContainerConfigError` rather than CrashLoopBackOff, and Events show `Error: secret "db-credential" not found`. The kubelet keeps retrying until the Secret appears ([Secret docs](https://kubernetes.io/docs/concepts/configuration/secret/)). You get CrashLoopBackOff when the Secret exists but a key name differs, so the env var you expected from `envFrom` is empty and the app dies during its startup config check. `-o yaml` prints the Secret values (base64) to your terminal, so `describe` is the safer way to check keys. **Before (Secret name typo → CreateContainerConfigError)** ```yaml envFrom: - secretRef: name: db-credential # missing 's' ``` **After** ```yaml envFrom: - secretRef: name: db-credentials ``` The real fix: deploy Secrets/ConfigMaps **before** the Pod, and cross-check names and keys with `kubectl get secret`. Also confirm they live in the same namespace. ### Cause 4. Liveness probe failure The container itself is fine, but the probe fails before the app is fully up, so kubelet kills it. ``` Warning Unhealthy kubelet Liveness probe failed: HTTP probe failed with statuscode: 500 Normal Killing kubelet Container failed liveness probe, will be restarted ``` **Before (no startup grace period)** ```yaml livenessProbe: httpGet: path: /healthz port: 8080 initialDelaySeconds: 1 failureThreshold: 1 ``` **After (startup grace + relaxed thresholds)** ```yaml startupProbe: # startup-only probe (GA in 1.20): holds off liveness until it passes httpGet: path: /healthz port: 8080 failureThreshold: 30 periodSeconds: 5 livenessProbe: httpGet: path: /healthz port: 8080 initialDelaySeconds: 10 failureThreshold: 3 periodSeconds: 10 ``` The real fix: split slow-starting apps onto a `startupProbe`, and configure liveness conservatively so it only fires when the process is actually dead. startupProbe went GA in 1.20 ([feature gate list](https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates-removed/)), and it defers liveness and readiness checks until it succeeds ([probe docs](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/)). The example above waits up to 30×5 s = 150 s for startup. **Probe field defaults** | Field | Default | What to look at when tuning | |---|---|---| | `initialDelaySeconds` | 0 | With a startupProbe in place, liveness can stay at 0 | | `periodSeconds` | 10 (min 1) | Too short and brief stalls pile up failures quickly | | `timeoutSeconds` | 1 (min 1) | Health endpoints that query a DB or external API easily exceed 1 s | | `failureThreshold` | 3 (min 1) | For a startupProbe, `failureThreshold × periodSeconds` must exceed the worst-case startup time | | `successThreshold` | 1 | Must be 1 for liveness and startup probes | Defaults are from the [Probe API reference](https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#Probe). Rather than copying the numbers as-is, measure your real startup time and health-endpoint latency and set the fields from those. ### Cause 5. Dependent service (DB) connection failure ```bash kubectl logs --previous # "connection refused" / "dial tcp 10.x:5432" ``` The app comes up before DB or Redis is ready and dies on a connection failure. **Fix: wait for the dependency with an initContainer** ```yaml initContainers: - name: wait-for-db image: busybox:1.36 command: ['sh', '-c', 'until nc -z postgres 5432; do echo waiting; sleep 2; done'] ``` Fundamentally, putting **backoff retry logic in the app** is the most robust approach. An initContainer guarantees startup order, but resilience when the DB blips during operation is the app-level retry's job. ### Cause 6. ConfigMap mount path error ```bash kubectl logs --previous # "no such file or directory: /config/app.yaml" kubectl describe pod # check Volumes / Mounts ``` **Before (no subPath — mounting a directory overwrites existing files)** ```yaml volumeMounts: - name: config mountPath: /app/config/app.yaml # mounts a directory where a file is expected ``` **After (inject a single file with subPath)** ```yaml volumeMounts: - name: config mountPath: /app/config/app.yaml subPath: app.yaml volumes: - name: config configMap: name: app-config ``` The real fix: use `subPath` when injecting a single file, and confirm mountPath matches the path the app actually reads. Note that a file mounted with `subPath` does not receive ConfigMap updates, so restart the Pod after changing the ConfigMap ([ConfigMap docs](https://kubernetes.io/docs/concepts/configuration/configmap/)). ### Cause 7. Wrong command / args If you override the image entrypoint incorrectly, you'll see `exec: "xxx": executable file not found`. **Before** ```yaml command: ["python3"] args: ["app.py"] # app.py is not in the working directory → exits immediately ``` **After** ```yaml command: ["python3"] args: ["/app/main.py"] ``` The real fix: check the image's default ENTRYPOINT/CMD, and override only when you actually need to. ## Exit code cheat sheet | Exit Code | Signal | Meaning | Look here first | |---|---|---|---| | 0 | - | Exited cleanly but still restarting | restartPolicy / entrypoint is not running as a daemon | | 1 | - | Generic app exception | `logs --previous` stack trace | | 2 | - | Misuse of shell builtins / bad arguments (shell convention) | Entrypoint script syntax, option typos | | 126 | - | Command found but not executable | Execute permission (`chmod +x`), file format | | 127 | - | Command not found | `command`/`args` path, `PATH`, whether the binary exists in the image | | 137 | SIGKILL | OOM or forced kill | memory limit, `kubectl top` | | 139 | SIGSEGV | Segmentation fault | Native library / architecture (arm vs amd) | | 143 | SIGTERM | Graceful termination signal | Graceful shutdown handling, external kill | If you get Exit 0 but it still restarts, it's usually a "run once and exit" script deployed as a Deployment. Switch one-shot work to a Job. 126 and 127 are returned by shells that follow the [Bash exit-status convention](https://www.gnu.org/software/bash/manual/html_node/Exit-Status.html) (127 if a command is not found, 126 if it is found but not executable). You mostly see them when the entrypoint is a shell script and a command inside it fails. If the entrypoint binary itself is missing from the image, no shell is involved: the container runtime fails to start the process, `Reason` shows `StartError`, and the message contains `executable file not found in $PATH` (the exit code in that case depends on the runtime). ### Exit 0 (`Completed`) but it keeps restarting In a Deployment with `restartPolicy: Always`, even a clean exit (0) gets restarted. If it isn't a one-off job and still exits 0, the main process (PID 1) usually isn't staying in the foreground. - **A daemon that forks into the background**: run nginx in the foreground, e.g. `nginx -g 'daemon off;'`. The official nginx image's default CMD already does this. - **An entrypoint script that starts the server with `&` and exits**: change the last line to `exec ` so the server becomes PID 1. With `exec`, SIGTERM also reaches the server process directly. ```bash #!/bin/sh # entrypoint.sh — do setup, then run the server as PID 1 set -e /app/migrate.sh exec /app/server --port=8080 ``` ### `exec format error`: image and node CPU architecture mismatch If `logs --previous` shows only `exec format error` and the container dies immediately, the image was most likely built for a different architecture than the node — typically an image built on an arm64 Mac running on amd64 nodes. ```bash # node architecture kubectl get nodes -L kubernetes.io/arch # platforms included in the image docker buildx imagetools inspect /: ``` If the node architecture is missing from the image's list, rebuild for that platform. If your cluster mixes architectures, build and push a multi-platform image ([Docker multi-platform build docs](https://docs.docker.com/build/building/multi-platform/)). ```bash docker buildx build --platform linux/amd64,linux/arm64 -t /: --push . ``` ## Debugging instant exits that leave no logs The most frustrating case. Even `--previous` returns empty logs, and the container dies too fast to `exec` in. In that situation, **keep the container alive on purpose** and inspect it from the inside. ```yaml # temporarily override the entrypoint with sleep to keep the container alive command: ["sleep", "3600"] ``` Deploy it that way, then exec in and run the process by hand. ```bash kubectl exec -it -- sh # run it by hand inside to see the real error message /app/entrypoint.sh ``` If you don't want to touch the original image, attaching an ephemeral container with `kubectl debug` is also a good option. ```bash kubectl debug -it --image=busybox:1.36 --target=app -- sh ``` If the container dies too fast to attach with `--target`, make a copy of the Pod with the command replaced by a shell. The original Pod stays as it is, and you can run the entrypoint by hand inside the copy ([official debugging docs](https://kubernetes.io/docs/tasks/debug/debug-application/debug-running-pod/)). ```bash kubectl debug -it --copy-to=-debug --container=app -- sh # delete the copy when you're done kubectl delete pod -debug ``` > **A note from the field:** In overnight incident response, the CrashLoopBackOff causes I ran into most often were "wrong env var or secret key" and "OOMKilled". So when I get paged, I always start with the Exit Code from `describe`. 137 means memory, a config-related message means Secrets/ConfigMaps, otherwise I go straight to `logs --previous`. ## Conclusion: diagnostic checklist 1. `kubectl get pod` → check RESTARTS count and STATUS 2. `kubectl describe pod` → check Events for `Back-off restarting...` and the Exit Code 3. `kubectl logs --previous` → capture the dead container's logs 4. Branch on Exit Code: 137 → memory, 1 → app exception, secret/env → config, probe → probes 5. If there are no logs, override with `command: sleep` or exec in via `kubectl debug` For ImagePullBackOff (image never pulled), see **Part 2**. For Pending (stuck at scheduling), see **Part 1**. If the container is up but you can't reach it from outside, that continues in **Part 4: troubleshooting Service/Endpoint connection failures**. ## References: official docs The primary source for the behavior, settings, and errors covered in this article is the official documentation below. Check it for version-specific options and exact behavior. - [Pod lifecycle: container restarts and backoff](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-restarts) - [Debug running Pods (kubectl debug)](https://kubernetes.io/docs/tasks/debug/debug-application/debug-running-pod/) - [Configure liveness, readiness, and startup probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/) - [Resource management for containers (OOMKilled)](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) - [Secrets](https://kubernetes.io/docs/concepts/configuration/secret/) · [ConfigMaps](https://kubernetes.io/docs/concepts/configuration/configmap/) - [Probe API reference (field defaults)](https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#Probe) - [Bash exit status (126, 127)](https://www.gnu.org/software/bash/manual/html_node/Exit-Status.html) · [Docker multi-platform builds](https://docs.docker.com/build/building/multi-platform/) ## FAQ **Q. I ran `kubectl logs --previous` and got "previous terminated container not found".** A. Either the container hasn't restarted yet (right after the first crash), or the node already cleaned up the previous container. Wait until RESTARTS increments and try again, or check the termination reason in the `Last State` block of `kubectl describe`. **Q. Exit Code is 137, but `kubectl top pod` shows memory below the limit. Why did it OOM?** A. It may have spiked over the limit and died, and you're looking at the post-death measurement. `top` only shows the current value, so check the peak of `container_memory_working_set_bytes` with [Prometheus](/blog/uptime-kuma-vs-netdata-vs-prometheus-소규모-서버-모니터링-추천) or similar and raise the limit above that. Because 137 is SIGKILL, also consider an external kill (e.g. node resource pressure). **Q. In CrashLoopBackOff, the restart interval keeps getting longer. Is that normal?** A. Yes. The kubelet doubles the restart delay (10 s, 20 s, 40 s, …) and caps it at 300 s (5 minutes). Once a container runs for 10 minutes without problems, the backoff timer resets; the RESTARTS number itself stays ([Pod lifecycle docs](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-restarts)). From 1.35, the `KubeletCrashLoopBackOffMax` feature (beta, enabled by default) lets you tune the maximum delay per node ([feature gate list](https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/)). A long interval does not mean a worse outage — focus on the **exit code and logs**, not the gap.

#kubectl logs

Solving CrashLoopBackOff: Diagnosing 7 Causes of Infinite Pod Restarts