# When a Kubernetes Pod Gets Stuck in Terminating: 5 Causes and Safe Force Deletion You ran `kubectl delete pod`, but minutes later STATUS still hasn't budged from `Terminating`. The CI pipeline stalls, ArgoCD never finishes syncing, and the on-call pager goes off. This article is **part 5 of the K8s_Troubleshooting_Guide series**, focused entirely on this "stuck in Terminating forever" situation. > 📌 In parts 1–4 we covered **startup/runtime states** such as `Pending` (unschedulable), `CrashLoopBackOff` (container restart loop), and `ImagePullBackOff`. This part is the opposite: **the termination phase only**, when a Pod is dying. The symptoms can look similar, but the diagnostic points are completely different—don't mix them up. ## kubectl delete is a request, not a command The first thing to understand is that `kubectl delete pod` is **not** a command that immediately kills the Pod. It is only a request to the API server to set the Pod's `metadata.deletionTimestamp` field. Once deletionTimestamp is set, this lifecycle starts: 1. `deletionTimestamp` is set and the Pod is shown as `Terminating` 2. kubelet runs the `preStop` hook on the container and sends `SIGTERM` 3. It waits for `terminationGracePeriodSeconds` (default 30 seconds) for shutdown 4. When time expires, it force-kills with `SIGKILL` 5. **All `finalizers` must be removed** before the API server actually deletes the Pod object So if Terminating never finishes, something is stuck in one of these five steps. If you hit `--force` without knowing the cause, you can corrupt data in a StatefulSet. The key is first figuring out **where it is stuck**. ## Diagnostic matrix for the 5 causes | # | Cause | Symptoms | Diagnostic command | Key YAML field | Resolution | |---|------|------|-----------|----------------|-----------| | 1 | **Process ignores SIGTERM** | Survives exactly for the grace period (30s), then disappears | Check shutdown logs with `kubectl logs ` | `spec.terminationGracePeriodSeconds` | Add a signal handler in the app; check for PID 1 issues | | 2 | **preStop hook delayed/hung** | Stays Terminating past the grace period | `kubectl describe pod` events, `lifecycle.preStop` | `spec.containers[].lifecycle.preStop` | Shorten the hook timeout; check the sleep value | | 3 | **Leftover finalizer** | deletionTimestamp is set but the Pod never disappears | `kubectl get pod -o yaml \| grep -A5 finalizers` | `metadata.finalizers` | Recover the controller, or remove the finalizer manually | | 4 | **Node [NotReady](/blog/kubectl-get-nodes-notready-원인-6가지-진단복구-5분-가이드) (kubelet unreachable)** | Many Pods on a specific node stuck Terminating | `kubectl describe node ` | `status.conditions` (Ready=Unknown) | Recover the node first; force-delete only if that fails | | 5 | **Volume detach failure** | Only Pods with PVCs won't disappear; VolumeAttachment remains | `kubectl get volumeattachment` | `status` / CSI events | Check CSI driver and storage status | ## Diagnostic routine checklist Before you hit `--force`, spend one minute checking the following in order. (Again: this diagnosis is **Terminating-state only**.) ```bash # 1. deletionTimestamp가 정말 찍혀 있는지 (= 삭제 요청은 들어갔는지) kubectl get pod -o jsonpath='{.metadata.deletionTimestamp}' # 2. finalizer가 남아있는지 (원인 3의 핵심) kubectl get pod -o yaml | grep -A5 finalizers # 3. grace period 설정값 확인 kubectl get pod -o jsonpath='{.spec.terminationGracePeriodSeconds}' # 4. 이벤트로 preStop / SIGKILL 흐름 확인 kubectl describe pod # 5. Pod가 떠 있던 노드의 상태 확인 (원인 4) kubectl describe node | grep -A5 Conditions # 6. 볼륨 detach 잔류 확인 (원인 5) kubectl get volumeattachment | grep ``` **How to decide**: If the finalizers array has values → cause 3. If the node Condition is `Ready=Unknown` → cause 4. If both are clean and it holds exactly for the grace period → causes 1 and 2. If a VolumeAttachment remains → cause 5. ## Safe force-deletion procedure Once diagnosis has narrowed the cause, it's time to clean up. First understand exactly what the command does. ```bash kubectl delete pod --grace-period=0 --force ``` This command sets the grace period to 0 and **immediately removes the object from the API server**. The problem is that if the node is still alive, the container may still be running, while the control plane believes the Pod is gone. > ⚠️ **Warning — do not casually use `--force` on a StatefulSet.** > StatefulSets guarantee **unique IDs and PVCs**, like `web-0`, `web-1`. If you force-delete while the node is still alive, the controller will start a Pod with the same ID on a new node **while the old container may still be writing to the same volume**. Concurrent writes from two instances cause split-brain and data corruption. This is especially fatal for stateful workloads like DBs, Kafka, and etcd. ### When a leftover finalizer is the cause Finalizers often linger due to controller bugs or GitOps environments (ArgoCD and similar), stalling sync. The correct procedure is to **recover the controller responsible for the finalizer**, but if the controller is already gone, you need to remove it manually. ```bash kubectl patch pod -p '{"metadata":{"finalizers":null}}' --type=merge ``` > ⚠️ **Warning — removing a finalizer means "skipping cleanup logic."** > Finalizers are usually a **safety mechanism that blocks object deletion until post-processing finishes**, such as "volume detach complete" or "deregister from an external LB." If you force-remove them, the object disappears with that post-processing skipped. Always confirm **what the finalizer is for** before removing it. ### When the node is NotReady (cause 4) This is where the most common mistake happens. If the node is `NotReady`, kubelet cannot talk to the control plane and cannot clean up the Pod—but **the container may still be running on that node.** The correct order is: 1. **Try to recover the node first** (network, kubelet restart). Once recovered, the Pod will clean itself up. 2. If recovery is impossible and you are sure the node is dead → `kubectl delete node ` to remove the node, which will also clean up Pods on it. 3. If Kubernetes 1.30+'s **Graceful Node Shutdown** feature is enabled, kubelet terminates Pods gracefully first on a planned node shutdown, which reduces this problem in the first place. > 💬 **A note from production**: In incident retrospectives, more than half of our "stuck Terminating" cases were **node failures (cause 4)**. On-call engineers habitually hit `--force` first, and we had two data-consistency incidents on stateful workloads. After we made "before force, spend 3 seconds checking finalizers and node status with `kubectl get pod -o yaml`" a mandatory runbook step, the recurrence stopped. One line of diagnosis protects your data. ## Preventing recurrence: design for graceful shutdown Force deletion is first aid only. The real fix is **designing the Pod so it shuts down cleanly on SIGTERM**. **Before — no grace period or preStop configured (killed roughly with [SIGKILL](/blog/npm-err-code-elifecycle-해결법-errno-1134sigkill-원인별-진단)):** ```yaml spec: containers: - name: api image: myapp:1.0 # terminationGracePeriodSeconds 없음 → 기본 30초 # preStop 없음 → LB가 트래픽 보내는 중에 종료될 수 있음 ``` **After — graceful shutdown applied:** ```yaml spec: terminationGracePeriodSeconds: 45 # 앱 종료 시간 + 여유 containers: - name: api image: myapp:1.0 lifecycle: preStop: exec: # LB가 엔드포인트에서 빠질 시간 확보 후 종료 command: ["/bin/sh", "-c", "sleep 5"] ``` Three core principles: - **Implement a SIGTERM handler in the app**: reject new requests and finish in-flight requests before exiting - **Tune terminationGracePeriodSeconds**: slightly longer than the app's actual cleanup time. Too long and force deletion takes forever; too short and you get cut off by SIGKILL - **Use preStop to allow time for LB deregistration**: something like `sleep 5` so endpoint removal can propagate ## Conclusion Stuck Terminating looks like one symptom—"the Pod won't delete"—but the causes split into five paths: **ignored SIGTERM, delayed preStop, leftover finalizer, node NotReady, and volume detach failure**. `--force` is not a magic button; it is **a last-resort knife you use after you know the cause**. Put diagnose → safe cleanup → graceful shutdown design into your runbook. Next, **part 6** covers troubleshooting `OOMKilled`, when a container dies after exceeding its memory limit. We'll dig into requests/limits tuning and the cgroup perspective. ## References: official docs The primary sources for the behavior, settings, and errors covered in this article are the following official docs. Check version-specific options and exact behavior there. - [Kubernetes official documentation](https://kubernetes.io/docs/home/) ## FAQ **Q. Does `--grace-period=0 --force` wipe data?** A. The command itself does not delete data. But if you force-delete while the node is still alive, the container may keep writing to the volume while the same Pod is started elsewhere, causing **corruption from concurrent writes**. The risk is not the command—it is split-brain. **Q. Why shouldn't you force-delete on a StatefulSet?** A. StatefulSets guarantee a **single instance** of a given identity and PVC. Force-deleting while the node is alive can make the controller start a new Pod with the same ID and volume, so two instances coexist. For a DB, that breaks data consistency. Proceed only when you are sure the old container is dead. **Q. The node is NotReady and the Pod is Terminating. What now?** A. kubelet cannot communicate, so it cannot clean up. **Try recovering the node first**; if the node is definitely dead, remove it with `kubectl delete node`. Force-deleting only the Pod without confirming whether the node is alive is the most dangerous move. **Q. Can I just delete the finalizer?** A. Finalizers guarantee post-processing such as volume detach and external resource cleanup. If you set `finalizers:null` without knowing what they are for, that post-processing is skipped. Recovering the controller is first priority; manual removal is a last resort when the controller is already gone.

#Pod Terminating

Kubernetes Pod가 Terminating에서 멈췄을 때: 원인 5가지와 안전한 강제 삭제