When a Kubernetes Pod Gets Stuck in Terminating: 5 Causes and Safe Force Deletion

When a Kubernetes Pod Gets Stuck in Terminating: 5 Causes and Safe Force Deletion

You ran kubectl delete pod, but minutes later STATUS still hasn't budged from Terminating. The CI pipeline stalls, ArgoCD never finishes syncing, and the on-call pager goes off. This article is part 5 of the K8s_Troubleshooting_Guide series, focused entirely on this "stuck in Terminating forever" situation.

📌 In parts 1–4 we covered startup/runtime states such as Pending (unschedulable), CrashLoopBackOff (container restart loop), and ImagePullBackOff. This part is the opposite: the termination phase only, when a Pod is dying. The symptoms can look similar, but the diagnostic points are completely different—don't mix them up.

kubectl delete is a request, not a command

The first thing to understand is that kubectl delete pod is not a command that immediately kills the Pod. It is only a request to the API server to set the Pod's metadata.deletionTimestamp field. Once deletionTimestamp is set, this lifecycle starts:

1. deletionTimestamp is set and the Pod is shown as Terminating
2. kubelet runs the preStop hook on the container and sends SIGTERM
3. It waits for terminationGracePeriodSeconds (default 30 seconds) for shutdown
4. When time expires, it force-kills with SIGKILL
5. All finalizers must be removed before the API server actually deletes the Pod object

So if Terminating never finishes, something is stuck in one of these five steps. If you hit --force without knowing the cause, you can corrupt data in a StatefulSet. The key is first figuring out where it is stuck.

Diagnostic matrix for the 5 causes

Diagnostic routine checklist

Before you hit --force, spend one minute checking the following in order. (Again: this diagnosis is Terminating-state only.)

# 1. deletionTimestamp가 정말 찍혀 있는지 (= 삭제 요청은 들어갔는지)
kubectl get pod <name> -o jsonpath='{.metadata.deletionTimestamp}'

# 2. finalizer가 남아있는지 (원인 3의 핵심)
kubectl get pod <name> -o yaml | grep -A5 finalizers

# 3. grace period 설정값 확인
kubectl get pod <name> -o jsonpath='{.spec.terminationGracePeriodSeconds}'

# 4. 이벤트로 preStop / SIGKILL 흐름 확인
kubectl describe pod <name>

# 5. Pod가 떠 있던 노드의 상태 확인 (원인 4)
kubectl describe node <node> | grep -A5 Conditions

# 6. 볼륨 detach 잔류 확인 (원인 5)
kubectl get volumeattachment | grep <pv-name>

How to decide: If the finalizers array has values → cause 3. If the node Condition is Ready=Unknown → cause 4. If both are clean and it holds exactly for the grace period → causes 1 and 2. If a VolumeAttachment remains → cause 5.

Safe force-deletion procedure

Once diagnosis has narrowed the cause, it's time to clean up. First understand exactly what the command does.

kubectl delete pod <name> --grace-period=0 --force

This command sets the grace period to 0 and immediately removes the object from the API server. The problem is that if the node is still alive, the container may still be running, while the control plane believes the Pod is gone.

⚠️ Warning — do not casually use --force on a StatefulSet. StatefulSets guarantee unique IDs and PVCs, like web-0, web-1. If you force-delete while the node is still alive, the controller will start a Pod with the same ID on a new node while the old container may still be writing to the same volume. Concurrent writes from two instances cause split-brain and data corruption. This is especially fatal for stateful workloads like DBs, Kafka, and etcd.

When a leftover finalizer is the cause

Finalizers often linger due to controller bugs or GitOps environments (ArgoCD and similar), stalling sync. The correct procedure is to recover the controller responsible for the finalizer, but if the controller is already gone, you need to remove it manually.

kubectl patch pod <name> -p '{"metadata":{"finalizers":null}}' --type=merge

⚠️ Warning — removing a finalizer means "skipping cleanup logic." Finalizers are usually a safety mechanism that blocks object deletion until post-processing finishes, such as "volume detach complete" or "deregister from an external LB." If you force-remove them, the object disappears with that post-processing skipped. Always confirm what the finalizer is for before removing it.

When the node is NotReady (cause 4)

This is where the most common mistake happens. If the node is NotReady, kubelet cannot talk to the control plane and cannot clean up the Pod—but the container may still be running on that node. The correct order is:

1. Try to recover the node first (network, kubelet restart). Once recovered, the Pod will clean itself up.
2. If recovery is impossible and you are sure the node is dead → kubectl delete node <node> to remove the node, which will also clean up Pods on it.
3. If Kubernetes 1.30+'s Graceful Node Shutdown feature is enabled, kubelet terminates Pods gracefully first on a planned node shutdown, which reduces this problem in the first place.

💬 A note from production: In incident retrospectives, more than half of our "stuck Terminating" cases were node failures (cause 4). On-call engineers habitually hit --force first, and we had two data-consistency incidents on stateful workloads. After we made "before force, spend 3 seconds checking finalizers and node status with kubectl get pod -o yaml" a mandatory runbook step, the recurrence stopped. One line of diagnosis protects your data.

Preventing recurrence: design for graceful shutdown

Force deletion is first aid only. The real fix is designing the Pod so it shuts down cleanly on SIGTERM.

Before — no grace period or preStop configured (killed roughly with SIGKILL):

spec:
  containers:
    - name: api
      image: myapp:1.0
      # terminationGracePeriodSeconds 없음 → 기본 30초
      # preStop 없음 → LB가 트래픽 보내는 중에 종료될 수 있음

After — graceful shutdown applied:

spec:
  terminationGracePeriodSeconds: 45   # 앱 종료 시간 + 여유
  containers:
    - name: api
      image: myapp:1.0
      lifecycle:
        preStop:
          exec:
            # LB가 엔드포인트에서 빠질 시간 확보 후 종료
            command: ["/bin/sh", "-c", "sleep 5"]

Three core principles:

• Implement a SIGTERM handler in the app: reject new requests and finish in-flight requests before exiting
• Tune terminationGracePeriodSeconds: slightly longer than the app's actual cleanup time. Too long and force deletion takes forever; too short and you get cut off by SIGKILL
• Use preStop to allow time for LB deregistration: something like sleep 5 so endpoint removal can propagate

Conclusion

Stuck Terminating looks like one symptom—"the Pod won't delete"—but the causes split into five paths: ignored SIGTERM, delayed preStop, leftover finalizer, node NotReady, and volume detach failure. --force is not a magic button; it is a last-resort knife you use after you know the cause. Put diagnose → safe cleanup → graceful shutdown design into your runbook.

Next, part 6 covers troubleshooting OOMKilled, when a container dies after exceeding its memory limit. We'll dig into requests/limits tuning and the cgroup perspective.

References: official docs

The primary sources for the behavior, settings, and errors covered in this article are the following official docs. Check version-specific options and exact behavior there.

• Kubernetes official documentation

FAQ

Q. Does --grace-period=0 --force wipe data? A. The command itself does not delete data. But if you force-delete while the node is still alive, the container may keep writing to the volume while the same Pod is started elsewhere, causing corruption from concurrent writes. The risk is not the command—it is split-brain.

Q. Why shouldn't you force-delete on a StatefulSet? A. StatefulSets guarantee a single instance of a given identity and PVC. Force-deleting while the node is alive can make the controller start a new Pod with the same ID and volume, so two instances coexist. For a DB, that breaks data consistency. Proceed only when you are sure the old container is dead.

Q. The node is NotReady and the Pod is Terminating. What now? A. kubelet cannot communicate, so it cannot clean up. Try recovering the node first; if the node is definitely dead, remove it with kubectl delete node. Force-deleting only the Pod without confirming whether the node is alive is the most dangerous move.

Q. Can I just delete the finalizer? A. Finalizers guarantee post-processing such as volume detach and external resource cleanup. If you set finalizers:null without knowing what they are for, that post-processing is skipped. Recovering the controller is first priority; manual removal is a last resort when the controller is already gone.