/인프라/7 Causes of Kubernetes Pod Pending: A Practical Guide to Complete Diagnosis with kubectl describe
InfrastructureKubernetesDevOps

7 Causes of Kubernetes Pod Pending: A Practical Guide to Complete Diagnosis with kubectl describe

When a Pod is stuck in Pending and causing a service outage, stop guessing. This practical guide uses kubectl describe and a 3-step checklist to systematically diagnose and resolve every Pending cause—from resource shortages to Taint/Tolera

7 Causes of Kubernetes Pod Pending: A Practical Guide to Complete Diagnosis with kubectl describe

Stop Guessing About Kubernetes Pod Pending: 7 Causes and a Real-Time Diagnosis Guide

If you run services on Kubernetes, you have almost certainly seen a Pod stuck in Pending. The deploy looked successful and the application logs looked fine, but no traffic was reaching the service. That Pending state feels like the system's heart has stopped, and it steals the golden window for incident response.

Pending is more than “hasn’t started yet.” It is an important signal that the Kubernetes scheduler hit a constraint while trying to place the Pod on a node. This article replaces vague guessing with a systematic diagnosis process: it breaks Pending root causes into 7 scenarios and gives a practical guide you can apply immediately in production.

Step 1: The magic command that uncovers the status — kubectl describe

When a Pod is Pending, the first thing to do is not rely on gut feel, but read the Events the system recorded. Those logs contain the decisive clues about why the scheduler refused to place the Pod.

Bash
kubectl describe pod [pod-name] -n [namespace]

Focus on the Events: section of the output. You will find clear error messages such as FailedScheduling, Insufficient CPU, or Node(s) had taint {key: value} that the Pod cannot tolerate. That event message is the first clue to the problem you need to solve.

Step 2: 3-step checklist for diagnosing Pod Pending (diagnosis flow)

When a Pod is Pending, randomly digging through every setting is inefficient. Checking in this 3-step logical flow is the fastest and most accurate approach.

StepArea to checkMain causeCommand / log to check
Step 1Resource constraintsInsufficient allocatable CPU/Memory on nodesEvents in kubectl describe pod, Insufficient... messages
Step 2Policy conflictsTaint and Toleration mismatchkubectl describe node and the Pod Spec
Step 3Scheduling constraintsNode Selector, Affinity, or PriorityClass mismatchNode-Selectors section of kubectl describe pod

If you check in this order, most Pending problems are resolved in steps 1–2.

Step 3: Diagnosing resource constraints

One of the most common causes is a shortage of physical or logical resources on nodes. It happens when no node in the cluster can satisfy the resources the Pod requested (Requests).

🚨 Example of the key error message:

CODE
Events:
  Type     Reason     From                Message
  ----     ------     ----                -------
  Warning  Failed     kube-scheduler     Pod cannot be scheduled: insufficient cpu

This message means the scheduler cannot find the CPU the Pod requested on any node in the current cluster.

💡 Remediation:

  1. Scale Down: Delete unnecessary Pods, or reduce the replica count of deployed Pods.
  2. Adjust Resource Requests: Lower the requests values in the Pod Spec to match current node capacity.
  3. Check Cluster Autoscaler: In a cloud environment, verify that the Autoscaler that adds nodes when capacity is short is actually working.

Step 4: Policy conflicts (Taints & Tolerations)

One of the trickiest but most important Kubernetes concepts is Taints and Tolerations. A Taint marks a node with a constraint (“this node should only be used for a specific purpose”), and a Toleration is the Pod saying it is willing to accept that constraint.

❌ Wrong scenario: If a node (Node-A) has the taint dedicated=gpu:NoSchedule and the Pod does not declare a toleration that ignores that taint, the scheduler will refuse to place the Pod on Node-A.

✅ Fix snippet (edit the Pod Spec): To let the Pod ignore the taint and be placed on that node, add the following to the Pod spec:

YAML
spec:
  tolerations:
  - key: "dedicated"
    operator: "Equal"
    value: "gpu"
    effect: "NoSchedule"

Step 5: Scheduling constraints (Affinity & Selector)

If a Pod is restricted to a specific group of nodes (Node Affinity, Node Selector) and no node in the cluster satisfies those conditions, it stays Pending.

  • Node Selector: Requires the Pod to be scheduled only on nodes with a specific label. (The simplest restriction)
  • Node Affinity: Used for more complex conditions (for example, schedule only on 3 or more nodes that have a certain label).

If you used NodeSelector and no node has that label, the Pod will stay Pending forever. Get in the habit of checking actual node labels first with kubectl get nodes --show-labels.


🛠️ Tip from the field: Automating incident response

In real production, having humans repeatedly run kubectl describe itself becomes a bottleneck. I strongly recommend building a process—especially if you already have a GitOps workflow—that automatically fires an alert when a Pod goes Pending, captures that Pod’s describe output, and attaches it to a Jira ticket. In other words, move incident response from “manual diagnosis” to “automated observability.”

💡 Final check: Pod Pending troubleshooting checklist

  1. [Required] Run kubectl describe pod [pod-name] and inspect the Events section. (Look here first)
  2. Resources: Confirm CPU/Memory requests and limits (Requests/Limits) are not too large or missing.
  3. Policy: Check whether the namespace has restriction policies such as LimitRange or ResourceQuota.
  4. Policy: Check whether scheduling constraints such as PodDisruptionBudget or PodAntiAffinity are causing conflicts.

This checklist will resolve most scheduling problems.

References: Official docs

The primary source for the behavior, configuration, and errors covered in this article is the official documentation below. Check version-specific options and exact behavior there.

확인 정보
✦ ✦ ✦
편집 검토 · Editorial Review

Nodelog는 모든 콘텐츠의 내용과 출처를 공개 전에 검토합니다. 환경(OS·버전)에 따라 결과가 달라질 수 있는 기술 정보는 공식 문서와 함께 확인하며, 검토 기준과 정정 원칙은 편집 정책에서 안내합니다. 오류를 발견하시면 이메일로 제보해 주세요 — 확인 후 신속히 정정합니다.

편집 책임 · Nodelog 기술 편집팀·발행 · ·업데이트 ·
관련 공식 문서Kubernetes 공식 문서

Comments

Be the first to comment.