Pod Pending FailedScheduling 0/3 nodes: A 30-Second Diagnosis and Recovery Runbook

Pod Pending FailedScheduling 0/3 nodes: A 30-Second Diagnosis and Recovery Runbook

K8s_Troubleshooting_Guide, Part 4. You deployed, but STATUS stays Pending and kubectl get events only shows FailedScheduling 0/3 nodes are available. This runbook takes that one line, splits the cause in 30 seconds, and recovers with copy-paste commands.

"It never becomes Running — it just stays Pending"

A Pod in Pending means the scheduler could not find a node that will accept this Pod. The container did not crash — it was never placed on a node at all. The scheduler rejects nodes for four main reasons:

1. Resource shortage — insufficient cpu / insufficient memory
2. Taint rejection — node(s) had untolerated taint
3. Label/affinity mismatch — didn't match Pod's node affinity/selector
4. Volume zone mismatch — had volume node affinity conflict

This post focuses on 1–3 (resource / node-matching rejections). Storage binding issues where the PVC itself is Pending are covered in a separate article. The one thing that matters is splitting "is there really no room (resources) vs. does it not match (taint/labels)".

30-second diagnosis table: branch on a single Events line

The first command to run is this one.

kubectl describe pod <pod> | grep -A20 Events

Map that one line to the table below.

You may also see multiple reasons separated by commas, like 0/3 nodes are available: 2 Insufficient cpu, 1 node(s) had untolerated taint. Adding the numbers gives the total node count (3 here). That means "2 nodes lack CPU, 1 has a taint." How to read this is covered in the FAQ.

Check available resources and node status

If you got a resource-shortage message, first confirm whether it is actually short. It is common for the scheduler to reject based on the sum of requests while actual usage still has plenty of headroom.

# Per-node requests/limits allocation (what the scheduler sees)
kubectl describe nodes | grep -A5 "Allocated resources"

# Actual usage (requires metrics-server)
kubectl top nodes

# Node labels at a glance (nodeSelector debugging)
kubectl get nodes --show-labels

# Inspect taints on a specific node
kubectl describe node <node> | grep Taints

If Allocated resources shows CPU Requests at 95% but kubectl top nodes shows actual usage at 30% — it is not that there is no room; requests are over-provisioned. In that case the fix is shrinking requests, not adding nodes.

Recovery commands by cause

① Shrink over-provisioned requests

The most common case: requests are so large relative to actual usage that scheduling is blocked.

kubectl patch deployment <name> --type='json' -p='[
  {"op":"replace","path":"/spec/template/spec/containers/0/resources/requests/cpu","value":"250m"},
  {"op":"replace","path":"/spec/template/spec/containers/0/resources/requests/memory","value":"256Mi"}
]'

Field tip: On an incident, I first drop requests to actual usage + 30% to get the Pod running, then after things stabilize I look at a few days of kubectl top pods data and set the real values. "Running first" → "tune later" is what determines recovery speed.

② Add a toleration for the taint

GPU/spot nodes are usually protected by taints. To place a Pod there on purpose, add a toleration.

spec:
  template:
    spec:
      tolerations:
      - key: "nvidia.com/gpu"
        operator: "Exists"
        effect: "NoSchedule"

③ Fix nodeSelector/affinity labels

The Pod requires disktype=ssd but no node has that label. Either label the node:

kubectl label nodes <node> disktype=ssd

or fix the manifest affinity to match the actual labels.

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: disktype
          operator: In
          values: ["ssd"]

④ Scale out nodes / check the Autoscaler

If you cannot shrink requests (they are already right-sized) and there truly is no room, scale out. If you have Cluster Autoscaler/Karpenter, check whether it is actually working.

kubectl -n kube-system logs deploy/cluster-autoscaler | tail -30

Decision tree: scale out vs. shrink vs. toleration

FailedScheduling occurs
   │
   ├─ Are resources actually short? (check with top nodes)
   │     ├─ Y → Can you shrink requests?
   │     │        ├─ Y → Shrink requests (①)
   │     │        └─ N → Scale out nodes / Autoscaler (④)
   │     └─ N (matching problem)
   │            ├─ Taint rejection? → Add toleration (②)
   │            └─ nodeSelector mismatch? → Fix labels (③)

Preventing recurrence

• LimitRange + default requests: Enforce default requests on the namespace so you avoid the two extremes of "requests unset → over/under-provisioned."
• 2026 trend: On K8s 1.30+, in-place Pod resize and Karpenter-based auto scale-out have become common. Auto scale-out still only works if requests are a trustworthy signal, so requests hygiene is the starting point for everything.

FAQ

Q. If I set requests to 0, will it always schedule? A. Yes. But it is dangerous. With requests at 0 the scheduler packs the Pod onto a node with no resource guarantee, causing over-commit. Under memory pressure that spreads into OOMKilled or a node going NotReady. Fine for a temporary recovery; set the real values from actual usage.

Q. Can I schedule Pods onto control-plane nodes? A. Control-plane nodes usually have the node-role.kubernetes.io/control-plane:NoSchedule taint. On a single-node test cluster you can add a toleration and schedule there, but it is not recommended in production. Keep dedicated worker nodes for workloads.

Q. What if reasons are mixed, like 0/3 nodes: 2 Insufficient cpu, 1 had taint? A. The numbers add up to the total node count (2+1=3). Each node was rejected for a different reason, so handle the largest share first. In this example CPU shortage on 2 nodes is the main cause, so try shrinking requests first; if that still fails, add a toleration for the tainted node.

Q. I have Cluster Autoscaler/Karpenter — why is it still Pending? A. Typical causes: ① the node group hit its max, ② Pod requests fit no instance type (e.g. 8 CPU requested but only 4 CPU nodes are provisioned), ③ spot capacity is exhausted, ④ the Autoscaler cannot find a node group that satisfies the nodeSelector/taint. Check Autoscaler logs first — you will see a concrete reason such as no node group can be scaled up.

Coming next — a diagnosis runbook for when a node goes NotReady (NodeNotReady · kubelet). Not Pending: the node itself is dying.