ImagePullBackOff diagnostic runbook

ImagePullBackOff / ErrImagePull means the kubelet could not fetch the container image. Unlike CrashLoopBackOff, the process often never started.

30-second checklist

1. kubectl get pod <name> -o wide — confirm ImagePullBackOff / ErrImagePull
2. kubectl describe pod <name> — copy the bottom Events (Failed / BackOff)
3. Validate the image string: registry/repo:tag, digest, avoid relying on latest
4. For private registries: imagePullSecrets, node IAM, network (firewall/DNS)
5. Reproduce on the worker with crictl pull / ctr images pull

Classify from Events

kubectl describe pod <pod> -n <ns> | sed -n '/Events:/,$p'
kubectl get events -n <ns> --field-selector involvedObject.name=<pod> --sort-by=.lastTimestamp

Diagnostic commands

kubectl get pod <pod> -n <ns> -o jsonpath='{range .spec.containers[*]}{.name}{"\t"}{.image}{"\n"}{end}'
kubectl get pod <pod> -n <ns> -o jsonpath='{.spec.imagePullSecrets[*].name}{"\n"}'

IMAGE=$(kubectl get pod <pod> -n <ns> -o jsonpath='{.spec.containers[0].image}')
NODE=$(kubectl get pod <pod> -n <ns> -o jsonpath='{.spec.nodeName}')
kubectl debug node/$NODE -it --image=busybox -- chroot /host crictl pull "$IMAGE"

Fixes by cause

Wrong tag or registry

Pin a real tag or digest. Do not redeploy on a floating latest without verification.

Private registry auth

kubectl create secret docker-registry regcred \
  --docker-server=ghcr.io \
  --docker-username=<user> \
  --docker-password=<token> \
  -n <ns>

Attach via Pod imagePullSecrets or the ServiceAccount. On EKS, confirm the node role / IRSA can call ECR GetAuthorizationToken and BatchGetImage.

Network and DNS

• getent hosts registry.example.com on the node
• Configure containerd proxy if required
• Trust private CAs on the node

Rate limits

Authenticate pulls, use a mirror, or a pull-through cache.

Prevent recurrence

• Pin digests in production
• CI verifies image existence before rollout
• Prefer digest over mutable tags with IfNotPresent
• Cluster-level registry mirror/cache

ImagePullBackOff is a supply-path failure, not an app crash. One clear Events line usually splits the tree in minutes.