Fix ImagePullBackOff · ErrImagePull: Diagnose the Cause from Events and Finish with Copy-Paste

Fix ImagePullBackOff · ErrImagePull: Diagnose the Cause from Events Messages and Finish with Copy-Paste

K8s_Troubleshooting_Guide Part 17

If the deploy pipeline is green but the Pod is stuck on ImagePullBackOff again, you've already cleared PVCs and network policies. You're blocked at the next step: kubelet pulling the image from the registry. This post focuses on that one step. The goal is to look at a single Events line from kubectl describe pod, pinpoint which of seven causes it is, and recover in five minutes with copy-paste commands for each cause.

ImagePullBackOff vs ErrImagePull: They're the Same Problem

Let's clear up these two confusing statuses first. They aren't separate problems — they're stages of the same failure.

• ErrImagePull: The state right after kubelet first tries to pull the image and fails
• ImagePullBackOff: Failures have repeated, so kubelet is waiting while exponentially increasing the retry interval (back-off)

In other words, ErrImagePull shows up briefly, then hardens into ImagePullBackOff. Both have the same root cause and the same diagnostic method. Don't obsess over the status name — go straight to Events.

Step 1: Diagnosis — How to Read Events Messages

90% of causes are written in the last Events line. These two commands are enough.

#
kubectl get events --sort-by=.lastTimestamp -n <namespace>

# Pod Events
kubectl describe pod <pod> -n <namespace> | grep -A10 Events

Match the message text on the Warning Failed line against the diagnosis table below, as-is.

Cause Diagnosis Table

With this one table, it's "see the message, go here."

Step 2: Fixes — Copy-Paste Commands by Cause

Cause 1: Wrong Tag or Missing Tag

If you see not found, it's the most common — and most anticlimactic — cause. First check whether that tag actually exists in the registry.

# manifest check
kubectl get pod <pod> -n <ns> -o jsonpath='{.spec.containers[*].image}'

#
docker pull <registry>/<image>:<tag>

If it was a typo, replace it with kubectl set image deployment/<deploy> <container>=<image>:<correct-tag> and you're done.

Cause 2: Missing Auth — imagePullSecrets

If you see unauthorized, you don't have credentials. Create a secret and attach it to the Pod.

# 1) check
docker login <registry>

# 2) docker-registry when (when !)
kubectl create secret docker-registry regcred \
  --docker-server=<registry> \
  --docker-username=<user> \
  --docker-password=<password> \
  --docker-email=<email> \
  -n <namespace>

Attach the secret to the Pod (or the Deployment's podTemplate).

spec:
  imagePullSecrets:
    - name: regcred
  containers:
    - name: app
      image: <registry>/<image>:<tag>

If you don't want to add it to every workload, attach it once to the ServiceAccount and apply it across the namespace.

kubectl patch serviceaccount default -n <namespace> \
  -p '{"imagePullSecrets":[{"name":"regcred"}]}'

Cause 3: Docker Hub Rate Limit

Docker Hub limits image pulls per 6-hour window. Per the official docs (checked 2026-10-04), unauthenticated users get 100 pulls per IPv4 address or IPv6 /64 subnet per 6 hours, authenticated Personal accounts get 200, and pulls are unlimited on Pro, Team and Business plans. Going over the limit returns toomanyrequests. Because the unauthenticated limit is per IP, clusters and CI runners behind the same NAT IP share it and get hit especially hard. There are two ways to respond.

(A) Pull with an authenticated account — Once authenticated, the limit applies per account instead of per IP (Personal: 200 per 6 hours; unlimited on Pro, Team and Business plans). Create the regcred above with a Docker Hub account and attach it to the ServiceAccount for immediate relief.

(B) Switch to a mirror / pull-through cache — This is the real fix. Stand up a Harbor proxy cache or ECR pull-through cache and change only the image path.

# : nginx:1.27 (Docker Hub )
# : <account>.dkr.ecr.<region>.amazonaws.com/dockerhub/library/nginx:1.27
image: harbor.mycorp.com/dockerhub-proxy/library/nginx:1.27

If changing image paths is hard, you can point containerd on each node at a Docker Hub mirror instead. With config_path set (see Cause 6 below), put a hosts.toml in the docker.io directory.

# /etc/containerd/certs.d/docker.io/hosts.toml
server = "https://registry-1.docker.io"

[host."https://mirror.example.com"]
  capabilities = ["pull", "resolve"]

The mirrors listed under host are tried first, and server is used as the fallback if they all fail (containerd registry configuration). This is per-node configuration, so bake it into the node image or bootstrap script.

War story: On one cluster, a nightly batch spun up dozens of Pods at once and they all died with toomanyrequests. Attaching an authenticated account put out the fire, but recurrence only stopped after we installed a Harbor proxy cache. If CI and production share the same egress IP, a cache isn't optional — it's required.

Cause 4: Registry Host Resolution Failure

lookup ... no such host means kubelet could not resolve the registry domain via DNS. (General DNS troubleshooting is covered in a separate post, so here we only look at it from the registry-host angle.) Check directly on the node.

#
nslookup <registry-host>
curl -v https://<registry-host>/v2/

For an internal registry, check whether the node /etc/hosts or private DNS has the record, and whether private endpoint routing is in place.

Cause 5: Node Disk Full

no space left on device means there isn't enough disk space to pull image layers.

# check
kubectl get pod <pod> -n <ns> -o wide

# check
kubectl describe node <node> | grep -A5 Conditions # DiskPressure check
# after
df -h /var/lib/containerd # or /var/lib/docker
crictl rmi --prune #

When image-filesystem usage exceeds imageGCHighThresholdPercent, the kubelet deletes unused images to bring usage down to imageGCLowThresholdPercent (garbage collection docs). If that cleanup fails, the node records an ImageGCFailed event, and once free space falls below the eviction threshold the node gets DiskPressure, which blocks new Pods from being scheduled there (node-pressure eviction docs). crictl rmi --prune is a stopgap; if it keeps happening, fix node disk size or image size.

kubectl get events -A --field-selector reason=ImageGCFailed

Cause 6: x509 — Private CA Not Trusted

failed to pull and unpack image ... x509: certificate signed by unknown authority

This is not a credentials problem: containerd on the node does not trust the CA that issued the registry certificate. It shows up mostly with private registries such as Harbor or Nexus whose certificates come from an internal PKI. On containerd nodes, per-registry settings live in hosts.toml files under certs.d.

First check that /etc/containerd/config.toml sets config_path. The section name depends on the containerd version.

# containerd 2.x (config version 3)
[plugins."io.containerd.cri.v1.images".registry]
  config_path = "/etc/containerd/certs.d"

# containerd 1.x (config version 2)
[plugins."io.containerd.grpc.v1.cri".registry]
  config_path = "/etc/containerd/certs.d"

Create a directory named after the registry host and put the CA certificate and hosts.toml in it.

# /etc/containerd/certs.d/harbor.example.com/hosts.toml
server = "https://harbor.example.com"

[host."https://harbor.example.com"]
  capabilities = ["pull", "resolve"]
  ca = "/etc/containerd/certs.d/harbor.example.com/ca.crt"
  # skip_verify = true  ← disables verification. Use only to confirm a repro; never leave it in production

sudo cp internal-root-ca.crt /etc/containerd/certs.d/harbor.example.com/ca.crt
sudo systemctl restart containerd   # required if you changed config.toml

The format follows containerd hosts.md. The old registry.mirrors / registry.configs style is deprecated, so don't mix the two; consolidate on config_path. On nodes running Docker Engine, put the CA at /etc/docker/certs.d/<registry host>/ca.crt, and include the port in the directory name if the registry uses one (e.g. registry.example.com:5000, Docker docs).

If x509 happens on only one node: check which node the failing Pod landed on with kubectl get pod -o wide. A newly added node missing the CA / hosts.toml rollout is a common cause. Distribute the same files to every node through the node image, bootstrap script, or configuration management, and add it to your new-node checklist.

Cause 7: no match for platform — Architecture Mismatch

failed to pull and unpack image ...: no match for platform in manifest: not found

The tag exists, but it has no image for the node architecture (amd64/arm64). You see this when you deploy an amd64-only image to a cluster that mixes in arm64 nodes.

kubectl get nodes -L kubernetes.io/arch
docker buildx imagetools inspect <registry>/<image>:<tag>   # platforms included

Either rebuild as a multi-platform image (docker buildx build --platform linux/amd64,linux/arm64 ... --push, Docker docs) or use a nodeSelector so the Pod only lands on nodes whose architecture the image supports.

spec:
  nodeSelector:
    kubernetes.io/arch: amd64

Step 3: Common Pitfalls and Cloud Registries

Pitfall 1: Secret Namespace Mismatch

This is the landmine people step on most. If you create the secret only in default while the app runs in the prod namespace, auth will never apply. imagePullSecrets only reference secrets in the same namespace as the Pod.

# prod ns
kubectl create secret docker-registry regcred ... -n prod

Pitfall 2: The Debugging Hell of latest + imagePullPolicy: Always

If you use the latest tag with an Always policy, every restart can pull a different image, which produces unreproducible failures of the "it worked yesterday, not today" kind. The recommended strategy is immutable tags.

• ❌ myapp:latest + imagePullPolicy: Always
• ✅ myapp:1.4.2 or myapp:git-a1b2c3d (commit/build-based pinned tag)

With immutable tags, imagePullPolicy: IfNotPresent is also safe, and cache hit rates go up.

Pitfall 3: The registry address stored in the secret doesn't match the image host

If you attached imagePullSecrets and still get unauthorized, check which registry address the secret actually contains.

kubectl get secret regcred -n <namespace> \
  -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d | jq '.auths | keys'

The keys under auths (registry addresses) must match the host in the manifest's image:. For a registry on a port such as harbor.example.com:8443, the port must match too, and it's safest to pass only the host, without a scheme, to --docker-server. Printing the whole document without jq shows the base64-encoded credentials (auth), so check only the keys.

Reproduce on the node: bypass the kubelet with crictl

After a fix, reproducing the pull directly on the node tells you whether the cause is on the node side or in the Kubernetes configuration.

# from a node shell
sudo crictl pull --creds '<user>:<password>' harbor.example.com/app/image:v1.2.3

If crictl pull succeeds, the node's CA, network, and credentials are fine, and what remains is the secret wiring (namespace, ServiceAccount) or the manifest. If crictl pull also fails, a node-level problem such as Cause 4 or 6 is still there. Clear your shell history afterwards so the credentials don't linger.

Auto-Renewing Expired Tokens per Cloud

Managed registries have tokens that expire. If auth that worked yesterday turns into unauthorized today, suspect token expiry.

If you use ECR with a static secret (regcred), it will always break after 12 hours. Prefer keyless auth with IRSA / Workload Identity; if that's hard, add a token-refresh CronJob.

# ECR CronJob (11when)
schedule: "0 */11 * * *"
# command example
# kubectl delete secret regcred -n app --ignore-not-found
# kubectl create secret docker-registry regcred \
#   --docker-server=<acct>.dkr.ecr.<region>.amazonaws.com \
#   --docker-username=AWS \
#   --docker-password=$(aws ecr get-login-password --region <region>) -n app

Conclusion: Recurrence-Prevention Checklist

• ✅ First action is always kubectl describe pod | grep -A10 Events
• ✅ Create the secret in the same namespace as the app, attach it to the ServiceAccount to propagate
• ✅ Ban latest; standardize on immutable tags + IfNotPresent
• ✅ Instead of pulling Docker Hub directly, use an authenticated account + pull-through cache
• ✅ For ECR/AR/ACR, prefer IRSA/Workload Identity; if you use a static secret, automate refresh
• ✅ Put private-registry CAs and hosts.toml into node provisioning so new nodes get them automatically
• ✅ In mixed-architecture clusters, use multi-platform images or a kubernetes.io/arch nodeSelector

In Part 18, we cover the next stage: the image pulled fine and the container started, but it infinitely restarts with CrashLoopBackOff — how to split the cause using logs, exit codes, and probes.

References: Official Docs

The primary sources for the behavior, settings, and errors covered in this post are the official docs below. Check them for version-specific options and exact behavior.

• Images, imagePullPolicy, and pull back-off
• Pull an image from a private registry (imagePullSecrets)
• Garbage collection: containers and images
• containerd registry host configuration (hosts.toml)
• Docker: registry certificates

FAQ

Q. Which is more serious, ImagePullBackOff or ErrImagePull? A. It's not a difference in severity — it's a difference in stage. ErrImagePull is immediately after a pull failure; ImagePullBackOff is waiting to retry after repeated failures. Cause and fix are identical, so go straight to the Events message.

Q. I definitely created imagePullSecrets, but I still get unauthorized. A. Nine times out of ten it's a namespace mismatch. The secret must be in the same namespace as the Pod to be referenced. Confirm it exists with kubectl get secret regcred -n <app-namespace>, and if you're on ECR, also suspect the 12-hour token expiry.

Q. Docker Hub toomanyrequests — is attaching auth enough? A. Auth accounts will ease an urgent situation, but if CI and production share the same egress IP, it will come back. The real fix is to put a mirror in front — Harbor proxy cache or ECR pull-through cache — and point image paths at the cache.

Sources · checked 2026-10-04

• Docker Docs, Docker Hub usage and limits — pull limits per 6 hours: 100 per IPv4 address or IPv6 /64 unauthenticated, Personal 200, unlimited on Pro, Team and Business plans
• containerd, Registry Configuration (hosts.md) — config_path section for containerd 2.x and 1.x, server/host/ca in hosts.toml, registry.mirrors/registry.configs deprecated (checked 2026-10-06)

Limits can change with Docker’s policy, so re-check this page before changing production settings.