ImagePullBackOff/ErrImagePull: A Guide to Diagnosing and Fixing 6 Causes

A Complete Guide to Fixing kubectl ImagePullBackOff/ErrImagePull: 6 Causes

"Pending is solved—now it's ImagePullBackOff?"

In Part 1 (Diagnosing Pod Pending), we fixed Pods stuck in Pending because the scheduler couldn't find a node. Now the Pod is scheduled, but it fails while pulling the container image. That's ErrImagePull and ImagePullBackOff.

Let's start by clarifying the difference between the two states.

• ErrImagePull: The kubelet just attempted an image pull and failed immediately.
• ImagePullBackOff: After repeated pull failures, the kubelet is waiting with an increasing retry interval (back-off). In other words, it's the accumulated result of ErrImagePull.

Both share the same root cause: "the image cannot be pulled." We'll start with a diagnosis that narrows it down in about a minute.

Symptoms and Immediate Diagnosis: Read the Error Message

First, check the status.

kubectl get pods
# NAME                     READY   STATUS             RESTARTS   AGE
# web-7d9f8c6b5-abcde      0/1     ImagePullBackOff   0          2m

The key is Events. Use describe to inspect the latest messages.

kubectl describe pod web-7d9f8c6b5-abcde

Events:
  Type     Reason     Age   From     Message
  ----     ------     ----  ----     -------
  Normal   Pulling    2m    kubelet  Pulling image "myreg.io/app:v1.2"
  Warning  Failed     2m    kubelet  Failed to pull image "myreg.io/app:v1.2":
                                     rpc error: code = ... pull access denied
  Warning  Failed     2m    kubelet  Error: ErrImagePull
  Normal   BackOff    1m    kubelet  Back-off pulling image "myreg.io/app:v1.2"

To see the full timeline:

kubectl get events --sort-by=.lastTimestamp

Error Message to Cause Mapping Table

The wording of the Failed message alone identifies about 70% of cases.

Checklist and Fixes for the 6 Causes

Cause 1. Image Name/Tag Typo or Nonexistent Tag

Signature: Failed to pull image ...: not found, manifest unknown

This is the most common—and most embarrassing—cause. Verify the image: value in the manifest yourself.

# 태그가 실제 존재하는지 확인 (docker가 있는 환경에서)
docker manifest inspect myreg.io/app:v1.2

It's usually a typo (nginx:latset) or a tag the build pipeline hasn't pushed yet. Fix the manifest and redeploy.

Cause 2. Missing Private Registry Authentication

Signature: pull access denied, unauthorized: authentication required

This happens when imagePullSecrets is missing or incorrect. The full walkthrough is in section 4 below.

Cause 3. Docker Hub Rate Limit (429)

Signature: toomanyrequests: You have reached your pull rate limit

Docker Hub's free tier has a tight limit based on anonymous IPs. It hits especially often in clusters that share node IPs.

The fix is to add authentication or switch to a mirror registry. See section 4 as well.

Cause 4. Private Registry TLS / Network and DNS

Signature: x509: certificate signed by unknown authority

This happens with Harbor, Nexus, and similar registries that use self-signed certificates. The node's container runtime does not trust that CA.

# 노드에서 직접 확인 (containerd 기준)
crictl pull myreg.io/app:v1.2
# x509: certificate signed by unknown authority

You need to register the CA certificate so the runtime trusts it (see the deep dive below).

Cause 5. Node Disk Full (ImageFS Eviction)

Signature: no space left on device, node condition DiskPressure=True

kubectl describe node <node>
# Conditions:
#   DiskPressure   True   ... ImageGCFailed

SSH into the node and check actual usage and images.

df -h /var/lib/containerd
crictl images          # 쌓인 이미지 확인
crictl rmi --prune     # 미사용 이미지 정리

Cause 6. Registry Down / Firewall / DNS

Signature: dial tcp ... i/o timeout, connection refused

Check the network path from the node to the registry.

nslookup myreg.io          # DNS 해석 확인
curl -v https://myreg.io/v2/   # 443 도달 여부

Check with the infra team whether firewall, proxy, or internal network policies have changed.

Deep Dive: Core Fix Examples

Creating and Attaching imagePullSecrets (Fix for Cause 2)

First, create a docker-registry type Secret.

kubectl create secret docker-registry regcred \
  --docker-server=myreg.io \
  --docker-username=<id> \
  --docker-password=<pw> \
  --docker-email=<email>

There are two ways to attach it.

Method A — Attach directly in the Pod (or Deployment) spec

spec:
  imagePullSecrets:
    - name: regcred
  containers:
    - name: app
      image: myreg.io/app:v1.2

Method B — Attach by default on the ServiceAccount (applies to the whole namespace)

kubectl patch serviceaccount default \
  -p '{"imagePullSecrets":[{"name":"regcred"}]}'

Practical tip: In production clusters, defaulting to Method B reduces incidents. It's surprisingly common to forget imagePullSecrets on every new deploy and hit ImagePullBackOff. Just remember that in multi-tenant environments with separate SAs, you need to apply it to each SA.

Bypassing Docker Hub Rate Limits (Fix for Cause 3)

The simplest fix is to create an auth Secret with a Docker Hub account and attach it the same way as above.

kubectl create secret docker-registry dockerhub \
  --docker-server=https://index.docker.io/v1/ \
  --docker-username=<dockerhub_id> \
  --docker-password=<token>

For a more fundamental fix, we recommend a pull-through cache (mirror registry). With containerd, configure a registry mirror so Docker Hub requests are redirected to an internal Harbor/ECR.

# /etc/containerd/certs.d/docker.io/hosts.toml
server = "https://registry-1.docker.io"

[host."https://mirror.myreg.io"]
  capabilities = ["pull", "resolve"]

Trusting a Private Registry CA (Fix for Cause 4)

Register the CA with containerd.

# 각 노드에서
mkdir -p /etc/containerd/certs.d/myreg.io
cp ca.crt /etc/containerd/certs.d/myreg.io/ca.crt
systemctl restart containerd

Conclusion: One-Page Summary and Preventing Recurrence

The diagnostic flow in one pass:

1. Check status with kubectl get pods
2. Read the Events messages from kubectl describe pod
3. Branch to one of the 6 causes using the mapping table above
4. Apply the matching command/YAML and redeploy

Recurrence-Prevention Code

Ban :latest and pin to a digest to stop "it worked yesterday, not today."

containers:
  - name: app
    # 태그 대신 digest로 불변 고정
    image: myreg.io/app@sha256:9f86d081884c7d659a2feaa0c55ad015...
    imagePullPolicy: IfNotPresent

Also know the imagePullPolicy differences.

Add a registry mirror / pull-through cache on top of that, and the cluster won't wobble even when Docker Hub is down or rate-limited.

References: Official Docs

The primary source for the behavior, settings, and errors in this post is the official documentation below. Check there for version-specific options and exact behavior.

• Kubernetes official documentation

FAQ

Q. What's the difference between ErrImagePull and ImagePullBackOff? A. ErrImagePull is the state right after a pull just failed. ImagePullBackOff is when failures have repeated and the kubelet is waiting with an increasing retry interval. The cause is the same, so diagnose from the Events messages in kubectl describe pod.

Q. I set imagePullSecrets but still get pull access denied. A. Check that the Secret is in the same namespace as the Pod, and that the --docker-server address exactly matches the registry in the image path. For Docker Hub, you must use https://index.docker.io/v1/. If you attached it to an SA, also verify the Pod is using that SA.

Q. How do I diagnose a full disk when I can't SSH to the node? A. First check the DiskPressure condition and ImageGCFailed events with kubectl describe node <node>. If you need more debugging, spin up a node debug container with kubectl debug node/<node> and run df -h and crictl images.

Coming next — Part 3: CrashLoopBackOff — Tracing Why a Container Keeps Restarting from Logs. If the image pulled fine but the container keeps crashing, see you in the next post.