Fix ImagePullBackOff · ErrImagePull: Diagnose the Cause from Events and Finish with Copy-Paste

Fix ImagePullBackOff · ErrImagePull: Diagnose the Cause from Events Messages and Finish with Copy-Paste

K8s_Troubleshooting_Guide Part 17

If the deploy pipeline is green but the Pod is stuck on ImagePullBackOff again, you've already cleared PVCs and network policies. You're blocked at the next step: kubelet pulling the image from the registry. This post focuses on that one step. The goal is to look at a single Events line from kubectl describe pod, pinpoint which of five causes it is, and recover in five minutes with copy-paste commands for each cause.

ImagePullBackOff vs ErrImagePull: They're the Same Problem

Let's clear up these two confusing statuses first. They aren't separate problems — they're stages of the same failure.

• ErrImagePull: The state right after kubelet first tries to pull the image and fails
• ImagePullBackOff: Failures have repeated, so kubelet is waiting while exponentially increasing the retry interval (back-off)

In other words, ErrImagePull shows up briefly, then hardens into ImagePullBackOff. Both have the same root cause and the same diagnostic method. Don't obsess over the status name — go straight to Events.

Step 1: Diagnosis — How to Read Events Messages

90% of causes are written in the last Events line. These two commands are enough.

# 최신순으로 이벤트 모아보기
kubectl get events --sort-by=.lastTimestamp -n <namespace>

# 특정 Pod의 Events만 빠르게 추출
kubectl describe pod <pod> -n <namespace> | grep -A10 Events

Match the message text on the Warning Failed line against the diagnosis table below, as-is.

Cause Diagnosis Table

With this one table, it's "see the message, go here."

Step 2: Fixes — Copy-Paste Commands by Cause

Cause 1: Wrong Tag or Missing Tag

If you see not found, it's the most common — and most anticlimactic — cause. First check whether that tag actually exists in the registry.

# 매니페스트에 박힌 이미지 문자열 확인
kubectl get pod <pod> -n <ns> -o jsonpath='{.spec.containers[*].image}'

# 로컬에서 동일 태그 풀 가능 여부 테스트
docker pull <registry>/<image>:<tag>

If it was a typo, replace it with kubectl set image deployment/<deploy> <container>=<image>:<correct-tag> and you're done.

Cause 2: Missing Auth — imagePullSecrets

If you see unauthorized, you don't have credentials. Create a secret and attach it to the Pod.

# 1) 레지스트리 로그인이 되는지 먼저 확인
docker login <registry>

# 2) docker-registry 타입 시크릿 생성 (반드시 앱과 같은 네임스페이스에!)
kubectl create secret docker-registry regcred \
  --docker-server=<registry> \
  --docker-username=<user> \
  --docker-password=<password> \
  --docker-email=<email> \
  -n <namespace>

Attach the secret to the Pod (or the Deployment's podTemplate).

spec:
  imagePullSecrets:
    - name: regcred
  containers:
    - name: app
      image: <registry>/<image>:<tag>

If you don't want to add it to every workload, attach it once to the ServiceAccount and apply it across the namespace.

kubectl patch serviceaccount default -n <namespace> \
  -p '{"imagePullSecrets":[{"name":"regcred"}]}'

Cause 3: Docker Hub Rate Limit

Since 2024–2025, Docker Hub has tightened anonymous pull rate limits, and toomanyrequests has exploded on CI and production nodes. Clusters sharing the same NAT IP get hit especially hard. There are two ways to respond.

(A) Pull with an authenticated account — Authenticated users get a much higher limit. Create the regcred above with a Docker Hub account and attach it to the ServiceAccount for immediate relief.

(B) Switch to a mirror / pull-through cache — This is the real fix. Stand up a Harbor proxy cache or ECR pull-through cache and change only the image path.

# 기존: nginx:1.27  (Docker Hub 직접)
# 변경: <account>.dkr.ecr.<region>.amazonaws.com/dockerhub/library/nginx:1.27
image: harbor.mycorp.com/dockerhub-proxy/library/nginx:1.27

War story: On one cluster, a nightly batch spun up dozens of Pods at once and they all died with toomanyrequests. Attaching an authenticated account put out the fire, but recurrence only stopped after we installed a Harbor proxy cache. If CI and production share the same egress IP, a cache isn't optional — it's required.

Cause 4: Registry Host Resolution Failure

lookup ... no such host means kubelet could not resolve the registry domain via DNS. (General DNS troubleshooting is covered in a separate post, so here we only look at it from the registry-host angle.) Check directly on the node.

# 노드에 들어가 레지스트리 도메인 해석 테스트
nslookup <registry-host>
curl -v https://<registry-host>/v2/

For an internal registry, check whether the node /etc/hosts or private DNS has the record, and whether private endpoint routing is in place.

Cause 5: Node Disk Full

no space left on device means there isn't enough disk space to pull image layers.

# 어느 노드에 떴는지 확인
kubectl get pod <pod> -n <ns> -o wide

# 노드 디스크 압박 상태 확인
kubectl describe node <node> | grep -A5 Conditions   # DiskPressure 확인
# 노드 접속 후
df -h /var/lib/containerd   # 또는 /var/lib/docker
crictl rmi --prune          # 미사용 이미지 정리

Step 3: Common Pitfalls and Cloud Registries

Pitfall 1: Secret Namespace Mismatch

This is the landmine people step on most. If you create the secret only in default while the app runs in the prod namespace, auth will never apply. imagePullSecrets only reference secrets in the same namespace as the Pod.

# prod ns에 똑같이 만들어 줘야 함
kubectl create secret docker-registry regcred ... -n prod

Pitfall 2: The Debugging Hell of latest + imagePullPolicy: Always

If you use the latest tag with an Always policy, every restart can pull a different image, which produces unreproducible failures of the "it worked yesterday, not today" kind. The recommended strategy is immutable tags.

• ❌ myapp:latest + imagePullPolicy: Always
• ✅ myapp:1.4.2 or myapp:git-a1b2c3d (commit/build-based pinned tag)

With immutable tags, imagePullPolicy: IfNotPresent is also safe, and cache hit rates go up.

Auto-Renewing Expired Tokens per Cloud

Managed registries have tokens that expire. If auth that worked yesterday turns into unauthorized today, suspect token expiry.

If you use ECR with a static secret (regcred), it will always break after 12 hours. Prefer keyless auth with IRSA / Workload Identity; if that's hard, add a token-refresh CronJob.

# ECR 토큰 갱신 CronJob 핵심 로직 (11시간마다)
schedule: "0 */11 * * *"
# 컨테이너 command 예시
# kubectl delete secret regcred -n app --ignore-not-found
# kubectl create secret docker-registry regcred \
#   --docker-server=<acct>.dkr.ecr.<region>.amazonaws.com \
#   --docker-username=AWS \
#   --docker-password=$(aws ecr get-login-password --region <region>) -n app

Conclusion: Recurrence-Prevention Checklist

• ✅ First action is always kubectl describe pod | grep -A10 Events
• ✅ Create the secret in the same namespace as the app, attach it to the ServiceAccount to propagate
• ✅ Ban latest; standardize on immutable tags + IfNotPresent
• ✅ Instead of pulling Docker Hub directly, use an authenticated account + pull-through cache
• ✅ For ECR/AR/ACR, prefer IRSA/Workload Identity; if you use a static secret, automate refresh

In Part 18, we cover the next stage: the image pulled fine and the container started, but it infinitely restarts with CrashLoopBackOff — how to split the cause using logs, exit codes, and probes.

References: Official Docs

The primary sources for the behavior, settings, and errors covered in this post are the official docs below. Check them for version-specific options and exact behavior.

• Kubernetes official documentation

FAQ

Q. Which is more serious, ImagePullBackOff or ErrImagePull? A. It's not a difference in severity — it's a difference in stage. ErrImagePull is immediately after a pull failure; ImagePullBackOff is waiting to retry after repeated failures. Cause and fix are identical, so go straight to the Events message.

Q. I definitely created imagePullSecrets, but I still get unauthorized. A. Nine times out of ten it's a namespace mismatch. The secret must be in the same namespace as the Pod to be referenced. Confirm it exists with kubectl get secret regcred -n <app-namespace>, and if you're on ECR, also suspect the 12-hour token expiry.

Q. Docker Hub toomanyrequests — is attaching auth enough? A. Auth accounts will ease an urgent situation, but if CI and production share the same egress IP, it will come back. The real fix is to put a mirror in front — Harbor proxy cache or ECR pull-through cache — and point image paths at the cache.