# Fix ImagePullBackOff · ErrImagePull: Diagnose the Cause from Events Messages and Finish with Copy-Paste > K8s_Troubleshooting_Guide Part 17 If the deploy pipeline is green but the Pod is stuck on `ImagePullBackOff` again, you've already cleared PVCs and network policies. You're blocked at the next step: **kubelet pulling the image from the registry**. This post focuses on that one step. The goal is to look at a **single Events line** from `kubectl describe pod`, pinpoint which of seven causes it is, and recover in five minutes with copy-paste commands for each cause. ## ImagePullBackOff vs ErrImagePull: They're the Same Problem Let's clear up these two confusing statuses first. They aren't separate problems — they're **stages of the same failure**. - **ErrImagePull**: The state right after kubelet first tries to pull the image and fails - **ImagePullBackOff**: Failures have repeated, so kubelet is waiting while exponentially increasing the retry interval (back-off) In other words, `ErrImagePull` shows up briefly, then hardens into `ImagePullBackOff`. Both have the **same root cause and the same diagnostic method**. Don't obsess over the status name — go straight to Events. ## Step 1: Diagnosis — How to Read Events Messages 90% of causes are written in the last Events line. These two commands are enough. ```bash # 최신순으로 이벤트 모아보기 kubectl get events --sort-by=.lastTimestamp -n # 특정 Pod의 Events만 빠르게 추출 kubectl describe pod -n | grep -A10 Events ``` Match the message text on the `Warning Failed` line against the diagnosis table below, as-is. ### Cause Diagnosis Table | Events message (key phrase) | Cause | Fix section | |---|---|---| | `Failed to pull image ... not found` / `manifest unknown` | Wrong tag / tag does not exist | → [Wrong tag](#cause-1-wrong-tag-or-missing-tag) | | `unauthorized: authentication required` / `pull access denied` | Missing auth (private registry) | → [Missing auth](#cause-2-missing-auth--imagepullsecrets) | | `toomanyrequests: You have reached your pull rate limit` | Docker Hub rate limit | → [rate limit](#cause-3-docker-hub-rate-limit) | | `dial tcp: lookup ... no such host` / `i/o timeout` | Registry host resolution / reachability failure | → [Host resolution](#cause-4-registry-host-resolution-failure) | | `failed to ... no space left on device` | Node disk full | → [Disk](#cause-5-node-disk-full) | | `x509: certificate signed by unknown authority` | Node does not trust the private registry CA | → [x509](#cause-6-x509--private-ca-not-trusted) | | `no match for platform in manifest` | The tag has no image for the node architecture | → [Platform](#cause-7-no-match-for-platform--architecture-mismatch) | With this one table, it's "see the message, go here." ## Step 2: Fixes — Copy-Paste Commands by Cause ### Cause 1: Wrong Tag or Missing Tag If you see `not found`, it's the most common — and most anticlimactic — cause. First check whether that tag actually exists in the registry. ```bash # 매니페스트에 박힌 이미지 문자열 확인 kubectl get pod -n -o jsonpath='{.spec.containers[*].image}' # 로컬에서 동일 태그 풀 가능 여부 테스트 docker pull /: ``` If it was a typo, replace it with `kubectl set image deployment/ =:` and you're done. ### Cause 2: Missing Auth — imagePullSecrets If you see `unauthorized`, you don't have credentials. Create a secret and attach it to the Pod. ```bash # 1) 레지스트리 로그인이 되는지 먼저 확인 docker login # 2) docker-registry 타입 시크릿 생성 (반드시 앱과 같은 네임스페이스에!) kubectl create secret docker-registry regcred \ --docker-server= \ --docker-username= \ --docker-password= \ --docker-email= \ -n ``` Attach the secret to the Pod (or the Deployment's podTemplate). ```yaml spec: imagePullSecrets: - name: regcred containers: - name: app image: /: ``` If you don't want to add it to every workload, attach it once to the [ServiceAccount](/blog/k8s-forbidden-오류-rbac부터-serviceaccount까지-5단계로-완벽-진단하는-방법) and apply it across the namespace. ```bash kubectl patch serviceaccount default -n \ -p '{"imagePullSecrets":[{"name":"regcred"}]}' ``` ### Cause 3: Docker Hub Rate Limit Docker Hub limits image pulls per 6-hour window. Per the official docs (checked 2026-10-04), unauthenticated users get 100 pulls per IPv4 address or IPv6 /64 subnet per 6 hours, authenticated Personal accounts get 200, and pulls are unlimited on Pro, Team and Business plans. Going over the limit returns `toomanyrequests`. Because the unauthenticated limit is per IP, clusters and CI runners behind the same NAT IP share it and get hit especially hard. There are two ways to respond. **(A) Pull with an authenticated account** — Once authenticated, the limit applies per account instead of per IP (Personal: 200 per 6 hours; unlimited on Pro, Team and Business plans). Create the `regcred` above with a Docker Hub account and attach it to the ServiceAccount for immediate relief. **(B) Switch to a mirror / pull-through cache** — This is the real fix. Stand up a Harbor proxy cache or ECR pull-through cache and change only the image path. ```yaml # 기존: nginx:1.27 (Docker Hub 직접) # 변경: .dkr.ecr..amazonaws.com/dockerhub/library/nginx:1.27 image: harbor.mycorp.com/dockerhub-proxy/library/nginx:1.27 ``` If changing image paths is hard, you can point containerd on each node at a Docker Hub mirror instead. With `config_path` set (see Cause 6 below), put a `hosts.toml` in the `docker.io` directory. ```toml # /etc/containerd/certs.d/docker.io/hosts.toml server = "https://registry-1.docker.io" [host."https://mirror.example.com"] capabilities = ["pull", "resolve"] ``` The mirrors listed under `host` are tried first, and `server` is used as the fallback if they all fail ([containerd registry configuration](https://github.com/containerd/containerd/blob/main/docs/hosts.md)). This is per-node configuration, so bake it into the node image or bootstrap script. > War story: On one cluster, a nightly batch spun up dozens of Pods at once and they all died with `toomanyrequests`. Attaching an authenticated account put out the fire, but recurrence only stopped after we installed a Harbor proxy cache. If CI and production share the same egress IP, a cache isn't optional — it's required. ### Cause 4: Registry Host Resolution Failure `lookup ... no such host` means kubelet **could not resolve the registry domain via DNS**. (General DNS troubleshooting is covered in a separate post, so here we only look at it from the registry-host angle.) Check directly on the node. ```bash # 노드에 들어가 레지스트리 도메인 해석 테스트 nslookup curl -v https:///v2/ ``` For an internal registry, check whether the node `/etc/hosts` or private DNS has the record, and whether private endpoint routing is in place. ### Cause 5: Node Disk Full `no space left on device` means there isn't enough disk space to pull image layers. ```bash # 어느 노드에 떴는지 확인 kubectl get pod -n -o wide # 노드 디스크 압박 상태 확인 kubectl describe node | grep -A5 Conditions # DiskPressure 확인 # 노드 접속 후 df -h /var/lib/containerd # 또는 /var/lib/docker crictl rmi --prune # 미사용 이미지 정리 ``` When image-filesystem usage exceeds `imageGCHighThresholdPercent`, the kubelet deletes unused images to bring usage down to `imageGCLowThresholdPercent` ([garbage collection docs](https://kubernetes.io/docs/concepts/architecture/garbage-collection/#containers-images)). If that cleanup fails, the node records an `ImageGCFailed` event, and once free space falls below the eviction threshold the node gets `DiskPressure`, which blocks new Pods from being scheduled there ([node-pressure eviction docs](https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/)). `crictl rmi --prune` is a stopgap; if it keeps happening, fix node disk size or image size. ```bash kubectl get events -A --field-selector reason=ImageGCFailed ``` ### Cause 6: x509 — Private CA Not Trusted ``` failed to pull and unpack image ... x509: certificate signed by unknown authority ``` This is not a credentials problem: **containerd on the node does not trust the CA that issued the registry certificate**. It shows up mostly with private registries such as Harbor or Nexus whose certificates come from an internal PKI. On containerd nodes, per-registry settings live in `hosts.toml` files under `certs.d`. First check that `/etc/containerd/config.toml` sets `config_path`. The section name depends on the containerd version. ```toml # containerd 2.x (config version 3) [plugins."io.containerd.cri.v1.images".registry] config_path = "/etc/containerd/certs.d" # containerd 1.x (config version 2) [plugins."io.containerd.grpc.v1.cri".registry] config_path = "/etc/containerd/certs.d" ``` Create a directory named after the registry host and put the CA certificate and `hosts.toml` in it. ```toml # /etc/containerd/certs.d/harbor.example.com/hosts.toml server = "https://harbor.example.com" [host."https://harbor.example.com"] capabilities = ["pull", "resolve"] ca = "/etc/containerd/certs.d/harbor.example.com/ca.crt" # skip_verify = true ← disables verification. Use only to confirm a repro; never leave it in production ``` ```bash sudo cp internal-root-ca.crt /etc/containerd/certs.d/harbor.example.com/ca.crt sudo systemctl restart containerd # required if you changed config.toml ``` The format follows [containerd hosts.md](https://github.com/containerd/containerd/blob/main/docs/hosts.md). The old `registry.mirrors` / `registry.configs` style is deprecated, so don't mix the two; consolidate on `config_path`. On nodes running Docker Engine, put the CA at `/etc/docker/certs.d//ca.crt`, and include the port in the directory name if the registry uses one (e.g. `registry.example.com:5000`, [Docker docs](https://docs.docker.com/engine/security/certificates/)). **If x509 happens on only one node**: check which node the failing Pod landed on with `kubectl get pod -o wide`. A newly added node missing the CA / `hosts.toml` rollout is a common cause. Distribute the same files to every node through the node image, bootstrap script, or configuration management, and add it to your new-node checklist. ### Cause 7: no match for platform — Architecture Mismatch ``` failed to pull and unpack image ...: no match for platform in manifest: not found ``` The tag exists, but **it has no image for the node architecture (amd64/arm64)**. You see this when you deploy an amd64-only image to a cluster that mixes in arm64 nodes. ```bash kubectl get nodes -L kubernetes.io/arch docker buildx imagetools inspect /: # platforms included ``` Either rebuild as a multi-platform image (`docker buildx build --platform linux/amd64,linux/arm64 ... --push`, [Docker docs](https://docs.docker.com/build/building/multi-platform/)) or use a `nodeSelector` so the Pod only lands on nodes whose architecture the image supports. ```yaml spec: nodeSelector: kubernetes.io/arch: amd64 ``` ## Step 3: Common Pitfalls and Cloud Registries ### Pitfall 1: Secret Namespace Mismatch This is the landmine people step on most. If you create the secret only in `default` while the app runs in the `prod` namespace, auth will never apply. **imagePullSecrets only reference secrets in the same namespace as the Pod**. ```bash # prod ns에 똑같이 만들어 줘야 함 kubectl create secret docker-registry regcred ... -n prod ``` ### Pitfall 2: The Debugging Hell of `latest` + `imagePullPolicy: Always` If you use the `latest` tag with an `Always` policy, every restart can pull a different image, which produces unreproducible failures of the "it worked yesterday, not today" kind. The recommended strategy is **immutable tags**. - ❌ `myapp:latest` + `imagePullPolicy: Always` - ✅ `myapp:1.4.2` or `myapp:git-a1b2c3d` (commit/build-based pinned tag) With immutable tags, `imagePullPolicy: IfNotPresent` is also safe, and cache hit rates go up. ### Pitfall 3: The registry address stored in the secret doesn't match the image host If you attached `imagePullSecrets` and still get `unauthorized`, check which registry address the secret actually contains. ```bash kubectl get secret regcred -n \ -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d | jq '.auths | keys' ``` The keys under `auths` (registry addresses) must match the host in the manifest's `image:`. For a registry on a port such as `harbor.example.com:8443`, the port must match too, and it's safest to pass only the host, without a scheme, to `--docker-server`. Printing the whole document without `jq` shows the base64-encoded credentials (`auth`), so check only the keys. ### Reproduce on the node: bypass the kubelet with crictl After a fix, reproducing the pull directly on the node tells you whether the cause is on the node side or in the Kubernetes configuration. ```bash # from a node shell sudo crictl pull --creds ':' harbor.example.com/app/image:v1.2.3 ``` If `crictl pull` succeeds, the node's CA, network, and credentials are fine, and what remains is the secret wiring (namespace, ServiceAccount) or the manifest. If `crictl pull` also fails, a node-level problem such as Cause 4 or 6 is still there. Clear your shell history afterwards so the credentials don't linger. ### Auto-Renewing Expired Tokens per Cloud Managed registries have tokens that **expire**. If auth that worked yesterday turns into `unauthorized` today, suspect token expiry. | Registry | Auth command | Expiry / recommendation | |---|---|---| | AWS ECR | `aws ecr get-login-password \| docker login ...` | Token expires in 12 hours → the standard is granting IAM to the node/Pod via **IRSA**. If you must, have a CronJob recreate the secret | | GCP GCR/AR | `gcloud auth configure-docker` | Workload Identity recommended; tokens are short-lived | | Azure ACR | `az acr login --name ` | AAD tokens are short-lived → on AKS, use `--attach-acr` for automatic auth | If you use ECR with a static secret (`regcred`), it will always break after 12 hours. Prefer keyless auth with **IRSA / Workload Identity**; if that's hard, add a token-refresh CronJob. ```yaml # ECR 토큰 갱신 CronJob 핵심 로직 (11시간마다) schedule: "0 */11 * * *" # 컨테이너 command 예시 # kubectl delete secret regcred -n app --ignore-not-found # kubectl create secret docker-registry regcred \ # --docker-server=.dkr.ecr..amazonaws.com \ # --docker-username=AWS \ # --docker-password=$(aws ecr get-login-password --region ) -n app ``` ## Conclusion: Recurrence-Prevention Checklist - ✅ First action is always `kubectl describe pod | grep -A10 Events` - ✅ Create the secret in the **same namespace as the app**, attach it to the ServiceAccount to propagate - ✅ Ban `latest`; standardize on **immutable tags + IfNotPresent** - ✅ Instead of pulling Docker Hub directly, use an **authenticated account + pull-through cache** - ✅ For ECR/AR/ACR, prefer **IRSA/Workload Identity**; if you use a static secret, automate refresh - ✅ Put private-registry CAs and `hosts.toml` into node provisioning so **new nodes get them automatically** - ✅ In mixed-architecture clusters, use **multi-platform images** or a `kubernetes.io/arch` nodeSelector In **Part 18**, we cover the next stage: the image pulled fine and the container started, but it infinitely restarts with `CrashLoopBackOff` — how to split the cause using logs, exit codes, and probes. ## References: Official Docs The primary sources for the behavior, settings, and errors covered in this post are the official docs below. Check them for version-specific options and exact behavior. - [Images, imagePullPolicy, and pull back-off](https://kubernetes.io/docs/concepts/containers/images/) - [Pull an image from a private registry (imagePullSecrets)](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/) - [Garbage collection: containers and images](https://kubernetes.io/docs/concepts/architecture/garbage-collection/#containers-images) - [containerd registry host configuration (hosts.toml)](https://github.com/containerd/containerd/blob/main/docs/hosts.md) - [Docker: registry certificates](https://docs.docker.com/engine/security/certificates/) ## FAQ **Q. Which is more serious, ImagePullBackOff or ErrImagePull?** A. It's not a difference in severity — it's a difference in stage. ErrImagePull is immediately after a pull failure; ImagePullBackOff is waiting to retry after repeated failures. Cause and fix are identical, so go straight to the Events message. **Q. I definitely created imagePullSecrets, but I still get unauthorized.** A. Nine times out of ten it's a namespace mismatch. The secret must be in the same namespace as the Pod to be referenced. Confirm it exists with `kubectl get secret regcred -n `, and if you're on ECR, also suspect the 12-hour token expiry. **Q. Docker Hub toomanyrequests — is attaching auth enough?** A. Auth accounts will ease an urgent situation, but if CI and production share the same egress IP, it will come back. The real fix is to put a mirror in front — Harbor proxy cache or ECR pull-through cache — and point image paths at the cache. ## Sources · checked 2026-10-04 - [Docker Docs, Docker Hub usage and limits](https://docs.docker.com/docker-hub/usage/) — pull limits per 6 hours: 100 per IPv4 address or IPv6 /64 unauthenticated, Personal 200, unlimited on Pro, Team and Business plans - [containerd, Registry Configuration (hosts.md)](https://github.com/containerd/containerd/blob/main/docs/hosts.md) — `config_path` section for containerd 2.x and 1.x, `server`/`host`/`ca` in `hosts.toml`, `registry.mirrors`/`registry.configs` deprecated (checked 2026-10-06) Limits can change with Docker’s policy, so re-check this page before changing production settings.

#Docker Hub rate limit

Fix ImagePullBackOff · ErrImagePull: Diagnose the Cause from Events and Finish with Copy-Paste