ImagePullBackOff, x509, unauthorized: Diagnose Private Registry Errors in 5 Minutes

ImagePullBackOff, x509, unauthorized: Diagnose Private Registry Errors in 5 Minutes

K8s Troubleshooting Guide, Part 15

The deployment manifest applied just fine, but the Pod is stuck red on ImagePullBackOff and will not move. This error shows up especially often with private registries such as Harbor, Nexus, or an in-house registry. This post is for people who landed here by searching the error text as-is: a routine that branches on a single line from kubectl describe pod Events and fixes the issue in 5 minutes with copy-paste commands per cause.

What's the difference between ErrImagePull and ImagePullBackOff

They are two stages of the same problem.

• ErrImagePull: kubelet just failed while pulling the image.
• ImagePullBackOff: failures have repeated, so kubelet is waiting with exponentially increasing retry intervals (up to 5 minutes).

In other words, ImagePullBackOff means "still failing"—not a new root cause. The real cause is always inside the Failed message in Events.

Private registries fail often because three issues stack: ① the node does not trust a certificate issued by an in-house PKI private CA (x509), ② missing credentials (imagePullSecrets) (401), ③ after K8s 1.24+ removed dockershim and containerd became the standard, people miss that registry config moved from /etc/docker to /etc/containerd/certs.d. On top of that, tighter Docker Hub free-tier rate limits led teams to add an in-house mirror, which adds another layer of config.

Step 1 — Identify the exact failure reason from Events

There is only one thing to do first.

kubectl describe pod <pod-name> -n <namespace> | sed -n '/Events:/,$p'

Branch on the words that appear in the Warning Failed line.

Keep this table open beside you and you can jump straight to ③ whenever you see that line.

① manifest unknown — tag or path typo

The most anticlimactic, and the most common. First check whether that tag actually exists in the registry.

# 태그 목록 조회 (Harbor/Nexus의 v2 API)
curl -sk -u <user>:<pw> https://harbor.example.com/v2/app/image/tags/list

Most of the time you specified latest but only actually pushed v1.2.3, or you omitted the project path (app/). Compare the manifest image: value to the API response character by character.

② unauthorized — create imagePullSecrets

If you see 401, credentials are missing or wrong. Create a docker-registry type secret.

kubectl create secret docker-registry regcred \
  --docker-server=harbor.example.com \
  --docker-username=<user> \
  --docker-password=<pw> \
  --docker-email=<mail> \
  -n <namespace>

Note: For --docker-server, omit the scheme (https://) and write only the registry host. For Docker Hub, https://index.docker.io/v1/ is the standard value.

There are two ways to attach the secret so the Pod uses it.

Method A — set it directly on the Deployment (Pod spec):

spec:
  template:
    spec:
      imagePullSecrets:
        - name: regcred
      containers:
        - name: app
          image: harbor.example.com/app/image:v1.2.3

Method B — attach it to the ServiceAccount so it applies to the whole namespace:

kubectl patch serviceaccount default -n <namespace> \
  -p '{"imagePullSecrets":[{"name":"regcred"}]}'

I recommend method B. You do not have to write imagePullSecrets on every workload; it applies automatically to every Pod that uses the same ServiceAccount. Existing Pods must be recreated for it to take effect.

③ x509 certificate signed by unknown authority — register the private CA

failed to pull and unpack image ... x509: certificate signed by unknown authority

If you see this line, the problem is certificate trust, not authentication. containerd on the node does not trust the certificate issued by the in-house PKI. On K8s 1.24+ with containerd as the standard, you use the certs.d directory.

First check that config_path is set in /etc/containerd/config.toml.

[plugins."io.containerd.grpc.v1.cri".registry]
  config_path = "/etc/containerd/certs.d"

Then create hosts.toml in the per-host directory.

# /etc/containerd/certs.d/harbor.example.com/hosts.toml
server = "https://harbor.example.com"

[host."https://harbor.example.com"]
  capabilities = ["pull", "resolve"]
  ca = "/etc/containerd/certs.d/harbor.example.com/ca.crt"
  # skip_verify = true  ← 인증서 검증을 끄는 옵션. 운영 환경에서는 절대 비권장!

Copy the in-house CA root certificate to the ca.crt path above, then restart containerd.

cp my-internal-ca.crt /etc/containerd/certs.d/harbor.example.com/ca.crt
systemctl restart containerd

On legacy Docker (dockershim) the path is different. Note that the port is included in the directory name.

mkdir -p /etc/docker/certs.d/harbor.example.com:443
cp my-internal-ca.crt /etc/docker/certs.d/harbor.example.com:443/ca.crt
systemctl restart docker

registry.mirrors (pointing at a mirror host) and config_path (using certs.d) serve different purposes. A mirror sends pull traffic to another host; certs.d holds per-host certificate and auth settings. Mix them up and you get "I clearly set a mirror but still see x509."

④ toomanyrequests — Docker Hub rate limit

toomanyrequests: You have reached your pull rate limit

You have hit the anonymous-IP limit of 100–200 pulls per 6 hours. As a stopgap, attach a Docker Hub auth secret to raise the quota. The real fix is to create a Harbor proxy cache project and pull through an in-house mirror like harbor.example.com/dockerhub-proxy/library/nginx. Once pulled, the image is cached and the external pull goes away.

⑤ no such host / i/o timeout — network and proxy

DNS, firewall, or proxy. Test from the node itself.

getent hosts harbor.example.com          # DNS 해석 확인
curl -vk https://harbor.example.com/v2/  # TLS 핸드셰이크·연결 확인

If containerd sits behind an in-house proxy, set HTTPS_PROXY/NO_PROXY in a systemd drop-in, and always put the in-house registry in NO_PROXY.

Verify on the node — reproduce with crictl, bypassing kubelet

Once you have applied a fix, the most reliable check is to reproduce the pull on the node without going through kubelet.

# 인증·인증서·네트워크를 한 번에 검증
crictl pull --creds <user>:<pw> harbor.example.com/app/image:v1.2.3

# 캐시에 들어왔는지 확인
crictl images | grep harbor.example.com

If crictl pull succeeds, certificates, auth, and network are all fine—the problem is secret wiring or the manifest. If crictl pull also fails, a node-level issue (③⑤) remains. This split diagnosis saves the most time.

Confirm the secret actually landed correctly

If you still get 401 after attaching imagePullSecrets, decode the secret and visually check that it matches the real registry.

kubectl get secret regcred -n <namespace> \
  -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d | jq .

Under the auths key, check that the server address exactly matches the host in the manifest image:, and that the auth value (base64-encoded user:pw) is valid. A common mistake is the secret using harbor.example.com while the image path has a port like harbor.example.com:443, so they do not match.

A field note

The trap I see most in the field is "only one node is broken." CA rollout automation was missing on a newly joined worker, so only Pods scheduled there died with x509. Always check the node name in Events, and force CA and hosts.toml onto every node with Ansible, a DaemonSet, or a node bootstrap script. If you fix one node by hand, the same incident comes back on the next scale-out.

5-minute diagnosis checklist

Prevention tips: bake the in-house CA and certs.d into node provisioning automation, route Docker Hub-dependent images through a Harbor proxy cache, and put imagePullSecrets on the ServiceAccount by default.

In the next part (16) we cover kubectl top and OOMKilled — diagnosing Pods that die at the memory limit.

References: official docs

The primary sources for the behavior, config, and errors in this post are the official docs below. Check them for version-specific options and exact behavior.

• Kubernetes official documentation

FAQ

Q. Which should I look at, ErrImagePull or ImagePullBackOff? A. They are different stages of the same failure, so the cause is the same. Always branch on the Events Failed message from kubectl describe pod. The status name itself is not diagnostic.

Q. Can I use skip_verify to disable certificate verification? A. Use it only for a quick reproduction, and never leave it in production. It exposes you to man-in-the-middle attacks. The correct approach is to register the in-house CA root certificate with the ca key.

Q. I created the secret but still get unauthorized. A. Check that ① the secret is in the same namespace as the Pod, ② imagePullSecrets is actually attached, and ③ the base64-decoded server address matches the image host (including whether a port is present). Existing Pods must be recreated after the secret is applied.