kubectl Unauthorized: 3-Minute Cause-by-Cause Diagnosis and Recovery Runbook (EKS Reissue)

kubectl "Unauthorized": 3-Minute Cause-by-Cause Diagnosis and Recovery Runbook (Including EKS Reissue)

K8s Troubleshooting Guide, Part 22

kubectl Worked Yesterday—Then It Suddenly Stopped

If you work in infra, you will hit this at least once: you run kubectl get pods to deploy, and this shows up.

error: You must be logged in to the server (Unauthorized)

Don't panic. kubectl's auth context (kubeconfig) only fails at five points, in this order: token → certificate → context mapping → RBAC → endpoint. Most of the time it's either "the short-lived token expired" or "you're pointing at the wrong context." Especially since EKS 1.24+ made aws eks get-token the default, and Kubernetes 1.24 dropped permanent ServiceAccount tokens, Unauthorized from bound (expiring) tokens has become noticeably more common.

This is not a theory lecture—it's a recovery-first runbook. Run the commands first; read the explanations after.

1. Diagnosis Table by Exact Error Text — Branch on the Message Immediately

The first thing to do is read the error verbatim. The wording alone splits the five causes.

The key distinction: Unauthorized (401) means "you failed to prove who you are"; Forbidden (403) means "I know who you are, but you don't have permission." Completely different directions.

2. 3-Second Check of Your Current Credentials

To narrow the cause, you need to see which user / cert / token kubectl is using right now.

# 지금 활성화된 context 이름
kubectl config current-context

# 전체 context 목록 — 별표(*)가 현재 사용 중
kubectl config get-contexts

# 현재 context의 cluster/user/endpoint 상세 (민감정보 주의!)
kubectl config view --minify

⚠️ kubectl config view --minify --raw dumps tokens and certificates in the clear. Always mask them before screen-sharing or pasting into logs.

A common trap here: a mapping error where the context is cluster A but the user belongs to cluster B. Check that the CLUSTER and AUTHINFO columns in get-contexts actually pair up.

3. Recovery in Practice, by Cause

(A) EKS — Reissue kubeconfig

On EKS, Unauthorized is resolved by this one line about 90% of the time.

aws eks update-kubeconfig --region ap-northeast-2 --name my-cluster

If it still fails after reissue, inspect the exec block in ~/.kube/config. EKS 1.24+ uses aws eks get-token as shown below (older versions used aws-iam-authenticator).

users:
- name: arn:aws:eks:ap-northeast-2:123456789012:cluster/my-cluster
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      command: aws
      args:
        - eks
        - get-token
        - --cluster-name
        - my-cluster

The most common real cause is a profile mismatch. If the AWS profile that created the kubeconfig differs from the profile in your current shell, auth goes out as a different IAM identity and you get Unauthorized.

aws sts get-caller-identity          # 지금 내 IAM 신원
echo $AWS_PROFILE                     # 셸 프로파일
aws --version                         # 1.16 이하 구버전이면 get-token 미지원

If the result of aws sts get-caller-identity doesn't match an identity registered in the cluster's aws-auth ConfigMap, that's the cause.

(B) Check and Renew an Expired client-certificate (kubeadm / on-prem)

If you're on certificate auth, start by checking the expiry date.

# kubeconfig에서 client 인증서 추출 → 만료일 확인
kubectl config view --raw -o jsonpath='{.users[0].user.client-certificate-data}' \
  | base64 -d | openssl x509 -noout -enddate
# 출력 예: notAfter=Jul  3 09:00:00 2026 GMT

If it's expired, on kubeadm check and renew like this.

kubeadm certs check-expiration      # 전체 인증서 만료 현황
kubeadm certs renew admin.conf      # admin kubeconfig 인증서 갱신
# 갱신 후 새 admin.conf를 ~/.kube/config로 복사
sudo cp /etc/kubernetes/admin.conf $HOME/.kube/config

(C) RBAC — Draw a Hard Line Between 401 and 403

If you got Forbidden, authentication already succeeded. You only need to check permissions.

# 내가 지금 누구로 인식되는지 (Kubernetes 1.28+)
kubectl auth whoami

# 특정 동작 가능 여부
kubectl auth can-i create deployments -n prod
kubectl auth can-i '*' '*' --all-namespaces   # 관리자급인지

If can-i returns no, you need to add a RoleBinding/ClusterRoleBinding. If kubectl auth whoami shows an identity you didn't expect, loop back to the profile/context issues.

A Note from the Field

Most cases that stay Unauthorized even after a reissue were multiple files merged via the KUBECONFIG env var, with a stale user still winning in the higher-priority file. Get in the habit of printing echo $KUBECONFIG first, then comparing which merged file actually won with kubectl config view --minify. If it only fails in CI, suspect an expired bound token on the runner's ServiceAccount.

Unauthorized 3-Minute Recovery Checklist

1. Read the error verbatim → distinguish 401 (Unauthorized) vs 403 (Forbidden)
2. Confirm the context/user pairing with kubectl config current-context / get-contexts
3. On EKS, reissue with aws eks update-kubeconfig and compare identity/profile with aws sts get-caller-identity
4. On certificate auth, check expiry with openssl x509 -noout -enddate
5. On 403, inspect RBAC with kubectl auth can-i / auth whoami

Part 23 covers The connection to the server ... was refused — endpoint/network problems where you can't reach the API server at all.

References: Official Docs

The primary sources for the behavior, settings, and errors in this post are the official docs below. Check them for version-specific options and exact behavior.

• Kubernetes official documentation

FAQ

Q. I reissued with update-kubeconfig and still get Unauthorized. A. Check two things. First, multiple files merged in KUBECONFIG so a stale user is winning. Second, the identity from aws sts get-caller-identity is not registered in the cluster's aws-auth ConfigMap (or an EKS Access Entry). The identity itself has to be mapped into the cluster.

Q. I merge several clusters and keep hitting the wrong context. A. Check where the asterisk (*) is with kubectl config get-contexts, then switch with kubectl config use-context <name>. Also verify that the AUTHINFO (user) and CLUSTER columns point at the same cluster so you don't get a mapping error.

Q. Unauthorized only happens on the CI/CD runner. A. Since Kubernetes 1.24, permanent ServiceAccount tokens were removed and bound (expiring) tokens are the default, so a stale token cached by the runner is a likely culprit. Change the pipeline to request a short-lived token via the TokenRequest API on every run.