# 6 Ways to Fix kubectl "You must be logged in to the server (Unauthorized)" `kubectl get pods` was working fine yesterday. Today it suddenly spits this out: ```text error: You must be logged in to the server (Unauthorized) ``` Palms start sweating. Bottom line first: this is **not** "you don't have permission (403)" — it is **"the cluster could not prove who you are (401)"**. That is an authentication problem. Before you go digging through [RBAC](/blog/k8s-forbidden-오류-rbac부터-serviceaccount까지-5단계로-완벽-진단하는-방법) settings, check whether **your ID (token or certificate) is still valid**. Earlier posts in this series (K8s_Troubleshooting_Guide) covered workload issues like scheduling, images, and networking. This 12th installment moves into **authn/authz**. If you landed here by searching the error text, the goal is to pinpoint the cause and recover with copy-paste commands in five minutes. ## 401 vs 403: the distinction you have to get right first Half of troubleshooting is just figuring out which neighborhood the problem lives in. | Error message | HTTP code | Meaning | Cause area | First check | |---|---|---|---|---| | `You must be logged in to the server (Unauthorized)` | 401 | Failed to prove who you are (authentication) | authn | Token/cert expiry, context, cloud credentials | | `Forbidden ... cannot list resource ...` | 403 | We know who you are, but you lack permission (authorization) | authz | RBAC Role/RoleBinding | Core analogy: **401 is the badge itself expired or lost**; **403 is you have a badge but no permission to enter that room**. This post focuses on 401. For 403 (RBAC) we only mark the boundary under cause 5. Also tell apart messages that look similar but have different causes. | Actual output | Likely cause | First action | |---|---|---| | `You must be logged in to the server (Unauthorized)` | Expired token/credentials (on kubeadm, including an expired client certificate), wrong user | (1)–(4) below | | `You must be logged in to the server (the server has asked for the client to provide credentials)` | Credentials are empty or the exec plugin failed | Check the kubeconfig `exec` block and `AWS_PROFILE` | | `Unable to connect to the server: x509: certificate has expired or is not yet valid` | Usually the API server's serving certificate has expired, or client/server clock skew. An expired client certificate usually shows up as `Unauthorized`, not this message | Server certificate expiry, `kubeadm certs check-expiration`, `timedatectl` | | `Error from server (Forbidden): ... cannot ...` | Authentication succeeded, authorization (RBAC) failed | (5) | | `Unable to connect to the server: dial tcp ...` | Endpoint or network problem | Check the cluster endpoint | ## Six cause-by-cause diagnoses and fixes (copy-paste commands) ### (1) kubeconfig token/certificate expired The most common culprit. Short-lived (STS) tokens are now the norm, so the "it suddenly broke a few hours later" pattern is frequent. ```bash # 진단: 현재 사용자에 박힌 token / 인증서 정보 확인 kubectl config view --raw --minify ``` In the output, look under `user:` for a `token:` or an `exec:` block. If a static token is embedded, it has likely expired. Fix: reissue the token from the issuer and refresh it (if this is a cloud cluster, go to cause 3). ⚠️ `--raw` prints tokens and certificates in plain text. Mask them before screen-sharing or pasting into issues or chat. To see just the structure, `kubectl config view --minify` without `--raw` is enough. ### (2) Wrong context/cluster selected In multi-cluster setups, hitting prod with dev credentials happens more often than you'd think. ```bash # 진단 kubectl config current-context # 지금 어디를 보고 있나 kubectl config get-contexts # * 표시가 현재 context # 해결 kubectl config use-context <원하는-context-이름> ``` Confirm the `CLUSTER`/`AUTHINFO` columns from `get-contexts` match what you intended. If it was the wrong context, this alone fixes it. ### (3) Cloud IAM token expired (EKS / GKE) EKS issues short-lived STS tokens via `aws-iam-authenticator` or `aws eks get-token` (exec credential plugin). When the AWS session expires, the AWS CLI errors out while issuing the token, or you get a straight 401. ```bash # EKS: kubeconfig 자격증명 갱신 aws eks update-kubeconfig --region <리전> --name <클러스터명> # 프로파일을 따로 쓴다면 aws eks update-kubeconfig --region <리전> --name <클러스터명> --profile <프로파일> # GKE gcloud container clusters get-credentials <클러스터명> --region <리전> ``` The most common EKS trap: first confirm **the AWS credentials themselves are valid** with `aws sts get-caller-identity`. If the AWS session is dead, `update-kubeconfig` still 401s at token issuance. ```bash aws sts get-caller-identity # if this errors, refresh your AWS credentials first ``` If the credentials are valid and you still get 401, check whether that IAM identity is mapped into the cluster. EKS accepts only identities registered in the `aws-auth` ConfigMap or in EKS access entries. Ask a cluster admin to confirm that the ARN from `aws sts get-caller-identity` (the role ARN, if you assumed a role) is registered. ```bash # Run from a side that has admin access aws eks list-access-entries --cluster-name kubectl -n kube-system get configmap aws-auth -o yaml ``` ### (4) client-certificate permissions and expiry On-prem kubeadm clusters usually use client-certificate auth. Certificates expire after one year by default — easy to forget. ```bash # 인증서 만료일 디코딩 openssl x509 -in ~/.kube/client.crt -noout -enddate # 출력 예) notAfter=Jun 16 09:00:00 2026 GMT # kubeconfig에 base64로 박혀 있다면 kubectl config view --raw -o jsonpath='{.users[0].user.client-certificate-data}' | base64 -d | openssl x509 -noout -enddate ``` If `notAfter` is in the past, the cert is expired. Also check the cert file path and permissions (`ls -l`; 600 is recommended). On kubeadm, renew with `kubeadm certs renew`. If you see `x509: certificate has expired or is not yet valid`, check **the API server certificate and the clock** before the client certificate. It means kubectl failed while verifying the certificate the server presented. ```bash # Validity of the certificate the API server presents (kubeadm default port 6443) echo | openssl s_client -connect :6443 2>/dev/null | openssl x509 -noout -dates # On a control plane node, check and renew all certificates sudo kubeadm certs check-expiration sudo kubeadm certs renew all # Renewed certificates take effect only after the control plane static Pods restart: # move the manifests out of /etc/kubernetes/manifests/, wait for the Pods to stop, then move them back # Copy the renewed admin.conf to your kubeconfig sudo cp /etc/kubernetes/admin.conf $HOME/.kube/config ``` [Kubernetes docs: Certificate Management with kubeadm](https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/) ### (5) Missing RBAC — you get 403 here, not 401 This is the 401 vs 403 boundary. **If authentication succeeded but you lack permission, you should get 403, not 401**. So if you are seeing 401, RBAC is usually not the culprit. ```bash # 내가 특정 동작을 할 수 있는지 확인 kubectl auth can-i list pods kubectl auth can-i get pods --namespace prod ``` If the output is `no`, it is a 403 (authorization) problem. If the message is still 401, you were already blocked at authentication — go back to causes 1–4. ```bash kubectl auth whoami # who the cluster thinks you are (kubectl 1.28+) ``` If `whoami` shows a different user than you expect, it's an authentication/context problem, not a permissions problem. ### (6) System clock skew JWT/STS tokens have `NotBefore`/`Expiry` baked in. If the client clock is wrong, a perfectly valid token fails verification as "not yet valid" or "expired". Surprisingly common on VMs and in containers. ```bash # 진단 timedatectl # System clock synchronized: yes 인지 확인 # 해결 (NTP 동기화) sudo timedatectl set-ntp true sudo systemctl restart systemd-timesyncd ``` If you see `System clock synchronized: no`, or the clock is off by more than a few minutes, sync and retry. ## 5-minute quick diagnosis flow When you are stuck, work top to bottom. 1. **Check context** → `kubectl config current-context` to see if you are pointed at the wrong place (cause 2) 2. **Check token/cert expiry** → `kubectl config view --raw --minify` + `openssl x509 -enddate` (causes 1 and 4) 3. **If cloud, refresh credentials** → `aws sts get-caller-identity` → `aws eks update-kubeconfig ...` or `gcloud ... get-credentials` (cause 3) 4. **Check the clock** → `timedatectl` sync status (cause 6) 5. **If you got this far, it is the permission layer** → `kubectl auth can-i ...` to check for 403 (cause 5) > Field tip: when a 401 is reported, make checking the context (cause 2) and `aws sts get-caller-identity` (cause 3) your first habit; recovery gets faster. ## Prevention is troubleshooting — auto token refresh and context isolation Recovery matters, but stopping recurrence is the real skill. - **Auto-refresh with an exec credential plugin**: do not embed a static token. Use `user.exec` in kubeconfig so a fresh token is fetched on every request. This is the recommended EKS approach. ```yaml users: - name: my-eks user: exec: apiVersion: client.authentication.k8s.io/v1beta1 command: aws args: - eks - get-token - --cluster-name - my-cluster - --region - ap-northeast-2 ``` - **Separate contexts and kubeconfigs per cluster**: do not mix prod and dev in one file. Split with `KUBECONFIG=~/.kube/prod:~/.kube/dev`, or show the current context in the shell prompt (kube-ps1) to cut down on misfires. - **Check KUBECONFIG merge precedence**: when you merge several files, for a user or context with the same name, **the value from the file listed first** wins. If you still get 401 after reissuing, check the file order with `echo $KUBECONFIG` and compare the user actually picked with `kubectl config view --minify`. - **Use short-lived tokens on CI/CD runners**: since Kubernetes 1.24, ServiceAccount token Secrets are no longer auto-generated, and Pods get bound tokens that expire. Don't rely on a token the runner copied long ago; issue one per run with `kubectl create token --duration=1h` (TokenRequest API), or use your cloud's OIDC integration. Command recap: confirm location with `current-context` → check expiry with `config view --raw` / `openssl x509` → refresh with `aws eks update-kubeconfig` → check the clock with `timedatectl` → separate permissions with `auth can-i`. Coming next: **K8s_Troubleshooting_Guide Part 13 — Troubleshooting 403 Forbidden and RBAC permission design**, which digs into the authorization (authz) area we only touched here. ## Reference: official docs Primary sources for the behavior, settings, and errors in this post are the official docs below. Check version-specific options and exact behavior there. - [Kubernetes official documentation](https://kubernetes.io/docs/home/) ## FAQ **Q. I ran `aws eks update-kubeconfig` and still get 401.** A. The AWS session itself is expired. First check that credentials are alive with `aws sts get-caller-identity`. If they are expired, SSO login (`aws sso login`) or refresh keys, then run `update-kubeconfig` again. **Q. I have multiple profiles and regions and it is confusing.** A. Pass `--profile` and `--region` explicitly, and put the same values in the exec plugin args. If the `AWS_PROFILE` environment variable and the kubeconfig exec settings disagree, you mint a token with the wrong credentials and get 401. **Q. Could a 401 still be an RBAC problem?** A. Almost never. Missing RBAC happens after authentication succeeds, so it usually shows up as 403. If it is 401, token, certificate, context, or clock issues are far more likely — start with steps 1–4 of the 5-minute flow.

#EKS aws-iam-authenticator token expired

6 Ways to Fix kubectl Unauthorized (You must be logged in) 401