**Errors this covers (workloads inside a Pod):** `open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory`, an API response of `401 Unauthorized`, or `the server has asked for the client to provide credentials`. If you get `403 Forbidden` (`User "system:serviceaccount:..." cannot ...`), authentication already succeeded; go to [diagnosing RBAC Forbidden](/blog/k8s-forbidden-오류-rbac부터-serviceaccount까지-5단계로-완벽-진단하는-방법). | Cause | Check | Fix | |---|---|---| | Token automount disabled (SA or Pod) | `kubectl -n get pod -o jsonpath='{.spec.automountServiceAccountToken}'` | Set `automountServiceAccountToken: true` only on Pods that call the API | | App reads the token once at startup | Works right after a restart, then 401 later | Re-read the token file periodically (client-go and the official Python client do this automatically) | | No SA token Secret on 1.24+ | `kubectl get secret` shows no token Secret | `kubectl create token --duration=1h` | | Audience/issuer mismatch (e.g. using the IRSA token) | Decode the token's `aud` (diagnostic command 2 below) | Use `/var/run/secrets/kubernetes.io/serviceaccount/token` for API calls | ## Part 3 is 403; this part is 401 and "token file not found" Let's start with the conclusion: **401 Unauthorized and "token file not found" are not RBAC problems.** They live on an entirely different layer. - **401 Unauthorized / missing token** → *authentication* failure. You couldn't answer "who are you?" No amount of RoleBinding fixes will ever resolve this. - **403 Forbidden** → *authorization* failure. Identity was verified, but you lack permission. That needs the approach in [Diagnosing Kubernetes RBAC Forbidden 403](/blog/k8s-forbidden-오류-rbac부터-serviceaccount까지-5단계로-완벽-진단하는-방법). One more boundary: the previously published [6 Ways to Fix kubectl Unauthorized (You must be logged in) 401](/blog/kubectl-unauthorizedyou-must-be-logged-in-401-해결-6가지) covers **broken kubeconfig authentication on your laptop**. This article is about the ServiceAccount token that **workloads running inside a Pod use when calling the API server**. The audience is code that uses `rest.InClusterConfig()` from inside a container — operators, sidecars, ingress controllers, in-cluster CI runners, and the like. The reason this symptom exploded after 2022 is clear. Kubernetes 1.24 **stopped auto-creating Secrets for ServiceAccounts**, and the BoundServiceAccountTokenVolume migration that had already been underway turned tokens into **short-lived, expiring credentials**. If you take a manifest that worked on 1.23 or earlier and apply it unchanged to a 1.24+ cluster, pipelines that directly referenced `secretName` fail silently. ## 30-second triage table: reverse index by error text Find the exact string you typed into the search box in the table below. The "Jump" column on each row points to the matching fix section. | Error text / symptom | Primary cause | Jump | |---|---|---| | `open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory` | automount disabled (SA or Pod level) | [automount precedence](#a-automount-precedence-the-pod-wins-over-the-sa) | | `Unauthorized` (the single word, as-is in the body) | Token expired, or the app is caching an old token | [token caching](#c-the-trap-of-reading-the-token-once-and-caching-it) | | `the server has asked for the client to provide credentials` | Token never attached to the request (wrong path or empty file) | [automount precedence](#a-automount-precedence-the-pod-wins-over-the-sa) | | `token is expired` / `Token has expired` | Token expired (see the expiry note below the version table) + the app never re-reads it | [token caching](#c-the-trap-of-reading-the-token-once-and-caching-it) | | `serviceaccounts "xxx" not found` | SA never created, or namespace mismatch | [two-line diagnostic cut](#version-behavior-differences-and-a-two-line-diagnostic-cut) | | You created an SA but `kubectl get secret` shows no token Secret | Policy change in 1.24+ (auto-creation removed) | [when you need a long-lived token](#d-when-you-need-a-long-lived-token-for-ci-or-external-systems) | | You applied every fix and still get 401 | Audience mismatch / issuer config / node clock skew | [still getting 401](#still-getting-401-failure-branch-checklist) | ## Version behavior differences and a two-line diagnostic cut The answer here is **"start by checking your cluster's minor version."** The same YAML behaves completely differently depending on the version. | Version | Token form | Expiry | SA Secret auto-creation | Who renews | |---|---|---|---|---| | ~1.20 | Secret-based legacy JWT | None (never expires) | Yes (automatic on SA creation) | None | | 1.21 | Projected bound token mounted by default (BoundServiceAccountTokenVolume beta) | kubelet requests 3607 s; up to 1 year when extended (see below) | Yes | kubelet | | 1.22~1.23 | Same (GA) | Same | Yes | kubelet | | 1.24~1.25 | Projected bound token | Same | **No** (LegacyServiceAccountTokenNoAutoGeneration beta) — use `kubectl create token` | kubelet | | 1.26~1.28 | Same | Same | No (GA). From 1.28 the legacy-token last-used label `kubernetes.io/legacy-token-last-used` is GA | kubelet | | 1.29+ | Same | Same | No. Cleanup of auto-generated legacy tokens (1.29 beta, 1.30 GA) | kubelet + cleanup controller | Feature stages follow the [feature gate list](https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates-removed/); as of October 2026 the supported releases are 1.35, 1.36, and 1.37 ([Kubernetes Releases](https://kubernetes.io/releases/)). **Watch the expiry**: kubelet requests the default projected token with `expirationSeconds: 3607`, but when kube-apiserver's `--service-account-extend-token-expiration` (default true) is on, admission-injected tokens are extended up to 1 year ([kube-apiserver options](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/), [Managing Service Accounts](https://kubernetes.io/docs/reference/access-authn-authz/service-accounts-admin/)). So on a cluster with default settings, a decoded token's `exp` may be far later than one hour. Don't diagnose on the assumption of a 1-hour expiry; check the real `exp` with the command below. Managed services may change this flag, so check your provider's docs too. **What the legacy cleanup covers**: from 1.29 the cleanup controller only handles **auto-generated** legacy token Secrets (the ones referenced from the ServiceAccount's `secrets` field). If unused for one year by default, they are labeled `kubernetes.io/legacy-token-invalid-since` and invalidated, then deleted after another year ([Managing Service Accounts](https://kubernetes.io/docs/reference/access-authn-authz/service-accounts-admin/)). Hand-made `kubernetes.io/service-account-token` Secrets are not cleaned up, but they never expire either, so pipelines that rely on them take a large hit if one leaks. ### Three diagnostic commands (copy-paste ready) **1) Check SA-level automount settings** ```bash kubectl -n get sa -o jsonpath='{.metadata.name}{" automount="}{.automountServiceAccountToken}{"\n"}' ``` Expected healthy result: `my-sa automount=` (empty means the default of true, so it is mounted). If you see `automount=false` → go to the automount branch. **2) Confirm the token file actually exists inside the Pod, and decode the expiry** ```bash kubectl -n exec -- ls -l /var/run/secrets/kubernetes.io/serviceaccount/ kubectl -n exec -- cat /var/run/secrets/kubernetes.io/serviceaccount/token \ | cut -d. -f2 | tr '_-' '/+' \ | awk '{n=length($0)%4; if(n) $0=$0 substr("==",1,4-n); print}' \ | base64 -d | jq '.exp, .iat, .aud, .sub' ``` Expected healthy result: you see the three files `ca.crt`, `namespace`, and `token`, and the JSON prints `exp` (expiry epoch), `aud` (audience: the cluster's `--api-audiences`, or the issuer if unset; the kubeadm default is `https://kubernetes.default.svc.cluster.local`, [kubeadm source](https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/phases/controlplane/manifests.go)), and `sub` (`system:serviceaccount::`). Branches: - `No such file or directory` → automount is disabled. Section (a). - The file's `exp` is in the future but the app gets 401 → the app is likely holding a pre-rotation token in memory. Section (c). - The file's own `exp` is already in the past → kubelet failed to rotate the token. Check that node's kubelet logs and clock. - `aud` contains only an external value like `sts.amazonaws.com` → this is not an API-server audience. See the failure-branch section. - `base64: invalid input` or a jq parse error → the token file is empty or not a JWT. The JWT payload is unpadded base64url, so feeding it straight to `base64 -d` fails even for a valid token; that's why the command above runs `tr` and adds padding. - If the image has no `cat` (distroless), attach an ephemeral container with `kubectl debug`, or first confirm issuance with `kubectl create token`. **3) Verify that token issuance itself works** ```bash kubectl -n create token --duration=30m ``` If healthy, you get a JWT starting with `eyJ...`. If you get `serviceaccounts "xxx" not found`, the SA itself is missing or you looked at the wrong namespace — that's a missing-object problem, not RBAC. ## Fixes by cause ### (a) automount precedence: the Pod wins over the SA The answer here is to remember that **"the Pod spec overrides the ServiceAccount setting."** Even if you turned it on at the SA, it will not be mounted if the Pod turned it off. | SA `automountServiceAccountToken` | Pod spec `automountServiceAccountToken` | Result | |---|---|---| | Unspecified (default true) | Unspecified | Mounted ✅ | | `false` | Unspecified | **Not mounted** ❌ | | `false` | `true` | Mounted ✅ (Pod wins) | | Unspecified / `true` | `false` | **Not mounted** ❌ (Pod wins) | Command to see both values at once: ```bash kubectl -n get pod -o jsonpath='{.spec.serviceAccountName}{" podAutomount="}{.spec.automountServiceAccountToken}{"\n"}' ``` A common cause is security hardening (Kyverno/OPA policies and the like) that bulk-disables automount on the `default` SA. [Pod Security Standards](https://kubernetes.io/docs/concepts/security/pod-security-standards/) (including restricted) do not check this field, so enabling PSS alone does not remove the token. Keep the policy, but carve out a Pod-level exception only for workloads that actually need to call the API. ```yaml spec: serviceAccountName: my-operator-sa automountServiceAccountToken: true # exception for this Pod even if policy sets the SA to false ``` ### (b) Control expiry and audience directly with a projected volume The answer here is **"don't rely on automount — declare a projected volume."** You can pin expiry and audience to the values your code expects. ```yaml apiVersion: v1 kind: Pod metadata: name: api-caller spec: serviceAccountName: my-operator-sa automountServiceAccountToken: false # turn off the default mount containers: - name: app image: my/app:1.0 volumeMounts: - name: sa-token mountPath: /var/run/secrets/tokens readOnly: true volumes: - name: sa-token projected: sources: - serviceAccountToken: path: token expirationSeconds: 3600 # min 600; may be capped by the API server's --service-account-max-token-expiration # audience omitted → the API server's default audience (for API calls). Set it only for other recipients - configMap: name: kube-root-ca.crt # default mount is off, so add the CA as well items: - key: ca.crt path: ca.crt ``` If you omit `audience`, it defaults to the API server's identifier ([Projected Volumes](https://kubernetes.io/docs/concepts/storage/projected-volumes/)). A value that is not in the API server's `--api-audiences` makes the token fail API calls with 401. Also, `rest.InClusterConfig()` only reads `token` and `ca.crt` from `/var/run/secrets/kubernetes.io/serviceaccount/` ([client-go source](https://github.com/kubernetes/client-go/blob/master/rest/config.go)), so if you move the path as above, the app must point to the token and CA paths itself. To keep using the default client unchanged, set mountPath to `/var/run/secrets/kubernetes.io/serviceaccount`. Conversely, a token with `audience` set is valid only for that audience. EKS IRSA and GKE Workload Identity use exactly this mechanism — they inject a cloud-STS audience token at a separate path. That's why **reusing the IRSA token path for API calls produces 401**. The two tokens serve different purposes. ### (c) The trap of reading the token once and caching it The answer here is **"re-read the file periodically."** kubelet renews the token file before it expires, but if the application reads it once at process start and holds it in memory, it will never see the renewed copy. The result is `Unauthorized` or `token is expired` when the token actually expires (the `expirationSeconds` you set, or up to 1 year for admission-injected tokens when extension is on). kubelet rotates the token once it is older than 80% of its TTL or 24 hours, and reloading it is the application's job ([Configure Service Accounts](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/)). - **client-go**: `rest.InClusterConfig()` sets `BearerTokenFile`, and the client re-reads that file every minute ([client-go source](https://github.com/kubernetes/client-go/blob/master/transport/token_source.go)). This path is usually fine. - **Code that `os.ReadFile`s once and stuffs it into a header / curl scripts / homegrown HTTP clients**: they never see the renewal. This is where most incidents happen. Minimal Go pattern that re-reads on every request: ```go const tokenPath = "/var/run/secrets/kubernetes.io/serviceaccount/token" type reloadingTransport struct{ base http.RoundTripper } func (t *reloadingTransport) RoundTrip(req *http.Request) (*http.Response, error) { b, err := os.ReadFile(tokenPath) // re-read on every request (cheap: it's tmpfs) if err != nil { return nil, fmt.Errorf("reading SA token: %w", err) } r := req.Clone(req.Context()) r.Header.Set("Authorization", "Bearer "+strings.TrimSpace(string(b))) return t.base.RoundTrip(r) } ``` If you call from Bash or a sidecar, don't stash the token in a variable — just `$(cat ...)` every time and you're done. ```bash curl -sS --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt \ -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \ https://kubernetes.default.svc/api/v1/namespaces/default/pods ``` The official Python `kubernetes` client's `config.load_incluster_config()` re-reads the token file every minute by default (`try_refresh_token=True`) ([source](https://github.com/kubernetes-client/python/blob/master/kubernetes/config/incluster_config.py)). Code that pulls the token string out and puts it into another HTTP client's header still has to re-read it each time, as in the Go example. ### (d) When you need a long-lived token for CI or external systems The answer here is **"periodic `kubectl create token` instead of a hand-made Secret."** ```bash # issue a short-lived token per CI pipeline step (recommended) TOKEN=$(kubectl -n ci create token ci-runner-sa --duration=1h) ``` | Method | Expiry | Revocation on leak | Audit trail | Recommended | |---|---|---|---|---| | `kubectl create token --duration` | Specified short period | Natural expiry | Issuance event is recorded | ✅ | | Manual `kubernetes.io/service-account-token` Secret | Effectively never | Immediate via Secret deletion | Hard to track usage | ⚠️ last resort | | Cloud workload identity (IRSA / Pod Identity / GKE WI) | Short-lived, auto-renewed | Controlled by IAM policy | Cloud audit logs | ✅ (cloud only) | `--duration` can be capped by the API server's `--service-account-max-token-expiration`. The issued token may be shorter than requested, so decode `exp` after issuance and check the actual value. If a manual Secret is truly unavoidable (e.g., a legacy external system that cannot refresh tokens), create a dedicated least-privilege SA, record the lack of expiry in your risk register, and put a regular rotation schedule on the calendar. That's the minimum defensive line. ## Security: disable, revoke, track **Disable.** Turning off automount on the `default` SA is a good default. You stop spraying credentials onto the majority of workloads that never call the API. ```bash kubectl -n patch serviceaccount default -p '{"automountServiceAccountToken": false}' ``` Blast radius: any workload in this namespace that uses the `default` SA and calls the API will break immediately. Before applying, scan which SA each Pod in the namespace uses. ```bash kubectl -n get pods -o custom-columns='POD:.metadata.name,SA:.spec.serviceAccountName' ``` **Revoke.** This is the important part — **bound tokens cannot be invalidated individually.** | Token type | How to revoke on leak | |---|---| | Projected bound token | No per-token revoke API. **Deleting the Pod** the token is bound to invalidates that token. **Deleting and recreating the SA** invalidates every token for that SA, and all Pods using it must be restarted. Waiting for expiry can take up to 1 year because of the extension setting above | | Legacy Secret token | Delete the Secret → immediately invalid | | Node-bound token | Automatically invalid when the Pod/node is deleted (the binding target is gone) | Bound tokens are tied to a Pod via `.spec.boundObjectRef`, so when the Pod disappears the token loses validity. In incident response, "delete the Pod" is itself a partial revocation action. **Track.** Query pattern to extract a given SA's usage history from the audit log: ```bash # extract only one SA's calls from audit-log JSON lines jq -c 'select(.user.username == "system:serviceaccount:prod:my-operator-sa") | {ts: .requestReceivedTimestamp, verb, uri: .requestURI, ip: .sourceIPs[0], code: .responseStatus.code}' \ audit.log | head -50 ``` If `responseStatus.code` is 401 it's an authentication failure; 403 is authorization — one log line tells you whether this article or part 3 applies. Unexpected `sourceIPs` are grounds to suspect a token leak. ## Still getting 401: failure-branch checklist If you've applied every fix above and still get 401, check these in order. 1. **Audience mismatch.** Confirm the token's `aud` is a value the API server accepts. Check the default audience with `kubectl create token | cut -d. -f2 | tr '_-' '/+' | awk '{n=length($0)%4; if(n) $0=$0 substr("==",1,4-n); print}' | base64 -d | jq .aud` and compare it to the Pod's token. On clusters using IRSA/Workload Identity, mistakenly using the cloud token for API calls is a common case. 2. **API server flags.** Check that `--service-account-issuer`, `--api-audiences`, and `--service-account-key-file` are consistent on both the issue and verify sides. Managed clusters (EKS/GKE/AKS) cannot be edited directly, so if something looks off at this step you need the cloud vendor's official docs. 3. **Node clock skew.** If `iat`/`nbf` is in the future, the API server rejects the token. Compare `kubectl exec -- date -u` with control-plane time, and check the node's NTP (chronyd/systemd-timesyncd) sync status. A skew of just a few minutes is enough to reproduce this. 4. **Auth webhook/proxy.** Check whether an in-house auth proxy or service mesh is overwriting or stripping the `Authorization` header. This often shows up in setups where an mTLS sidecar rewrites headers. 5. **Newlines/whitespace in the token file.** If a newline sneaks into a header assembled in the shell, the header breaks. Always use `tr -d '\n'` or `TrimSpace`. ### Preventing recurrence - Set `automountServiceAccountToken` **explicitly** in deploy templates so default-value changes don't shake you. - For apps that call the API, put "does it re-read the token?" on the code-review checklist. - In staging, run with a short expiry like `expirationSeconds: 600` so expiry-handling bugs surface early. - Before a cluster upgrade, inventory every hand-made SA token Secret and every `secretName` reference. ```bash kubectl get secrets -A --field-selector type=kubernetes.io/service-account-token ``` If this returns results, you are still depending on the legacy path in a 1.24+ environment. Treat the output as your migration backlog. ## FAQ **Q1. After upgrading to 1.24, creating an SA no longer creates a Secret. Is this a bug?** No — it's an intentional change. From 1.24, creating a ServiceAccount no longer auto-creates a token Secret. If you need a token, issue a short-lived one with `kubectl create token `, or use a projected volume (the default automount) inside the Pod. **Q2. How do I tell 401 from 403 from logs alone?** Split on the HTTP status code. 401 is "I don't know who you are" (missing/expired token or signature verification failure); 403 is "I know who you are, but you don't have permission." A 403 message usually includes the identity, like `User "system:serviceaccount:ns:sa" cannot list resource ...`. If the identity is printed, authentication succeeded — look at RBAC. **Q3. The token changes periodically — do I have to restart the app each time?** No. kubelet renews the token file before it expires. The app just has to re-read the file. If a restart is required, that's a caching bug — apply the re-read pattern above. **Q4. Can I set `expirationSeconds` very long (e.g. one year)?** You can request it, but the API server policy cap may truncate it, and it is not recommended for security. Short-lived credentials are a baseline assumption of a zero-trust model. If an external integration truly needs long-lived credentials, prefer cloud workload identity or a periodic re-issue pipeline over a long-lived token. **Q5. A token leaked. Can I invalidate it immediately?** Bound tokens have no per-token cancel API. Realistic options are (1) delete and recreate the SA to invalidate every existing token and restart related Pods, (2) if the token is bound to a Pod, delete that Pod, or (3) wait for expiry (admission-injected tokens can last up to 1 year because of the extension setting). For a legacy Secret token, deleting the Secret revokes it immediately. In parallel, always pull the call history for the leak window from the audit log. **Q6. What breaks if I turn off automount on the `default` SA?** Every Pod in that namespace that uses the `default` SA and calls the API server loses its token file. Typical casualties include some monitoring agents, apps that do service discovery, and apps with their own leader-election logic. Before applying, query the namespace's Pod-to-SA mapping and grant a Pod-level `automountServiceAccountToken: true` exception where needed. **Q7. I'm using IRSA on EKS and API server calls return 401.** The token IRSA injects (`AWS_WEB_IDENTITY_TOKEN_FILE`) has audience `sts.amazonaws.com` — it is AWS STS only. Kubernetes API calls must use `/var/run/secrets/kubernetes.io/serviceaccount/token`. First check that you didn't mix up the two paths. --- In part 5 we'll cover **network-level blocking — diagnosing NetworkPolicy and mTLS failures**. When authentication and authorization both pass but the connection itself is blocked, we'll use the same style of lookup table to split `connection refused` from `i/o timeout` and reverse-engineer the policy rules.

#쿠버네티스 토큰 갱신

쿠버네티스 ServiceAccount 토큰 없음·401 Unauthorized 해결 (1.24+)