Fixing Missing Kubernetes ServiceAccount Tokens and 401 Unauthorized (1.24+)

Part 3 is 403; this part is 401 and "token file not found"

Let's start with the conclusion: 401 Unauthorized and "token file not found" are not RBAC problems. They live on an entirely different layer.

• 401 Unauthorized / missing token → authentication failure. You couldn't answer "who are you?" No amount of RoleBinding fixes will ever resolve this.
• 403 Forbidden → authorization failure. Identity was verified, but you lack permission. That needs the approach in Diagnosing Kubernetes RBAC Forbidden 403.

One more boundary: the previously published kubectl Unauthorized: 3-minute diagnosis and recovery runbook (EKS reissue) covers broken kubeconfig authentication on your laptop. This article is about the ServiceAccount token that workloads running inside a Pod use when calling the API server. The audience is code that uses rest.InClusterConfig() from inside a container — operators, sidecars, ingress controllers, in-cluster CI runners, and the like.

The reason this symptom exploded after 2022 is clear. Kubernetes 1.24 stopped auto-creating Secrets for ServiceAccounts, and the BoundServiceAccountTokenVolume migration that had already been underway turned tokens into short-lived, expiring credentials. If you take a manifest that worked on 1.23 or earlier and apply it unchanged to a 1.24+ cluster, pipelines that directly referenced secretName fail silently.

30-second triage table: reverse index by error text

Find the exact string you typed into the search box in the table below. The "Jump" column on each row points to the matching fix section.

Version behavior differences and a two-line diagnostic cut

The answer here is "start by checking your cluster's minor version." The same YAML behaves completely differently depending on the version.

From 1.29 onward, unused legacy SA tokens become cleanup candidates after a period of inactivity, and last-used time is tracked via labels/metrics. In other words, any pipeline that still depends on "a token Secret you created by hand a long time ago" is a ticking time bomb.

Three diagnostic commands (copy-paste ready)

1) Check SA-level automount settings

kubectl -n <ns> get sa <sa-name> -o jsonpath='{.metadata.name}{"  automount="}{.automountServiceAccountToken}{"\n"}'

Expected healthy result: my-sa automount= (empty means the default of true, so it is mounted). If you see automount=false → go to the automount branch.

2) Confirm the token file actually exists inside the Pod, and decode the expiry

kubectl -n <ns> exec <pod> -- ls -l /var/run/secrets/kubernetes.io/serviceaccount/
kubectl -n <ns> exec <pod> -- sh -c 'cat /var/run/secrets/kubernetes.io/serviceaccount/token' | cut -d. -f2 | base64 -d 2>/dev/null | jq '.exp, .iat, .aud, .sub'

Expected healthy result: you see the three files ca.crt, namespace, and token, and the JSON prints exp (expiry epoch), aud (audience, usually ["https://kubernetes.default.svc"]), and sub (system:serviceaccount:<ns>:<sa>).

Branches:

• No such file or directory → automount is disabled. Section (a).
• File exists but exp is in the past → the app likely never re-read the renewed file. Section (c).
• aud contains only an external value like sts.amazonaws.com → this is not an API-server audience. See the failure-branch section.
• base64: invalid input → the token file is empty or in a legacy format.

3) Verify that token issuance itself works

kubectl -n <ns> create token <sa-name> --duration=30m

If healthy, you get a JWT starting with eyJ.... If you get serviceaccounts "xxx" not found, the SA itself is missing or you looked at the wrong namespace — that's a missing-object problem, not RBAC.

Fixes by cause

(a) automount precedence: the Pod wins over the SA

The answer here is to remember that "the Pod spec overrides the ServiceAccount setting." Even if you turned it on at the SA, it will not be mounted if the Pod turned it off.

Command to see both values at once:

kubectl -n <ns> get pod <pod> -o jsonpath='{.spec.serviceAccountName}{" podAutomount="}{.spec.automountServiceAccountToken}{"\n"}'

The most commonly reported cause in production is a security-hardening policy (PSS restricted profile, Kyverno/OPA rules) that bulk-disables automount on the default SA. Keep the policy, but carve out a Pod-level exception only for workloads that actually need to call the API.

spec:
  serviceAccountName: my-operator-sa
  automountServiceAccountToken: true   # 정책으로 SA가 false여도 이 Pod만 예외

(b) Control expiry and audience directly with a projected volume

The answer here is "don't rely on automount — declare a projected volume." You can pin expiry and audience to the values your code expects.

apiVersion: v1
kind: Pod
metadata:
  name: api-caller
spec:
  serviceAccountName: my-operator-sa
  automountServiceAccountToken: false     # 기본 마운트는 끄고
  containers:
    - name: app
      image: my/app:1.0
      volumeMounts:
        - name: sa-token
          mountPath: /var/run/secrets/tokens
          readOnly: true
  volumes:
    - name: sa-token
      projected:
        sources:
          - serviceAccountToken:
              path: token
              expirationSeconds: 3600      # 최소 600, 요청값은 클러스터 정책에 의해 조정될 수 있음
              audience: https://kubernetes.default.svc

Setting audience makes the token valid only for that audience. EKS IRSA and GKE Workload Identity use exactly this mechanism — they inject a cloud-STS audience token at a separate path. That's why reusing the IRSA token path for API calls produces 401. The two tokens serve different purposes.

(c) The trap of reading the token once and caching it

The answer here is "re-read the file periodically." kubelet renews the token file before it expires, but if the application reads it once at process start and holds it in memory, it will never see the renewed copy. The result is Unauthorized or token is expired exactly one hour later.

• Recent client-go: clients built via rest.InClusterConfig() detect token-file changes and re-read. This path is usually fine.
• Code that os.ReadFiles once and stuffs it into a header / curl scripts / homegrown HTTP clients: they never see the renewal. This is where most incidents happen.

Minimal Go pattern that re-reads on every request:

const tokenPath = "/var/run/secrets/kubernetes.io/serviceaccount/token"

type reloadingTransport struct{ base http.RoundTripper }

func (t *reloadingTransport) RoundTrip(req *http.Request) (*http.Response, error) {
	b, err := os.ReadFile(tokenPath) // 매 요청 재읽기 (tmpfs라 비용 낮음)
	if err != nil {
		return nil, fmt.Errorf("SA 토큰 읽기 실패: %w", err)
	}
	r := req.Clone(req.Context())
	r.Header.Set("Authorization", "Bearer "+strings.TrimSpace(string(b)))
	return t.base.RoundTrip(r)
}

If you call from Bash or a sidecar, don't stash the token in a variable — just $(cat ...) every time and you're done.

curl -sS --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
  -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
  https://kubernetes.default.svc/api/v1/namespaces/default/pods

Same for Python (kubernetes client): if you call config.load_incluster_config() once at process start and run as a long-lived daemon, you need periodic re-load or re-auth logic.

(d) When you need a long-lived token for CI or external systems

The answer here is "periodic kubectl create token instead of a hand-made Secret."

# CI 파이프라인 스텝마다 짧게 발급 (권장)
TOKEN=$(kubectl -n ci create token ci-runner-sa --duration=1h)

--duration can be capped by the API server's --service-account-max-token-expiration. The issued token may be shorter than requested, so decode exp after issuance and check the actual value.

If a manual Secret is truly unavoidable (e.g., a legacy external system that cannot refresh tokens), create a dedicated least-privilege SA, record the lack of expiry in your risk register, and put a regular rotation schedule on the calendar. That's the minimum defensive line.

Security: disable, revoke, track

Disable. Turning off automount on the default SA is a good default. You stop spraying credentials onto the majority of workloads that never call the API.

kubectl -n <ns> patch serviceaccount default -p '{"automountServiceAccountToken": false}'

Blast radius: any workload in this namespace that uses the default SA and calls the API will break immediately. Before applying, scan which SA each Pod in the namespace uses.

kubectl -n <ns> get pods -o custom-columns='POD:.metadata.name,SA:.spec.serviceAccountName'

Revoke. This is the important part — bound tokens cannot be invalidated individually.

Bound tokens are tied to a Pod via .spec.boundObjectRef, so when the Pod disappears the token loses validity. In incident response, "delete the Pod" is itself a partial revocation action.

Track. Query pattern to extract a given SA's usage history from the audit log:

# 감사로그 JSON 라인에서 특정 SA의 호출만 추출
jq -c 'select(.user.username == "system:serviceaccount:prod:my-operator-sa")
       | {ts: .requestReceivedTimestamp, verb, uri: .requestURI, ip: .sourceIPs[0], code: .responseStatus.code}' \
  audit.log | head -50

If responseStatus.code is 401 it's an authentication failure; 403 is authorization — one log line tells you whether this article or part 3 applies. Unexpected sourceIPs are grounds to suspect a token leak.

Still getting 401: failure-branch checklist

If you've applied every fix above and still get 401, check these in order.

1. Audience mismatch. Confirm the token's aud is a value the API server accepts. Check the default audience with kubectl create token <sa> | cut -d. -f2 | base64 -d | jq .aud and compare it to the Pod's token. On clusters using IRSA/Workload Identity, mistakenly using the cloud token for API calls is a common case.
2. API server flags. Check that --service-account-issuer, --api-audiences, and --service-account-key-file are consistent on both the issue and verify sides. Managed clusters (EKS/GKE/AKS) cannot be edited directly, so if something looks off at this step you need the cloud vendor's official docs.
3. Node clock skew. If iat/nbf is in the future, the API server rejects the token. Compare kubectl exec <pod> -- date -u with control-plane time, and check the node's NTP (chronyd/systemd-timesyncd) sync status. A skew of just a few minutes is enough to reproduce this.
4. Auth webhook/proxy. Check whether an in-house auth proxy or service mesh is overwriting or stripping the Authorization header. This often shows up in setups where an mTLS sidecar rewrites headers.
5. Newlines/whitespace in the token file. If a newline sneaks into a header assembled in the shell, the header breaks. Always use tr -d '\n' or TrimSpace.

Preventing recurrence

• Set automountServiceAccountToken explicitly in deploy templates so default-value changes don't shake you.
• For apps that call the API, put "does it re-read the token?" on the code-review checklist.
• In staging, run with a short expiry like expirationSeconds: 600 so expiry-handling bugs surface early.
• Before a cluster upgrade, inventory every hand-made SA token Secret and every secretName reference.

kubectl get secrets -A --field-selector type=kubernetes.io/service-account-token

If this returns results, you are still depending on the legacy path in a 1.24+ environment. Treat the output as your migration backlog.

FAQ

Q1. After upgrading to 1.24, creating an SA no longer creates a Secret. Is this a bug? No — it's an intentional change. From 1.24, creating a ServiceAccount no longer auto-creates a token Secret. If you need a token, issue a short-lived one with kubectl create token <sa>, or use a projected volume (the default automount) inside the Pod.

Q2. How do I tell 401 from 403 from logs alone? Split on the HTTP status code. 401 is "I don't know who you are" (missing/expired token or signature verification failure); 403 is "I know who you are, but you don't have permission." A 403 message usually includes the identity, like User "system:serviceaccount:ns:sa" cannot list resource .... If the identity is printed, authentication succeeded — look at RBAC.

Q3. The token expires every hour — do I have to restart the app every hour? No. kubelet renews the token file before it expires. The app just has to re-read the file. If a restart is required, that's a caching bug — apply the re-read pattern above.

Q4. Can I set expirationSeconds very long (e.g. one year)? You can request it, but the API server policy cap may truncate it, and it is not recommended for security. Short-lived credentials are a baseline assumption of a zero-trust model. If an external integration truly needs long-lived credentials, prefer cloud workload identity or a periodic re-issue pipeline over a long-lived token.

Q5. A token leaked. Can I invalidate it immediately? Bound tokens have no per-token cancel API. Realistic options are (1) delete and recreate the SA to invalidate every existing token and restart related Pods, (2) if the token is bound to a Pod, delete that Pod, or (3) wait for expiry. For a legacy Secret token, deleting the Secret revokes it immediately. In parallel, always pull the call history for the leak window from the audit log.

Q6. What breaks if I turn off automount on the default SA? Every Pod in that namespace that uses the default SA and calls the API server loses its token file. Typical casualties include some monitoring agents, apps that do service discovery, and apps with their own leader-election logic. Before applying, query the namespace's Pod-to-SA mapping and grant a Pod-level automountServiceAccountToken: true exception where needed.

Q7. I'm using IRSA on EKS and API server calls return 401. The token IRSA injects (AWS_WEB_IDENTITY_TOKEN_FILE) has audience sts.amazonaws.com — it is AWS STS only. Kubernetes API calls must use /var/run/secrets/kubernetes.io/serviceaccount/token. First check that you didn't mix up the two paths.

In part 5 we'll cover network-level blocking — diagnosing NetworkPolicy and mTLS failures. When authentication and authorization both pass but the connection itself is blocked, we'll use the same style of lookup table to split connection refused from i/o timeout and reverse-engineer the policy rules.