Fixing Missing Kubernetes ServiceAccount Tokens and 401 Unauthorized (1.24+)

Errors this covers (workloads inside a Pod): open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory, an API response of 401 Unauthorized, or the server has asked for the client to provide credentials. If you get 403 Forbidden (User "system:serviceaccount:..." cannot ...), authentication already succeeded; go to diagnosing RBAC Forbidden.

Part 3 is 403; this part is 401 and "token file not found"

Let's start with the conclusion: 401 Unauthorized and "token file not found" are not RBAC problems. They live on an entirely different layer.

• 401 Unauthorized / missing token → authentication failure. You couldn't answer "who are you?" No amount of RoleBinding fixes will ever resolve this.
• 403 Forbidden → authorization failure. Identity was verified, but you lack permission. That needs the approach in Diagnosing Kubernetes RBAC Forbidden 403.

One more boundary: the previously published 6 Ways to Fix kubectl Unauthorized (You must be logged in) 401 covers broken kubeconfig authentication on your laptop. This article is about the ServiceAccount token that workloads running inside a Pod use when calling the API server. The audience is code that uses rest.InClusterConfig() from inside a container — operators, sidecars, ingress controllers, in-cluster CI runners, and the like.

The reason this symptom exploded after 2022 is clear. Kubernetes 1.24 stopped auto-creating Secrets for ServiceAccounts, and the BoundServiceAccountTokenVolume migration that had already been underway turned tokens into short-lived, expiring credentials. If you take a manifest that worked on 1.23 or earlier and apply it unchanged to a 1.24+ cluster, pipelines that directly referenced secretName fail silently.

30-second triage table: reverse index by error text

Find the exact string you typed into the search box in the table below. The "Jump" column on each row points to the matching fix section.

Version behavior differences and a two-line diagnostic cut

The answer here is "start by checking your cluster's minor version." The same YAML behaves completely differently depending on the version.

Feature stages follow the feature gate list; as of October 2026 the supported releases are 1.35, 1.36, and 1.37 (Kubernetes Releases).

Watch the expiry: kubelet requests the default projected token with expirationSeconds: 3607, but when kube-apiserver's --service-account-extend-token-expiration (default true) is on, admission-injected tokens are extended up to 1 year (kube-apiserver options, Managing Service Accounts). So on a cluster with default settings, a decoded token's exp may be far later than one hour. Don't diagnose on the assumption of a 1-hour expiry; check the real exp with the command below. Managed services may change this flag, so check your provider's docs too.

What the legacy cleanup covers: from 1.29 the cleanup controller only handles auto-generated legacy token Secrets (the ones referenced from the ServiceAccount's secrets field). If unused for one year by default, they are labeled kubernetes.io/legacy-token-invalid-since and invalidated, then deleted after another year (Managing Service Accounts). Hand-made kubernetes.io/service-account-token Secrets are not cleaned up, but they never expire either, so pipelines that rely on them take a large hit if one leaks.

Three diagnostic commands (copy-paste ready)

1) Check SA-level automount settings

kubectl -n <ns> get sa <sa-name> -o jsonpath='{.metadata.name}{"  automount="}{.automountServiceAccountToken}{"\n"}'

Expected healthy result: my-sa automount= (empty means the default of true, so it is mounted). If you see automount=false → go to the automount branch.

2) Confirm the token file actually exists inside the Pod, and decode the expiry

kubectl -n <ns> exec <pod> -- ls -l /var/run/secrets/kubernetes.io/serviceaccount/
kubectl -n <ns> exec <pod> -- cat /var/run/secrets/kubernetes.io/serviceaccount/token \
  | cut -d. -f2 | tr '_-' '/+' \
  | awk '{n=length($0)%4; if(n) $0=$0 substr("==",1,4-n); print}' \
  | base64 -d | jq '.exp, .iat, .aud, .sub'

Expected healthy result: you see the three files ca.crt, namespace, and token, and the JSON prints exp (expiry epoch), aud (audience: the cluster's --api-audiences, or the issuer if unset; the kubeadm default is https://kubernetes.default.svc.cluster.local, kubeadm source), and sub (system:serviceaccount:<ns>:<sa>).

Branches:

• No such file or directory → automount is disabled. Section (a).
• The file's exp is in the future but the app gets 401 → the app is likely holding a pre-rotation token in memory. Section (c).
• The file's own exp is already in the past → kubelet failed to rotate the token. Check that node's kubelet logs and clock.
• aud contains only an external value like sts.amazonaws.com → this is not an API-server audience. See the failure-branch section.
• base64: invalid input or a jq parse error → the token file is empty or not a JWT. The JWT payload is unpadded base64url, so feeding it straight to base64 -d fails even for a valid token; that's why the command above runs tr and adds padding.
• If the image has no cat (distroless), attach an ephemeral container with kubectl debug, or first confirm issuance with kubectl create token.

3) Verify that token issuance itself works

kubectl -n <ns> create token <sa-name> --duration=30m

If healthy, you get a JWT starting with eyJ.... If you get serviceaccounts "xxx" not found, the SA itself is missing or you looked at the wrong namespace — that's a missing-object problem, not RBAC.

Fixes by cause

(a) automount precedence: the Pod wins over the SA

The answer here is to remember that "the Pod spec overrides the ServiceAccount setting." Even if you turned it on at the SA, it will not be mounted if the Pod turned it off.

Command to see both values at once:

kubectl -n <ns> get pod <pod> -o jsonpath='{.spec.serviceAccountName}{" podAutomount="}{.spec.automountServiceAccountToken}{"\n"}'

A common cause is security hardening (Kyverno/OPA policies and the like) that bulk-disables automount on the default SA. Pod Security Standards (including restricted) do not check this field, so enabling PSS alone does not remove the token. Keep the policy, but carve out a Pod-level exception only for workloads that actually need to call the API.

spec:
  serviceAccountName: my-operator-sa
  automountServiceAccountToken: true   # exception for this Pod even if policy sets the SA to false

(b) Control expiry and audience directly with a projected volume

The answer here is "don't rely on automount — declare a projected volume." You can pin expiry and audience to the values your code expects.

apiVersion: v1
kind: Pod
metadata:
  name: api-caller
spec:
  serviceAccountName: my-operator-sa
  automountServiceAccountToken: false     # turn off the default mount
  containers:
    - name: app
      image: my/app:1.0
      volumeMounts:
        - name: sa-token
          mountPath: /var/run/secrets/tokens
          readOnly: true
  volumes:
    - name: sa-token
      projected:
        sources:
          - serviceAccountToken:
              path: token
              expirationSeconds: 3600      # min 600; may be capped by the API server's --service-account-max-token-expiration
              # audience omitted → the API server's default audience (for API calls). Set it only for other recipients
          - configMap:
              name: kube-root-ca.crt       # default mount is off, so add the CA as well
              items:
                - key: ca.crt
                  path: ca.crt

If you omit audience, it defaults to the API server's identifier (Projected Volumes). A value that is not in the API server's --api-audiences makes the token fail API calls with 401. Also, rest.InClusterConfig() only reads token and ca.crt from /var/run/secrets/kubernetes.io/serviceaccount/ (client-go source), so if you move the path as above, the app must point to the token and CA paths itself. To keep using the default client unchanged, set mountPath to /var/run/secrets/kubernetes.io/serviceaccount.

Conversely, a token with audience set is valid only for that audience. EKS IRSA and GKE Workload Identity use exactly this mechanism — they inject a cloud-STS audience token at a separate path. That's why reusing the IRSA token path for API calls produces 401. The two tokens serve different purposes.

(c) The trap of reading the token once and caching it

The answer here is "re-read the file periodically." kubelet renews the token file before it expires, but if the application reads it once at process start and holds it in memory, it will never see the renewed copy. The result is Unauthorized or token is expired when the token actually expires (the expirationSeconds you set, or up to 1 year for admission-injected tokens when extension is on). kubelet rotates the token once it is older than 80% of its TTL or 24 hours, and reloading it is the application's job (Configure Service Accounts).

• client-go: rest.InClusterConfig() sets BearerTokenFile, and the client re-reads that file every minute (client-go source). This path is usually fine.
• Code that os.ReadFiles once and stuffs it into a header / curl scripts / homegrown HTTP clients: they never see the renewal. This is where most incidents happen.

Minimal Go pattern that re-reads on every request:

const tokenPath = "/var/run/secrets/kubernetes.io/serviceaccount/token"

type reloadingTransport struct{ base http.RoundTripper }

func (t *reloadingTransport) RoundTrip(req *http.Request) (*http.Response, error) {
	b, err := os.ReadFile(tokenPath) // re-read on every request (cheap: it's tmpfs)
	if err != nil {
		return nil, fmt.Errorf("reading SA token: %w", err)
	}
	r := req.Clone(req.Context())
	r.Header.Set("Authorization", "Bearer "+strings.TrimSpace(string(b)))
	return t.base.RoundTrip(r)
}

If you call from Bash or a sidecar, don't stash the token in a variable — just $(cat ...) every time and you're done.

curl -sS --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
  -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
  https://kubernetes.default.svc/api/v1/namespaces/default/pods

The official Python kubernetes client's config.load_incluster_config() re-reads the token file every minute by default (try_refresh_token=True) (source). Code that pulls the token string out and puts it into another HTTP client's header still has to re-read it each time, as in the Go example.

(d) When you need a long-lived token for CI or external systems

The answer here is "periodic kubectl create token instead of a hand-made Secret."

# issue a short-lived token per CI pipeline step (recommended)
TOKEN=$(kubectl -n ci create token ci-runner-sa --duration=1h)

--duration can be capped by the API server's --service-account-max-token-expiration. The issued token may be shorter than requested, so decode exp after issuance and check the actual value.

If a manual Secret is truly unavoidable (e.g., a legacy external system that cannot refresh tokens), create a dedicated least-privilege SA, record the lack of expiry in your risk register, and put a regular rotation schedule on the calendar. That's the minimum defensive line.

Security: disable, revoke, track

Disable. Turning off automount on the default SA is a good default. You stop spraying credentials onto the majority of workloads that never call the API.

kubectl -n <ns> patch serviceaccount default -p '{"automountServiceAccountToken": false}'

Blast radius: any workload in this namespace that uses the default SA and calls the API will break immediately. Before applying, scan which SA each Pod in the namespace uses.

kubectl -n <ns> get pods -o custom-columns='POD:.metadata.name,SA:.spec.serviceAccountName'

Revoke. This is the important part — bound tokens cannot be invalidated individually.

Bound tokens are tied to a Pod via .spec.boundObjectRef, so when the Pod disappears the token loses validity. In incident response, "delete the Pod" is itself a partial revocation action.

Track. Query pattern to extract a given SA's usage history from the audit log:

# extract only one SA's calls from audit-log JSON lines
jq -c 'select(.user.username == "system:serviceaccount:prod:my-operator-sa")
       | {ts: .requestReceivedTimestamp, verb, uri: .requestURI, ip: .sourceIPs[0], code: .responseStatus.code}' \
  audit.log | head -50

If responseStatus.code is 401 it's an authentication failure; 403 is authorization — one log line tells you whether this article or part 3 applies. Unexpected sourceIPs are grounds to suspect a token leak.

Still getting 401: failure-branch checklist

If you've applied every fix above and still get 401, check these in order.

1. Audience mismatch. Confirm the token's aud is a value the API server accepts. Check the default audience with kubectl create token <sa> | cut -d. -f2 | tr '_-' '/+' | awk '{n=length($0)%4; if(n) $0=$0 substr("==",1,4-n); print}' | base64 -d | jq .aud and compare it to the Pod's token. On clusters using IRSA/Workload Identity, mistakenly using the cloud token for API calls is a common case.
2. API server flags. Check that --service-account-issuer, --api-audiences, and --service-account-key-file are consistent on both the issue and verify sides. Managed clusters (EKS/GKE/AKS) cannot be edited directly, so if something looks off at this step you need the cloud vendor's official docs.
3. Node clock skew. If iat/nbf is in the future, the API server rejects the token. Compare kubectl exec <pod> -- date -u with control-plane time, and check the node's NTP (chronyd/systemd-timesyncd) sync status. A skew of just a few minutes is enough to reproduce this.
4. Auth webhook/proxy. Check whether an in-house auth proxy or service mesh is overwriting or stripping the Authorization header. This often shows up in setups where an mTLS sidecar rewrites headers.
5. Newlines/whitespace in the token file. If a newline sneaks into a header assembled in the shell, the header breaks. Always use tr -d '\n' or TrimSpace.

Preventing recurrence

• Set automountServiceAccountToken explicitly in deploy templates so default-value changes don't shake you.
• For apps that call the API, put "does it re-read the token?" on the code-review checklist.
• In staging, run with a short expiry like expirationSeconds: 600 so expiry-handling bugs surface early.
• Before a cluster upgrade, inventory every hand-made SA token Secret and every secretName reference.

kubectl get secrets -A --field-selector type=kubernetes.io/service-account-token

If this returns results, you are still depending on the legacy path in a 1.24+ environment. Treat the output as your migration backlog.

FAQ

Q1. After upgrading to 1.24, creating an SA no longer creates a Secret. Is this a bug? No — it's an intentional change. From 1.24, creating a ServiceAccount no longer auto-creates a token Secret. If you need a token, issue a short-lived one with kubectl create token <sa>, or use a projected volume (the default automount) inside the Pod.

Q2. How do I tell 401 from 403 from logs alone? Split on the HTTP status code. 401 is "I don't know who you are" (missing/expired token or signature verification failure); 403 is "I know who you are, but you don't have permission." A 403 message usually includes the identity, like User "system:serviceaccount:ns:sa" cannot list resource .... If the identity is printed, authentication succeeded — look at RBAC.

Q3. The token changes periodically — do I have to restart the app each time? No. kubelet renews the token file before it expires. The app just has to re-read the file. If a restart is required, that's a caching bug — apply the re-read pattern above.

Q4. Can I set expirationSeconds very long (e.g. one year)? You can request it, but the API server policy cap may truncate it, and it is not recommended for security. Short-lived credentials are a baseline assumption of a zero-trust model. If an external integration truly needs long-lived credentials, prefer cloud workload identity or a periodic re-issue pipeline over a long-lived token.

Q5. A token leaked. Can I invalidate it immediately? Bound tokens have no per-token cancel API. Realistic options are (1) delete and recreate the SA to invalidate every existing token and restart related Pods, (2) if the token is bound to a Pod, delete that Pod, or (3) wait for expiry (admission-injected tokens can last up to 1 year because of the extension setting). For a legacy Secret token, deleting the Secret revokes it immediately. In parallel, always pull the call history for the leak window from the audit log.

Q6. What breaks if I turn off automount on the default SA? Every Pod in that namespace that uses the default SA and calls the API server loses its token file. Typical casualties include some monitoring agents, apps that do service discovery, and apps with their own leader-election logic. Before applying, query the namespace's Pod-to-SA mapping and grant a Pod-level automountServiceAccountToken: true exception where needed.

Q7. I'm using IRSA on EKS and API server calls return 401. The token IRSA injects (AWS_WEB_IDENTITY_TOKEN_FILE) has audience sts.amazonaws.com — it is AWS STS only. Kubernetes API calls must use /var/run/secrets/kubernetes.io/serviceaccount/token. First check that you didn't mix up the two paths.

In part 5 we'll cover network-level blocking — diagnosing NetworkPolicy and mTLS failures. When authentication and authorization both pass but the connection itself is blocked, we'll use the same style of lookup table to split connection refused from i/o timeout and reverse-engineer the policy rules.