kubectl Error from server (Forbidden) 403: 5 Causes and How to Fix RBAC

It's 403, not 401 — authentication succeeded, so why were you blocked?

Five minutes before a deploy, this single line in the CI pipeline log makes your stomach drop.

Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:dev:ci-deployer" cannot list resource "pods" in API group "" in the namespace "prod"

The most common accident at this moment is "just slap on cluster-admin and ship the deploy." And that binding is almost never revoked. The point of this post is to break that habit. Read one error line, decide in 30 seconds who / what / where was blocked, narrow it to one of five causes, and recover with least privilege.

First, draw a hard line: 401 and 403 are different problems

If the message is in the Unauthorized family, this post is the wrong one — start with kubectl Unauthorized: 3-minute cause-by-cause diagnosis and recovery runbook (EKS reissue) or 6 ways to fix kubectl Unauthorized (You must be logged in) 401. If authentication never succeeds, authorization never even starts.

If the API server is unreachable entirely and you get The connection to the server localhost:8080 was refused, that belongs in kubectl localhost:8080 refused: 30-second diagnosis and recovery runbook.

Where this sits in the series

A Kubernetes API request flows Authentication → Authorization (RBAC) → Admission.

kubectl / SDK
   │
   ├─ ① Authn  : who is the subject of this request     → fail → 401
   ├─ ② Authz  : is that subject allowed to do this     → fail → 403 (this post)
   └─ ③ Admission: should this object spec be allowed   → PSA violation → create denied

Part 2 of the series covered Pod Security Admission — step ③, "what are we allowed to run." This post is part 3, step ②, "who can do what." Even though both use the word Forbidden, a PSA denial includes a phrase like violates PodSecurity "restricted:latest", so you can tell them apart.

Anatomy of the raw error: the answer is already in that one line

Scope: Kubernetes v1.24+ (RBAC v1 API), kubectl 1.24+, common to EKS, GKE, AKS, and kubeadm clusters. Sample output assumes a typical Linux environment.

Three raw variants

① ServiceAccount subject (most common from CI / inside a pod)

Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:dev:ci-deployer" cannot list resource "pods" in API group "" in the namespace "prod"

② Anonymous subject (the token was never sent)

Error from server (Forbidden): nodes is forbidden: User "system:anonymous" cannot get path "/api/v1/nodes"

③ EKS IAM subject (never mapped to a K8s identity)

Error from server (Forbidden): deployments.apps is forbidden: User "arn:aws:iam::123456789012:role/eks-dev-role" cannot create resource "deployments" in API group "apps" at the cluster scope

Field-by-field anatomy table

Each fragment of the message maps 1:1 to a field in RBAC YAML.

One-line rule: if the message ends with in the namespace X, create a RoleBinding in that namespace; if it says at the cluster scope, you need a ClusterRoleBinding. Following just this line eliminates half of the wasted effort.

30-second triage table — branch on the subject prefix

If you see system:anonymous and you never opened anonymous access, this is a credential-delivery problem, not a permission-grant problem. Creating Roles will never fix it.

Five cause branches: your Forbidden is one of these five

(a) Scope mismatch — the most common cause

There are four Role/ClusterRole × RoleBinding/ClusterRoleBinding combinations, and each covers a different range.

Two easy misses:

• Cluster-scoped resources such as nodes, persistentvolumes, namespaces, storageclasses, and clusterroles cannot be reached no matter how you bind them with a RoleBinding.
• "I want to list pods across all namespaces" also needs a ClusterRoleBinding. Pods themselves are namespaced, but kubectl get pods -A is evaluated as a cluster-scoped request.

(b) Wrong apiGroups

The in API group "" string in the error must match apiGroups in the Role YAML exactly.

If resource-to-group mapping is fuzzy, asking the cluster is the most accurate approach.

kubectl api-resources -o wide | head -20

NAME          SHORTNAMES   APIVERSION   NAMESPACED   KIND         VERBS
configmaps    cm           v1           true         ConfigMap    create,delete,get,list,patch,update,watch
pods          po           v1           true         Pod          create,delete,get,list,patch,update,watch
deployments   deploy       apps/v1      true         Deployment   create,delete,get,list,patch,update,watch

If APIVERSION is v1, it is core (""); if apps/v1, it is apps. If the NAMESPACED column is false, it is a cluster-scoped resource.

(c) subjects typo — fails silently with no apply error

The nastiest RBAC trap: a typo still lets the resource create successfully. The mismatch only shows up at runtime as 403.

# Wrong — this YAML applies successfully but never matches
subjects:
  - kind: ServiceAccount
    name: system:serviceaccount:dev:ci-deployer   # ← do not put the full name here

# Correct
subjects:
  - kind: ServiceAccount
    name: ci-deployer      # SA name only
    namespace: dev         # ServiceAccount subjects require namespace

Summary:

• kind: ServiceAccount → name is the SA name only; namespace is required. Omit it and matching fails silently.
• kind: User / kind: Group → do not set namespace. In this form, putting the full name system:serviceaccount:dev:ci-deployer in name is valid (referring to an SA as a User).
• apiGroup values: User/Group use rbac.authorization.k8s.io; ServiceAccount uses "" (omittable).

(d) You are not the subject you think you are

The permissions are correct — you just are not that subject.

kubectl config current-context
kubectl config view --minify -o jsonpath='{.contexts[0].context.user}{"\n"}'

On EKS, the IAM principal must be mapped to a K8s identity. With no mapping, the error exposes the IAM ARN as-is. Check:

• Missing mapRoles / mapUsers entries in the aws-auth ConfigMap
• IAM Role ARN includes a path like /aws-reserved/sso.amazonaws.com/... and fails to match (SSO roles are generally known to match only after the path is stripped — confirm against the official docs for your EKS version)
• Running EKS Access Entry (newer) and aws-auth in parallel, so it is unclear which one actually applies. The center of gravity has been shifting toward Access Entry / Access Policy, so during diagnosis you must check both paths.

(e) Missing subresources

Permission on pods does not include pods/log. That is by design, not a bug.

Log collectors suddenly unable to read logs, or HPA unable to scale without deployments/scale, are among the most frequently reported production cases.

Copy-paste diagnosis → least-privilege recovery → impersonation verification

Step 1: Dump the subject's current permissions

The --as flag lets you query as if you were another subject (impersonation). Your own account running this command must have the impersonate permission.

# Full permissions a given ServiceAccount has in the prod namespace
kubectl auth can-i --list \
  --as=system:serviceaccount:dev:ci-deployer \
  -n prod

Healthy-looking (but actually empty) sample:

Resources                                       Non-Resource URLs   Resource Names   Verbs
selfsubjectaccessreviews.authorization.k8s.io   []                  []               [create]
selfsubjectrulesreviews.authorization.k8s.io    []                  []               [create]
                                                [/api/*]            []               [get]
                                                [/healthz]          []               [get]

If you only see selfsubject* and /healthz, this SA effectively has no permissions (you are only seeing system:basic-user and system:discovery, which every authenticated subject gets). Suspect cause (c) subjects typo, or a missing binding altogether.

Step 2: Check individual verbs

kubectl auth can-i list pods --as=system:serviceaccount:dev:ci-deployer -n prod
# → no

kubectl auth can-i create deployments.apps --as=system:serviceaccount:dev:ci-deployer -n prod
# → yes

# Subresources can be checked directly
kubectl auth can-i get pods/log --as=system:serviceaccount:dev:ci-deployer -n prod
# → no

# Check by group
kubectl auth can-i list secrets --as=dev@corp.com --as-group=platform-team -n prod

If can-i says yes but the real call still fails, RBAC passed and you were blocked at admission (PSA, ValidatingWebhook) or by another authorizer.

Step 3: Reverse-trace bindings attached to this subject

SUBJECT="ci-deployer"
SUBJECT_NS="dev"

kubectl get clusterrolebinding,rolebinding -A -o json \
| jq -r --arg n "$SUBJECT" --arg ns "$SUBJECT_NS" '
  .items[]
  | select(
      (.subjects // [])
      | any(
          (.name == $n and (.kind == "ServiceAccount") and (.namespace == $ns))
          or (.name == ("system:serviceaccount:" + $ns + ":" + $n))
        )
    )
  | "\(.kind)\t\(.metadata.namespace // "-")/\(.metadata.name)\t-> \(.roleRef.kind)/\(.roleRef.name)"
' | column -t

Sample healthy output:

RoleBinding         prod/ci-deployer-deploy   -> Role/ci-deployer
ClusterRoleBinding  -/metrics-reader          -> ClusterRole/view

If nothing prints, there is no binding or the subjects notation is wrong → cause (c).

Step 4: Inspect the actual rules

# ClusterRole rules in human-readable form
kubectl describe clusterrole view | head -30

# Namespace Role source
kubectl get role ci-deployer -n prod -o yaml

If describe shows pods in the Resources column but not pods/log, that confirms (e) missing subresource.

Step 5: Check EKS mapping

kubectl -n kube-system get configmap aws-auth -o yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system
data:
  mapRoles: |
    - rolearn: arn:aws:iam::123456789012:role/eks-node-role
      username: system:node:{{EC2PrivateDNSName}}
      groups:
        - system:bootstrappers
        - system:nodes

If eks-dev-role is missing here, that IAM role cannot be converted into a K8s identity. Also check Access Entry in parallel.

aws eks list-access-entries --cluster-name my-cluster
aws eks list-associated-access-policies --cluster-name my-cluster \
  --principal-arn arn:aws:iam::123456789012:role/eks-dev-role

Step 6: Find denials in the audit log

If API server audit logs are enabled, filter on "decision":"forbid" and you get the denied request's subject, verb, and resource as-is.

# On a control-plane node that has the audit log file
jq -c 'select(.annotations["authorization.k8s.io/decision"]=="forbid")
       | {user: .user.username, verb, uri: .requestURI,
          reason: .annotations["authorization.k8s.io/reason"]}' \
  /var/log/kubernetes/audit.log | tail -20

On EKS, query the same fields in the CloudWatch Logs audit log group.

Three least-privilege recovery YAML examples

① Read-only (risk: low)

For developers inspecting the state of their team's namespace.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: ns-readonly
  namespace: prod
rules:
  - apiGroups: [""]                       # core group
    resources: ["pods", "services", "configmaps", "events",
                "persistentvolumeclaims"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources: ["deployments", "replicasets", "statefulsets", "daemonsets"]
    verbs: ["get", "list", "watch"]
  # secrets intentionally omitted — a single get exposes every DB password and token.
  # pods/log also omitted — logs commonly contain PII and tokens.
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ns-readonly-dev-team
  namespace: prod
subjects:
  - kind: Group
    name: dev-team                        # User/Group have no namespace field
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: ns-readonly
  apiGroup: rbac.authorization.k8s.io

② Debugging (risk: high — time-box it)

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: ns-debugger
  namespace: prod
  annotations:
    security.internal/expires-at: "2026-08-05T00:00:00Z"   # explicit revocation deadline
    security.internal/ticket: "OPS-1234"
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get"]
  - apiGroups: [""]
    resources: ["pods/exec", "pods/portforward"]
    verbs: ["create"]                     # exec/portforward use the create verb
  # delete omitted — prevents a debug session from deleting pods and making the outage worse
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ns-debugger-oncall
  namespace: prod
subjects:
  - kind: User
    name: oncall@corp.com
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: ns-debugger
  apiGroup: rbac.authorization.k8s.io

Why this is dangerous: pods/exec is effectively a shell inside the container. You can read Secret files and the ServiceAccount token mounted in that pod, which is equivalent to inheriting the pod's SA permissions. Always grant it with an expiry ticket and leave an audit trail.

③ CI deploy (risk: medium)

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: ci-deployer
  namespace: prod
rules:
  - apiGroups: ["apps"]
    resources: ["deployments", "replicasets"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
  - apiGroups: ["apps"]
    resources: ["deployments/scale"]      # kubectl scale / HPA
    verbs: ["get", "update", "patch"]
  - apiGroups: [""]
    resources: ["services", "configmaps"]
    verbs: ["get", "list", "create", "update", "patch"]
  - apiGroups: [""]
    resources: ["pods", "pods/log"]
    verbs: ["get", "list"]                # to inspect rollout failures
  # create secrets omitted — creating an SA token Secret opens a privilege-escalation
  # path to steal another SA's permissions. Deliver secrets via External Secrets or similar.
  # delete deployments omitted — rollback via update/patch is enough.
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ci-deployer
  namespace: dev
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ci-deployer-prod
  namespace: prod                          # ns where the permission applies
subjects:
  - kind: ServiceAccount
    name: ci-deployer
    namespace: dev                         # ns where the SA actually lives (required)
roleRef:
  kind: Role
  name: ci-deployer
  apiGroup: rbac.authorization.k8s.io

A common point of confusion: RoleBinding metadata.namespace (where the permission applies) and the subject's namespace (where the SA lives) can differ. The example above is a valid setup that grants a SA in dev permissions in prod.

Verification loop

kubectl apply -f rbac.yaml

kubectl auth can-i --list \
  --as=system:serviceaccount:dev:ci-deployer -n prod

kubectl auth can-i update deployments.apps/scale \
  --as=system:serviceaccount:dev:ci-deployer -n prod

Branch on the result:

To verify with the real token from inside a pod:

kubectl exec -it <pod> -n dev -- sh -c '
  TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token);
  curl -s -o /dev/null -w "%{http_code}\n" \
    --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
    -H "Authorization: Bearer $TOKEN" \
    https://kubernetes.default.svc/api/v1/namespaces/prod/pods
'

200 is healthy, 403 is missing RBAC, 401 is a token problem (bound-token expiry or audience mismatch). Recent clusters default to bound ServiceAccount tokens with a lifetime and audience, so an old CI script using a cached token can produce 401 rather than 403.

What not to do

Taboo 1 — temporary cluster-admin

# Never do this
kubectl create clusterrolebinding temp-fix \
  --clusterrole=cluster-admin \
  --serviceaccount=dev:ci-deployer

The typical incident path: grant in a hurry → deploy succeeds → no revocation ticket → left for months → CI runner container compromised, SA token leaked → Secrets readable in every namespace → full cluster takeover. If you did grant it, record the expiry in both an annotation and a ticket, and audit regularly with:

kubectl get clusterrolebinding -o json \
| jq -r '.items[]
    | select(.roleRef.name=="cluster-admin")
    | "\(.metadata.name)\t\((.subjects // []) | map(.kind + ":" + .name) | join(","))"' \
| column -t

Taboo 2 — wildcards

verbs: ["*"], resources: ["*"], apiGroups: ["*"] are frequently flagged in ISMS-P, SOC 2, and similar audits as least-privilege violations (confirm the exact criteria against the relevant audit standard). Manage these verbs separately in particular.

Response checklist

1. Split the raw error into five fields: subject / verb / resource / apiGroup / scope
2. 30-second triage by subject prefix (SA / anonymous / IAM ARN / User)
3. Narrow to one of the five causes (scope · apiGroup · subjects · identity · subresource)
4. Add only a least-privilege Role/Binding (no wildcards, no cluster-admin)
5. Verify with impersonation via kubectl auth can-i --list --as=...
6. Record the grant history and expiry in Git (RBAC as Code)

Next in the series

Part 4 covers Secrets and ServiceAccount token management: blocking unnecessary token mounts with automountServiceAccountToken: false, bound-token lifetime and audience, and wiring external secrets via External Secrets Operator. "Why we omitted create secrets from the CI Role" in this post is the starting point of part 4.

Official references: Kubernetes docs "Using RBAC Authorization" and "Authorization Overview", plus the cluster access management (Access Entry) section of the AWS EKS user guide.

FAQ

Q. I ran kubectl auth can-i --list --as=... and my account says cannot impersonate. A. Impersonation is itself a separate permission. You need the impersonate verb on the users, groups, and serviceaccounts resources. That permission is powerful enough to impersonate other users, so prefer limiting it to audited admin accounts, or verify directly with the target SA's token via --token.

Q. I created a RoleBinding but kubectl get pods -A is still Forbidden. A. -A (all namespaces) is evaluated as a cluster-scoped request, so a RoleBinding cannot cover it. You need ClusterRole + ClusterRoleBinding. If you only need one namespace, it is much safer from a permissions standpoint to change the script to call with -n <ns>.

Q. I set up an EKS Access Entry but the error still prints the IAM ARN as-is. A. If aws-auth ConfigMap and Access Entry are both in use, which one actually applies depends on the cluster authentication mode (API, API_AND_CONFIG_MAP, CONFIG_MAP). Check accessConfig.authenticationMode with aws eks describe-cluster first, and also compare whether the ARN string includes an SSO path (/aws-reserved/...).