Production Kubernetes Cluster Checklist: HA, Security, and Observability

Production K8s vs. Development Environments

Running Kubernetes in production is a completely different problem from local Minikube. You need to consider high availability, security, monitoring, and cost optimization.

Managed vs. Self-Managed

Node Group Design

nodeGroups:
  - name: system
    instanceType: m5.large
    desiredCapacity: 3
    minSize: 3
    maxSize: 5

  - name: app-general
    instanceType: c5.2xlarge
    desiredCapacity: 3
    minSize: 2
    maxSize: 20
 spot: true #

Pod Anti-Affinity (High Availability)

affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    - labelSelector:
        matchLabels:
          app: my-app
      topologyKey: kubernetes.io/hostname
topologySpreadConstraints:
- maxSkew: 1
  topologyKey: topology.kubernetes.io/zone
  whenUnsatisfiable: DoNotSchedule

PodDisruptionBudget

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: my-app-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: my-app

Resource Settings (Required)

resources:
  requests:
    cpu: "250m"
    memory: "256Mi"
  limits:
    cpu: "500m"
    memory: "512Mi"

Always set requests and limits. Without them, noisy-neighbor issues can take down an entire node.

Security Hardening

securityContext:
  runAsNonRoot: true
  runAsUser: 1000
  fsGroup: 2000
containers:
- name: app
  securityContext:
    allowPrivilegeEscalation: false
    readOnlyRootFilesystem: true
    capabilities:
      drop: ["ALL"]

Monitoring Stack

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace

Essential alerts: node CPU > 80%, memory > 85%, Pod restarts > 5/hour, PVC utilization > 80%

The core of production Kubernetes operations is automation. Build a self-healing environment with HPA, Cluster Autoscaler, and GitOps.

When Services Drop During Node Drains or Upgrades

Outages during planned work (node replacement, version upgrades) in production clusters are mostly caused by missing settings.

Causes

• Single-replica Deployments: a gap until the replacement Pod is up
• PDB minAvailable equals the replica count, so ALLOWED DISRUPTIONS is 0 and the drain waits forever
• No readinessProbe, so traffic hits new Pods before they're ready
• The app doesn't handle SIGTERM, so in-flight requests are cut

Reproduce and verify

# PDBs with zero allowed disruptions
kubectl get pdb -A
# simulate a drain without evicting
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data --dry-run=server
# single-replica workloads
kubectl get deploy -A -o jsonpath='{range .items[?(@.spec.replicas==1)]}{.metadata.namespace}/{.metadata.name}{"\n"}{end}'
# missing readinessProbe
kubectl get deploy <name> -o jsonpath='{.spec.template.spec.containers[*].readinessProbe}'

Fix

• Give user-facing workloads at least two replicas and a PDB like maxUnavailable: 1.
• Set a readinessProbe and terminationGracePeriodSeconds, and if needed a short sleep in preStop so the load balancer has time to remove the endpoint.
• Use kubectl version to check versions (the --short flag was removed in recent kubectl).

Prevention

• Encode these checks as policy-engine rules (Kyverno, OPA Gatekeeper) so single-replica or probe-less manifests are warned or rejected at deploy time.

References: Official Documentation

The primary source for the behavior, configuration, and errors covered in this post is the official documentation below. Check there for version-specific options and exact behavior.

• Kubernetes Official Documentation