# Fixing Helm "another operation in progress": Unlock a Pending Lock in 5 Minutes Your CI pipeline is stuck in red, and the last line of the log is this: ``` Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress ``` Hitting retry produces the same error. This post is for you if a deploy is blocked right now. Bottom line: **you can almost always unlock it in about five minutes with no data loss.** There is one thing you must never do. > ⚠️ **Read this first**: Do not run `helm delete` (i.e. `helm uninstall`) in a panic. That does not unlock the release—it **deletes the entire release and its workloads (Deployments, Services, etc.)**. Plenty of production outages started with someone trying to "clear the lock" this way. ## Why the lock happens: Helm 3 release state In Helm 2, Tiller stored release state in a ConfigMap. **Helm 3 stores it in a Secret by default.** The name pattern is fixed: ``` sh.helm.release.v1..v ``` Every deploy creates a new revision Secret, each with a status label. The happy path looks like this: | When | Secret status label | |------|-------------------| | Install/upgrade starts | `pending-install` / `pending-upgrade` | | Completes successfully | `deployed` | | Explicit uninstall | `uninstalled` | The problem is when a deploy **dies mid-flight**. Timeout, pod kill, CI job abort, concurrent triggers—if Helm never finishes with `deployed`, that revision Secret stays `pending-*` forever. - `pending-install` — **first install** was interrupted (no previous deployed revision) - `pending-upgrade` — **upgrade** was interrupted (a previous deployed revision exists) - `pending-rollback` — **rollback** was interrupted Before starting the next operation, Helm treats a pending last revision as "another operation in progress" and blocks. That one-line error is the result. GitOps tools like [Argo CD](/blog/gitops와-argocd로-쿠버네티스-배포-완전-자동화하기) and Flux use Helm under the hood, so they hit the same lock. ## Diagnose: find where your release is stuck in 5 minutes Always check state **before** you recover. You need the last healthy (`deployed`) revision so you can unlock safely. ```bash # 1) 리비전 이력 — STATUS 컬럼을 보세요 helm history -n # 2) 현재 릴리스 상태 helm status -n # 3) 실제 Secret을 시간순으로 확인 kubectl get secret -n -l owner=helm,name= \ --sort-by=.metadata.creationTimestamp ``` `helm history` output looks like this: ``` REVISION STATUS CHART DESCRIPTION 3 deployed myapp-1.2.0 Upgrade complete 4 pending-upgrade myapp-1.3.0 Preparing upgrade ``` Two things matter: 1. **The last `deployed` revision number** → `3` in the example. That is your rollback target. 2. **The revision stuck in pending** → `4`. That is the lock. `kubectl get secret` will show Secret `sh.helm.release.v1.myapp.v4` labeled `status=pending-upgrade`. That Secret is the culprit. ## Three recovery paths, in priority order Try them in this order—safest first. If one fails, drop to the next. ### 1. Roll back to the last deployed revision (safest) If a previous healthy revision exists (i.e. you are in `pending-upgrade`), rollback is the right move. Helm cleans up the pending Secret and restores a clean state. ```bash # 위에서 찾은 '마지막 deployed' 리비전 번호 사용 helm rollback 3 -n ``` When rollback succeeds, the lock is gone and you can `helm upgrade` again. Cleanest path, no data loss. ### 2. Delete the pending Secret directly (no rollback target, or rollback failed) On `pending-install` there is no deployed revision to roll back to. Rollback can also fail on its own. In those cases, **delete only the pending Secret**. ```bash # 라벨로 pending 상태 Secret만 삭제 (status를 정확히 지정!) kubectl delete secret -n \ -l owner=helm,name=,status=pending-upgrade # 또는 리비전 번호로 특정 Secret만 삭제 kubectl delete secret sh.helm.release.v1..v -n ``` After deleting, redeploy; the lock is gone and the operation proceeds. ```bash helm upgrade --install ./chart -n ``` ### 3. Clean up when state is tangled If retries piled up several pending Secrets, **never touch a deployed Secret**—delete only those with a pending label, then redeploy. Glance at `kubectl get secret` one more time before you delete; that habit prevents accidents. > ⚠️ **Rules (follow these)** > - ❌ Deleting a `deployed` Secret breaks release history. Delete only `pending-*`. > - ❌ `helm delete` / `helm uninstall` is not an unlock. Workloads go with it. > - ✅ Always pass the namespace (`-n`) and revision, and double-check the target before deleting. ## When timeout is the root cause In production, the most common cause of a pending lock is a **timeout**. With `--wait`, Helm waits until pods are Ready; if health checks never pass, you get: ``` Error: UPGRADE FAILED: timed out waiting for the condition ``` The release stays `pending-upgrade` and the next deploy is blocked. Tight readinessProbes or slow image pulls make this common. Unlocking alone **just repeats the same failure next time**. The real fix is changing deploy flags. ```bash helm upgrade --install ./chart -n \ --atomic \ # 실패 시 자동 롤백 → pending이 남지 않음 --cleanup-on-fail \ # 실패 시 생성된 리소스 정리 --wait \ --timeout 10m # 환경에 맞게 충분히 ``` `--atomic` is the key. On failure Helm rolls back to the previous state automatically, so a pending lock never remains. Adding this one flag to your CI deploy command cuts how often you see this error. ## Wrap-up: prevention checklist + copy-paste summary Diagnosis through recovery on one page: ```bash # ── 진단 ── helm history -n # deployed / pending 리비전 확인 helm status -n kubectl get secret -n -l owner=helm,name= \ --sort-by=.metadata.creationTimestamp # ── 복구 1순위: 롤백 ── helm rollback <마지막 deployed REVISION> -n # ── 복구 2순위: pending Secret 직접 삭제 ── kubectl delete secret -n -l owner=helm,name=,status=pending-upgrade helm upgrade --install ./chart -n # ── 재발 방지: 배포 옵션 ── helm upgrade --install ./chart -n \ --atomic --cleanup-on-fail --wait --timeout 10m ``` Prevention checklist: - [ ] Add `--atomic` + `--cleanup-on-fail` to the deploy command - [ ] Set `--timeout` high enough for your environment - [ ] **Block concurrent/duplicate deploys** in CI (GitHub Actions `concurrency`, GitLab `resource_group`) - [ ] Make the job timeout longer than Helm `--timeout` (if the job dies first, you get pending) ## References: official docs Primary source for the behavior, flags, and errors in this post. Check version-specific options and exact semantics there. - [Helm documentation](https://helm.sh/docs/) ## FAQ **Q. If I delete the pending Secret, do running pods go away too?** A. No. `sh.helm.release.v1.*` Secrets hold only release **metadata (status / manifest history)**. Workloads (Deployments, Pods, etc.) are separate resources, so deleting one pending Secret does not affect running pods. Never delete a `deployed` Secret. **Q. I deploy with Argo CD and get the same error. Why?** A. Argo CD and Flux use Helm templating/release internals, so the same pending lock can happen. Diagnosis and recovery are identical. Clean up the pending Secret with `kubectl`, then retrigger Sync. **Q. `helm rollback` also fails with "another operation in progress".** A. Either the last revision is `pending-install` (no deployed revision to return to) or state is tangled. Use path 2: `kubectl delete secret` for the pending Secret only, then redeploy with `helm upgrade --install`.

#Helm

How to Fix the Helm "another operation in progress" Error (Pending Lock)