Fixing Helm "another operation in progress": Unlock a Pending Lock in 5 Minutes
Your CI pipeline is stuck in red, and the last line of the log is this:
Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progressHitting retry produces the same error. This post is for you if a deploy is blocked right now. Bottom line: you can almost always unlock it in about five minutes with no data loss. There is one thing you must never do.
⚠️ Read this first: Do not run
helm delete(i.e.helm uninstall) in a panic. That does not unlock the release—it deletes the entire release and its workloads (Deployments, Services, etc.). Plenty of production outages started with someone trying to "clear the lock" this way.
Why the lock happens: Helm 3 release state
In Helm 2, Tiller stored release state in a ConfigMap. Helm 3 stores it in a Secret by default. The name pattern is fixed:
sh.helm.release.v1.<release-name>.v<revision-number>Every deploy creates a new revision Secret, each with a status label. The happy path looks like this:
| When | Secret status label |
|---|---|
| Install/upgrade starts | pending-install / pending-upgrade |
| Completes successfully | deployed |
| Explicit uninstall | uninstalled |
The problem is when a deploy dies mid-flight. Timeout, pod kill, CI job abort, concurrent triggers—if Helm never finishes with deployed, that revision Secret stays pending-* forever.
pending-install— first install was interrupted (no previous deployed revision)pending-upgrade— upgrade was interrupted (a previous deployed revision exists)pending-rollback— rollback was interrupted
Before starting the next operation, Helm treats a pending last revision as "another operation in progress" and blocks. That one-line error is the result. GitOps tools like Argo CD and Flux use Helm under the hood, so they hit the same lock.
Diagnose: find where your release is stuck in 5 minutes
Always check state before you recover. You need the last healthy (deployed) revision so you can unlock safely.
# 1) 리비전 이력 — STATUS 컬럼을 보세요
helm history <release> -n <ns>
# 2) 현재 릴리스 상태
helm status <release> -n <ns>
# 3) 실제 Secret을 시간순으로 확인
kubectl get secret -n <ns> -l owner=helm,name=<release> \
--sort-by=.metadata.creationTimestamphelm history output looks like this:
REVISION STATUS CHART DESCRIPTION
3 deployed myapp-1.2.0 Upgrade complete
4 pending-upgrade myapp-1.3.0 Preparing upgradeTwo things matter:
- The last
deployedrevision number →3in the example. That is your rollback target. - The revision stuck in pending →
4. That is the lock.
kubectl get secret will show Secret sh.helm.release.v1.myapp.v4 labeled status=pending-upgrade. That Secret is the culprit.
Three recovery paths, in priority order
Try them in this order—safest first. If one fails, drop to the next.
1. Roll back to the last deployed revision (safest)
If a previous healthy revision exists (i.e. you are in pending-upgrade), rollback is the right move. Helm cleans up the pending Secret and restores a clean state.
# 위에서 찾은 '마지막 deployed' 리비전 번호 사용
helm rollback <release> 3 -n <ns>When rollback succeeds, the lock is gone and you can helm upgrade again. Cleanest path, no data loss.
2. Delete the pending Secret directly (no rollback target, or rollback failed)
On pending-install there is no deployed revision to roll back to. Rollback can also fail on its own. In those cases, delete only the pending Secret.
# 라벨로 pending 상태 Secret만 삭제 (status를 정확히 지정!)
kubectl delete secret -n <ns> \
-l owner=helm,name=<release>,status=pending-upgrade
# 또는 리비전 번호로 특정 Secret만 삭제
kubectl delete secret sh.helm.release.v1.<release>.v<N> -n <ns>After deleting, redeploy; the lock is gone and the operation proceeds.
helm upgrade --install <release> ./chart -n <ns>3. Clean up when state is tangled
If retries piled up several pending Secrets, never touch a deployed Secret—delete only those with a pending label, then redeploy. Glance at kubectl get secret one more time before you delete; that habit prevents accidents.
⚠️ Rules (follow these)
- ❌ Deleting a
deployedSecret breaks release history. Delete onlypending-*.- ❌
helm delete/helm uninstallis not an unlock. Workloads go with it.- ✅ Always pass the namespace (
-n) and revision, and double-check the target before deleting.
When timeout is the root cause
In production, the most common cause of a pending lock is a timeout. With --wait, Helm waits until pods are Ready; if health checks never pass, you get:
Error: UPGRADE FAILED: timed out waiting for the conditionThe release stays pending-upgrade and the next deploy is blocked. Tight readinessProbes or slow image pulls make this common. Unlocking alone just repeats the same failure next time. The real fix is changing deploy flags.
helm upgrade --install <release> ./chart -n <ns> \
--atomic \ # 실패 시 자동 롤백 → pending이 남지 않음
--cleanup-on-fail \ # 실패 시 생성된 리소스 정리
--wait \
--timeout 10m # 환경에 맞게 충분히--atomic is the key. On failure Helm rolls back to the previous state automatically, so a pending lock never remains. Adding this one flag to your CI deploy command cuts how often you see this error.
Wrap-up: prevention checklist + copy-paste summary
Diagnosis through recovery on one page:
# ── 진단 ──
helm history <release> -n <ns> # deployed / pending 리비전 확인
helm status <release> -n <ns>
kubectl get secret -n <ns> -l owner=helm,name=<release> \
--sort-by=.metadata.creationTimestamp
# ── 복구 1순위: 롤백 ──
helm rollback <release> <마지막 deployed REVISION> -n <ns>
# ── 복구 2순위: pending Secret 직접 삭제 ──
kubectl delete secret -n <ns> -l owner=helm,name=<release>,status=pending-upgrade
helm upgrade --install <release> ./chart -n <ns>
# ── 재발 방지: 배포 옵션 ──
helm upgrade --install <release> ./chart -n <ns> \
--atomic --cleanup-on-fail --wait --timeout 10mPrevention checklist:
- Add
--atomic+--cleanup-on-failto the deploy command - Set
--timeouthigh enough for your environment - Block concurrent/duplicate deploys in CI (GitHub Actions
concurrency, GitLabresource_group) - Make the job timeout longer than Helm
--timeout(if the job dies first, you get pending)
References: official docs
Primary source for the behavior, flags, and errors in this post. Check version-specific options and exact semantics there.
FAQ
Q. If I delete the pending Secret, do running pods go away too?
A. No. sh.helm.release.v1.* Secrets hold only release metadata (status / manifest history). Workloads (Deployments, Pods, etc.) are separate resources, so deleting one pending Secret does not affect running pods. Never delete a deployed Secret.
Q. I deploy with Argo CD and get the same error. Why?
A. Argo CD and Flux use Helm templating/release internals, so the same pending lock can happen. Diagnosis and recovery are identical. Clean up the pending Secret with kubectl, then retrigger Sync.
Q. helm rollback also fails with "another operation in progress".
A. Either the last revision is pending-install (no deployed revision to return to) or state is tangled. Use path 2: kubectl delete secret for the pending Secret only, then redeploy with helm upgrade --install.
Nodelog는 모든 콘텐츠의 내용과 출처를 공개 전에 검토합니다. 환경(OS·버전)에 따라 결과가 달라질 수 있는 기술 정보는 공식 문서와 함께 확인하며, 검토 기준과 정정 원칙은 편집 정책에서 안내합니다. 오류를 발견하시면 이메일로 제보해 주세요 — 확인 후 신속히 정정합니다.
Comments
Be the first to comment.