Elasticsearch FORBIDDEN/12 Index Read-Only Error: A 5-Minute Recovery Guide

An alert fires in the middle of the night. Application logs are flooded with messages like this.

CODE

ClusterBlockException[index [logs-2026.06.14] blocked by:
[TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark,
index has read-only-allow-delete block];]

Or on the client side, it looks like this.

CODE

FORBIDDEN/12/index read-only / allow delete (api)

Indexing is completely blocked. Search still works, but writes are dead. Bottom line: nine times out of ten, the culprit is exceeding the disk watermark flood_stage (95%). This post is written for operators who are in the middle of an incident—emergency recovery commands first, then root-cause removal below.

Emergency flowchart: what do I do right now?

CODE

[Writes blocked]
   │
   ├─ 1. Check disk usage (GET _cat/allocation?v)
   │       └─ Over 95%? → Almost certainly: automatic flood_stage block
   │
   ├─ 2. Free disk first (delete old indices / expand volume)
   │
   ├─ 3. Clear the read_only_allow_delete block
   │
   └─ 4. Prevent recurrence with monitoring, alerts, and ILM

⚠️ Order is everything. If you lift the block without freeing disk first, the watermark check (every 30 seconds by default) will immediately re-block you. I'll emphasize this again below.

5-minute diagnosis: error text → cause classification table

The number in the error code tells you the cause. Start by classifying the type of block with this table.

Error text	Trigger	Clear key	Notes
`FORBIDDEN/12 index read-only / allow delete (api)`	Automatic (disk) (flood_stage exceeded)	`index.blocks.read_only_allow_delete`	Most common. Deletes are still allowed
`FORBIDDEN/8 index write (api)`	Manual / snapshot restore / etc. write block	`index.blocks.write`	Writes only are blocked
`FORBIDDEN/5 index read-only (api)`	Manual read-only setting	`index.blocks.read_only`	Set by an operator
`cluster_block_exception ... no master`	Cluster-level block	Recover master/node state	May not be disk-related

Diagnosis commands are provided for both curl and Kibana Dev Tools.

curl version

Bash

# 노드별 디스크 사용률 (가장 중요)
curl -s "localhost:9200/_cat/allocation?v"

# 클러스터 상태
curl -s "localhost:9200/_cluster/health?pretty"

# 어떤 인덱스에 read_only 블록이 걸렸는지 확인
curl -s "localhost:9200/_all/_settings?flat_settings=true&filter_path=**.read_only*&pretty"

Kibana Dev Tools version

CODE

GET _cat/allocation?v
GET _cluster/health
GET _all/_settings?flat_settings=true&filter_path=**.read_only*

If the disk.percent column in _cat/allocation is over 95, diagnosis is done. This is an automatic flood_stage block.

How the 3 disk watermark stages work

Elasticsearch/OpenSearch monitors node disk usage in three stages.

Stage	Default	Behavior
low	85%	Do not allocate new shards to this node
high	90%	Attempt to move existing shards to other nodes
flood_stage	95%	Automatically apply `read_only_allow_delete` to all indices on that node

The key is flood_stage. The moment usage exceeds 95%, ES blocks writes on its own to prevent data loss (index corruption from a full disk). As the name implies, reads and deletes still work, but indexing new documents is blocked. This is a safeguard, not a bug.

Immediate recovery: clearing the write block

I'll say it again. Free disk space first, then run the commands below.

curl version

Bash

curl -X PUT "localhost:9200/_all/_settings" \
  -H 'Content-Type: application/json' -d '
{
  "index.blocks.read_only_allow_delete": null
}'

Kibana Dev Tools version

CODE

PUT _all/_settings
{
  "index.blocks.read_only_allow_delete": null
}

Setting it to null removes the explicitly applied block (prefer null over false — it returns the setting to automatic management). After clearing, run GET _cluster/health again to confirm indexing is healthy.

💡 Practitioner tip: During incident response I always keep _cat/allocation open and wait until I see disk % drop below 90 before I run the unblock command. If you unblock without freeing disk, the command returns success—then 30 seconds later you're blocked again, and you waste time wondering why it didn't stick. This is the most common trap.

Root-cause removal & practical disk reclamation

1) What's eating the disk?

Bash

# 데이터 디렉터리에서 용량 큰 것부터
du -sh /var/lib/elasticsearch/* | sort -rh | head
df -h

2) Delete old indices (the fastest fix)

For log-style data, old date-based indices usually account for most of the space.

CODE

# 와일드카드로 과거 인덱스 일괄 삭제
DELETE logs-2024.*

Disk space is freed immediately after deletion; run the unblock command above and writes recover right away.

3) Automate with ILM/rollover (prevent recurrence)

Manual deletion is a stopgap. Put an ILM (Index Lifecycle Management) policy in place for automatic deletion.

CODE

PUT _ilm/policy/logs-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": { "rollover": { "max_size": "50gb", "max_age": "1d" } }
      },
      "delete": {
        "min_age": "30d",
        "actions": { "delete": {} }
      }
    }
  }
}

On OpenSearch, use ISM (Index State Management) instead of ILM to configure the same rollover/delete policy.

4) Adjust watermark thresholds (carefully)

Sometimes you need to raise the thresholds temporarily before you can expand disk. Remember this is not a root-cause fix.

CODE

PUT _cluster/settings
{
  "transient": {
    "cluster.routing.allocation.disk.watermark.low": "90%",
    "cluster.routing.allocation.disk.watermark.high": "95%",
    "cluster.routing.allocation.disk.watermark.flood_stage": "97%"
  }
}

On small disks, absolute values like "50gb" are more predictable than percentages.

Conclusion: recurrence-prevention checklist

Set disk-usage alerts at 80%, below the watermarks (so you notice before low is reached)
Apply rollover and auto-delete policies with ILM (ES) / ISM (OpenSearch)
Move cold data to cheaper storage with Data Tiers and Searchable Snapshots
Document watermark settings and make them explicit per environment
Record the recovery order (free disk → unblock → monitor) in your runbook

As log and observability data explode, disk saturation has become the most common incident for ES/OpenSearch operators. Remember two things and you can recover in five minutes: flood_stage is not the enemy—it's the last line of defense for your data—and unblocking alone will not fix it.

FAQ

Q. It says read-only, but I have plenty of disk. Why? A. In some versions (especially older ES before 7.x), once flood_stage is exceeded the block does not automatically lift even after disk usage drops. After freeing disk, you must manually clear it with index.blocks.read_only_allow_delete: null. It could also be a manual read_only (FORBIDDEN/5) setting—check settings.

Q. It looks like only some nodes are blocked. A. flood_stage operates per node. If only one node is full, only indices that have shards on that node are blocked. Check per-node usage with _cat/allocation, then rebalance shards or clean up disk on that node.

Q. Does this apply to AWS OpenSearch Service (managed) too? A. The mechanism is the same, but in a managed environment you cannot SSH in or run df, so you cannot free disk directly. Check storage usage in the console and respond by deleting indices or increasing node storage (or scaling the instance). Moving old data to UltraWarm or Cold Storage is the real fix.

확인 정보

#Elasticsearch #OpenSearch #index read-only #disk watermark #cluster_block_exception

✦ ✦ ✦

편집 검토 · Editorial Review

Nodelog는 모든 콘텐츠의 내용과 출처를 공개 전에 검토합니다. 환경(OS·버전)에 따라 결과가 달라질 수 있는 기술 정보는 공식 문서와 함께 확인하며, 검토 기준과 정정 원칙은 편집 정책에서 안내합니다. 오류를 발견하시면 이메일로 제보해 주세요 — 확인 후 신속히 정정합니다.

편집 책임 · Nodelog 기술 편집팀·발행 · Jun 13, 2026·업데이트 · Sep 17, 2026

Elasticsearch FORBIDDEN/12 Index Read-Only Error: A 5-Minute Recovery Guide