Elasticsearch FORBIDDEN/12 Index Read-Only Error: A 5-Minute Recovery Guide

Elasticsearch FORBIDDEN/12 Index Read-Only Error: A 5-Minute Recovery Guide

An alert fires in the middle of the night. Application logs are flooded with messages like this.

ClusterBlockException[index [logs-2026.06.14] blocked by:
[TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark,
index has read-only-allow-delete block];]

Or on the client side, it looks like this.

FORBIDDEN/12/index read-only / allow delete (api)

Indexing is completely blocked. Search still works, but writes are dead. Bottom line: nine times out of ten, the culprit is exceeding the disk watermark flood_stage (95%). This post is written for operators who are in the middle of an incident—emergency recovery commands first, then root-cause removal below.

Emergency flowchart: what do I do right now?

[Writes blocked]
   │
   ├─ 1. Check disk usage (GET _cat/allocation?v)
   │       └─ Over 95%? → Almost certainly: automatic flood_stage block
   │
   ├─ 2. Free disk first (delete old indices / expand volume)
   │
   ├─ 3. Clear the read_only_allow_delete block
   │
   └─ 4. Prevent recurrence with monitoring, alerts, and ILM

⚠️ Order is everything. If you lift the block without freeing disk first, the watermark check (every 30 seconds by default) will immediately re-block you. I'll emphasize this again below.

5-minute diagnosis: error text → cause classification table

The number in the error code tells you the cause. Start by classifying the type of block with this table.

Diagnosis commands are provided for both curl and Kibana Dev Tools.

curl version

#
curl -s "localhost:9200/_cat/allocation?v"

#
curl -s "localhost:9200/_cluster/health?pretty"

# read_only check
curl -s "localhost:9200/_all/_settings?flat_settings=true&filter_path=**.read_only*&pretty"

Kibana Dev Tools version

GET _cat/allocation?v
GET _cluster/health
GET _all/_settings?flat_settings=true&filter_path=**.read_only*

If the disk.percent column in _cat/allocation is over 95, diagnosis is done. This is an automatic flood_stage block.

How the 3 disk watermark stages work

Elasticsearch/OpenSearch monitors node disk usage in three stages.

The key is flood_stage. The moment usage exceeds 95%, ES blocks writes on its own to prevent data loss (index corruption from a full disk). As the name implies, reads and deletes still work, but indexing new documents is blocked. This is a safeguard, not a bug.

Immediate recovery: clearing the write block

I'll say it again. Free disk space first, then run the commands below.

curl version

curl -X PUT "localhost:9200/_all/_settings" \
  -H 'Content-Type: application/json' -d '
{
  "index.blocks.read_only_allow_delete": null
}'

Kibana Dev Tools version

PUT _all/_settings
{
  "index.blocks.read_only_allow_delete": null
}

Setting it to null removes the explicitly applied block (prefer null over false — it returns the setting to automatic management). After clearing, run GET _cluster/health again to confirm indexing is healthy.

💡 Practitioner tip: During incident response I always keep _cat/allocation open and wait until I see disk % drop below 90 before I run the unblock command. If you unblock without freeing disk, the command returns success—then 30 seconds later you're blocked again, and you waste time wondering why it didn't stick. This is the most common trap.

Root-cause removal & practical disk reclamation

1) What's eating the disk?

# from
du -sh /var/lib/elasticsearch/* | sort -rh | head
df -h

2) Delete old indices (the fastest fix)

For log-style data, old date-based indices usually account for most of the space.

#
DELETE logs-2024.*

Disk space is freed immediately after deletion; run the unblock command above and writes recover right away.

3) Automate with ILM/rollover (prevent recurrence)

Manual deletion is a stopgap. Put an ILM (Index Lifecycle Management) policy in place for automatic deletion.

PUT _ilm/policy/logs-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": { "rollover": { "max_size": "50gb", "max_age": "1d" } }
      },
      "delete": {
        "min_age": "30d",
        "actions": { "delete": {} }
      }
    }
  }
}

On OpenSearch, use ISM (Index State Management) instead of ILM to configure the same rollover/delete policy.

4) Adjust watermark thresholds (carefully)

Sometimes you need to raise the thresholds temporarily before you can expand disk. Remember this is not a root-cause fix.

PUT _cluster/settings
{
  "transient": {
    "cluster.routing.allocation.disk.watermark.low": "90%",
    "cluster.routing.allocation.disk.watermark.high": "95%",
    "cluster.routing.allocation.disk.watermark.flood_stage": "97%"
  }
}

On small disks, absolute values like "50gb" are more predictable than percentages.

Conclusion: recurrence-prevention checklist

• Set disk-usage alerts at 80%, below the watermarks (so you notice before low is reached)
• Apply rollover and auto-delete policies with ILM (ES) / ISM (OpenSearch)
• Move cold data to cheaper storage with Data Tiers and Searchable Snapshots
• Document watermark settings and make them explicit per environment
• Record the recovery order (free disk → unblock → monitor) in your runbook

As log and observability data explode, disk saturation has become the most common incident for ES/OpenSearch operators. Remember two things and you can recover in five minutes: flood_stage is not the enemy—it's the last line of defense for your data—and unblocking alone will not fix it.

FAQ

Q. It says read-only, but I have plenty of disk. Why? A. In some versions (especially older ES before 7.x), once flood_stage is exceeded the block does not automatically lift even after disk usage drops. After freeing disk, you must manually clear it with index.blocks.read_only_allow_delete: null. It could also be a manual read_only (FORBIDDEN/5) setting—check settings.

Q. It looks like only some nodes are blocked. A. flood_stage operates per node. If only one node is full, only indices that have shards on that node are blocked. Check per-node usage with _cat/allocation, then rebalance shards or clean up disk on that node.

Q. Does this apply to AWS OpenSearch Service (managed) too? A. The mechanism is the same, but in a managed environment you cannot SSH in or run df, so you cannot free disk directly. Check storage usage in the console and respond by deleting indices or increasing node storage (or scaling the instance). Moving old data to UltraWarm or Cold Storage is the real fix.