Kafka CommitFailedException Infinite Rebalance: Diagnose and Fix the 5 Root Causes

Kafka CommitFailedException Infinite Rebalance: Diagnostic and Fix Guide by the 5 Root Causes

The pager goes off in the middle of the night. Consumer lag is exploding, and the same message floods the logs.

org.apache.kafka.clients.consumer.CommitFailedException:
Commit cannot be completed since the group has already rebalanced
and assigned the partitions to another member. This means that the
time between subsequent calls to poll() was longer than the configured
max.poll.interval.ms, which typically implies that the poll loop is
spending too much time message processing.

"This consumer was fine until yesterday—why now?" This post is action-first so an engineer in the middle of an incident can recover by following from the top.

What's actually happening: the anatomy of the infinite loop

CommitFailedException is not a standalone error—it is one slice of a vicious cycle. The flow looks like this:

poll() called → fetch one batch (e.g. 500 records)
   ↓
processing delay (DB, external APIs, etc. exceeds max.poll.interval.ms)
   ↓
coordinator decides "this member is dead" → kicked from the group
   ↓
processing finishes, commit attempted → "already rebalanced" → CommitFailedException
   ↓
offset commit fails → another member receives the same messages again
   ↓
reassignment (rebalance) occurs → poll again → processing delay again → repeat ♾️

The core problem: commits fail, so offsets never advance, and the same range is processed over and over—lag does not shrink, it grows. While rebalancing is in progress consumers barely process messages, so throughput converges on zero and the lag graph trends up and to the right.

The 5 causes that trigger rebalance: which one is yours?

Most incidents narrow down to these five.

Group them as ①② = "processing is heavy", ③ = "config mismatch", ④⑤ = "membership operations", and the mental model clicks into place.

Pin the cause with data: 3-step diagnosis

Don't guess—name the culprit with logs and metrics.

Step 1 — Search consumer log keywords

Member ... sending LeaveGroup     → client left on its own (suspect processing delay, ①②)
was removed from the group        → coordinator evicted (session/processing exceeded, ①③)
Attempt to heartbeat failed       → heartbeat dropped (③)
Revoke previously assigned ...    → confirm when rebalance actually happened

Step 2 — Key metrics

• records-lag-max (consumer-lag): check whether lag is trending up and to the right
• rebalance-rate-per-hour: rebalances per hour. Double digits or more is abnormal
• rebalance-latency-avg: how long each rebalance stalls processing
• commit-rate: whether commits are actually happening (near 0 means commits are failing)

Step 3 — Measure processing-time distribution

Measure p99 latency of one batch and compare it to max.poll.interval.ms. If p99 exceeds half the interval, that's a red flag.

On MSK, use CloudWatch MaxOffsetLag and EstimatedMaxTimeLag plus broker logs (open monitoring / Prometheus if needed). On Confluent Cloud, use the Consumer Lag and Rebalance panels in Control Center for the same metrics.

The diagnostic flow:

Lag trending up? → yes → commit-rate ≈ 0? → yes → error mentions max.poll.interval.ms?
   ├ yes → processing p99 > interval? → yes: cause ① / no: batch too large → cause ②
   └ no → heartbeat failed logs? → yes: cause ③
Only right after deploy/scale? → cause ④ / whole group shakes on every rolling deploy? → cause ⑤

Copy-paste prescriptions by cause

①② Processing time and oversized batches

The rule of thumb: set max.poll.interval.ms to 2× the worst-case (p99) processing time of one batch.

# Formula: max.poll.interval.ms ≈ (batch p99 processing time) × max.poll.records × 2
# e.g. per-record p99=20ms, 100 records → 2s × 2 = 4s headroom → keep a generous 300000 (5 min) but shrink the batch
max.poll.records=100          # stepwise reduction 500 → 100
max.poll.interval.ms=300000   # increase if processing is truly long, but watch for leftover zombie members
fetch.max.bytes=5242880

Rather than blindly increasing batch size, the textbook move is shrink it so each cycle stays light.

③ Session and heartbeat settings

session.timeout.ms should be about 3× heartbeat.interval.ms.

heartbeat.interval.ms=3000    # heartbeat interval
session.timeout.ms=10000      # ~3× heartbeat (within group.min/max.session.timeout.ms range)

④⑤ Static membership + CooperativeStickyAssignor

Full rebalances on every rolling deploy are solved with static membership and cooperative rebalancing.

# Fixed, unique ID per instance (Pod name/ordinal, etc.)
group.instance.id=consumer-order-0
# Static members skip rebalance if they rejoin within session timeout → slightly generous
session.timeout.ms=45000
# Eager → Cooperative: incremental reassignment without a full stop
partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor

CooperativeStickyAssignor rolling migration procedure (do not switch from Eager in a single cutover):

1. First deploy: register both in partition.assignment.strategy → keep the existing strategy alongside, e.g. [CooperativeStickyAssignor, RangeAssignor]
2. Wait until every instance has completed the first deploy (mixed state is safe)
3. Second deploy: remove the old Range/Sticky from the list and leave only Cooperative
4. From then on, only changed partitions move on rebalance—no full stop

Idempotent processing + manual commit (commitSync)

Auto-commit risks committing offsets that were never processed (message loss). Manual commit + idempotent handling is safer.

Properties props = new Properties();
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, 100);

KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(List.of("orders"));

try {
    while (running) {
        ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
        for (ConsumerRecord<String, String> record : records) {
            // Idempotent handling: skip if this key was already processed (e.g. processed-history table/Redis)
            if (alreadyProcessed(record.key(), record.offset())) continue;
            handle(record);        // business logic
            markProcessed(record); // mark as processed
        }
        // Sync commit at end of batch → immediately detect collision with rebalance
        consumer.commitSync();
    }
} catch (CommitFailedException e) {
    log.warn("Already rebalanced → absorb via idempotent reprocessing on the next batch", e);
} finally {
    consumer.close(); // even for static members, graceful close keeps things clean
}

Move heavy work out of the poll loop

The highest-leverage structural fix is offloading heavy work to a separate worker thread/queue so the poll loop always stays light. If processing slows, apply backpressure with consumer.pause(partitions) and resume() when workers drain. That decouples processing time from the poll interval, so ①② disappear structurally.

A note from the trenches

A commonly reported incident pattern got worse from the wrong-direction prescription: "lag spiked, so bump max.poll.records." Bigger batches make each cycle longer and trigger more rebalances. The right answer was always shrink the batch and move heavy work out of the loop. And after Kafka 3.x, just switching to CooperativeStickyAssignor plus static membership cut deploy-time stalls by about 90% in practice. Once the next-gen protocol KIP-848 (broker-driven rebalancing) stabilizes in 4.x, the rebalance cost clients currently absorb should drop further—if you run on managed platforms like MSK Serverless or Confluent Cloud, check supported protocol versions ahead of time.

Conclusion: 5-step checklist to escape the rebalance loop

1. Emergency mitigation: cut max.poll.records in half and redeploy → shorten one cycle
2. Diagnose: log keywords + rebalance-rate-per-hour and commit-rate to pin cause ①–⑤
3. Align configs: session.timeout.ms = heartbeat.interval.ms × 3, size the interval from processing p99
4. Structural improvement: manual commit + idempotent handling; offload heavy work to workers
5. Stabilize at the root: roll out static membership + CooperativeStickyAssignor; alert on rebalance-rate

References: official docs

The primary sources for the behavior, settings, and errors covered here are the following official docs. Check version-specific options and exact behavior there.

• Apache Kafka documentation

FAQ

Q. Will just turning on auto-commit (enable.auto.commit=true) fix this? A. No. Auto-commit only hides CommitFailedException; the root cause—processing delay—remains. Worse, unprocessed offsets can get committed, causing message loss. Manual commit + idempotent handling is the safe path.

Q. Does adding partitions reduce rebalances? A. It can help spread throughput, but adding partitions itself triggers a rebalance, and it does nothing if you don't have enough consumers. First, shorten per-batch processing time.

Q. Do static membership and autoscaling conflict? A. They don't conflict, but group.instance.id must be fixed and unique per instance (e.g. StatefulSet ordinal). When scale-out introduces a new ID, one rebalance happens as expected. On scale-in, a zombie member can linger for session.timeout.ms, so send LeaveGroup via graceful shutdown.