How Kafka consumer group rebalancing works
When a consumer joins or leaves a group, Kafka reassigns partitions across the remaining members. During an eager rebalance every consumer stops, gives up all its partitions, and waits — which is why rebalances can feel like a mini outage.
Why rebalances happen
- A consumer joins or leaves (deploys, autoscaling).
- A consumer misses heartbeats for longer than
session.timeout.ms. - Processing takes longer than
max.poll.interval.ms, so the broker thinks the consumer is dead.
⚠️ The sneaky one
Slow processing inside the poll loop is the most common cause I’ve seen in production. The consumer is alive, but Kafka can’t tell.
Cooperative rebalancing
With the CooperativeStickyAssignor, consumers only give up the partitions that actually move. Everyone else keeps working.
props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
CooperativeStickyAssignor.class.getName());
Related: retries and idempotency — a rebalance can redeliver messages, so handlers must be safe to repeat.