Effective consumers (Kafka lag)
Effective consumers is the actual number of consumer instances actively processing partitions, capped at the partition count: effective_consumers = min(running_consumers, partition_count). Consumers beyond the partition count are assigned zero partitions and sit idle, contributing no throughput. This is the fundamental reason why adding consumer instances does not reduce lag once you have reached consumer saturation.
Why it matters in practice
Consumer saturation is the point where running_consumers > partition_count. At saturation, the only way to increase group throughput is to increase partition count. Since partitions cannot be decreased after creation, partition count planning must account for expected future scale. A common pattern: provision 20–30% more partitions than current consumer count to allow horizontal scaling without topic recreation.
Common mistakes
- •Adding consumer instances to reduce lag without checking partition count — if you have 12 partitions and 12 consumers, the 13th consumer will be idle.
- •Not monitoring idle consumers — idle consumers still participate in rebalances, adding latency to every scaling event without contributing throughput.
- •Assuming throughput scales linearly with consumer count — it does up to partition count, then plateaus completely.