Consumer group (Kafka)
A Kafka consumer group is a set of consumer instances that cooperatively consume a topic. Kafka assigns each partition to exactly one consumer in the group at any time, ensuring that each message is processed by exactly one consumer and that messages within a partition are processed in order. Adding consumers to a group increases parallelism up to the partition count; beyond that, additional consumers are idle.
Why it matters in practice
Consumer groups are how Kafka enables horizontal scaling of message processing. To double throughput, double the number of consumers and partitions in concert. Consumer groups also enable independent processing: two different applications (billing service and analytics service) can each have their own consumer group on the same topic, both receiving all messages independently without coordinating. This is a core Kafka design pattern for fan-out to multiple consumers.
Common mistakes
- •Using the same consumer group ID for two different applications that should process messages independently — they will share partitions and each receive only a fraction of the messages.
- •Not monitoring the number of active consumers in the group — a consumer that crashes silently may not trigger alerting, but its partitions will be reassigned and lag will grow until the next rebalance.