Consumer offset (Kafka)
A Kafka consumer offset is the position in a partition log that a consumer group has committed as successfully processed. It represents the next message the consumer will read on restart — not the last message read. Offsets are stored in the internal __consumer_offsets topic. The gap between the partition's log end offset and the committed offset is the consumer lag.
Why it matters in practice
Offset commits are the durability mechanism for consumer progress. If a consumer crashes, it will resume from the last committed offset, not from where it left off. This means processing must be idempotent or exactly-once semantics must be used, because the consumer may reprocess messages between the last commit and the crash. Manual offset management enables at-least-once processing with custom checkpoint logic; automatic commits are simpler but may cause duplicate processing on crash.
Common mistakes
- •Confusing committed offset with last processed message — committed offset is the next message to read, not the last successfully processed one.
- •Not considering offset commit latency in processing time calculations — commits add network round-trip time to each batch.
- •Using enable.auto.commit=true with manual offset management — this creates race conditions where automatic commits overwrite manual commits.