Kafka claim check pattern
The Kafka claim check pattern (also called the external payload pattern) stores large message payloads in external object storage (AWS S3, Azure Blob Storage, Google Cloud Storage) and puts only a reference pointer — typically a URL or object key plus metadata — in the Kafka record. Consumers fetch the full payload from object storage when processing. This sidesteps Kafka's message size limits and keeps the Kafka cluster's memory and network usage bounded.
Formula
When to use: Messages > 1 MB that don't need to be in Kafka's retention window, binary assets (images, ML model outputs, large JSON documents), payloads consumed by only one downstream system (no fan-out benefit from Kafka storage).Why it matters in practice
Kafka is optimised for small, fast messages — the default limit is 1 MB for good reason. Large messages hurt compression efficiency, consume more broker memory for in-flight batches, and slow down partition leader elections (more data to transfer to the new leader). The claim check pattern keeps Kafka lean: the pointer is tiny (typically < 1 KB), compresses well, and retains the event stream semantics (ordering, consumer groups, replay) while offloading the heavy data to storage that is optimised for large objects.
Common mistakes
- •Using claim check for small messages (< 100 KB) — the overhead of external storage calls outweighs the benefits.
- •Not implementing cleanup for external storage objects — orphaned objects can accumulate indefinitely, increasing storage costs.
- •Not handling object storage failures gracefully — if S3 is unavailable, the consumer cannot process the message even though the Kafka record is accessible.