A key takeaway is that batching alone isn’t a silver bullet; effective processing requires cohesive design of batching, offset management, ordering, and exception handling to avoid data loss or blocked partitions.

To reduce lag and preserve order, the setup kept the same partitioning and active consumers, while batching up to 2,000 events or 500 milliseconds, whichever comes first.

This production-focused case study demonstrates scaling a Kafka consumer from 4,000 to 25,000 events per second through batch processing, while maintaining order and delivery guarantees.

The approach avoids a dead-letter queue by retrying transient failures and recording terminal failure details (event, partition, offset, and error) in a database for later investigation, allowing continued processing without blocking partitions.

Offsets are committed only after successful batch processing; if a batch fails, the system reverts to record-by-record processing to advance offsets safely.

Before dispatching a batch to the business logic, the consumer retained partition and offset metadata for each record; after batch processing, it committed only the offsets that corresponded to successfully processed events.

The error handling differentiates transient from non-transient failures: transient errors trigger indefinite retries without advancing offsets, while non-transient errors are logged as terminal, yet offsets are advanced to prevent blocking, preserving at-least-once delivery for recoverable events.

Throughput rose from about 4,000 to roughly 25,000 events per second, surpassing the production service level goal of 8,000, with lag stabilizing as backlog fell.