Kafka
Focused kafka articles with clear context, practical examples, source links where needed, and honest limits.
13 articles in this section.
Measure Kafka consumer processing age, not only offset lag
Offset count lacks business time, while processing age reveals whether the oldest unhandled event is already missing its deadline.
Choose a Kafka message key from the ordering requirement
The message key should express the smallest business entity that requires ordering, balancing correctness against partition distribution.
Kafka replay without sending duplicate customer notifications
Replay is safe only when projections can rebuild and irreversible side effects can recognize historical duplicates.
Pause and resume Kafka consumers for downstream backpressure
Pausing fetches can protect a slow dependency while heartbeats continue, but long processing must still respect poll and rebalance settings.
Detect Kafka partition skew before adding consumers
Average lag can hide one hot partition whose key distribution limits throughput no matter how many idle consumers are added.
Transactional outbox with Kafka: publish database changes reliably
An outbox writes business state and an event record in one database transaction, then a relay publishes and marks it safely.
Kafka schema evolution: compatible changes without wishful thinking
Schema compatibility rules help, but consumers also need defaults and behavior that make old and new messages meaningful.
Design a Kafka dead-letter topic that supports recovery
A dead-letter topic needs original payload, source coordinates, failure category, attempt history, and a controlled replay path.
Exactly-once Kafka does not make external APIs exactly once
Kafka transactions can coordinate Kafka reads and writes, but they cannot atomically include an unrelated payment, email, or database API.
Kafka cooperative rebalancing: reduce stop-the-world pauses
Cooperative rebalancing lets consumers move partitions incrementally, but handlers still need safe revoke and assignment behavior.
Kafka partition count planning: throughput is only one constraint
Partitions control parallelism, ordering, recovery work, and metadata cost, which makes a single messages-per-second formula.
Kafka consumer lag is not enough: measure processing age too
A consumer can have low offset lag while processing old events, so operations need both queue depth and event-age.
Kafka KRaft migration: a rollback-aware plan for production clusters
Moving Kafka metadata from ZooKeeper to KRaft is an operational migration, not a configuration rename, and each phase needs a stop.