-
Late-arriving data: when the event shows up after the report closed
Event time is not arrival time. How late data breaks closed reports and incremental loads, and the patterns that make it harmless: watermarking, lookback reprocessing, idempotent partitions, and versioned event logs.
streamingevent-timewatermarksidempotencydata-quality -
Break your CDC pipeline on purpose
A lab that kills the consumer mid-batch, poisons the topic, and replays a consumer group — plus the bug this pipeline shipped with and how I found it.
kafkadlqobservabilitytestingstreaming -
Effectively-once: making a Kafka → ClickHouse sink idempotent
Kafka only promises at-least-once. Here is how write-then-commit, source-timestamp versions, and ReplacingMergeTree make duplicates harmless — without claiming exactly-once.
clickhousekafkaidempotencyexactly-oncestreaming -
Change data capture, explained by building it
Why a nightly sync is never enough — and how Debezium, Kafka, and a Python consumer turn the write-ahead log into a live replica of your database.
cdcstreamingdebeziumkafkapostgres