Notes
Data engineering, explained slowly
Working notes on storage, pipelines, and system design — the fundamentals I keep coming back to. English-first; Indonesian translations to follow.
-
Late-arriving data: when the event shows up after the report closed
Event time is not arrival time. How late data breaks closed reports and incremental loads, and the patterns that make it harmless: watermarking, lookback reprocessing, idempotent partitions, and versioned event logs.
streamingevent-timewatermarksidempotencydata-quality -
Break your CDC pipeline on purpose
A lab that kills the consumer mid-batch, poisons the topic, and replays a consumer group — plus the bug this pipeline shipped with and how I found it.
kafkadlqobservabilitytestingstreaming -
Effectively-once: making a Kafka → ClickHouse sink idempotent
Kafka only promises at-least-once. Here is how write-then-commit, source-timestamp versions, and ReplacingMergeTree make duplicates harmless — without claiming exactly-once.
clickhousekafkaidempotencyexactly-oncestreaming -
Change data capture, explained by building it
Why a nightly sync is never enough — and how Debezium, Kafka, and a Python consumer turn the write-ahead log into a live replica of your database.
cdcstreamingdebeziumkafkapostgres -
File storage vs block storage vs object storage
The differences that actually matter to a data engineer — and why your lakehouse lives on objects while your database lives on blocks.
storagefundamentalssystem-design