System designs and architecture decision records (ADRs) for data platforms. These are written the way I'd present them in a design review: requirements and numbers first, then the options, the trade-offs, what fails, and how we'd know.
The scenarios, team sizes and numbers are illustrative. They're not taken from any employer's systems; they're realistic stand-ins chosen to make the trade-offs concrete.
They complement the working code in lakehouse-medallion-platform, realtime-cdc-pipeline and dq-contracts.
| # | Design | Core tension |
|---|---|---|
| 01 | Real-time ad click aggregation | Exactly-once counts vs latency at 1M events/s |
| 02 | Multi-tenant self-serve data platform | Team autonomy vs governance and cost control |
| 03 | Batch → streaming migration | Migrating without a correctness regression or a big-bang cutover |
| 04 | Lakehouse cost optimization | Reducing spend substantially without slowing anyone down |
| 05 | Data reliability: SLAs, observability, incidents | Treating data as a product with measurable reliability |
| ADR | Decision | Status |
|---|---|---|
| 0001 | Open table format: Iceberg vs Delta vs Hudi | Accepted |
| 0002 | Orchestrator: Airflow vs Dagster vs managed workflows | Accepted |
| 0003 | Enforce data contracts in producer CI, not only in the warehouse | Accepted |
| 0004 | Every pipeline step must be idempotent and replayable | Accepted |
- Pin the numbers. Events/sec (avg and peak), payload size, retention, freshness SLO, query concurrency. Most bad designs come from skipping this step.
- Name the correctness model. At-least-once + idempotent sink? Exactly-once within a boundary? What's the source of truth when real-time and batch disagree?
- Draw the write path and the read path separately. They have different scaling problems.
- Plan for failure. Late data, duplicates, replays, poison records, partial outages, backfills. For each one: what happens and who finds out.
- Cost and ownership. Who pays, who's paged, and what does it cost at 10x?