Skip to content

About

Data platform system designs and ADRs: real-time aggregation, multi-tenant platforms, batch-to-CDC migration, lakehouse cost, data SLAs and incident response.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

1 Commit

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 

Repository files navigation

data-platform-design

System designs and architecture decision records (ADRs) for data platforms. These are written the way I'd present them in a design review: requirements and numbers first, then the options, the trade-offs, what fails, and how we'd know.

The scenarios, team sizes and numbers are illustrative. They're not taken from any employer's systems; they're realistic stand-ins chosen to make the trade-offs concrete.

They complement the working code in lakehouse-medallion-platform, realtime-cdc-pipeline and dq-contracts.

System designs

# Design Core tension
01 Real-time ad click aggregation Exactly-once counts vs latency at 1M events/s
02 Multi-tenant self-serve data platform Team autonomy vs governance and cost control
03 Batch → streaming migration Migrating without a correctness regression or a big-bang cutover
04 Lakehouse cost optimization Reducing spend substantially without slowing anyone down
05 Data reliability: SLAs, observability, incidents Treating data as a product with measurable reliability

Architecture decision records

ADR Decision Status
0001 Open table format: Iceberg vs Delta vs Hudi Accepted
0002 Orchestrator: Airflow vs Dagster vs managed workflows Accepted
0003 Enforce data contracts in producer CI, not only in the warehouse Accepted
0004 Every pipeline step must be idempotent and replayable Accepted

Playbooks

How I approach a data system design

  1. Pin the numbers. Events/sec (avg and peak), payload size, retention, freshness SLO, query concurrency. Most bad designs come from skipping this step.
  2. Name the correctness model. At-least-once + idempotent sink? Exactly-once within a boundary? What's the source of truth when real-time and batch disagree?
  3. Draw the write path and the read path separately. They have different scaling problems.
  4. Plan for failure. Late data, duplicates, replays, poison records, partial outages, backfills. For each one: what happens and who finds out.
  5. Cost and ownership. Who pays, who's paged, and what does it cost at 10x?

About

Data platform system designs and ADRs: real-time aggregation, multi-tenant platforms, batch-to-CDC migration, lakehouse cost, data SLAs and incident response.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors