Open Source data engineering demo project using dbt, DuckDB, dlt, Dagster and Metabase.
-
Updated
Oct 28, 2026 - Python
Open Source data engineering demo project using dbt, DuckDB, dlt, Dagster and Metabase.
A modern banking data pipeline built with Dagster and DBT!
Fortune-500-grade banking analytics platform: OLTP -> medallion lakehouse -> Kimball star schema -> semantic layer -> 9-tab executive dashboard + 5 ML models (churn, fraud, segmentation, forecasting). Production-ready, governed, fully tested.
Governed ARR, retention, and AI-adoption metrics with SCD2 modeling, role-based access, and safe SQL compilation.
Analytics data warehouse for a production livestock ERP — dbt + BigQuery, 5-layer architecture, DAMA-DMBOK governance implemented as executable tests.
PySpark bronze/silver/gold CDC pipeline with SCD Type 2, data-quality gates and idempotent runs
Modern data stack reference: dbt + BigQuery + Airflow (Cloud Composer) with medallion layering, SCD2 snapshots, exposures, freshness SLAs, and 45× cost reduction via partition + cluster + incremental tuning.
One dbt project, three platforms: a tested star schema with SCD2 history and an AI-ready vector layer, built on PostgreSQL, Snowflake and Databricks from synthetic private-markets data.
An end-to-end analytics engineering pipeline that transforms raw API telemetry into actionable business metrics. Built with Python, DBT, and DuckDB to model usage, monitor latency, and calculate tiered billing.
Convention-aware integrity auditor for SCD Type 2 dimensions. 16 SQL checks covering overlaps, gaps, current-flag drift, sentinel drift and point-in-time referential breaks, with findings ranked by the fact rows and revenue at risk and idempotent repair SQL in DuckDB and Snowflake via sqlglot. 1.28M version rows audited in 3.95s.
Production-style data platform for an invoice-factoring company: Postgres/MongoDB → Databricks (Spark + Delta) medallion pipeline with data contracts, quarantine and SCD2 → Snowflake (Snowpipe, streams/tasks, dbt marts) orchestrated by Airflow. Includes Terraform, CI, pytest + behave tests, and ADRs.
dbt e-commerce analytics on DuckDB: staging, intermediate and mart layers with tests, an incremental fact table and an SCD2 snapshot.
Batch medallion lakehouse: raw -> bronze/silver Delta (delta-rs) -> dbt gold star schema with a full SCD2 customer dimension and point-in-time fact resolution. Great Expectations guards each layer and a reconciler proves gold equals silver in dollars. Ships Spark jobs, an Airflow DAG, and Terraform.
Medallion lakehouse on Spark + Delta Lake + Airflow: idempotent bronze, MERGE/SCD2 silver with quarantine, point-in-time gold with partition-scoped rebuilds.
Read-only temporal-integrity validator for SCD Type 2 dimensions. Runs seven DuckDB window-function checks (overlaps, gaps, multiple current rows, and more) and reports each violation's blast radius in keys, fact rows, and dollars.
Spark batch and streaming ETL pipeline on 59M rows of Walmart M5 sales with a quality gate, SCD2 price history, Structured Streaming dedup and a measured Spark tuning lab
Batch retail data lakehouse on Databricks: Delta Live Tables (bronze → silver → gold), Unity Catalog, synthetic data generator, and an executive analytics dashboard.
To associate your repository with the scd2 topic, visit your repo's landing page and select "manage topics."