Skip to content
View CodeWithSubramani's full-sized avatar

Block or report CodeWithSubramani

Block user

Prevent this user from interacting with your repositories and sending you notifications. Learn more about blocking users.

You must be logged in to block users.

Content in all repositories owned by your account will be closed.
Maximum 250 characters. Please don’t include any personal information such as legal names or email addresses. This note will only be visible to you.
Report abuse

Contact GitHub support about this user’s behavior. Learn more about reporting abuse.

Report abuse
CodeWithSubramani/README.md

Hi, I'm Subramani 👋

Lead Data Engineer in Hyderabad, India, with 9+ years building data platforms at Atlassian, PayPal and Deloitte. I lead a small data engineering team and stay hands-on: lakehouse architecture on Databricks, batch and streaming pipelines, and the reliability work that keeps them correct (data contracts, observability, idempotent backfills).

  • 🏗️ Lakehouse platforms: Databricks (Unity Catalog, Delta Live Tables, Delta Lake), Spark, Iceberg
  • 🌊 Streaming & CDC: Kafka, Debezium, Spark Structured Streaming, Flink
  • ✅ Data reliability: contracts, quality gates, SLAs/SLOs, lineage, incident response
  • 👥 Leading engineers: design reviews, mentoring, hiring, DataOps practices

Featured projects

Project What it shows
lakehouse-medallion-platform Spark + Delta medallion lakehouse: idempotent bronze (Delta txn versions), MERGE-if-newer and SCD2 silver with quarantine, point-in-time gold with partition-scoped rebuilds. End-to-end tests reconcile gold to silver to the cent.
realtime-cdc-pipeline Postgres → Debezium → Kafka → warehouse with LSN-guarded upserts. Tests prove every event permutation converges; DLQ, retry-without-commit, event-time windows with watermarks.
dq-contracts Data contracts library: SQL-pushdown validation, robust (median/MAD) volume anomaly detection, and a semver gate that blocks breaking schema changes in producer CI.
data-platform-design System designs and ADRs: ad-click aggregation at 1M events/s, multi-tenant platform, batch→CDC migration, lakehouse cost, data SLAs.

How these fit together

flowchart LR
    CDC[realtime-cdc-pipeline<br/>OLTP → Kafka → sink] --> LH[lakehouse-medallion-platform<br/>bronze → silver → gold]
    DQ[dq-contracts<br/>contracts + gates] -.validates.-> CDC
    DQ -.validates.-> LH
    D[data-platform-design<br/>the why behind the how] -.-> CDC & LH & DQ
Loading

Toolbox

Python SQL PySpark Scala · Databricks Delta Lake Iceberg Spark Flink Trino · Kafka Debezium · Airflow Databricks Workflows · AWS (S3, EMR, Glue, Kinesis) GCP (BigQuery, Pub/Sub, Dataflow) · CI/CD Docker IaC

Connect

LinkedIn

Pinned Loading

  1. data-platform-design data-platform-design Public

    Data platform system designs and ADRs: real-time aggregation, multi-tenant platforms, batch-to-CDC migration, lakehouse cost, data SLAs and incident response.

  2. dq-contracts dq-contracts Public

    Declarative, versioned data contracts: SQL-pushdown validation, robust volume anomaly detection, and a semver gate that blocks breaking schema changes in CI.

    Python

  3. lakehouse-medallion-platform lakehouse-medallion-platform Public

    Medallion lakehouse on Spark + Delta Lake + Airflow: idempotent bronze, MERGE/SCD2 silver with quarantine, point-in-time gold with partition-scoped rebuilds.

    Python

  4. realtime-cdc-pipeline realtime-cdc-pipeline Public

    Postgres -> Debezium -> Kafka -> idempotent, order-safe warehouse sink. LSN-guarded upserts, DLQ, retry without offset commit, event-time windows.

    Python