Skip to content

Suggest article: testing a change-safety check on a 2M-row Delta table - #79

Open
Brahmawritings wants to merge 1 commit into
Data-Engineering-Weekly:mainfrom
Brahmawritings:main
Open

Brahmawritings wants to merge 1 commit into
Data-Engineering-Weekly:mainfrom
Brahmawritings:main

Conversation

@Brahmawritings

Copy link
Copy Markdown

Suggesting an article for the weekly: field notes from stress-testing a data change-safety check on a 2-million-row Delta table on Databricks serverless.

What readers can take from it without using any tool:

  • Unity Catalog lineage keeps reporting tables that were dropped
  • joining keys with = turns NULL-key rows into fake adds and removes
  • shallow clones work as a free sandbox for risky writes
  • a 10% sampled diff hid a one-row delete until a full-table checksum was added

Disclosure, given the no-vendor-promotion rule: I wrote both the article and the open-source (Apache-2.0) tool it tests. There is no company or paid product behind it. I understand if it doesn't fit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant