Skip to content

feat(postgres): set up dual-writes - #6316

Open
michaelkedar wants to merge 4 commits into
google:masterfrom
michaelkedar:💽✍️✍️

Hidden character warning

The head ref may contain hidden characters: "\ud83d\udcbd\u270d\ufe0f\u270d\ufe0f"
Open

michaelkedar wants to merge 4 commits into
google:masterfrom
michaelkedar:💽✍️✍️

Conversation

@michaelkedar

Copy link
Copy Markdown
Member

(Sorry this PR is so big, a lot of things ended up having to be done at the same time)
Set up dual-writing infrastructure for Postgres in the places where vulns get written:

  • Implemented the remaining VulnerabilityStore for postgres (splitting the queries.sql into files corresponding to the go files they're used in)
  • Propagated data needed from importer (and recoverer) to worker to enable raw_vulnerabilities and historic_vulnerabilities to be written with all needed fields
    • Doing this meant passing a compressed JSON object instead of a compressed proto over the Pub/Sub for the raw vulnerability. We need to make sure there are no jobs in Pub/Sub while this is being merged (i.e. pause the importers).
  • Implemented Dual-write wrapper, to write to Datastore/GCS, then Postgres. Using this for workers and vanir signatures.
  • Updated relations cron-job to also mutate in Postgres (this wasn't using the database interfaces, and I have plans to change how this is done when postgres becomes the source of truth)
  • Configured all the GKE images with the POSTGRES_* env vars (I've already set up the GKE secret for the db password)
  • tests and such

I may have to make a follow-up PR fairly quickly to bump the CPUs of the postgres if affected commits are too plentiful.

In a future PR, I plan to re-jig the tests so they spin up postgres in a docker container instead of trying to mock all the endpoints, so we can make sure our queries actually work properly.

@codecov-commenter

codecov-commenter commented Nov 16, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 65.89698% with 192 lines in your changes missing coverage. Please review.
✅ Project coverage is 55.53%. Comparing base (305ba68) to head (834c165).
⚠️ Report is 2 commits behind head on master.

Files with missing lines Patch % Lines
go/internal/database/postgres/vulnerability.go 77.55% 66 Missing ⚠️
go/cmd/relations/update.go 26.92% 19 Missing ⚠️
go/cmd/worker/main.go 0.00% 19 Missing ⚠️
go/cmd/relations/relations.go 0.00% 17 Missing ⚠️
go/internal/worker/subscriber.go 33.33% 12 Missing ⚠️
go/cmd/vanir_signatures/main.go 0.00% 11 Missing ⚠️
go/internal/database/dualwrite/vulnerability.go 72.72% 9 Missing ⚠️
go/internal/database/postgres/affected_versions.go 70.96% 9 Missing ⚠️
go/internal/recoverer/recoverer.go 83.33% 7 Missing ⚠️
go/internal/importer/importer.go 76.92% 6 Missing ⚠️
... and 5 more
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6316      +/-   ##
==========================================
+ Coverage   55.21%   55.53%   +0.32%     
==========================================
  Files         222      223       +1     
  Lines       22782    23297     +515     
==========================================
+ Hits        12579    12938     +359     
- Misses      10200    10356     +156     
  Partials        3        3              
Flag Coverage Δ
go 59.06% <65.89%> (+0.35%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants