Skip to content

perf: add prelimit CTE to getMapKeys query + store clickhouse settings in shared cache - #1187

Closed
wrn14897 wants to merge 4 commits into
mainfrom
warren/opt-metadata-field-queries
Closed

wrn14897 wants to merge 4 commits into
mainfrom
warren/opt-metadata-field-queries

Conversation

@wrn14897

@wrn14897 wrn14897 commented Sep 19, 2025 •

Copy link
Copy Markdown
Contributor

Optimize the query performance of the getMapKeys method to prevent excessive resource usage in ClickHouse, even when max_rows_to_read is specified.

Ref: HDX-2411

I believe the reason Anthropic was hitting the memory error is that the `metadataMaxRowsToRead` setting was applied only to a specific query (and did not include the getKeyValues method). Now, since ClickHouse settings are cached and shared across all metadata instances, every method within the Metadata class should respect these settings.

This PR also adds another CTE filtering on top of queries that touch array aggregation functions like `groupUniqArray` and `groupUniqArrayArray`

I will port this fix to OSS later

Ref: HDX-2321
@changeset-bot

changeset-bot Bot commented Sep 19, 2025 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 5c7a5e2

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages
Name Type
@hyperdx/common-utils Patch
@hyperdx/app Patch
@hyperdx/api Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercel Bot commented Sep 19, 2025 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Preview Comments Updated (UTC)
hyperdx-v2-oss-app Ready Ready Preview Comment Sep 23, 2025 9:51pm

@github-actions

Copy link
Copy Markdown
Contributor

Stably Runner - Test Suite - 'Smoke Test'

Test Suite Run Result: 🔴 Failure (1/4 tests failed) [dashboard]

Failed Tests:


This comment was generated from stably-runner-action

@wrn14897
wrn14897 requested review from a team and SpencerTorres and removed request for a team and SpencerTorres September 19, 2025 02:22
...chartConfig,
with: [
{
name: 'sampledData',

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This actually introduces the regression noted in HDX-2440. Before we merge this PR we should have a fix prepared to land on top so we can smoothly fix the regression without too much weirdness in downstream git history while not breaking the next build.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good. I will take a look at it

@MikeShi42 MikeShi42 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See HDX-2440

@wrn14897

wrn14897 commented Sep 23, 2025 •

Copy link
Copy Markdown
Contributor Author

See HDX-2440

Fix PR for HDX-2431 is ready (#1201)

@github-actions

Copy link
Copy Markdown
Contributor

E2E Test Results

❌ 1 test failed • 20 passed • 3 skipped • 259s

Status Count
✅ Passed 20
❌ Failed 1
⚠️ Flaky 0
⏭️ Skipped 3

View full report →

@wrn14897

Copy link
Copy Markdown
Contributor Author

Its great the E2E picks up the filter errors :)

@wrn14897

Copy link
Copy Markdown
Contributor Author

Raised a new PR with the fix #1201

@wrn14897 wrn14897 closed this Sep 23, 2025
@wrn14897
wrn14897 deleted the warren/opt-metadata-field-queries branch September 23, 2025 21:56

This branch was successfully deployed

1 active deployment
Preview — 5c7a5e20 Deployed Sep 23, 2025 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants