feat: attribute ClickHouse queries to the surface that issued them

.changeset/query-log-attribution.md

@@ -0,0 +1,40 @@
+---
+'@hyperdx/common-utils': minor
+'@hyperdx/api': minor
+'@hyperdx/app': minor
+---
+
+feat: attribute ClickHouse queries to the dashboard, tile, search or alert that
+issued them
+
+A row in `system.query_log` previously couldn't be attributed to the part of
+HyperDX that produced it. Now every query HyperDX runs has a small JSON
+`log_comment`, plus a `query_id` starting with `hdx-`.
+
+Attribute load after the fact:
+
+```sql
+SELECT
+    JSONExtractString(log_comment, 'surface') AS surface,
+    JSONExtractString(log_comment, 'dashboard') AS dashboard,
+    JSONExtractString(log_comment, 'tile') AS tile,
+    count(),
+    sum(read_rows)
+FROM system.query_log
+WHERE type = 'QueryFinish' AND log_comment != '' AND query_id like 'hdx-%'
+GROUP BY ALL
+ORDER BY sum(read_rows) DESC
+```
+
+Or spot a query while it is still running, in `system.processes`, where the
+`query_id` names the surface with no JSON parsing needed.
+
+The payload records which part of the product asked (a dashboard, a search, an
+alert, an MCP tool, the chart explorer, session replay, field lookups, and so
+on), the dashboard and tile or saved search id, the source id, and on the server
+the trace id of the request. Field lookups and autocomplete are labelled too, so
+they can be told apart from a user's chart queries.
+
+The exception is queries that skip settings processing: the `system.settings`,
+server version and Cloud-detection probes, and the onboarding connection check.
+They get a `query_id` but an empty `log_comment`.

agent_docs/observability.md

@@ -35,6 +35,9 @@ aligned with the
 5. **Use the shared helpers** in
    [`packages/api/src/utils/instrumentation.ts`](../packages/api/src/utils/instrumentation.ts).
    Don't hand-roll tracer/meter lifecycle.
+6. **A new surface that queries ClickHouse declares its query attribution.**
+   One provider or one `withQueryAttribution` scope at the entry point, not a
+   change at each query. See [Query attribution](#query-attribution).
 
 ## The shared helper library
 
@@ -232,6 +235,84 @@ for this on functionality with real failure modes worth a target (external
 dependencies, query proxies, AI calls) — not on thin CRUD that the HTTP
 auto-instrumentation and error middleware already cover.
 
+## Query attribution
+
+Every ClickHouse query HyperDX runs is tagged with what asked for it, so a row
+in the customer's `system.query_log` points back to a dashboard tile, a saved
+search, an alert, or an MCP tool. This is the one piece of our instrumentation
+that lands in *their* data rather than ours, so it stays small, carries no
+identity, and must never be able to fail a query.
+
+Two channels carry it, set by `BaseClickhouseClient.query`:
+
+| Where | Shape | What it answers |
+| --- | --- | --- |
+| `log_comment` | JSON payload, e.g. `{"v":1,"surface":"dashboard","dashboard":"…","tile":"…"}` | After the fact: `GROUP BY JSONExtractString(log_comment, 'tile')` over `system.query_log` |
+| `query_id` | `hdx-<surface>-<uuid>` | Right now: reading `system.processes` while a query is still running |
+
+The vocabulary lives in
+[`packages/common-utils/src/clickhouse/attribution.ts`](../packages/common-utils/src/clickhouse/attribution.ts).
+`QUERY_SURFACES` is a fixed, short list, because it goes in the `query_id` and
+people group by it. Anything naming one specific dashboard, tile or search goes
+in the id fields instead.
+
+### Declaring it
+
+Not at the call site that runs the query: there are roughly sixty of those, and
+`Metadata` accounts for two dozen on its own. Set it once where the reason for
+the query is known, and everything underneath picks it up.
+
+**Browser** — React context, read by `useClickhouseClient`:
+
+```tsx
+<QueryAttributionProvider
+  attribution={{ surface: 'dashboard', dashboard: dashboardId, tile: chart.id }}
+>
+  {children}
+</QueryAttributionProvider>
+```
+
+Providers nest, and the innermost value wins, so a dashboard sets its id once
+and each tile adds only its own. A page that needs nothing finer than "which
+page" can use `withAppNavForSurface('…')` as its `getLayout` instead.
+
+Do **not** put attribution on the chart config: the config is the React Query
+cache key, so two surfaces asking the identical question would stop sharing a
+cache entry and query volume would double.
+
+**Server** — a scope around the request or job (`AsyncLocalStorage`, node only):
+
+```ts
+await withQueryAttribution({ surface: 'alert', alert: alert.id }, async () =>
+  processAlert(...),
+);
+```
+
+Wrap the whole request or job, not the query. One call at the top covers
+everything below it, including the materialized-view `EXPLAIN` probes.
+
+Order of precedence: the client's default, then the scope around the current
+request or job, then anything passed with an individual query.
+
+### Rules
+
+- **Values from the browser are hints, not proof.** The client controls them.
+  `sanitizeField` drops control characters and caps the length, and the payload
+  only ever becomes JSON strings, never SQL.
+- **It must never throw.** A tag describes the work; it does not gate it. Read
+  fields with `?.` even where the type says they are required, and keep helpers
+  like `getActiveTraceId` from throwing. A half-populated object must not be
+  able to stop an alert from running.
+- **Set it where the reason is known, not just on the page.** A page-level
+  surface is a fallback. A modal or panel rendered outside a tile only gets the
+  page's surface, so give it its own (`chart-preview`) when it has ids to
+  report.
+- **Keep the payload small.** It rides in the URL query string on every browser
+  query. The cap is 1 KB; real payloads are 100 to 200 bytes. Today's fields
+  cannot fill the budget even at full length, so nothing is ever shed. If you
+  add a field, check that still holds: over budget, fields are dropped whole
+  from the end rather than truncated, so the value stays parseable JSON.
+
 ## Where to look for examples
 
 - Generic span + status handling: `packages/api/src/utils/instrumentation.ts`
@@ -260,3 +341,9 @@ auto-instrumentation and error middleware already cover.
 - End-to-end + sub-operation SLO metrics on a background job (alert evaluation
   and its data fetch): `packages/api/src/tasks/checkAlerts/index.ts`
   (`alerts.evaluate`, `alerts.query`)
+- Query attribution scope at the auth boundary:
+  `packages/api/src/middleware/auth.ts` (`nextWithQueryAttribution`)
+- Query attribution around one alert evaluation:
+  `packages/api/src/tasks/checkAlerts/index.ts` (`alertQueryAttribution`)
+- Query attribution per React subtree: `packages/app/src/queryAttribution.tsx`,
+  mounted per tile in `packages/app/src/DBDashboardPage.tsx`

packages/api/src/mcp/utils/tracing.ts

@@ -1,10 +1,12 @@
+import { withQueryAttribution } from '@hyperdx/common-utils/dist/clickhouse/node';
 import type { CallToolResult } from '@modelcontextprotocol/sdk/types.js';
 
 import { recordOnboardingTaskCompletion } from '@/controllers/user';
 import type { McpContext, ToolResult } from '@/mcp/tools/types';
 import type { McpErrorCategory, McpErrorResult } from '@/mcp/utils/errors';
 import { getErrorCategory } from '@/mcp/utils/errors';
 import {
+  getActiveTraceId,
   getCounter,
   getHistogram,
   SpanStatusCode,
@@ -88,7 +90,16 @@ export function withToolTracing<TArgs>(
         logger.info(logContext, `MCP tool invoked: ${toolName}`);
 
         try {
-          const result = await handler(args);
+          // Around the whole handler, so every query the tool makes carries
+          // the tool name and who called it.
+          const result = await withQueryAttribution(
+            {
+              surface: 'mcp',
+              label: toolName,
+              trace: getActiveTraceId(),
+            },
+            async () => handler(args),
+          );
           const durationMs = Date.now() - startTime;
 
           if (result.isError) {

packages/api/src/middleware/__tests__/queryAttribution.int.test.ts

@@ -0,0 +1,98 @@
+import * as config from '@/config';
+import {
+  DEFAULT_DATABASE,
+  dropTimeSeriesTable,
+  getLoggedInAgent,
+  getServer,
+  getTestFixtureClickHouseClient,
+  seedTimeSeriesTagsTable,
+} from '@/fixtures';
+import Connection from '@/models/connection';
+
+// Covers the wiring the unit tests can't: auth middleware opens the
+// AsyncLocalStorage scope, and a query issued deep inside the route handler
+// still reads it. If that scope is lost, queries fall back to hdx-unknown-*.
+describe('request query attribution', () => {
+  const server = getServer();
+  // Unique per run so query_log rows from earlier runs can't match.
+  const TABLE = `query_attribution_${Date.now()}`;
+
+  beforeAll(async () => {
+    await server.start();
+    await seedTimeSeriesTagsTable({
+      table: TABLE,
+      series: [
+        {
+          metricName: 'attribution_metric',
+          tags: { job: 'api' },
+          startSec: 1700000000,
+          endSec: 1700003600,
+        },
+      ],
+    });
+  });
+
+  afterEach(async () => {
+    await server.clearDBs();
+  });
+
+  afterAll(async () => {
+    await dropTimeSeriesTable({ table: TABLE });
+    await server.stop();
+  });
+
+  // query_log is written asynchronously, so flush and poll.
+  async function findRouteQueries() {
+    const client = await getTestFixtureClickHouseClient();
+    for (let attempt = 0; attempt < 10; attempt++) {
+      await client.command({ query: 'SYSTEM FLUSH LOGS' });
+      const result = await client.query({
+        query: `
+          SELECT query_id, log_comment
+          FROM system.query_log
+          WHERE type = 'QueryFinish'
+            AND query_kind = 'Select'
+            AND position(query, {table:String}) > 0`,
+        // Match on the text: timeSeriesTags() reads an inner table, so
+        // `tables` lists `.inner_id.tags.<uuid>`, not TABLE.
+        query_params: { table: TABLE },
+        format: 'JSON',
+      });
+      const { data }: { data: { query_id: string; log_comment: string }[] } =
+        await result.json();
+      if (data.length > 0) return data;
+      await new Promise(resolve => setTimeout(resolve, 250));
+    }
+    return [];
+  }
+
+  it('tags queries from an authenticated route with the api surface', async () => {
+    const { agent, team } = await getLoggedInAgent(server);
+    const conn = await Connection.create({
+      team: team._id,
+      name: 'CH',
+      host: config.CLICKHOUSE_HOST,
+      username: config.CLICKHOUSE_USER,
+      password: config.CLICKHOUSE_PASSWORD,
+    });
+
+    await agent
+      .get('/v1/prometheus/labels')
+      .query({
+        connectionId: conn._id.toString(),
+        database: DEFAULT_DATABASE,
+        table: TABLE,
+      })
+      .expect(200);
+
+    const logged = await findRouteQueries();
+    expect(logged.length).toBeGreaterThan(0);
+    for (const row of logged) {
+      expect(row.query_id).toMatch(/^hdx-api-/);
+      expect(JSON.parse(row.log_comment)).toMatchObject({
+        surface: 'api',
+        label: '/v1/prometheus',
+      });
+    }
+  });
+});

packages/api/src/middleware/auth.ts

@@ -1,3 +1,4 @@
+import { withQueryAttribution } from '@hyperdx/common-utils/dist/clickhouse/node';
 import { Connection } from '@hyperdx/common-utils/dist/types';
 import type { NextFunction, Request, Response } from 'express';
 import { serializeError } from 'serialize-error';
@@ -6,6 +7,7 @@ import * as config from '@/config';
 import { findUserByAccessKey } from '@/controllers/user';
 import type { UserDocument } from '@/models/user';
 import {
+  getActiveTraceId,
   getStaticFeatureFlags,
   setBusinessContext,
 } from '@/utils/instrumentation';
@@ -109,7 +111,23 @@ export async function validateUserAccessKey(
     ...getStaticFeatureFlags(),
   });
 
-  next();
+  nextWithQueryAttribution(req, next);
+}
+
+/**
+ * Continue the request with its route recorded on any ClickHouse queries it
+ * makes. Here because every authenticated route passes through this point.
+ */
+function nextWithQueryAttribution(req: Request, next: NextFunction): void {
+  withQueryAttribution(
+    {
+      surface: 'api',
+      // Mount path, not the full URL, to keep request ids out of the log.
+      label: req.baseUrl || undefined,
+      trace: getActiveTraceId(),
+    },
+    () => next(),
+  );
 }
 
 export function isUserAuthenticated(
@@ -133,7 +151,7 @@ export function isUserAuthenticated(
       'hyperdx.local_mode': true,
       ...getStaticFeatureFlags(),
     });
-    return next();
+    return nextWithQueryAttribution(req, next);
   }
 
   if (req.isAuthenticated()) {
@@ -145,7 +163,7 @@ export function isUserAuthenticated(
       ...getStaticFeatureFlags(),
     });
 
-    return next();
+    return nextWithQueryAttribution(req, next);
   }
   res.sendStatus(401);
 }