[vinzee]

Problem

High-throughput services can produce millions of spans per second. Storing every span is expensive, so we run the OpenTelemetry Collector's tail-sampling processor to keep only 1-in-N spans. Each kept span carries a SampleRate attribute recording N.

Once data is sampled, naive aggregations are wrong: count() returns N-x fewer events than actually occurred, sum()/avg() are biased, and percentiles shift. Dashboards show misleadingly low request counts, throughput, and error rates, making capacity planning and alerting unreliable.

Why Materialized Views Cannot Solve This Alone

A materialized view that pre-aggregates sampled spans is a useful performance optimization for known dashboard queries, but it cannot replace a sampling-aware query engine.

Fixed dimensions. A materialized view pre-aggregates by a fixed set of GROUP BY keys (e.g. ServiceName, SpanName, StatusCode, TimestampBucket). Trace exploration requires slicing by arbitrary span attributes -- http.target, k8s.pod.name, custom business tags -- in combinations that cannot be predicted at view creation time. Grouping by a different dimension either requires going back to raw table or a separate materialized views for every possible dimension combination. If you try to work around the fixed-dimensions problem by adding high-cardinality span attributes to the GROUP BY, the materialized table approaches a 1:1 row ratio with the raw table. You end up doubling storage without meaningful compression.

Fixed aggregation fields. A typical MV only aggregates a single numeric column like Duration. Users want weighted aggregations over any numeric attribute: request body sizes, queue depths, retry counts, custom metrics attached to spans. Each new field requires adding more AggregateFunction columns and recreating the view.

Industry precedent. Platforms that rely solely on pre-aggregation (Datadog, Splunk, New Relic, Elastic) get accurate RED dashboards but cannot correct ad-hoc queries over sampled span data. Only query-engine weighting (Honeycomb) produces correct results for arbitrary ad-hoc queries, including weighted percentiles and heatmaps.

A better solution is making the query engine itself sampling-aware, so that all queries from dashboards, alerts, an ad-hoc searches, automatically weights by SampleRate regardless of which dimensions or fields the user picks. Materialized views remain a useful complement for accelerating known, fixed-dimension dashboard panels, but they are not a substitute for correct query-time weighting.

Summary

TraceSourceSchema gets a new optional field sampleRateExpression - the ClickHouse expression that evaluates to the per-span sample rate (e.g. SpanAttributes['SampleRate']). When not configured, all queries are unchanged.
When set, the query builder rewrites SQL aggregations to weight each span by its sample rate:

aggFn	Before	After (sample-corrected)	Overhead
count	count()	sum(weight)	~1x
count + cond	countIf(cond)	sumIf(weight, cond)	~1x
avg	avg(col)	sum(col * weight) / sum(weight)	~2x
sum	sum(col)	sum(col * weight)	~1x
quantile(p)	quantile(p)(col)	quantileTDigestWeighted(p)(col, toUInt32(weight))	~1.5x
min/max	unchanged	unchanged	1x
count_distinct	unchanged	unchanged (cannot correct)	1x

Types:

• Add sampleRateExpression to TraceSourceSchema + Mongoose model
• Add sampleWeightExpression to ChartConfig schema

Query builder:

• sampleWeightExpression is wrapped as greatest(toUInt64OrZero(toString(expr)), 1) so
  spans without a SampleRate attribute default to weight 1 (unsampled
  data produces identical results to the original queries).
• Rewrite aggFnExpr in renderChartConfig.ts when sampleWeightExpression
  is set, with safe default-to-1 wrapping

Integration (propagate sampleWeightExpression to all chart configs):

• ChartEditor/utils.ts, DBSearchPage, ServicesDashboardPage, sessions
• DBDashboardPage (raw SQL + builder branches)
• AlertPreviewChart
• SessionSubpanel
• ServiceDashboardEndpointPerformanceChart
• ServiceDashboardSlowestEventsTile (p95 query + events table)
• ServiceDashboardEndpointSidePanel (error rate + throughput)
• ServiceDashboardDbQuerySidePanel (total query time + throughput)
• External API v2 charts, AI controller, alerts (index + template)

UI:

• Add Sample Rate Expression field to trace source admin form

Screenshots or video

Before	After

How to test locally or on Vercel

References

• Linear Issue:
• Related PRs:

[vercel]

@vinzee is attempting to deploy a commit to the HyperDX Team on Vercel.

A member of the Team first needs to authorize it.

[changeset-bot]

🦋 Changeset detected

Latest commit: 9c789c2

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages

Name	Type
@hyperdx/common-utils	Minor
@hyperdx/api	Minor
@hyperdx/app	Minor
@hyperdx/otel-collector	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[github-actions]

PR Review

feat: support sample-weighted aggregations for sampled trace data (#1963)

Overall this is a well-implemented, well-tested feature. No critical bugs or security issues.

• ⚠️ toUInt32(greatest(toUInt64OrZero(...), 1)) in quantileTDigestWeighted silently truncates weights > 4,294,967,295 (renderChartConfig.ts:437). Impractical in real deployments (would require 4B+ sampling rate), but consider a guard or toUInt64 if ClickHouse supports it for this function.
• ⚠️ Functions like any/last_value silently fall through to unweighted behavior when sampleWeightExpression is set (renderChartConfig.ts:443-447). This is probably correct, but there's no user-visible indication in the UI/docs that these functions are unaffected by the sample weight setting. Acceptable as-is if intentional.
• ℹ️ The checkAlerts/index.ts mixes two patterns for the same concern: ...pickSampleWeightExpressionProps(source) (line 460) vs explicit getSampleWeightExpression + named key (lines 482/495). Functionally equivalent but slightly inconsistent style within the same file.

✅ Core logic, test coverage (unit snapshots + ClickHouse integration tests), migration safety (IF NOT EXISTS, greatest(..., 1) defaulting unsampled rows to weight 1), and propagation across chart configs all look correct.

[teeohhem]

If all rows are filtered out or all have NULL values, the denominator could be 0.

[teeohhem]

@vinzee Thank you for the PR. I left some feedback and will be testing this now.

[teeohhem]

Review: Additional Integration Test Coverage & Division-by-Zero Edge Case

Nice work on this feature — the core math and plumbing are solid. We ran the existing integration test suite against real ClickHouse and all 18 tests pass. We wrote 12 additional integration tests to cover scenarios the current suite doesn't exercise. All 30 pass.

Suggested additional test cases

Below are the tests we'd recommend adding to sampleWeightedAggregations.int.test.ts. They cover:

Empty result sets (3 tests):

• Weighted avg on empty result → documents the division-by-zero behavior (see bug below)
• Weighted sum on empty result → returns 0
• Weighted count on empty result → returns 0

groupBy with weighted aggregations (3 tests):

• Weighted count per ServiceName: api=23, web=17 (hand-computed)
• Weighted avg per ServiceName: api≈153.48, web≈158.82
• Weighted sum per ServiceName: api=3530, web=2700

Time-series with granularity (2 tests):

• Weighted count per time bucket (all rows in 1 bucket, total=40)
• Weighted count per service per time bucket (combines granularity + groupBy)

Edge cases with a separate table (4 tests):

• All SampleRate=1: proves weighted = unweighted (identity property)
• Non-numeric SpanAttributes 'abc' → clamped to weight 1 via toUInt64OrZero
• Large SampleRate (1,000,000): no overflow, exact expected values
• Large SampleRate avg: high-weight row correctly dominates the weighted average

Click to expand: full test code to add

Add an executeChartConfigAllRows helper alongside the existing executeChartConfig:

// Helper: render chart config to SQL, execute it, return all rows
async function executeChartConfigAllRows(
  config: ChartConfigWithOptDateRange,
): Promise<Record<string, string>[]> {
  const generatedSql = await renderChartConfig(
    config,
    metadata,
    querySettings,
  );
  const sql = parameterizedQueryToSql(generatedSql);

  const result = await client.query({
    query: sql,
    format: 'JSONEachRow',
  });
  return (await result.json()) as Record<string, string>[];
}

Then add these tests after the existing quantile with where condition test:

it('weighted avg on empty result set: documents division-by-zero behavior', async () => {
  const result = await executeChartConfig({
    ...baseConfig,
    select: [
      {
        aggFn: 'avg',
        valueExpression: 'Duration',
        aggCondition: "ServiceName = 'nonexistent'",
        aggConditionLanguage: 'sql',
      },
    ],
  });
  const value = Object.values(result)[0];
  const numValue = Number(value);
  // ClickHouse may return NaN, inf, or 0 for 0/0 depending on column types.
  // Any of these is acceptable — the key point is it doesn't error.
  // A future improvement would wrap this in if(denom = 0, NULL, num/denom).
  expect(
    !Number.isFinite(numValue) || numValue === 0,
  ).toBe(true);
});

it('weighted sum on empty result set: should return 0', async () => {
  const result = await executeChartConfig({
    ...baseConfig,
    select: [
      {
        aggFn: 'sum',
        valueExpression: 'Duration',
        aggCondition: "ServiceName = 'nonexistent'",
        aggConditionLanguage: 'sql',
      },
    ],
  });
  const value = Number(Object.values(result)[0]);
  expect(value).toBe(0);
});

it('weighted count on empty result set: should return 0', async () => {
  const result = await executeChartConfig({
    ...baseConfig,
    select: [
      {
        aggFn: 'count',
        valueExpression: '',
        aggCondition: "ServiceName = 'nonexistent'",
        aggConditionLanguage: 'sql',
      },
    ],
  });
  const value = Number(Object.values(result)[0]);
  expect(value).toBe(0);
});

it('groupBy ServiceName: weighted count per group', async () => {
  // api rows weights: 1+5+10+1+1+5 = 23
  // web rows weights: 1+5+10+1 = 17
  const rows = await executeChartConfigAllRows({
    ...baseConfig,
    groupBy: 'ServiceName',
    select: [
      {
        aggFn: 'count',
        valueExpression: '',
        aggCondition: '',
        alias: 'weighted_count',
      },
    ],
  });

  const byService = Object.fromEntries(
    rows.map(r => [r['ServiceName'], Number(r['weighted_count'])]),
  );
  expect(byService['api']).toBe(23);
  expect(byService['web']).toBe(17);
});

it('groupBy ServiceName: weighted avg(Duration) per group', async () => {
  // api: sum=3530, weight=23, avg=153.478...
  // web: sum=2700, weight=17, avg=158.823...
  const rows = await executeChartConfigAllRows({
    ...baseConfig,
    groupBy: 'ServiceName',
    select: [
      {
        aggFn: 'avg',
        valueExpression: 'Duration',
        aggCondition: '',
        alias: 'weighted_avg',
      },
    ],
  });

  const byService = Object.fromEntries(
    rows.map(r => [r['ServiceName'], Number(r['weighted_avg'])]),
  );
  expect(byService['api']).toBeCloseTo(3530 / 23, 2);
  expect(byService['web']).toBeCloseTo(2700 / 17, 2);
});

it('groupBy ServiceName: weighted sum(Duration) per group', async () => {
  // api: 100*1+200*5+150*10+250*1+80*1+120*5 = 3530
  // web: 300*1+50*5+175*10+400*1 = 2700
  const rows = await executeChartConfigAllRows({
    ...baseConfig,
    groupBy: 'ServiceName',
    select: [
      {
        aggFn: 'sum',
        valueExpression: 'Duration',
        aggCondition: '',
        alias: 'weighted_sum',
      },
    ],
  });

  const byService = Object.fromEntries(
    rows.map(r => [r['ServiceName'], Number(r['weighted_sum'])]),
  );
  expect(byService['api']).toBe(3530);
  expect(byService['web']).toBe(2700);
});

it('time-series with granularity: weighted count per time bucket', async () => {
  const rows = await executeChartConfigAllRows({
    ...baseConfig,
    displayType: DisplayType.Line,
    granularity: '1 minute',
    select: [
      {
        aggFn: 'count',
        valueExpression: '',
        aggCondition: '',
        alias: 'weighted_count',
      },
    ],
  });

  expect(rows.length).toBeGreaterThanOrEqual(1);
  const totalCount = rows.reduce(
    (acc, r) => acc + Number(r['weighted_count']),
    0,
  );
  expect(totalCount).toBe(40);
});

it('time-series with groupBy: weighted count per service per time bucket', async () => {
  const rows = await executeChartConfigAllRows({
    ...baseConfig,
    displayType: DisplayType.Line,
    granularity: '1 minute',
    groupBy: 'ServiceName',
    select: [
      {
        aggFn: 'count',
        valueExpression: '',
        aggCondition: '',
        alias: 'weighted_count',
      },
    ],
  });

  const byService = new Map<string, number>();
  for (const r of rows) {
    const svc = r['ServiceName'];
    byService.set(svc, (byService.get(svc) ?? 0) + Number(r['weighted_count']));
  }
  expect(byService.get('api')).toBe(23);
  expect(byService.get('web')).toBe(17);
});

describe('additional edge cases', () => {
  const EDGE_TABLE = 'test_sample_weighted_edge';

  beforeAll(async () => {
    await client.command({
      query: `
        CREATE OR REPLACE TABLE ${DB}.${EDGE_TABLE} (
          Timestamp DateTime64(9) CODEC(Delta(8), ZSTD(1)),
          Duration Float64,
          SampleRate UInt64,
          ServiceName LowCardinality(String),
          SpanAttributes Map(LowCardinality(String), String)
        )
        ENGINE = MergeTree()
        ORDER BY (Timestamp)
      `,
    });

    await client.command({
      query: `
        INSERT INTO ${DB}.${EDGE_TABLE}
          (Timestamp, Duration, SampleRate, ServiceName, SpanAttributes)
        VALUES
          ('2025-06-01 00:00:01', 100, 1, 'api', {'SampleRate': '1'}),
          ('2025-06-01 00:00:02', 200, 1, 'api', {'SampleRate': '1'}),
          ('2025-06-01 00:00:03', 300, 1, 'web', {'SampleRate': '1'}),
          ('2025-06-01 00:00:04', 400, 1, 'web', {'SampleRate': 'abc'}),
          ('2025-06-01 00:00:05', 50,  1000000, 'api', {'SampleRate': '1000000'})
      `,
    });
  });

  afterAll(async () => {
    await client.command({
      query: `DROP TABLE IF EXISTS ${DB}.${EDGE_TABLE}`,
    });
  });

  const edgeConfig: ChartConfigWithOptDateRange = {
    displayType: DisplayType.Table,
    connection: 'test-connection',
    from: { databaseName: DB, tableName: EDGE_TABLE },
    select: [],
    where: '',
    whereLanguage: 'sql',
    timestampValueExpression: 'Timestamp',
    sampleWeightExpression: 'SampleRate',
    dateRange: [new Date('2025-01-01'), new Date('2025-12-31')],
  };

  it('all SampleRate=1: weighted results should equal unweighted results', async () => {
    const filterConfig = {
      ...edgeConfig,
      where: "SampleRate = 1",
      whereLanguage: 'sql' as const,
    };

    const weightedResult = await executeChartConfig({
      ...filterConfig,
      select: [
        { aggFn: 'count', valueExpression: '', aggCondition: '', alias: 'wcount' },
        { aggFn: 'avg', valueExpression: 'Duration', aggCondition: '', alias: 'wavg' },
        { aggFn: 'sum', valueExpression: 'Duration', aggCondition: '', alias: 'wsum' },
      ],
    });

    const unweightedResult = await executeChartConfig({
      ...filterConfig,
      sampleWeightExpression: undefined,
      select: [
        { aggFn: 'count', valueExpression: '', aggCondition: '', alias: 'count' },
        { aggFn: 'avg', valueExpression: 'Duration', aggCondition: '', alias: 'avg' },
        { aggFn: 'sum', valueExpression: 'Duration', aggCondition: '', alias: 'sum' },
      ],
    });

    expect(Number(weightedResult['wcount'])).toBe(Number(unweightedResult['count']));
    expect(Number(weightedResult['wavg'])).toBeCloseTo(Number(unweightedResult['avg']), 5);
    expect(Number(weightedResult['wsum'])).toBeCloseTo(Number(unweightedResult['sum']), 5);
  });

  it('non-numeric SampleRate in SpanAttributes: should clamp to weight 1', async () => {
    const result = await executeChartConfig({
      ...edgeConfig,
      sampleWeightExpression: "SpanAttributes['SampleRate']",
      select: [
        {
          aggFn: 'count',
          valueExpression: '',
          aggCondition: "ServiceName = 'web'",
          aggConditionLanguage: 'sql',
        },
      ],
    });
    const value = Number(Object.values(result)[0]);
    // web rows: SpanAttributes['SampleRate'] = '1' (weight 1) + 'abc' (weight 1) = 2
    expect(value).toBe(2);
  });

  it('very large SampleRate: should handle without overflow', async () => {
    const result = await executeChartConfig({
      ...edgeConfig,
      select: [
        { aggFn: 'count', valueExpression: '', aggCondition: '', alias: 'wcount' },
        { aggFn: 'sum', valueExpression: 'Duration', aggCondition: '', alias: 'wsum' },
        { aggFn: 'avg', valueExpression: 'Duration', aggCondition: '', alias: 'wavg' },
      ],
    });

    // Total weights: 1+1+1+1+1000000 = 1000004
    expect(Number(result['wcount'])).toBe(1000004);
    // Total weighted sum: 100*1+200*1+300*1+400*1+50*1000000 = 50001000
    expect(Number(result['wsum'])).toBe(50001000);
    // Weighted avg: 50001000/1000004 ≈ 50.000800...
    expect(Number(result['wavg'])).toBeCloseTo(50001000 / 1000004, 2);
  });

  it('very large SampleRate: weighted avg dominated by high-weight row', async () => {
    const result = await executeChartConfig({
      ...edgeConfig,
      select: [
        { aggFn: 'avg', valueExpression: 'Duration', aggCondition: '' },
      ],
    });
    const value = Number(Object.values(result)[0]);
    // With weight 1M on Duration=50, the avg should be very close to 50
    expect(value).toBeGreaterThan(49);
    expect(value).toBeLessThan(51);
  });
});

---

Bug: Division by zero in weighted avg for empty result sets

In renderChartConfig.ts, the weighted avg formula:

chSql`sumIf(${weightedVal}, ${cond}) / sumIf(${w}, ${cond})`

When no rows match the filter, both sumIf calls return 0, producing 0/0. ClickHouse returns 0 or nan depending on column types rather than NULL. This will display as 0 or NaN in the UI instead of "no data".

Suggested fix — wrap the denominator:

// In the avg branch of aggFnExpr, for both the where and non-where cases:
const denom = isWhereUsed
  ? chSql`sumIf(${w}, ${cond})`
  : chSql`sumIf(${w}, ${{ UNSAFE_RAW_SQL: nullCheck }})`;
const numer = isWhereUsed
  ? chSql`sumIf(${weightedVal}, ${cond})`
  : chSql`sumIf(${weightedVal}, ${{ UNSAFE_RAW_SQL: nullCheck }})`;

return chSql`if(${denom} = 0, NULL, ${numer} / ${denom})`;

This is a minor edge case (requires all rows to be filtered out), but worth fixing for correctness.

[vinzee]

@teeohhem Thanks for the review. addressed all comments.

[brandon-pereira]

nit: this is already in the gitignore (line 54)

[vinzee]

removed ✅

[brandon-pereira]

Change LGTM - I tested this by manually manipulating SampleRate['SampleRate'] locally and was able to see the AVG changing as I changed the weight.

I didn't do a full test of the feature across all screens (assuming that @teeohhem did that), but given this is a backwards compatible feature that shouldn't be an issue.

I have added one very very minor request - otherwise this is good to go. Thanks for another great addition @vinzee !