feat: optimized default otel-logs schema by knudtty · Pull Request #2125 · ClickHouse/ClickStack

knudtty · 2026-04-15T20:17:13Z

Summary

After weeks of benchmarking, this optimized logs schema proves to beat the existing schema in almost all queries. The primary key is now prepended with toStartOfFiveMinutes, which groups rows together well enough that TimestampTime is not needed. Additionally, full text search is added for all indexes as it shows superior performance with minimal ingest overhead.

How to test locally

If you have a .volumes directory locally, (re)move it
yarn dev
Poke around and make sure everything is working. Check the schema to ensure its the new one

References

Linear Issue: Closes HDX-4034

vercel · 2026-04-15T20:17:18Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
hyperdx-oss	Ready	Preview, Comment	May 5, 2026 3:31pm

changeset-bot · 2026-04-15T20:17:21Z

🦋 Changeset detected

Latest commit: adfd9aa

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages

Name	Type
@hyperdx/otel-collector	Minor
@hyperdx/api	Minor
@hyperdx/app	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

github-actions · 2026-04-15T20:17:26Z

🔴 Tier 4 — Critical

Touches auth, data models, config, tasks, OTel pipeline, ClickHouse, or CI/CD.

Why this tier:

Critical-path files (8):
- .github/workflows/main.yml
- docker/clickhouse/local/init-db-e2e.sh
- docker/hyperdx/Dockerfile
- docker/hyperdx/entry.local.base.sh
- docker/otel-collector/schema/seed/00002_otel_logs.sql
- docker/otel-collector/schema/seed/00002_otel_logs_compat.sql
- packages/otel-collector/cmd/migrate/main.go
- packages/otel-collector/cmd/migrate/main_test.go
Cross-layer change: touches frontend (packages/app) + backend (packages/api)

Review process: Deep review from a domain expert. Synchronous walkthrough may be required.
SLA: Schedule synchronous review within 2 business days.

Stats

Production files changed: 34
Production lines changed: 513 (+ 16 in test files, excluded from tier calculation)
Branch: aaron/new-otel-schema
Author: knudtty

To override this classification, remove the review/tier-4 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

github-actions · 2026-04-15T20:20:26Z

PR Review

⚠️ Breaking schema change with no automated migration → Existing deployments with data in otel_logs will silently lose it (table won't be recreated by goose if it already exists). Consider adding a goose -- +goose Down migration or a doc/warning for ops teams upgrading from the old schema; "rm .volumes" only works locally.
⚠️ EventName column added to schema but never referenced in any query, config, or API code → Confirm this is intentional (forward-looking for OTEL spec) or wire it up; silent dead columns cause confusion.
⚠️ enable_block_number_column / enable_block_offset_column settings added to new schema but absent from e2e init script (docker/clickhouse/local/init-db-e2e.sh:129) → E2E tests run against a schema that differs from production, which could mask bugs dependent on those settings.
⚠️ INDEX idx_trace_id TraceId TYPE text(tokenizer = 'array') — array tokenizer treats the value as an array of strings; TraceId is a single String (hex UUID), so this indexes it as a one-element array and won't help partial/prefix lookups. The old bloom_filter(0.001) was correct for exact TraceId lookups. Consider using text(tokenizer = 'default') or keeping bloom_filter for this specific index.
✅ Version-gated compat schema logic (Go), unit tests, and smoke tests for the compat path are solid.
✅ Typo fix in CI workflow (otel-ccollector → otel-collector) is a good catch.

github-actions · 2026-04-15T20:26:01Z

E2E Test Results

✅ All tests passed • 158 passed • 3 skipped • 1246s

Status	Count
✅ Passed	158
❌ Failed	0
⚠️ Flaky	5
⏭️ Skipped	3

Tests ran across 4 shards in parallel.

View full report →

knudtty · 2026-04-15T21:31:08Z

Edit: This is completed

I still need to add some logic that only adds FTS if the CH version is > 26.2. Will require some modification to the migration logic

26.2

wrn14897 · 2026-04-17T22:13:41Z

+  INDEX idx_scope_attr_value mapValues(ScopeAttributes) TYPE text(tokenizer = 'array'),
+  INDEX idx_log_attr_key mapKeys(LogAttributes) TYPE text(tokenizer = 'array'),
+  INDEX idx_log_attr_value mapValues(LogAttributes) TYPE text(tokenizer = 'array'),
+  INDEX idx_lower_body lower(Body) TYPE text(tokenizer = 'splitByNonAlpha')


What about adding FTS on all fields like

ADD INDEX fulltext_lower_v1_idx concatWithSeparator(';', TraceId, SpanId, TraceFlags, SeverityText, SeverityNumber, ServiceName, Body, ResourceSchemaUrl, ResourceAttributes, ScopeSchemaUrl, ScopeName, ScopeVersion, ScopeAttributes, LogAttributes) TYPE text(tokenizer = 'splitByNonAlpha', preprocessor = lower(concatWithSeparator(';', TraceId, SpanId, TraceFlags, SeverityText, SeverityNumber, ServiceName, Body, ResourceSchemaUrl, ResourceAttributes, ScopeSchemaUrl, ScopeName, ScopeVersion, ScopeAttributes, LogAttributes))) GRANULARITY 100000000

Add change the implicit search field to concatWithSeparator(';', TraceId, SpanId, TraceFlags, SeverityText, SeverityNumber, ServiceName, Body, ResourceSchemaUrl, ResourceAttributes, ScopeSchemaUrl, ScopeName, ScopeVersion, ScopeAttributes, LogAttributes). This is going to massively improve DX since developers won’t always need to look up the attributes. Any thoughts?

Yeah I'd like to, it isn't supported yet. Probably made it into 26.4

wrn14897 · 2026-04-17T22:21:50Z

@@ -1 +1 @@
-SELECT SeverityText FROM otel_logs WHERE ResourceAttributes['suite-id'] = 'normalize-severity' AND ResourceAttributes['test-id'] = 'text-case' ORDER BY TimestampTime FORMAT CSV
+SELECT SeverityText FROM otel_logs WHERE ResourceAttributes['suite-id'] = 'normalize-severity' AND ResourceAttributes['test-id'] = 'text-case' ORDER BY (toStartOfFiveMinutes(Timestamp), Timestamp) FORMAT CSV


For smoke tests, I’d suggest adding another ClickHouse instance with an older version (< 26.2) and ensuring the schema and SELECT queries still work correctly.

@knudtty Is that something you're working on?

wrn14897 · 2026-04-17T22:28:06Z

 ENABLE_SWAGGER=true
 DEFAULT_CONNECTIONS=[{"name":"Local ClickHouse","host":"http://localhost:${HDX_DEV_CH_HTTP_PORT}","username":"default","password":""}]
-DEFAULT_SOURCES=[{"from":{"databaseName":"default","tableName":"otel_logs"},"kind":"log","timestampValueExpression":"TimestampTime","name":"Logs","displayedTimestampValueExpression":"Timestamp","implicitColumnExpression":"Body","serviceNameExpression":"ServiceName","bodyExpression":"Body","eventAttributesExpression":"LogAttributes","resourceAttributesExpression":"ResourceAttributes","defaultTableSelectExpression":"Timestamp,ServiceName,SeverityText,Body","severityTextExpression":"SeverityText","traceIdExpression":"TraceId","spanIdExpression":"SpanId","connection":"Local ClickHouse","traceSourceId":"Traces","sessionSourceId":"Sessions","metricSourceId":"Metrics"},{"from":{"databaseName":"default","tableName":"otel_traces"},"kind":"trace","timestampValueExpression":"Timestamp","name":"Traces","displayedTimestampValueExpression":"Timestamp","implicitColumnExpression":"SpanName","serviceNameExpression":"ServiceName","eventAttributesExpression":"SpanAttributes","resourceAttributesExpression":"ResourceAttributes","defaultTableSelectExpression":"Timestamp,ServiceName,StatusCode,round(Duration/1e6),SpanName","traceIdExpression":"TraceId","spanIdExpression":"SpanId","durationExpression":"Duration","durationPrecision":9,"parentSpanIdExpression":"ParentSpanId","spanNameExpression":"SpanName","spanKindExpression":"SpanKind","statusCodeExpression":"StatusCode","statusMessageExpression":"StatusMessage","connection":"Local ClickHouse","logSourceId":"Logs","sessionSourceId":"Sessions","metricSourceId":"Metrics"},{"from":{"databaseName":"default","tableName":""},"kind":"metric","timestampValueExpression":"TimeUnix","name":"Metrics","resourceAttributesExpression":"ResourceAttributes","metricTables":{"gauge":"otel_metrics_gauge","histogram":"otel_metrics_histogram","sum":"otel_metrics_sum","_id":"682586a8b1f81924e628e808","id":"682586a8b1f81924e628e808"},"connection":"Local ClickHouse","logSourceId":"Logs","traceSourceId":"Traces","sessionSourceId":"Sessions"},{"from":{"databaseName":"default","tableName":"hyperdx_sessions"},"kind":"session","timestampValueExpression":"TimestampTime","name":"Sessions","displayedTimestampValueExpression":"Timestamp","implicitColumnExpression":"Body","serviceNameExpression":"ServiceName","bodyExpression":"Body","eventAttributesExpression":"LogAttributes","resourceAttributesExpression":"ResourceAttributes","defaultTableSelectExpression":"Timestamp,ServiceName,SeverityText,Body","severityTextExpression":"SeverityText","traceIdExpression":"TraceId","spanIdExpression":"SpanId","connection":"Local ClickHouse","logSourceId":"Logs","traceSourceId":"Traces","metricSourceId":"Metrics"},{"from":{"databaseName":"otel_json","tableName":"otel_logs"},"kind":"log","timestampValueExpression":"Timestamp","name":"JSON Logs","displayedTimestampValueExpression":"Timestamp","implicitColumnExpression":"Body","serviceNameExpression":"ServiceName","bodyExpression":"Body","eventAttributesExpression":"LogAttributes","resourceAttributesExpression":"ResourceAttributes","defaultTableSelectExpression":"Timestamp,ServiceName,SeverityText,Body","severityTextExpression":"SeverityText","traceIdExpression":"TraceId","spanIdExpression":"SpanId","connection":"Local ClickHouse","traceSourceId":"JSON Traces","metricSourceId":"JSON Metrics"},{"from":{"databaseName":"otel_json","tableName":"otel_traces"},"kind":"trace","timestampValueExpression":"Timestamp","name":"JSON Traces","displayedTimestampValueExpression":"Timestamp","implicitColumnExpression":"SpanName","serviceNameExpression":"ServiceName","eventAttributesExpression":"SpanAttributes","resourceAttributesExpression":"ResourceAttributes","defaultTableSelectExpression":"Timestamp,ServiceName,StatusCode,round(Duration/1e6),SpanName","traceIdExpression":"TraceId","spanIdExpression":"SpanId","durationExpression":"Duration","durationPrecision":9,"parentSpanIdExpression":"ParentSpanId","spanNameExpression":"SpanName","spanKindExpression":"SpanKind","statusCodeExpression":"StatusCode","statusMessageExpression":"StatusMessage","connection":"Local ClickHouse","logSourceId":"JSON Logs","metricSourceId":"JSON Metrics"},{"from":{"databaseName":"otel_json","tableName":""},"kind":"metric","timestampValueExpression":"TimeUnix","name":"JSON Metrics","resourceAttributesExpression":"ResourceAttributes","metricTables":{"gauge":"otel_metrics_gauge","histogram":"otel_metrics_histogram","sum":"otel_metrics_sum"},"connection":"Local ClickHouse","logSourceId":"JSON Logs","traceSourceId":"JSON Traces"}]
+DEFAULT_SOURCES=[{"from":{"databaseName":"default","tableName":"otel_logs"},"kind":"log","timestampValueExpression":"Timestamp","name":"Logs","displayedTimestampValueExpression":"Timestamp","implicitColumnExpression":"Body","serviceNameExpression":"ServiceName","bodyExpression":"Body","eventAttributesExpression":"LogAttributes","resourceAttributesExpression":"ResourceAttributes","defaultTableSelectExpression":"Timestamp,ServiceName,SeverityText,Body","severityTextExpression":"SeverityText","traceIdExpression":"TraceId","spanIdExpression":"SpanId","connection":"Local ClickHouse","traceSourceId":"Traces","sessionSourceId":"Sessions","metricSourceId":"Metrics"},{"from":{"databaseName":"default","tableName":"otel_traces"},"kind":"trace","timestampValueExpression":"Timestamp","name":"Traces","displayedTimestampValueExpression":"Timestamp","implicitColumnExpression":"SpanName","serviceNameExpression":"ServiceName","eventAttributesExpression":"SpanAttributes","resourceAttributesExpression":"ResourceAttributes","defaultTableSelectExpression":"Timestamp,ServiceName,StatusCode,round(Duration/1e6),SpanName","traceIdExpression":"TraceId","spanIdExpression":"SpanId","durationExpression":"Duration","durationPrecision":9,"parentSpanIdExpression":"ParentSpanId","spanNameExpression":"SpanName","spanKindExpression":"SpanKind","statusCodeExpression":"StatusCode","statusMessageExpression":"StatusMessage","connection":"Local ClickHouse","logSourceId":"Logs","sessionSourceId":"Sessions","metricSourceId":"Metrics"},{"from":{"databaseName":"default","tableName":""},"kind":"metric","timestampValueExpression":"TimeUnix","name":"Metrics","resourceAttributesExpression":"ResourceAttributes","metricTables":{"gauge":"otel_metrics_gauge","histogram":"otel_metrics_histogram","sum":"otel_metrics_sum","_id":"682586a8b1f81924e628e808","id":"682586a8b1f81924e628e808"},"connection":"Local ClickHouse","logSourceId":"Logs","traceSourceId":"Traces","sessionSourceId":"Sessions"},{"from":{"databaseName":"default","tableName":"hyperdx_sessions"},"kind":"session","timestampValueExpression":"TimestampTime","name":"Sessions","displayedTimestampValueExpression":"Timestamp","implicitColumnExpression":"Body","serviceNameExpression":"ServiceName","bodyExpression":"Body","eventAttributesExpression":"LogAttributes","resourceAttributesExpression":"ResourceAttributes","defaultTableSelectExpression":"Timestamp,ServiceName,SeverityText,Body","severityTextExpression":"SeverityText","traceIdExpression":"TraceId","spanIdExpression":"SpanId","connection":"Local ClickHouse","logSourceId":"Logs","traceSourceId":"Traces","metricSourceId":"Metrics"},{"from":{"databaseName":"otel_json","tableName":"otel_logs"},"kind":"log","timestampValueExpression":"Timestamp","name":"JSON Logs","displayedTimestampValueExpression":"Timestamp","implicitColumnExpression":"Body","serviceNameExpression":"ServiceName","bodyExpression":"Body","eventAttributesExpression":"LogAttributes","resourceAttributesExpression":"ResourceAttributes","defaultTableSelectExpression":"Timestamp,ServiceName,SeverityText,Body","severityTextExpression":"SeverityText","traceIdExpression":"TraceId","spanIdExpression":"SpanId","connection":"Local ClickHouse","traceSourceId":"JSON Traces","metricSourceId":"JSON Metrics"},{"from":{"databaseName":"otel_json","tableName":"otel_traces"},"kind":"trace","timestampValueExpression":"Timestamp","name":"JSON Traces","displayedTimestampValueExpression":"Timestamp","implicitColumnExpression":"SpanName","serviceNameExpression":"ServiceName","eventAttributesExpression":"SpanAttributes","resourceAttributesExpression":"ResourceAttributes","defaultTableSelectExpression":"Timestamp,ServiceName,StatusCode,round(Duration/1e6),SpanName","traceIdExpression":"TraceId","spanIdExpression":"SpanId","durationExpression":"Duration","durationPrecision":9,"parentSpanIdExpression":"ParentSpanId","spanNameExpression":"SpanName","spanKindExpression":"SpanKind","statusCodeExpression":"StatusCode","statusMessageExpression":"StatusMessage","connection":"Local ClickHouse","logSourceId":"JSON Logs","metricSourceId":"JSON Metrics"},{"from":{"databaseName":"otel_json","tableName":""},"kind":"metric","timestampValueExpression":"TimeUnix","name":"JSON Metrics","resourceAttributesExpression":"ResourceAttributes","metricTables":{"gauge":"otel_metrics_gauge","histogram":"otel_metrics_histogram","sum":"otel_metrics_sum"},"connection":"Local ClickHouse","logSourceId":"JSON Logs","traceSourceId":"JSON Traces"}]


A reminder that we also need to update helm-chart before the next release https://github.com/ClickHouse/ClickStack-helm-charts/blob/a7862878b89aa30d459fbda602087cb76defd33a/charts/clickstack/values.yaml#L103

wrn14897 · 2026-05-05T04:00:00Z

+PARTITION BY toDate(Timestamp)
+ORDER BY (toStartOfFiveMinutes(Timestamp), ServiceName, Timestamp)
+TTL toDateTime(Timestamp) + ${TABLES_TTL}
+SETTINGS index_granularity = 8192, ttl_only_drop_parts = 1;


do we need to add enable_block_number_column = 1, enable_block_offset_column = 1; here?

Good call! Added

wrn14897

great optimization!

New schema is introduced in ClickHouse/ClickStack#2125 The schema removes the `TimestampTime` column, so default sources should be updated appropriately

knudtty added 2 commits April 15, 2026 14:55

feat: optimized default otel-logs schema

93c530d

fix: upgrade clickhouse version; update DEFAULT_SOURCES

32fdb05

github-actions Bot added the review/tier-4 Critical — deep review + domain expert sign-off label Apr 15, 2026

vercel Bot deployed to Preview April 15, 2026 20:17

View deployment

fix: remove TimestampTime from order by in smoke tests

db5a56d

vercel Bot deployed to Preview April 15, 2026 20:33

View deployment

fix: e2e tests

1258457

vercel Bot deployed to Preview April 15, 2026 21:05

View deployment

knudtty requested review from a team, bot-hyperdx and dhable and removed request for a team and bot-hyperdx April 16, 2026 01:16

feat: only use full text index if clickhouse version is greater than

9ad1d75

26.2

vercel Bot deployed to Preview April 16, 2026 15:30

View deployment

chore: add testing for compatibility schema

508f369

vercel Bot deployed to Preview April 16, 2026 16:31

View deployment

knudtty force-pushed the aaron/new-otel-schema branch from d845174 to 8177a12 Compare April 17, 2026 18:11

vercel Bot deployed to Preview April 17, 2026 18:14

View deployment

feat: add EventName column (from upstream otel-collector-contrib)

2ddf9fd

knudtty force-pushed the aaron/new-otel-schema branch from 8177a12 to 2ddf9fd Compare April 17, 2026 18:15

vercel Bot deployed to Preview April 17, 2026 18:18

View deployment

wrn14897 reviewed Apr 17, 2026

View reviewed changes

vinzee reviewed Apr 17, 2026

View reviewed changes

Comment thread docker/clickhouse/local/init-db-e2e.sh Resolved

vinzee reviewed Apr 17, 2026

View reviewed changes

Comment thread docker/clickhouse/local/init-db-e2e.sh Resolved

add compat schema tests

bf582c6

vercel Bot deployed to Preview May 4, 2026 20:58

View deployment

Merge remote-tracking branch 'origin/main' into aaron/new-otel-schema

dfe8a64

knudtty force-pushed the aaron/new-otel-schema branch from 7468faa to dfe8a64 Compare May 4, 2026 21:22

vercel Bot deployed to Preview May 4, 2026 21:24

View deployment

chore: fix tests

613f82f

vercel Bot deployed to Preview May 4, 2026 22:11

View deployment

wrn14897 reviewed May 5, 2026

View reviewed changes

fix: add unique row settings to compat schema

ef954a7

vercel Bot deployed to Preview May 5, 2026 15:27

View deployment

Merge branch 'main' into aaron/new-otel-schema

adfd9aa

vercel Bot deployed to Preview May 5, 2026 15:31

View deployment

knudtty added the automerge label May 5, 2026

knudtty mentioned this pull request May 5, 2026

chore: update default sources based on new schema changes ClickHouse/ClickStack-helm-charts#213

Merged

wrn14897 approved these changes May 5, 2026

View reviewed changes

kodiakhq Bot merged commit aaba3e9 into main May 5, 2026
19 checks passed

kodiakhq Bot deleted the aaron/new-otel-schema branch May 5, 2026 16:23

brandon-pereira mentioned this pull request May 12, 2026

Logs Search View: Unknown expression or function identifier TimestampTime #2251

Closed

cflee mentioned this pull request May 18, 2026

Migrations for schema changes #2297

Closed

		@@ -1 +1 @@
		SELECT SeverityText FROM otel_logs WHERE ResourceAttributes['suite-id'] = 'normalize-severity' AND ResourceAttributes['test-id'] = 'text-case' ORDER BY TimestampTime FORMAT CSV
		SELECT SeverityText FROM otel_logs WHERE ResourceAttributes['suite-id'] = 'normalize-severity' AND ResourceAttributes['test-id'] = 'text-case' ORDER BY (toStartOfFiveMinutes(Timestamp), Timestamp) FORMAT CSV

Conversation

knudtty commented Apr 15, 2026

Summary

How to test locally

References

Uh oh!

vercel Bot commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

changeset-bot Bot commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

github-actions Bot commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔴 Tier 4 — Critical

Uh oh!

github-actions Bot commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Review

Uh oh!

github-actions Bot commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

E2E Test Results

Uh oh!

knudtty commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wrn14897 Apr 17, 2026

Choose a reason for hiding this comment

Uh oh!

knudtty May 4, 2026

Choose a reason for hiding this comment

Uh oh!

wrn14897 Apr 17, 2026

Choose a reason for hiding this comment

Uh oh!

dhable Apr 22, 2026

Choose a reason for hiding this comment

Uh oh!

wrn14897 Apr 17, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

wrn14897 May 5, 2026

Choose a reason for hiding this comment

Uh oh!

knudtty May 5, 2026

Choose a reason for hiding this comment

Uh oh!

wrn14897 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

vercel Bot commented Apr 15, 2026 •

edited

Loading

changeset-bot Bot commented Apr 15, 2026 •

edited

Loading

github-actions Bot commented Apr 15, 2026 •

edited

Loading

github-actions Bot commented Apr 15, 2026 •

edited

Loading

github-actions Bot commented Apr 15, 2026 •

edited

Loading

knudtty commented Apr 15, 2026 •

edited

Loading