Repository navigation
feat: fast and full autocomplete #2128
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| --- | ||
| "@hyperdx/common-utils": minor | ||
| "@hyperdx/api": minor | ||
| "@hyperdx/app": minor | ||
| --- | ||
|
|
||
| feat: introducing rollup and source support for full autocomplete |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,171 @@ | ||
| -- +goose Up | ||
|
Contributor
Author
There was a problem hiding this comment. The general idea:
The The same idea is applied to both log and trace tables |
||
|
|
||
| -- Map Attributes Key Value Rollup (all map columns in one table) | ||
| CREATE TABLE IF NOT EXISTS ${DATABASE}.otel_logs_kv_rollup_15m | ||
| ( | ||
| `Timestamp` DateTime, | ||
| `ColumnIdentifier` LowCardinality(String), | ||
| `Key` LowCardinality(String), | ||
| `Value` String, | ||
| `count` UInt64, | ||
| INDEX idx_count_minmax count TYPE minmax GRANULARITY 1, | ||
| INDEX idx_timestamp_minmax Timestamp TYPE minmax GRANULARITY 1 | ||
| ) | ||
| ENGINE = SummingMergeTree | ||
| PARTITION BY toDate(Timestamp) | ||
| ORDER BY (ColumnIdentifier, Key, Timestamp, Value) | ||
| TTL Timestamp + ${TABLES_TTL} | ||
| SETTINGS index_granularity = 8192, ttl_only_drop_parts = 1; | ||
|
|
||
| -- Key Only Rollup (derived from KV rollup) | ||
| CREATE TABLE IF NOT EXISTS ${DATABASE}.otel_logs_key_rollup_15m | ||
| ( | ||
| `Timestamp` DateTime, | ||
| `ColumnIdentifier` LowCardinality(String), | ||
| `Key` LowCardinality(String), | ||
| `count` UInt64, | ||
| INDEX idx_count_minmax count TYPE minmax GRANULARITY 1, | ||
| INDEX idx_timestamp_minmax Timestamp TYPE minmax GRANULARITY 1 | ||
| ) | ||
| ENGINE = SummingMergeTree | ||
| PARTITION BY toDate(Timestamp) | ||
| ORDER BY (ColumnIdentifier, Key, Timestamp) | ||
| TTL Timestamp + ${TABLES_TTL} | ||
| SETTINGS index_granularity = 8192, ttl_only_drop_parts = 1; | ||
|
|
||
| CREATE MATERIALIZED VIEW IF NOT EXISTS ${DATABASE}.otel_logs_key_rollup_15m_mv TO ${DATABASE}.otel_logs_key_rollup_15m | ||
| AS SELECT | ||
| Timestamp, | ||
| ColumnIdentifier, | ||
| Key, | ||
| sum(count) as count | ||
| FROM ${DATABASE}.otel_logs_kv_rollup_15m | ||
| GROUP BY ColumnIdentifier, Key, Timestamp; | ||
|
|
||
| -- Single MV: CTE with UNION ALL across all columns, then aggregate | ||
| CREATE MATERIALIZED VIEW IF NOT EXISTS ${DATABASE}.otel_logs_attr_kv_rollup_15m_mv TO ${DATABASE}.otel_logs_kv_rollup_15m | ||
| AS WITH elements AS ( | ||
| SELECT | ||
| 'ResourceAttributes' AS ColumnIdentifier, | ||
| toStartOfFifteenMinutes(Timestamp) AS Timestamp, | ||
| replaceRegexpAll(entry.1, '\\[\\d+\\]', '[*]') AS Key, | ||
| CAST(entry.2 AS String) AS Value | ||
| FROM ${DATABASE}.otel_logs | ||
| ARRAY JOIN ResourceAttributes AS entry | ||
| UNION ALL | ||
| SELECT | ||
| 'LogAttributes' AS ColumnIdentifier, | ||
| toStartOfFifteenMinutes(Timestamp) AS Timestamp, | ||
| replaceRegexpAll(entry.1, '\\[\\d+\\]', '[*]') AS Key, | ||
| CAST(entry.2 AS String) AS Value | ||
| FROM ${DATABASE}.otel_logs | ||
| ARRAY JOIN LogAttributes AS entry | ||
| UNION ALL | ||
| SELECT | ||
| 'ScopeAttributes' AS ColumnIdentifier, | ||
| toStartOfFifteenMinutes(Timestamp) AS Timestamp, | ||
| replaceRegexpAll(entry.1, '\\[\\d+\\]', '[*]') AS Key, | ||
| CAST(entry.2 AS String) AS Value | ||
| FROM ${DATABASE}.otel_logs | ||
| ARRAY JOIN ScopeAttributes AS entry | ||
| UNION ALL | ||
| SELECT | ||
| 'NativeColumn' AS ColumnIdentifier, | ||
| toStartOfFifteenMinutes(Timestamp) AS Timestamp, | ||
| 'SeverityText' as Key, | ||
| CAST(SeverityText AS String) as Value | ||
| FROM ${DATABASE}.otel_logs | ||
| UNION ALL | ||
| SELECT | ||
| 'NativeColumn' AS ColumnIdentifier, | ||
| toStartOfFifteenMinutes(Timestamp) AS Timestamp, | ||
| 'ServiceName' as Key, | ||
| CAST(ServiceName AS String) as Value | ||
| FROM ${DATABASE}.otel_logs | ||
| UNION ALL | ||
| SELECT | ||
| 'NativeColumn' AS ColumnIdentifier, | ||
| toStartOfFifteenMinutes(Timestamp) AS Timestamp, | ||
| 'ScopeName' as Key, | ||
| CAST(ScopeName AS String) as Value | ||
| FROM ${DATABASE}.otel_logs | ||
| UNION ALL | ||
| SELECT | ||
| 'NativeColumn' AS ColumnIdentifier, | ||
| toStartOfFifteenMinutes(Timestamp) AS Timestamp, | ||
| 'ScopeVersion' as Key, | ||
| CAST(ScopeVersion AS String) as Value | ||
| FROM ${DATABASE}.otel_logs | ||
| UNION ALL | ||
| SELECT | ||
| 'NativeColumn' AS ColumnIdentifier, | ||
| toStartOfFifteenMinutes(Timestamp) AS Timestamp, | ||
| 'ResourceSchemaUrl' as Key, | ||
| CAST(ResourceSchemaUrl AS String) as Value | ||
| FROM ${DATABASE}.otel_logs | ||
| UNION ALL | ||
| SELECT | ||
| 'NativeColumn' AS ColumnIdentifier, | ||
| toStartOfFifteenMinutes(Timestamp) AS Timestamp, | ||
| 'ScopeSchemaUrl' as Key, | ||
| CAST(ScopeSchemaUrl AS String) as Value | ||
| FROM ${DATABASE}.otel_logs | ||
| UNION ALL | ||
| SELECT | ||
| 'NativeColumn' AS ColumnIdentifier, | ||
| toStartOfFifteenMinutes(Timestamp) AS Timestamp, | ||
| '__hdx_materialized_k8s.cluster.name' as Key, | ||
| CAST(`__hdx_materialized_k8s.cluster.name` AS String) as Value | ||
| FROM ${DATABASE}.otel_logs | ||
| UNION ALL | ||
| SELECT | ||
| 'NativeColumn' AS ColumnIdentifier, | ||
| toStartOfFifteenMinutes(Timestamp) AS Timestamp, | ||
| '__hdx_materialized_k8s.container.name' as Key, | ||
| CAST(`__hdx_materialized_k8s.container.name` AS String) as Value | ||
| FROM ${DATABASE}.otel_logs | ||
| UNION ALL | ||
| SELECT | ||
| 'NativeColumn' AS ColumnIdentifier, | ||
| toStartOfFifteenMinutes(Timestamp) AS Timestamp, | ||
| '__hdx_materialized_k8s.deployment.name' as Key, | ||
| CAST(`__hdx_materialized_k8s.deployment.name` AS String) as Value | ||
| FROM ${DATABASE}.otel_logs | ||
| UNION ALL | ||
| SELECT | ||
| 'NativeColumn' AS ColumnIdentifier, | ||
| toStartOfFifteenMinutes(Timestamp) AS Timestamp, | ||
| '__hdx_materialized_k8s.namespace.name' as Key, | ||
| CAST(`__hdx_materialized_k8s.namespace.name` AS String) as Value | ||
| FROM ${DATABASE}.otel_logs | ||
| UNION ALL | ||
| SELECT | ||
| 'NativeColumn' AS ColumnIdentifier, | ||
| toStartOfFifteenMinutes(Timestamp) AS Timestamp, | ||
| '__hdx_materialized_k8s.node.name' as Key, | ||
| CAST(`__hdx_materialized_k8s.node.name` AS String) as Value | ||
| FROM ${DATABASE}.otel_logs | ||
| UNION ALL | ||
| SELECT | ||
| 'NativeColumn' AS ColumnIdentifier, | ||
| toStartOfFifteenMinutes(Timestamp) AS Timestamp, | ||
| '__hdx_materialized_k8s.pod.name' as Key, | ||
| CAST(`__hdx_materialized_k8s.pod.name` AS String) as Value | ||
| FROM ${DATABASE}.otel_logs | ||
| UNION ALL | ||
| SELECT | ||
| 'NativeColumn' AS ColumnIdentifier, | ||
| toStartOfFifteenMinutes(Timestamp) AS Timestamp, | ||
| '__hdx_materialized_k8s.pod.uid' as Key, | ||
| CAST(`__hdx_materialized_k8s.pod.uid` AS String) as Value | ||
| FROM ${DATABASE}.otel_logs | ||
| UNION ALL | ||
| SELECT | ||
| 'NativeColumn' AS ColumnIdentifier, | ||
| toStartOfFifteenMinutes(Timestamp) AS Timestamp, | ||
| '__hdx_materialized_deployment.environment.name' as Key, | ||
| CAST(`__hdx_materialized_deployment.environment.name` AS String) as Value | ||
| FROM ${DATABASE}.otel_logs | ||
| ) | ||
| SELECT Timestamp, ColumnIdentifier, Key, Value, count() AS count FROM elements | ||
| GROUP BY Timestamp, ColumnIdentifier, Key, Value; | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The change is to add the
metadataMaterializedViewsource by default