Skip to content

Releases

0.3.0 - 2026-09-07

Choose a tag to compare

@github-actions github-actions released this 07 Sep 15:13

Release Notes

execution, reuse, and capacity milestones (2026-09-06)

  • Prune unneeded nested object descendants across JSONL row, batch, gzip and parallel readers, while validating every consumed JSON value. Index column projection masks once per batch and retain safe whole-value fallbacks for arrays, wildcards and scoped paths.
  • Serialize JSON/NDJSON directly from borrowed batch columns, preserving field order, duplicate-alias replacement, Float32 spelling, lazy errors and output failure propagation. Add trusted Float32 column addition and constant multiplication kernels with per-operation rounding and scalar fallbacks.
  • Support HAVING aggregate calls, including hidden aggregate outputs; reject unsupported global and nested-aggregate forms clearly. Bind nested, array and aliased aggregate inputs to their materialized columns without overwriting grouping keys or other aggregate arguments.
  • Keep SQL constants out of the source-field namespace, apply FROM aliases to stdin consistently with files, and fix duplicate-alias full-sort/Top-N binding and a Top-N panic. Accept following clauses after IN, postfix and CAST expressions.
  • Add a same-engine fresh/reused-plan diagnostic and a bounded fixed-schema external-sort example with temporary-disk quotas and exact-output validation. Retain both as experiments: measurements do not justify a production session API, and production spilling still needs lossless dynamic-value storage and shared query accounting.
  • Add generated cross-path correctness checks and a generated conformance index checked in CI. Runtime numbers remain Int32/Float32.
  • Publish reproducible CLI, lifecycle, kernel and external-sort runners with independent answer checks, immutable manifests and archived source/binary identities. Record the completed plan, measured adoption decisions, and a proposed Int64/Float64 compatibility contract.

correctness, performance, and documentation audit (2026-09-06)

  • Read every member of concatenated gzip files, including rows split across members, and report corruption or truncation in later members. Keep JSON prefix LIMIT demand-driven for simple projections, filters, and DISTINCT.
  • Preserve literal-prefixed field names and scientific notation; allow spaced operators after CAST, CASE, and postfix predicates. Validate aggregate arity and percentile bounds during planning, accept case-insensitive aggregate names, and prevent zero-argument scalar calls or non-boolean arithmetic predicates from panicking.
  • Correct Unicode character lengths, extreme substring/split indexes, integer overflow and invalid bit shifts. Validate arity on public function-registry calls and give array_sort the same total ordering as ORDER BY. MIN/MAX and percentiles ignore NULL/MISSING; mixed numeric extrema, discrete percentile ranks/endpoints, and descending approximate percentiles now return correct results.
  • Propagate final output-write failures for every query output format and schema tables, reject unknown schema formats with a failing exit status, and accept underscores in table names.
  • Bound parallel newline searches for oversized log rows, reduce Levenshtein workspace to the shorter string, stop split_part after the requested segment, and reuse cached regex matching workspace. Add correctness and allocation regressions for these paths.
  • Protect benchmark result directories from accidental reuse so failed reruns cannot mix new samples with an earlier report.
  • Refresh current architecture, CLI examples, development checks, memory and numeric limits, and historical benchmark labels.
  • Make closed-stdout regressions deterministic when macOS tests launch subprocesses concurrently.

next performance milestones (2026-09-05)

  • Borrow nested batch values when resolving paths, retain movable columns in mixed projections, and evaluate supported complex predicates without copying unrelated columns. Preserve aliases, active masks, lazy errors and demand-driven LIMIT; budgeted mixed projections retain their existing materialization policy.
  • Add answer-checked narrow/wide, predicate-selectivity, plain/gzip and 1/8/32/125-file controls, with paired CPU/wall measurements and reproducible input manifests. Track independent follow-on work in the milestone plan.
  • Execute eligible JSONL shards through one bounded worker pool with per-file filtering, projection and partial aggregation, including independent gzip decoders. Preserve ordered merging and cancellation; non-regular inputs retain lazy sequential processing so later FIFOs cannot block earlier errors.
  • Use flate2's Rust zlib-rs backend after isolated decode/parse/CLI comparisons; keep the pinned flate2 version and Rust 1.85 compatibility requirement.
  • Pipeline eligible single-gzip full aggregations through bounded decoded chunks and worker-local filters, expressions and aggregate states. Count the decoder within --threads; preserve one-thread and memory-budget fallbacks, decoded-prefix errors, cancellation and worker teardown.
  • Bind batch calls to their actual function definitions and specialize trusted Float32 Plus chains into typed output, preserving per-step rounding, custom functions, masks, duplicate aliases and lazy errors. Serialize borrowed JSON/NDJSON records without rebuilding object trees, retaining exact public float formatting.
  • Add diagnostic aggregation, expression, reader/task and gzip phase probes. Keep plain aggregate task scheduling unchanged after skew controls; do not enable broader kernels from isolated timings alone.
  • Add opt-in Parquet/persisted ClickHouse reuse experiments with full representation checks, immutable prepared identities, preparation/storage costs and actual 1/10/100-query fresh-process and session controls. General transparent caching remains deferred outside the validated input contract.
  • Resolve the existing lru dependency to 0.18.2 for RUSTSEC-2026-0253 and pass the advisory gates.
  • Record final paired CLI controls, phase measurements, adoption decisions and limitations in the next-milestones report.

performance expansion (2026-09-05)

  • Build homogeneous JSONL primitive columns directly, avoiding per-row owned values and long-string allocations; preserve strict parsing, duplicate keys, mixed types and validity masks. Use a 64 KiB sequential JSON buffer.
  • Execute supported JSONL projections and computed aggregate inputs with bound batch expressions. Preserve active-row masks, lazy CASE branches, duplicate aliases and demand-driven expression LIMIT behavior; fixed-format expressions retain their existing reader semantics.
  • Correct computed SUM/AVG/COUNT inputs that previously read missing projection fields. Keep separate function occurrences independent and report COUNT/approximate cardinality outside the public Int32 range instead of silently narrowing, including the fixed-format COUNT scan fast path.
  • Add deterministic paired workloads with independent answer checking, thread/CPU/RSS controls and reproducible provenance. Add identical-kernel JSON reader and allocation probes; JSON dictionary construction remains experimental because measured total query cost did not show a stable benefit.
  • Track implementation and the acceptance criteria for subsequent aggregation, Top-K, shard scheduling, spilling, columnar reuse and numeric-width work in the expansion plan.
  • Recorded paired 100K/500K-row workloads, isolated scanner/allocation experiments and original JSONL/ELB/ClickHouse controls in the expansion report, including the remaining COUNT gap and CPU/RSS tradeoffs.

performance corrections (2026-09-05)

  • Parse JSONL directly into required root fields and typed batches while validating ignored input; preserve nested values, NULL/MISSING, duplicate keys and aliases. Keep unsupported dynamic projections on the strict row reader.
  • Execute scans with shared mmap ranges, lazy file opening, bounded queues and controlled workers. Small ordered scan tasks avoid queue stalls; worker-local COUNT/SUM/AVG states merge without rounding intermediate results. --threads 0 now resolves to available CPUs.
  • Preserve batch execution under --max-memory, charging queued batches and retained grouping, sorting, DISTINCT and bucket state through shared reservations. Failures cancel workers and return nonzero CLI status.
  • Correct COUNT(column), aggregate aliases and schemas, typed/Mixed grouping equality, NULL/MISSING predicates and aggregates, projection dependencies, stable mixed-value sorting, duplicate aliases and constant projections. Accumulate SUM in f64 and round once to the existing public f32 representation.
  • Skip unchanged memory reservations so a generous budget does not serialize fixed-size parallel aggregate updates.
  • Reuse LIKE regex search caches and dictionary matching, tokenizer offset storage and group key buffers. Compare TopK keys before constructing rejected payloads, and move full-sort output through its permutation.
  • Compile time-bucket intervals once and aggregate computed batch buckets without assuming raw logs are sorted. Preserve timezone offsets, fractional timestamps and arbitrary input ordering.
  • Make benchmark queries fail visibly, validate answers against an independent oracle, fix multi-key ORDER BY parsing and LIMIT timing, and record thread settings, data/query/binary hashes and build provenance.
  • Recorded answer-checked 100 MiB/1 GiB comparisons, 1/2/4/8-thread scaling and memory-ceiling controls in the performance report.

Install logq 0.3.0

Install prebuilt binaries via shell script

curl --proto '=https' --tlsv1.2 -LsSf https://github.com/MnO2/logq/releases/download/v0.3.0/logq-installer.sh | sh

Install...

Read more

0.2.0 - 2026-07-11

Choose a tag to compare

@github-actions github-actions released this 12 Jul 16:24

Release Notes

  • Query a substantially expanded PartiQL subset, including INNER/RIGHT joins, subqueries, set operations, three-valued NULL/MISSING logic, approximate aggregates, array/string/date functions, and completed time_bucket intervals.
  • Read gzip files transparently and scan deterministic globs or comma-separated shards without concatenating them first.
  • Query schema-free JSONL, custom named-capture regex formats, and built-in Apache/nginx clf and combined logs.
  • Stream newline-delimited JSON with --output ndjson; inspect batch/row selection with pipeline-aware explain; cap materializing queries with --max-memory.
  • Diagnose syntax, planning, schema, and runtime failures with source locations, carets, contextual hints, and did-you-mean suggestions.
  • Bound ORDER BY ... LIMIT memory to O(k), retain fixed-schema batch execution where supported, and publish reproducible end-to-end performance and memory measurements.
  • Require Rust 1.85 or newer and ship current, advisory-clean dependencies with multi-platform CI.

Install logq 0.2.0

Install prebuilt binaries via shell script

curl --proto '=https' --tlsv1.2 -LsSf https://github.com/MnO2/logq/releases/download/v0.2.0/logq-installer.sh | sh

Install prebuilt binaries via powershell script

powershell -ExecutionPolicy Bypass -c "irm https://github.com/MnO2/logq/releases/download/v0.2.0/logq-installer.ps1 | iex"

Download logq 0.2.0

File Platform Checksum
logq-aarch64-apple-darwin.tar.xz Apple Silicon macOS checksum
logq-x86_64-apple-darwin.tar.xz Intel macOS checksum
logq-x86_64-pc-windows-msvc.zip x64 Windows checksum
logq-x86_64-unknown-linux-musl.tar.xz x64 MUSL Linux checksum