Repository navigation
Releases
Releases · MnO2/logq
0.3.0 - 2026-09-07
Release Notes
execution, reuse, and capacity milestones (2026-09-06)
- Prune unneeded nested object descendants across JSONL row, batch, gzip and parallel readers, while validating every consumed JSON value. Index column projection masks once per batch and retain safe whole-value fallbacks for arrays, wildcards and scoped paths.
- Serialize JSON/NDJSON directly from borrowed batch columns, preserving field order, duplicate-alias replacement, Float32 spelling, lazy errors and output failure propagation. Add trusted Float32 column addition and constant multiplication kernels with per-operation rounding and scalar fallbacks.
- Support HAVING aggregate calls, including hidden aggregate outputs; reject unsupported global and nested-aggregate forms clearly. Bind nested, array and aliased aggregate inputs to their materialized columns without overwriting grouping keys or other aggregate arguments.
- Keep SQL constants out of the source-field namespace, apply FROM aliases to stdin consistently with files, and fix duplicate-alias full-sort/Top-N binding and a Top-N panic. Accept following clauses after IN, postfix and CAST expressions.
- Add a same-engine fresh/reused-plan diagnostic and a bounded fixed-schema external-sort example with temporary-disk quotas and exact-output validation. Retain both as experiments: measurements do not justify a production session API, and production spilling still needs lossless dynamic-value storage and shared query accounting.
- Add generated cross-path correctness checks and a generated conformance index checked in CI. Runtime numbers remain Int32/Float32.
- Publish reproducible CLI, lifecycle, kernel and external-sort runners with independent answer checks, immutable manifests and archived source/binary identities. Record the completed plan, measured adoption decisions, and a proposed Int64/Float64 compatibility contract.
correctness, performance, and documentation audit (2026-09-06)
- Read every member of concatenated gzip files, including rows split across members, and report corruption or truncation in later members. Keep JSON prefix LIMIT demand-driven for simple projections, filters, and DISTINCT.
- Preserve literal-prefixed field names and scientific notation; allow spaced operators after CAST, CASE, and postfix predicates. Validate aggregate arity and percentile bounds during planning, accept case-insensitive aggregate names, and prevent zero-argument scalar calls or non-boolean arithmetic predicates from panicking.
- Correct Unicode character lengths, extreme substring/split indexes, integer overflow and invalid bit shifts. Validate arity on public function-registry calls and give array_sort the same total ordering as ORDER BY. MIN/MAX and percentiles ignore NULL/MISSING; mixed numeric extrema, discrete percentile ranks/endpoints, and descending approximate percentiles now return correct results.
- Propagate final output-write failures for every query output format and schema tables, reject unknown schema formats with a failing exit status, and accept underscores in table names.
- Bound parallel newline searches for oversized log rows, reduce Levenshtein workspace to the shorter string, stop split_part after the requested segment, and reuse cached regex matching workspace. Add correctness and allocation regressions for these paths.
- Protect benchmark result directories from accidental reuse so failed reruns cannot mix new samples with an earlier report.
- Refresh current architecture, CLI examples, development checks, memory and numeric limits, and historical benchmark labels.
- Make closed-stdout regressions deterministic when macOS tests launch subprocesses concurrently.
next performance milestones (2026-09-05)
- Borrow nested batch values when resolving paths, retain movable columns in mixed projections, and evaluate supported complex predicates without copying unrelated columns. Preserve aliases, active masks, lazy errors and demand-driven LIMIT; budgeted mixed projections retain their existing materialization policy.
- Add answer-checked narrow/wide, predicate-selectivity, plain/gzip and 1/8/32/125-file controls, with paired CPU/wall measurements and reproducible input manifests. Track independent follow-on work in the milestone plan.
- Execute eligible JSONL shards through one bounded worker pool with per-file filtering, projection and partial aggregation, including independent gzip decoders. Preserve ordered merging and cancellation; non-regular inputs retain lazy sequential processing so later FIFOs cannot block earlier errors.
- Use flate2's Rust zlib-rs backend after isolated decode/parse/CLI comparisons; keep the pinned flate2 version and Rust 1.85 compatibility requirement.
- Pipeline eligible single-gzip full aggregations through bounded decoded chunks and worker-local filters, expressions and aggregate states. Count the decoder within
--threads; preserve one-thread and memory-budget fallbacks, decoded-prefix errors, cancellation and worker teardown. - Bind batch calls to their actual function definitions and specialize trusted Float32 Plus chains into typed output, preserving per-step rounding, custom functions, masks, duplicate aliases and lazy errors. Serialize borrowed JSON/NDJSON records without rebuilding object trees, retaining exact public float formatting.
- Add diagnostic aggregation, expression, reader/task and gzip phase probes. Keep plain aggregate task scheduling unchanged after skew controls; do not enable broader kernels from isolated timings alone.
- Add opt-in Parquet/persisted ClickHouse reuse experiments with full representation checks, immutable prepared identities, preparation/storage costs and actual 1/10/100-query fresh-process and session controls. General transparent caching remains deferred outside the validated input contract.
- Resolve the existing lru dependency to 0.18.2 for RUSTSEC-2026-0253 and pass the advisory gates.
- Record final paired CLI controls, phase measurements, adoption decisions and limitations in the next-milestones report.
performance expansion (2026-09-05)
- Build homogeneous JSONL primitive columns directly, avoiding per-row owned values and long-string allocations; preserve strict parsing, duplicate keys, mixed types and validity masks. Use a 64 KiB sequential JSON buffer.
- Execute supported JSONL projections and computed aggregate inputs with bound batch expressions. Preserve active-row masks, lazy CASE branches, duplicate aliases and demand-driven expression LIMIT behavior; fixed-format expressions retain their existing reader semantics.
- Correct computed SUM/AVG/COUNT inputs that previously read missing projection fields. Keep separate function occurrences independent and report COUNT/approximate cardinality outside the public Int32 range instead of silently narrowing, including the fixed-format COUNT scan fast path.
- Add deterministic paired workloads with independent answer checking, thread/CPU/RSS controls and reproducible provenance. Add identical-kernel JSON reader and allocation probes; JSON dictionary construction remains experimental because measured total query cost did not show a stable benefit.
- Track implementation and the acceptance criteria for subsequent aggregation, Top-K, shard scheduling, spilling, columnar reuse and numeric-width work in the expansion plan.
- Recorded paired 100K/500K-row workloads, isolated scanner/allocation experiments and original JSONL/ELB/ClickHouse controls in the expansion report, including the remaining COUNT gap and CPU/RSS tradeoffs.
performance corrections (2026-09-05)
- Parse JSONL directly into required root fields and typed batches while validating ignored input; preserve nested values, NULL/MISSING, duplicate keys and aliases. Keep unsupported dynamic projections on the strict row reader.
- Execute scans with shared mmap ranges, lazy file opening, bounded queues and controlled workers. Small ordered scan tasks avoid queue stalls; worker-local COUNT/SUM/AVG states merge without rounding intermediate results.
--threads 0now resolves to available CPUs. - Preserve batch execution under
--max-memory, charging queued batches and retained grouping, sorting, DISTINCT and bucket state through shared reservations. Failures cancel workers and return nonzero CLI status. - Correct COUNT(column), aggregate aliases and schemas, typed/Mixed grouping equality, NULL/MISSING predicates and aggregates, projection dependencies, stable mixed-value sorting, duplicate aliases and constant projections. Accumulate SUM in f64 and round once to the existing public f32 representation.
- Skip unchanged memory reservations so a generous budget does not serialize fixed-size parallel aggregate updates.
- Reuse LIKE regex search caches and dictionary matching, tokenizer offset storage and group key buffers. Compare TopK keys before constructing rejected payloads, and move full-sort output through its permutation.
- Compile time-bucket intervals once and aggregate computed batch buckets without assuming raw logs are sorted. Preserve timezone offsets, fractional timestamps and arbitrary input ordering.
- Make benchmark queries fail visibly, validate answers against an independent oracle, fix multi-key ORDER BY parsing and LIMIT timing, and record thread settings, data/query/binary hashes and build provenance.
- Recorded answer-checked 100 MiB/1 GiB comparisons, 1/2/4/8-thread scaling and memory-ceiling controls in the performance report.
Install logq 0.3.0
Install prebuilt binaries via shell script
curl --proto '=https' --tlsv1.2 -LsSf https://github.com/MnO2/logq/releases/download/v0.3.0/logq-installer.sh | shInstall...
0.2.0 - 2026-07-11
Release Notes
- Query a substantially expanded PartiQL subset, including INNER/RIGHT joins, subqueries, set operations, three-valued NULL/MISSING logic, approximate aggregates, array/string/date functions, and completed
time_bucketintervals. - Read gzip files transparently and scan deterministic globs or comma-separated shards without concatenating them first.
- Query schema-free JSONL, custom named-capture regex formats, and built-in Apache/nginx
clfandcombinedlogs. - Stream newline-delimited JSON with
--output ndjson; inspect batch/row selection with pipeline-awareexplain; cap materializing queries with--max-memory. - Diagnose syntax, planning, schema, and runtime failures with source locations, carets, contextual hints, and did-you-mean suggestions.
- Bound
ORDER BY ... LIMITmemory to O(k), retain fixed-schema batch execution where supported, and publish reproducible end-to-end performance and memory measurements. - Require Rust 1.85 or newer and ship current, advisory-clean dependencies with multi-platform CI.
Install logq 0.2.0
Install prebuilt binaries via shell script
curl --proto '=https' --tlsv1.2 -LsSf https://github.com/MnO2/logq/releases/download/v0.2.0/logq-installer.sh | shInstall prebuilt binaries via powershell script
powershell -ExecutionPolicy Bypass -c "irm https://github.com/MnO2/logq/releases/download/v0.2.0/logq-installer.ps1 | iex"Download logq 0.2.0
| File | Platform | Checksum |
|---|---|---|
| logq-aarch64-apple-darwin.tar.xz | Apple Silicon macOS | checksum |
| logq-x86_64-apple-darwin.tar.xz | Intel macOS | checksum |
| logq-x86_64-pc-windows-msvc.zip | x64 Windows | checksum |
| logq-x86_64-unknown-linux-musl.tar.xz | x64 MUSL Linux | checksum |