Repository navigation
Conversation
…updates Add a reusable apply_table_updates(&mut TableMetadata, &[TableUpdate], now_ms) that applies Iceberg REST metadata updates (properties, location, uuid, format-version, schema, current-schema, default-spec, sort-order, removals, statistics) to produce new table metadata server-side. Updates needing deeper machinery (add-spec binding, add-snapshot sequencing, snapshot-ref state) return NotImplemented rather than producing incorrect metadata. Lets a catalog (LakeCat) materialize new metadata on commit instead of relying on the client to author it.
IntBytesMapEntry.value is a Vec<u8> typed as Avro `bytes` in the manifest schema, but serde's default Vec<u8> serialization emits an Avro array. That array fails to resolve against the `bytes` schema, so the nullable lower_bounds /upper_bounds maps were silently written as null and every column bound was lost on read. Force byte-string (de)serialization on the entry value (tolerating the legacy array form on read for backward compatibility) so bounds survive the write/parse round-trip. sail-iceberg tests: 79 passed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UbNyd333y17hMC55Pk4CRk
Once manifest lower/upper bounds round-trip correctly, IcebergPruningStats feeds them to DataFusion's PruningPredicate. An Iceberg bound can decode to a primitive whose Arrow scalar type differs from how the predicate treats the column (the to_scalar fallback yields e.g. an Int32 for a value the predicate compares as Timestamp/Date), which trips DataFusion's "same data type are comparable" assertion on partition-transform reads. Wire up the previously-unused arrow_schema: min_values/max_values now skip stats whose array type does not match the logical column type, so pruning is only applied with well-typed bounds (no false pruning, no wrong results) and falls back to a full scan otherwise. sail-iceberg: 79 tests pass; clippy + nightly fmt clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UbNyd333y17hMC55Pk4CRk
…r seam
Surface the symbols a thin REST-catalog bridge (LakeCat) needs without forking
the generated models:
- sail-catalog-iceberg: make `models` public; add planning result helpers
(planning.rs: completed_planning[_with_id]_result_from_values,
fetch_scan_tasks_result_from_values); re-export LoadTableResult/TableMetadata;
expose load_table_result_to_status as a standalone `pub fn` (the existing
method now delegates to it).
- sail-catalog: add CommitTableOptions/GetTableCommitsOptions/TableCommitInfo/
GetTableCommitsResponse and default-impl CatalogProvider::{commit_table,
get_table_commits} (returning NotSupported), so existing providers are
unaffected.
sail-catalog + sail-catalog-iceberg build clean; existing providers compile.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UbNyd333y17hMC55Pk4CRk
Add LAKECAT-SAIL.md recording the bug fixes and enabling changes made to Sail for LakeCat: manifest-bounds round-trip (#2139), pruning type-safety (#2139), apply_table_updates server-side metadata (#2135, held), planning/commit-table provider seam (#2140), atop the merged TableUpdate/ ViewUpdate discriminated enums (#2134). Documents why the work is held on the lakecat branch (parallel provider redesign #2141, Iceberg decisions in #982) and how LakeCat consumes it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RJGCWiVM3DW1Keu4Pwuqjs
Establish this as a LakeCat fork of Sail: add lakecat/AGENTS.md with agent working discipline (PR intention/coordination, no-scope Conventional-Commits titles, design-before-code on shared trait surfaces, BDD .feature tests for gaps), pointing back to .github/copilot-instructions.md as authoritative. Add a README fork notice linking to it, and un-ignore lakecat/AGENTS.md (AGENTS.md is otherwise gitignored repo-wide). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RJGCWiVM3DW1Keu4Pwuqjs
Add an opt-in per-worker object-store read cache to sail-object-store, modeled on lancedb/ocra (Apache-2.0) with Moka swapped for Foyer, mirroring the existing RuntimeAwareObjectStore layer. Caches fixed-size pages keyed by (location, page_id) plus ObjectMeta, fetching misses through the inner store and coalescing concurrent misses via Foyer's get_or_fetch. object_store 0.13.2 note: get/get_range/head live in the non-overridable ObjectStoreExt blanket trait, so caching is intercepted at get_opts + get_ranges; invalidation in put/copy/rename/delete. Conditional/versioned reads bypass the cache. Opt-in and default-unchanged: enable with SAIL_OBJECT_STORE_CACHE (+ _PAGE_SIZE, _MEMORY, _METADATA). Defaults: 1 MiB pages, 1 GiB memory, 64 MiB metadata. In-memory tier only; Foyer HybridCache disk tiering is a follow-up. ocra is attributed in a header block and per-site `// Adapted from ocra ...` comments. foyer 0.22.3. 8 sail-object-store tests pass; clippy/fmt clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UbNyd333y17hMC55Pk4CRk
…_updates A catalog receiving an Iceberg `updateTable` commit from a stock client (pyiceberg/Spark append) applies an `add-snapshot` followed by a `set-snapshot-ref` for `main`; both previously returned NotImplemented, so a real snapshot append was rejected with HTTP 400. Implement them per the Iceberg spec in `apply_table_updates`: - add-snapshot: append the snapshot to the snapshot set, advance last-sequence-number, and advance the v3 next-row-id when the snapshot carries added-rows. It does not move the current ref (that is set-snapshot-ref's job). - set-snapshot-ref: insert/move the branch or tag ref; moving `main` advances current-snapshot-id and records a snapshot-log entry (timestamped with the commit time threaded through apply_one). Rejects a ref to an unknown snapshot-id. `add-spec` remains the only unsupported update (partition-spec binding). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UbNyd333y17hMC55Pk4CRk
Unit tests over a minimal v2 metadata fixture: - a stock append (add-snapshot + set-snapshot-ref main) adds the snapshot, moves current-snapshot-id + the main ref, logs a snapshot-log entry at the commit timestamp, and bumps last-sequence-number / last-updated-ms; - add-snapshot alone adds the snapshot but does not move the current ref or log; - set-snapshot-ref to an unknown snapshot-id is rejected; - a second append accumulates both snapshots and advances current + sequence. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UbNyd333y17hMC55Pk4CRk
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
LakeCat ↔ Sail: catalog & Iceberg changes (tracking)
This PR tracks the Sail-side changes that LakeCat (an Iceberg REST catalog
sitting close to the engine) depends on, integrated on this fork's
lakecatbranch, with
LAKECAT-SAIL.mdas the running change log.This is the development base for LakeCat catalog work —
querygraph/saillakecat→main. It is held here (rather than onlakehq/sail) to avoidchurning against the parallel upstream provider redesign
(
lakehq/sail#2141) and the Iceberg catalog-support decisions(
lakehq/sail#982). Correctness fixes are offered upstream separately; theprovider-surface changes stay on-branch pending design coordination.
What's in the branch (atop the merged
lakehq/sail#2134)lakehq/sail#2139lakehq/sail#2139apply_table_updates— evolveTableMetadataserver-sidelakehq/sail#2135lakehq/sail#2140docs(lakecat): addLAKECAT-SAIL.md,lakecat/AGENTS.md, fork noticeCorrectness fixes
lower_bounds/upper_boundswere silently written asnullbecauseserde encoded the
bytes-typed value as an Avro array — bounds were lost onread, defeating pruning. Forced byte-string (de)serialization, tolerant of the
legacy array form.
logical column type, avoiding DataFusion's comparability assertion on
partition-transform reads (no false pruning, no wrong results).
Enablement (coordinated with upstream)
apply_table_updates(&mut TableMetadata, &[TableUpdate], now_ms)materializesnew metadata server-side (properties, schema, spec, sort-order, statistics, …),
returning
NotImplementedfor updates needing deeper machinery. Held pendingIceberg catalog decisions (
lakehq/sail#982); originallylakehq/sail#2135.models, planning-result helpers,and default
CatalogProvider::{commit_table, get_table_commits}→NotSupported.commit_tableoverlaps existingcommit_lakehouse_tableThe seam adds
CatalogProvider::commit_table, butcommit_lakehouse_table(Xiaolong's) already covers that path and is implemented by the Iceberg
provider. The REST commit flow should reconcile with / route through the
existing method rather than add a parallel one;
get_table_commits(historylisting) may be genuinely new. Upstream has since pivoted
(
lakehq/sail#2138), so this seam needs reconciling against the new providershape regardless. No merge of the seam until a light design discussion settles
the direction.
Note: the bug fixes are independently mergeable
The two correctness fixes are independent of the provider redesign
(
lakehq/sail#2141) and the Iceberg decisions (lakehq/sail#982) — they shouldnot be blocked behind the enablement work, and can be re-spun as a focused
upstream fixes-only PR.
Superseded PRs (for additional context)
Consolidated here; their original descriptions, diffs, and review threads carry
extra detail:
lakehq/sail#2139— manifest-bounds round-trip + pruning type-safety (closed).lakehq/sail#2140— planning helpers + commit-table provider seam (closed).lakehq/sail#2135—apply_table_updatesserver-side metadata (closed, held).lakehq/sail#2134—TableUpdate/ViewUpdatediscriminated enums (merged base).See
LAKECAT-SAIL.mdfor the full write-up andlakecat/AGENTS.mdforagent working discipline on this fork.
🤖 Generated with Claude Code
https://claude.ai/code/session_01RJGCWiVM3DW1Keu4Pwuqjs