Skip to content

Promote non-unified prefill rotation onto the live engine line - #80

Merged
joelteply merged 4 commits into
889580fc/seq-state-across-stream-countsfrom
fable/prefill-fairness-test
Oct 28, 2026
Merged

joelteply merged 4 commits into
889580fc/seq-state-across-stream-countsfrom
fable/prefill-fairness-test

Conversation

@joelteply

Copy link
Copy Markdown

Promote #69's three commits onto the current engine line 0d48ef8. This lets persistent minds with short warm prefills receive service while a cold prompt is filling, without discarding their KV or cancelling their work.

Head 7be73db rotates compatible admitted text-prefill service only in non-unified KV mode. Unified KV keeps physical traversal. Includes the production-helper fixture and Ubuntu/Windows server test wiring.

Fable measured this exact cherry-pick on the M5: a 1.5k warm prefill behind an 18k cold prefill fell from 23.4s to 8.2s; the cold prefill stayed 25.4s. This is owner-provided dyno evidence; live pin adoption and its 15-minute read remain required.

Original #69 final head31d07d0d passed server CI39919145623 on Ubuntu and Windows. Exact promoted-head CI and Cormac's carry-forward source review are required before merge. No unified-KV fairness guarantee or live deployment claimed.

Preserve live slot and KV custody while bounding compatible text-prefill opportunities. This does not preempt pinned outer admission or promise wall-clock service across incompatible adapters.
@joelteply

Copy link
Copy Markdown
Author

PASS (source) carried from #69 to 7be73db (Cormac), conditional on server CI at this head.

Verified:

  1. Same change: the changed lines of git diff 0d48ef846 7be73db37 (merge-base with 889580fc/seq-state-across-stream-counts) and of git diff 4a79fed98 31d07d0dc (server: rotate admitted prefill service without moving KV #69 against fable/decode-time-share) are identical, three commits each: rotate after actual service, the CI fixture, and retain shared KV order.
  2. The claim re-checked on this base, since the code around it differs. In server-context.cpp at this head the prompt-slot lambda runs from 3651 (batch_before) to 4227 (served()). The only prompt-token add is batch.add at 4121, inside the token loop at 4103, and no return lies between 4103 and 4227. So on the live line too, the cursor advances exactly on real service. Traversal is at 3647, with unified KV kept in physical order.

My #69 notes carry too: image chunks bypass batch.size() and never advance the cursor. Fable's M5 measurement (1.5k warm behind 18k cold: 23.4 s -> 8.2 s, cold unchanged) is the call-site receipt, and the live read follows the pin bump.

@joelteply

Copy link
Copy Markdown
Author

PASS (source) carries to 0f91a5b (Cormac), conditional on CI at this head. Verified git diff 7be73db37 0f91a5bc: one file, tests/test-dup-quantized.cpp (+11/-2). The CPU backend now loads through the registry (ggml_backend_load_all + init_by_type(CPU), failing if absent) and computes through ggml_backend_graph_compute instead of the directly linked ggml_graph_compute_with_ctx, which is right for dynamically loaded backends. The assertions are unchanged, and cleanup now also runs on the compute-failure path. The prefill change I reviewed is untouched. One thing only CI answers: the graph's tensors live in a plain ggml_init context (no backend buffer). The CPU backend computes host tensors without one, so this should hold, but the Windows run is the receipt.

@joelteply
joelteply merged commit 08d6d35 into 889580fc/seq-state-across-stream-counts Oct 28, 2026
10 of 25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant