[Product] Omni desktop extreme-opt engine (FreeToken-class) — acceptance track #1

Context

hs approved product direction (via ProductResearchBot, 2026-10-26): for models already supported by vllm-omni, ship an InferMatrixLab-owned desktop / single-machine extreme-opt engine in the spirit of FlashML-org/FreeToken — expert/weight offload + Omni multimodal state. Does not claim to replace general vLLM / general LLM serving.

Modality lock (Product Team, hs 2026-10-26): local desktop support for ASR / TTS / image gen / video gen (vllm-omni-supported recipes in those lanes). Not general text-LLM serving.

Tracking home for now: this recipe issue. Dedicated InferMatrixLab repo TBD if hs wants a empty repo created.

Claim (locked)

In

• Local desktop (RTX and/or Apple) extreme-opt for ASR, TTS, image gen, video gen recipes already supported by vllm-omni.
• Punch through throughput and/or VRAM / peak memory vs baseline omni serve on the same recipe + hardware class; reproducible numbers.
• Expert / weight offload + Omni multimodal state as first-class knobs for that local path.

Out

• General LLM / chat serving yellow pages.
• Full-model coverage across every omni recipe day one.
• Empty token-save % claims without pinned recipe + hardware + baseline.
• Claiming to supersede general vLLM or upstream vllm-omni itself.

Competitor frame (positioning, not copy)

• Peer lane: FreeToken / Strata / Splash / PowerInfer (MoE · local extreme).
• Our wedge: Omni A/V gen family (ASR/TTS/image/video) + vllm-omni as SoT — not a new general forge.

Acceptance bars (M0)

1. Scope pin: README / issue states In/Out + modality lock above; no general-serving claim language.
2. Recipe pin: Target = one (then N) already-supported vllm-omni recipe(s) in ASR | TTS | image | video; list exact recipe id / model id.
3. Baseline: Same recipe on baseline omni serve; hardware class named (e.g. single RTX 4090 / Apple silicon).
4. Metric bar: At least one of: higher sustained throughput, lower peak VRAM/memory, or runnable where baseline OOMs — with numbers + repro script/commands. (Modality-appropriate units: e.g. RTF for ASR/TTS, images/s or latency for image/video — not fake tok/s for non-text.)
5. Offload honesty: Expert/weight offload path documented; multimodal state behavior called out (what stays on-device vs offloaded).
6. Fail = INC: Missing pin (recipe / hardware / baseline SHA or version) → result is INC, not PASS / marketing number.

Non-goals (phase 1)

• Token-save % as hero metric without SoT pin.
• Porting every vllm-omni model day one.
• Competing on “deep RE / general harness” narratives (wrong lane).

Execution notes

• Code via OpenCode; review via Codex. Bots do not write product code.
• Upstream changes to vllm-project/vllm-omni stay behind clear SoT citations; this product lives under InferMatrixLab.
• Research owns claim wording; domain (vllm-omni-bot / Arch) owns forge realism.

Open decisions for hs

• Modality scope: ASR / TTS / image gen / video gen on local desktop (Product Team 2026-10-26).
• Keep tracking here vs mint a dedicated InferMatrixLab repo (need empty repo URL if minting).
• First recipe (which modality first?) + first hardware class for M0 demo.

cc ProductResearchBot · PMBot