[x-Pepsy]

The server runs one request at a time (handle decodes until the reply is done), so four clients means four people waiting for their turn. This lets up to --parallel N replies make progress at once on one GPU.

The model already parked a conversation's state in host memory when another history replaced it, so I reused that: a reply prefills and parks its state when it arrives, then each turn of the scheduler adopts the state, generates PAR_QUANTUM tokens and parks it again. The tokens are unchanged; the only thing that moves is the state.

I picked the quantum by measuring. With four sequences, one token a turn loses about half the aggregate throughput to the state copies (23 vs 39 tok/s), four tokens gets most of it back (35), and eight is within noise of one at a time (39). So the default is 8.

Numbers on a 2060 6 GB:

• four streaming requests, 48 tokens: 4.7-7.3 s each, 1-3.8 s to the first byte, all done.
• four 64-token replies one at a time: 9.2 s total, the last client waits 8.7 s.
• the same four with --parallel 4: 9.8 s total, all four finish around 8-9 s, so every stream moves instead of queueing.

Limitations, all deliberate for a first cut:

• MTP is off in this mode. A parked state has the KV and the recurrent state but not the draft head's pending drafts.
• Prefill still runs when a request arrives, so the fourth client's first byte waits for the three prefills ahead of it.
• A split model falls back to serial: parking is per model and each part would need its own copy.
• --parallel defaults to 1, so nothing changes unless asked.

tests/concurrency.rs runs four sequences interleaved at q = 1, 4 and 8 and checks the tokens match generating them one at a time, so the state round-trip is covered.