[olliemnicholls]

Motivation

When the output needs a specific shape (JSON matching a schema, one of exactly two labels) the only lever today is the spec text; nothing enforces the shape at sampling time. llama.cpp already implements constrained decoding: Llama.sample() takes a logits_processor, and runtime_llamacpp.py already calls self._llm.sample(temp=...) once per token. The capability is in the runtime but isn't reachable from the public API, so the only way to use it today is to reach PawFunction._llm — a private attribute with no stability contract.

A logits_processor is stateless from llama.cpp's point of view (it receives the full input_ids on every call), so a straight per-token passthrough is enough; there is no sampler state to carry across tokens. In my own use the masking costs ~0.40 ms/token against llama.cpp's 151k-token vocabulary. The cost that matters is building the processor's own token tables up front, and that falls on the caller, not on the SDK.

What changed

• PawFunction.__call__ takes a new optional logits_processor
  (llama_cpp.LogitsProcessorList | None, default None), threaded through
  PawFunction._generate into the existing self._llm.sample(...) call, so it applies to
  every generated token in both compiled and adapter-free base mode.
• Nine added lines in programasweights/runtime_llamacpp.py; three tests in
  tests/test_base_interpreter.py; docstring, README.md,
  docs/api-reference/python-sdk.md and CHANGELOG.md updated.

Why no grammar

A GBNF grammar would be more attractive; enforced in C++, no Python callback per token, but it is not a straight passthrough. In llama-cpp-python 0.3.x, sample() builds a temporary sampler chain when none is attached and discards it after each call, and add_grammar re-initialises the grammar sampler each time, so a grammar passed per token would reset to its root rule on every token. Supporting it properly means creating one sampler per __call__, as Llama.generate() does. That is a larger change and a separate decision; happy to open it separately if you'd want it.

Testing

The hermetic suite CI runs goes from 132 to 135 passing, no failures before or after. The three new tests use the existing fake llama_cpp fixture: that a compiled program passes the processor list on every generated token and the processor is invoked once per token with a token sequence that grows by one each time; that base mode passes it through the same way; and that sample() still receives logits_processor=None when the argument is unset. Also checked end to end outside the suite against a real cached adapter on CPU (n_gpu_layers=0, llama-cpp-python 0.3.35): a processor masking everything except one token changed the output as expected, and unconstrained output was unchanged.

One thing worth a glance from your side — I verified against 0.3.35, but pyproject.toml allows >=0.3.0,<=0.3.19, so it would be worth confirming that logits_processor is on Llama.sample() at the 0.3.0 floor too.

Interest

I maintain a downstream project, paw-kit, whose schema-constrained decoding currently has to reach PawFunction._llm to get at the sampling loop. That is the use case that prompted this change, and it is why I would rather the hook were public than keep monkeypatching a private attribute.