decidekit

Try it in your browser · Watch it learn from an LLM · Launch post · Research: when the LLM is wrong

A decision model for the browser and Node: it picks one of your labels for a piece of text, with confidence you can trust, and learns from an LLM when it isn't sure. Your labels are the type. No server, no API key.

A System One model in the browser: like TypeSafe's Jev, but on the user's device, so each decision is free, private and a few milliseconds. Jev leaves unsure cases to your code; decidekit can ask an LLM about them and learn from the answers. decidekit is an independent project, not affiliated with TypeSafe.

import { createDecider } from "decidekit";

const d = await createDecider();
const r = await d.decide("my card never arrived", ["card_arrival", "refund", "other"]);
r.choice; // "card_arrival" | "refund" | "other"

Give it 5 examples per label and it reaches 84% on BANKING77 (77 intents); give it the full training set and it reaches 93.1%, with each answer taking about 8 ms (median) in Chrome. Training runs in the user's tab, so their data never leaves the device.

Zero-shot

Pass labels, or labels with descriptions. Labels like card_arrival are read as "card arrival".

await d.decide(text, ["question", "bug_report", "praise"]);
await d.decide(text, { refund: "wants their money back", bug: "something is broken" });

Zero-shot is for trying things out: 66% on BANKING77 and 73% on CLINC150 (dev splits), against 90-94% trained. Its confidence came out close to calibrated on those datasets (error 0.03-0.04), but check it on your own labels before relying on a threshold.

Train a head

A head is a small layer trained on your examples: 156 KB for 77 labels. Train it once and ship it as a static file, or train it in the page.

import { createDecider, saveHead, loadHead } from "decidekit";

const d = await createDecider();
const head = await d.train(examples); // { text, label }[]
await writeFile("intent.head.json", saveHead(head));

// later, in the app
const r = await d.decide(text, loadHead(json)); // choice is typed as your labels
const rs = await d.decideMany(texts, head); // many texts, encoded in batches

Each label starts at the average of its examples plus its own name, so a handful per label already works. If your labels are opaque ids (c17), pass { labelNames: false }. Training then fits a temperature on held-out examples, so confidence is calibrated: of the answers given at 0.9, about 90% are right.

Or train from the command line and ship the file:

npx decidekit train examples.csv -o intent.head.json   # CSV with text,label columns, or JSONL
npx decidekit decide intent.head.json "my card never arrived"

Add --types to also write intent.head.labels.ts, so loadHead<Label>(json) types the choice as your labels.

"None of these"

const r = await d.decide(text, head, { minConfidence: 0.9 });
r.choice; // "card_arrival" | "refund" | ... | null

Below the threshold, choice is null. Use it to send unclear messages to a person or a bigger model. It needs enough examples to mean something: with 3 examples per label, a head still gave "hello" 0.89. The numbers below come from CLINC150's 100 examples per label. With fewer, check the threshold on some off-topic text of your own, or add an other label trained on it.

Fall back to an LLM

const route = withFallback(d, head, { minConfidence: 0.9, ask: (text, labels) => askMyLlm(text, labels), examples });
const r = await route(text); // r.source: "local" | "fallback"

Below the threshold, ask answers and its answer becomes a training example. Simulated on CLINC150 from 5 examples per label with an LLM that is always right, the share sent to the LLM fell from 41% to 14-17% over 10,000 queries while the head went from 86% to 94%. With real Llama 3.1 8B answers, starting from 12 described labels alone, it fell from 30% to 19%.

Real LLMs are sometimes wrong, and withFallback() is built for that: it sets its confidence on its own sure answers rather than the LLM's, and learns the LLM's "none of these" (null) as a label, NONE, so it can say null itself for off-topic text. Details and numbers: docs/guide-fallback.md.

Coming from Jev

decidekit/typesafe has the shape of TypeSafe's SDK, for Choice questions:

import { choice, typesafeClient } from "decidekit/typesafe";

const client = typesafeClient(await createDecider());
const { answers } = await client.systemOne({
  state: "I was charged twice",
  questions: { category: choice("What is this about?", { billing: null, technical: null, other: null }) },
});

The guide covers what differs, including what confidence means.

switch

decidekit/switch makes decidekit a decision model for switch, the harness that runs a decision model through a task. Zero-shot, it won two of the three WikiRace races from switch's example on a laptop CPU. See the switch guide.

React

useDecider() and useDecision(decider, text, labels) from decidekit/react: docs/guide-react.md.

In the browser

Use the worker version. The encoder blocks whatever thread it runs on (we measured a 171 s freeze while training on the main thread; in a worker the page's longest stall was 18 ms).

import { createWorkerDecider, loadHead } from "decidekit/browser";

const head = loadHead(await (await fetch("/intent.head.json")).text());
const d = await createWorkerDecider({
  onProgress: ({ file, loaded, total }) => showProgress(file, loaded / total), // first visit only
});
const r = await d.decide(text, head);

Import loadHead from decidekit/browser too, not from decidekit: that keeps the encoder out of your page bundle.

Tested from npm in Vite 7, webpack 5 and Next.js 16 apps; see docs/bundlers.md for the config. Serve the page cross-origin isolated if you can:

Cross-Origin-Opener-Policy: same-origin
Cross-Origin-Embedder-Policy: credentialless

That lets wasm use several threads: in our runs with all-MiniLM-L6-v2 it cut p95 decision time from 14.3 ms to 7.8 ms. With the default bge-small, isolated, p95 is 13.8 ms.

Download on first visit: the model (34 MB) and onnxruntime's wasm (12.9 MB, 3.3 MB gzipped; 23.6 MB before 0.2.1), both cached after.

Models

decidekit runs the Decide models, each a sentence encoder plus a calibrated head:

Model	Id	Download	Reads	Upstream
Decide Small (default)	debuffjp/decide-small	34 MB	English	BAAI's bge-small-en-v1.5, identical to Xenova/bge-small-en-v1.5
Decide Multilingual	debuffjp/decide-multilingual	about 118 MB	76% on Japanese and Chinese intents	intfloat's multilingual-e5-small, identical to Xenova/multilingual-e5-small

Pass the id as model to createDecider(). The files download from Hugging Face on first use, pinned to a revision, and are then cached. To serve them yourself, pass modelHost: see self-hosting. onnxruntime's wasm comes from jsDelivr unless you pass wasmHost.

Before 0.1.1 the default id was Xenova/bge-small-en-v1.5. The old ids and heads recording them keep working: they resolve to the new ids, so d.model returns the new id.

Numbers

Decide Small (bge-small-en-v1.5, the default) at q8, without label names, Ryzen 7 260 laptop, Node 24 and headless Chrome 156, 2026-11-01. Accuracy and calibration error (ECE, lower is better) on each dataset's test split, mean of 3 seeds (one run for "all"): eval/results-bge-small.json.

Examples per label	5	10	25	all
BANKING77 (77 labels)	83.7%	87.5%	90.2%	93.1%
CLINC150 (150 labels)	90.2%	92.8%	94.3%	96.3%
ECE, BANKING77	0.050	0.015	0.013	0.013
ECE, CLINC150	0.020	0.015	0.016	0.008

Those are 0.2, with label names, from eval/results-h14.json. HWU64 with all data goes from 90.15% to 91.17%. Before 0.2 (SGD at every size, no label names, eval/results-bge-small.json):

Examples per label	5	10	25	all
BANKING77 (77 labels)	82.3%	83.8%	88.7%	90.7%
CLINC150 (150 labels)	89.6%	92.0%	93.4%	94.4%
ECE, BANKING77	0.071	0.029	0.011	0.017
ECE, CLINC150	0.028	0.010	0.015	0.008

Pre-release builds defaulted to all-MiniLM-L6-v2 (23.7 MB, 2026-10-31, eval/results.json):

Examples per label	5	10	25	all
BANKING77 (77 labels)	78.7%	83.0%	87.4%	90.6%
CLINC150 (150 labels)	87.3%	89.8%	92.8%	94.0%
ECE, BANKING77	0.065	0.048	0.025	0.019
ECE, CLINC150	0.029	0.013	0.012	0.012

Calibration is loosest at 5 examples per label on BANKING77. On small tasks (2 to 10 labels sampled from CLINC150, scored on its validation queries, eval/tiny.ts, measured with all-MiniLM-L6-v2), 1 example per label was right 93% of the time with calibration error 0.020, and 3 per label 98% with 0.016. Add examples before relying on a threshold there.

In Chrome (BANKING77, full training set, worker, cross-origin isolated, 16 threads): 7.7 ms median and 13.8 ms p95 per decision before 0.2, 228 s to train on 10,003 examples with no page stall longer than 18 ms. A 0.2 run on the same machine took 208 s, with a longest stall of 19 ms and 93.05% test accuracy (runs on this laptop vary: another 0.1 run took 147 s). all-MiniLM-L6-v2 in the same run: 4.2 ms median, 7.7 ms p95. In Node: 3.2 ms median for a single text, 1.1 ms per text batched (MiniLM 2.0 and 0.7). Earlier MiniLM run: bench/results.json.

Text that fits no label: CLINC150 has 1,000 test queries that match none of its 150 intents. Trained on in-scope examples only, with minConfidence 0.93 picked on the validation split, the shipped bge-small head returns null for 89.9% of them. With all-MiniLM-L6-v2 that was 86.3%, and 82.9% of in-scope queries were answered and right (94.0% right with no threshold).

The smaller model. createDecider({ model: "Xenova/all-MiniLM-L6-v2" }) (23.7 MB, the pre-release default) is about 1.8x faster per decision in Chrome and 3 to 5 points less accurate with few examples, about the same with all the data. Pass it to keep using heads trained on it. Before 0.1.0 a label name counted as 3 examples and the README suggested { labelNames: false } with bge-small. It now counts as 1, which helps.

Reproduce: npm run eval:suite (Node), npm run bench then npm run bench:run (Chrome).

Playground

Try it at debuff.jp/decide. To run it locally: npm run playground, then serve playground/dist cross-origin isolated (node bench/serve.mjs style headers). Try zero-shot labels, the ready-made heads, a head trained in the page and "none of these". Two lanes shows withFallback live: decidekit starts from twelve label names, asks a small LLM when it's unsure, learns from each answer, and replays the first 20 messages at the end.

Ready-made heads

heads/ has a BANKING77 support-intent head (90.6%) and a CLINC150 assistant-intent head (95.0%), so you can try a trained head before collecting examples.

What it's for

Routing support messages, triaging feedback, search and command intent, moderation hints while someone types, labelling a dataset in the browser, a cheap first pass before an LLM that learns from it, games, extensions and offline apps, and text that can't leave the device. Details, code and poor fits: docs/use-cases.md. Runnable examples are in examples/.

Compared with

	Runs in the browser	Trains in the browser	Typed result	Calibrated confidence
decidekit	yes (worker)	yes	yes	trained heads
jimothy	inference (worker)	no, Node	yes	yes
Jev (TypeSafe)	no, API	no	yes	yes
transformers.js zero-shot pipeline	yes	no	no	no
MediaPipe text classifier	yes	no, offline	no	no
LLM structured output (AI SDK Output.choice)	no, API	n/a	yes	no

Docs

debuff.jp/decide/docs (also in docs/README.md) indexes the guides (training, confidence and thresholds, the browser), the API reference, how it works and the FAQ.

Community

Questions and show-and-tell in Discussions, chat in the debuff Discord. Results on your own data, good or bad, help most: there's an issue template for them. See CONTRIBUTING.md.

Status

0.1. English with the default model, other languages with Decide Multilingual (FAQ); short texts (about a paragraph). Not done yet: WebGPU in the worker and fine-tuning the encoder (the likely route past about 91%).

Made by the debuff team. MIT.