TinyTask

TinyTask trains a local text classifier and measures which requests it can accept. Use it for repeated classification tasks, such as support queues or intent routing.

The question is practical: can this candidate handle enough requests at your target accuracy? TinyTask returns accept or review, then evaluates the saved candidate against separate test data.

This is an alpha Python package. The PyPI name is tiny-task-py. The import is tiny_task. The command is tiny-task.

Try the support inbox

Use Python 3.11 or later. Run these commands:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install tiny-task-py
tiny-task demo --output tiny-task-demo

Open tiny-task-demo/index.html. The demo trains a local classifier without an API key or model download. It routes cancellation, refund, delivery, and password messages. A hamster-haiku request goes to review.

The example data is authored. Its small test set returns needs_data, even when the example messages route correctly.

Use the saved candidate on your own CSV inbox:

tiny-task predict --model tiny-task-demo/model \
  --input inbox.csv --output predictions.jsonl

The CSV needs id and text columns. JSONL uses the same fields. The output keeps the ID, proposed label, decision, scores, and reason. It does not change the inbox or send messages to an external service. Use a new output path for each run. TinyTask refuses to replace existing output files.

Why choose TinyTask

• Get training, threshold selection, evaluation, and export through one small API.
• Keep training, calibration, and test data separate.
• Refuse known ID, group, and normalized-text overlap between splits.
• Send unsupported requests to review instead of forcing a label.
• Read accepted errors, coverage, unknown acceptance, and uncertainty intervals in one report.
• Bind the report to the saved model with file hashes.

The default backend uses TF-IDF and logistic regression. The optional backend does actual SetFit fine-tuning, then preserves the encoder and numeric classifier head.

TinyTask reuses existing learning methods. Its contribution is the decision workflow around a trained candidate. Read the comparison before a novelty claim.

Train your task

Get the repository for the example files and benchmark recipes:

git clone https://github.com/eduard-s/tiny-task.git
cd tiny-task

Keep the Python environment active. The installed package supplies the commands. Prepare three JSONL files. Keep related examples in one split. Each labeled row uses this format:

{"id":"train-1","text":"Cancel my subscription","label":"cancel","group_id":"conversation-1","label_source":"human"}

Use __unknown__ for unsupported requests in calibration and test data. Training data needs at least two known labels. Teacher labels can train the model. Calibration and test data need reviewed or dataset labels.

tiny-task train \
  --train examples/support/train.jsonl \
  --calibration examples/support/calibration.jsonl \
  --min-calibration-accepted 8 --output artifacts/my-task

tiny-task evaluate --model artifacts/my-task \
  --test examples/support/test.jsonl --output artifacts/evaluation.json

Evaluation exits with 0 for ready, 1 for reject, and 3 for needs_data. Execution errors use code 2. The supplied small example returns code 3.

Use a matching successful report to enforce its criteria during model load:

tiny-task predict --model artifacts/my-task \
  --release-report artifacts/evaluation.json --text "Please cancel my subscription"

The command refuses a rejected, incomplete, or mismatched report. Without --release-report, prediction uses a candidate without release approval.

Python API

from tiny_task import PolicyConfig, TinyTask, read_records

task = TinyTask.fit(
    read_records("examples/support/train.jsonl"),
    read_records("examples/support/calibration.jsonl"),
    policy_config=PolicyConfig(min_calibration_accepted=8),
)
task.save("artifacts/api-model")
report = task.evaluate(read_records("examples/support/test.jsonl"))
task.write_report("artifacts/api-report.json", report)
print(task.predict("Write a haiku about a sleepy hamster"))

Optional SetFit training

python -m pip install 'tiny-task-py[neural]'
tiny-task train --backend setfit --allow-download \
  --train examples/support/train.jsonl \
  --calibration examples/support/calibration.jsonl \
  --min-calibration-accepted 8 --output artifacts/neural-task

The first checkpoint download needs --allow-download. The checkpoint revision is fixed. Later inference uses local files on the CPU. SetFit can select a training accelerator. The benchmark records the actual training device.

Evidence and limits

The development benchmark uses a pinned CLINC150 cohort. It compares three acceptance policies on each trained classifier. The SetFit run includes changed encoder hashes and a training history.

Both current candidates fail the default release criteria. The report preserves that result. The result does not establish broad superiority over SetFit or Prompt2Model.

The first scope is English, one label per request, and short text. Near-duplicate paraphrases can cross a split without an explicit group ID. Model scores are not calibrated probabilities of correctness. Wilson intervals describe sample uncertainty under independent sampling assumptions. They do not establish future accuracy after a distribution change.

No live LLM baseline or measured API savings are included yet. TinyTask does not generate training labels, invoke a fallback model, or do business actions.

Documents

• Prompt2Model and SetFit comparison
• MVP plan and acceptance criteria
• Python stack and architecture
• Datasets and partition rules
• Benchmark protocol and results
• API and artifact contract
• Demo and user trial
• Development
• PyPI release procedure

MIT. Read the third-party notices.