modelstage

Safe staged rollout of local LLM model files (GGUF and similar) across a few machines, with verification and rollback. Python 3.10+, standard library only, plus ssh and rsync on the PATH.

Why

Copying a multi-gigabyte model onto a serving host by hand fails in quiet ways: a full disk, a truncated copy, a service restarted onto a broken file. modelstage copies into a staging directory, checks the hash on the far side, only then renames, keeps the old file, and only keeps a switch if the server answers.

Quick start

# 1. record the file (sha256, size, date) in the local catalog
python3 modelstage.py add ./my-model-q4.gguf --name my-model

# 2. see what would run, then do it
python3 modelstage.py --models-dir /srv/models push my-model --to host-a,host-b --dry-run
python3 modelstage.py --models-dir /srv/models push my-model --to host-a,host-b

# 3. switch one host: relink, restart, health-check, roll back on failure
python3 modelstage.py --models-dir /srv/models switch my-model \
    --host host-a --service llama.service --url http://127.0.0.1:8080

# 4. table of alias, sha match, current link and free disk per host
python3 modelstage.py --models-dir /srv/models status

Global options (--catalog, --models-dir) go before the subcommand.

Commands

• add <file> --name <alias> reads the GGUF magic and version (versions 1 to 3) and refuses other files unless --any is given. It stores alias, absolute path, sha256, size and date in the catalog.
• push <alias> --to h1[,h2...] [--dry-run] per host: create the staging dir, check free disk, rsync into <models-dir>/.stage/, run sha256sum on the host, then rename into <models-dir>. A failure on one host does not stop the others; the exit code is 1 if any failed.
• switch <alias> --host h --service unit repoints current.gguf (an atomic symlink swap), restarts the unit, then checks GET /v1/models (retried up to --wait seconds) and one 1-token POST /v1/completions, run with curl on the host. If anything fails it restores the old link, restarts again and exits 1.
• status [--hosts h1,h2] [--alias a] [--quick] prints one row per host and alias: sha match (ok, MISMATCH, missing, UNREACHABLE), whether current.gguf points at it, and free disk. --quick compares size instead of hashing. Exit 1 on a mismatch or unreachable host.

Safety rules

1. Verify before rename. The staged copy must match the catalog sha256 on the remote host, or it is deleted and nothing is replaced.
2. Keep the previous file. The existing file is hard-linked to <name>.prev before the new one is renamed over it, so there is no moment without a file.
3. Free-disk precheck. A push is refused if free space is under file size x 1.1.
4. Health-gated switch. The new link stays only if the server answers a model listing and a tiny completion; otherwise the previous link is restored.
5. Every remote action goes through one function, run, so tests replace it and never touch a network. --dry-run prints the commands without running them.

Configuration

Flag, then environment, then default:

Setting	Environment	Default
catalog file	MODELSTAGE_CATALOG	~/.modelstage/catalog.json
models dir on hosts	MODELSTAGE_MODELS_DIR	/var/lib/modelstage/models
ssh options	MODELSTAGE_SSH_OPTS	-o BatchMode=yes -o ConnectTimeout=10
service command	MODELSTAGE_SYSTEMCTL	systemctl (set sudo systemctl or systemctl --user)
endpoint seen from the host	MODELSTAGE_URL	http://127.0.0.1:8080

Hosts are anything ssh accepts (user@host, an alias from your ssh config). Your service unit should load <models-dir>/current.gguf.

Limits

• Needs key-based ssh, rsync, sha256sum, curl, GNU coreutils (mv -T, ln -f) and systemd on the hosts. The models dir and its .stage subdirectory must share a filesystem that supports hard links.
• One models dir for all hosts, and one file per alias. Sharded multi-file models are not handled.
• The remote name is the file's basename; two aliases with the same basename overwrite each other.
• Rollback restores the symlink and restarts the service; it does not check that the old model is healthy again.
• switch does not re-hash the file; run status first if in doubt.
• The health check is a liveness check, not a quality check.
• Not for untrusted hosts or concurrent pushes of the same alias.

Tests

python3 -m pytest -q

License

Apache-2.0. See LICENSE and NOTICE. This project does not accept outside contributions; see CONTRIBUTING.md.