Skip to content

Convert and profile sub-1B Decision Index models for Core ML - #102

Merged
Alex-Wengg merged 1 commit into
mainfrom
feat/sub1b-coreml
Sep 23, 2026
Merged

Alex-Wengg merged 1 commit into
mainfrom
feat/sub1b-coreml

Conversation

@Alex-Wengg

@Alex-Wengg Alex-Wengg commented Sep 22, 2026 •

Copy link
Copy Markdown
Member

The frozen Decision Index snapshot has 13 entries under 1B served parameters. This draft PR now contains pinned Core ML conversion toolkits and selected real-request validation for all 13. Eleven have public, model-specific Hugging Face repositories with Core ML weights. NanoJev and system-one-gemma remain source-only because trained-weight redistribution rights are unstated or the exact base model is gated. The scope table and ANE/compression matrix state the supported shapes and limits.

This update publishes optional, targeted int8 packages for GLiNER2.5 base, GLiNER2.5 multilingual, Kev 0.5B, Jeff, Kai, and Lex. Previously published GLiNER2 small and Kev 0.6B int8 variants remain available. Each new upload was checked against its local package hash; GLiNER2 base was also smoke-tested from a materialized Hub download.

The size reductions are scoped. GLiNER2 base classification shrank 389.0→291.1 MB and retained 100/100 selected native labels; multilingual classification shrank 576.5→385.2 MB and retained 100/100 paired FP16 choices (99/100 native, the same near-tie miss as FP16). Kev 0.5B embedding int8 shrank 989.6→853.9 MB and retained 19/19 eligible native choices. Jeff W8 shrank 922.8→488.4 MB and retained 4/4 native labels under its existing logit gate. Kai's three typed L128 packages each shrank 643.8→448.0 MB and passed 2/2 pinned native choices. Lex's Noul and Score paths did likewise; its Choice path stays FP16 because both int8 variants changed a pinned decision. The model cards and source reports contain exact errors, hashes, and supported request shapes. Default runtimes keep the validated FP16 route unless the caller selects an optional package.

ANE compute-plan compatibility did not translate into a reliable speedup. Kai Choice and Lex Noul W8 placed 926/938 executable operations on ANE under forced CPU+ANE, but small paired AC runs showed no repeatable end-to-end latency gain. GLiNER2 base/multilingual forced CPU+ANE was slower than automatic placement despite 496/516 ANE operations. Jeff's automatic/GPU route was substantially faster than forced CPU+ANE. Kev 0.5B forced ANE missed its probability gate. LFM2.5 asymmetric tied-embedding W8 saved 67 MB but exceeded its stricter 0.1 score-error release gate (0.291); symmetric W8 changed a decision. English Laya int8 also missed its parity gate and slowed requests. Those variants are documented, not published. This update does not claim every model is fully ANE-resident or that int8 improves speed.

The separately integrated GLiNER2 base/multilingual extraction paths retain FP32 for native structure parity. The earlier GLiNER2 small experimental int8 and Kev 0.6B W8 remain size options. Timing is exploratory on an Apple M5 Pro/24 GB Mac; newer GLiNER2 classification and Kai/Lex checks recorded AC power. No full 132,422-request Decision Index reproduction has been run, and the tracker snapshots are not proven identical to all pinned current upstream checkpoints. FluidUse #8 now includes an exploratory ten-seed 2048 comparison for GLiNER2 base/multilingual against GLiClass; it is not a comparison across all 13 entries.

Hub config follow-up: all 11 public weighted model repositories now have a root config.json. Eight missing files were added from pinned, SHA-256-checked upstream configs with a CONFIG_PROVENANCE.md on each page. For Kev, Laya English, Kai, Lex, and Jeff the root file describes the base/encoder component, not a standalone Transformers checkpoint. sync-hub-configs.py makes the metadata release reproducible and idempotent.

Validation for this update: targeted real-model tests passed for GLiNER2 base/multilingual, Kai/Lex optional release checks, Jeff W8 publication checks, Kev 0.5B E8 checks, and LFM release-integrity checks; Ruff, three config-release unit tests, a source-hash audit, and git diff --check passed on touched source. No model binaries are committed to Git. This targets main after #101 merged. The overlapping GLiNER extraction runtime from merged #103 is retained.
Review follow-up: the Kev 0.6B publisher now requires a CPU+ANE native-parity report whose SHA-256 inventory matches the exact FP16 or W8 package uploaded. The Laya English and Verdict publishers bind conversion, parity, and package hashes. Verdict was reverified against both published packages (L128 4/4, L512 5/5); the hash-bound reports are on its Hub page. The published Laya English package files match its existing parity hashes. Kev release-gate tests, Ruff, and git diff --check passed.

@Alex-Wengg Alex-Wengg changed the title Convert sub-1B Decision Index models to scoped Core ML artifacts Convert and profile sub-1B Decision Index models for Core ML Sep 22, 2026
Base automatically changed from feat/laya-coreml to main September 23, 2026 00:44
@Alex-Wengg
Alex-Wengg marked this pull request as ready for review September 23, 2026 00:46
@Alex-Wengg
Alex-Wengg merged commit b10771c into main Sep 23, 2026
@Alex-Wengg
Alex-Wengg deleted the feat/sub1b-coreml branch September 23, 2026 00:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant