Repository navigation
Convert and profile sub-1B Decision Index models for Core ML - #102
Merged
Merged
Conversation
Alex-Wengg
force-pushed
the
feat/sub1b-coreml
branch
from
September 23, 2026 00:46
df73e44 to
d3bb7d9
Compare
Alex-Wengg
marked this pull request as ready for review
September 23, 2026 00:46
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The frozen Decision Index snapshot has 13 entries under 1B served parameters. This draft PR now contains pinned Core ML conversion toolkits and selected real-request validation for all 13. Eleven have public, model-specific Hugging Face repositories with Core ML weights. NanoJev and system-one-gemma remain source-only because trained-weight redistribution rights are unstated or the exact base model is gated. The scope table and ANE/compression matrix state the supported shapes and limits.
This update publishes optional, targeted int8 packages for GLiNER2.5 base, GLiNER2.5 multilingual, Kev 0.5B, Jeff, Kai, and Lex. Previously published GLiNER2 small and Kev 0.6B int8 variants remain available. Each new upload was checked against its local package hash; GLiNER2 base was also smoke-tested from a materialized Hub download.
The size reductions are scoped. GLiNER2 base classification shrank 389.0→291.1 MB and retained 100/100 selected native labels; multilingual classification shrank 576.5→385.2 MB and retained 100/100 paired FP16 choices (99/100 native, the same near-tie miss as FP16). Kev 0.5B embedding int8 shrank 989.6→853.9 MB and retained 19/19 eligible native choices. Jeff W8 shrank 922.8→488.4 MB and retained 4/4 native labels under its existing logit gate. Kai's three typed L128 packages each shrank 643.8→448.0 MB and passed 2/2 pinned native choices. Lex's Noul and Score paths did likewise; its Choice path stays FP16 because both int8 variants changed a pinned decision. The model cards and source reports contain exact errors, hashes, and supported request shapes. Default runtimes keep the validated FP16 route unless the caller selects an optional package.
ANE compute-plan compatibility did not translate into a reliable speedup. Kai Choice and Lex Noul W8 placed 926/938 executable operations on ANE under forced CPU+ANE, but small paired AC runs showed no repeatable end-to-end latency gain. GLiNER2 base/multilingual forced CPU+ANE was slower than automatic placement despite 496/516 ANE operations. Jeff's automatic/GPU route was substantially faster than forced CPU+ANE. Kev 0.5B forced ANE missed its probability gate. LFM2.5 asymmetric tied-embedding W8 saved 67 MB but exceeded its stricter 0.1 score-error release gate (0.291); symmetric W8 changed a decision. English Laya int8 also missed its parity gate and slowed requests. Those variants are documented, not published. This update does not claim every model is fully ANE-resident or that int8 improves speed.
The separately integrated GLiNER2 base/multilingual extraction paths retain FP32 for native structure parity. The earlier GLiNER2 small experimental int8 and Kev 0.6B W8 remain size options. Timing is exploratory on an Apple M5 Pro/24 GB Mac; newer GLiNER2 classification and Kai/Lex checks recorded AC power. No full 132,422-request Decision Index reproduction has been run, and the tracker snapshots are not proven identical to all pinned current upstream checkpoints. FluidUse #8 now includes an exploratory ten-seed 2048 comparison for GLiNER2 base/multilingual against GLiClass; it is not a comparison across all 13 entries.
Hub config follow-up: all 11 public weighted model repositories now have a root
config.json. Eight missing files were added from pinned, SHA-256-checked upstream configs with aCONFIG_PROVENANCE.mdon each page. For Kev, Laya English, Kai, Lex, and Jeff the root file describes the base/encoder component, not a standalone Transformers checkpoint.sync-hub-configs.pymakes the metadata release reproducible and idempotent.Validation for this update: targeted real-model tests passed for GLiNER2 base/multilingual, Kai/Lex optional release checks, Jeff W8 publication checks, Kev 0.5B E8 checks, and LFM release-integrity checks; Ruff, three config-release unit tests, a source-hash audit, and
git diff --checkpassed on touched source. No model binaries are committed to Git. This targetsmainafter #101 merged. The overlapping GLiNER extraction runtime from merged #103 is retained.Review follow-up: the Kev 0.6B publisher now requires a CPU+ANE native-parity report whose SHA-256 inventory matches the exact FP16 or W8 package uploaded. The Laya English and Verdict publishers bind conversion, parity, and package hashes. Verdict was reverified against both published packages (L128 4/4, L512 5/5); the hash-bound reports are on its Hub page. The published Laya English package files match its existing parity hashes. Kev release-gate tests, Ruff, and
git diff --checkpassed.