Repository navigation
Description
Summary
Follow-up to #12 (fixed in 0.6.1): on Hermes 0.22.5 with plugin 0.6.4 (main @ d684003), the in-process loopback adapter starts and binds, the resolved runtime base URL points at it, yet every inference through --provider muse-code fails with FailoverReason.auth (authentication failed) and falls back to meta-ai. Running the same adapter code standalone and pointing Hermes at it via --base-url succeeds end-to-end and bills to the subscription.
Environment
- Hermes Agent v0.22.5 (linux x86_64), profile default
- Plugin TheStreamCode/hermes-muse-code @
d684003(v0.6.4), enabled muse1.4.14, logged in, tier High Usage (subscription active)- Credential cache
~/.hermes/muse-code-sub.jsonpresent (device-code login done); key also pinned asMUSE_CODE_SUB_TOKENin~/.hermes/.env(verified byte-identical to cache, len 48) - No
META_BASE_URLset; noplugins.isolationoverride in config
What works (verified)
- Key is valid:
GET https://api.meta.ai/v1/modelswithBearer <key>→ 200, 13 models incl.muse-spark-1.3. - Raw Responses call with the key → 200.
- Standalone adapter:
start_adapter(ports=[8920])in its own process, thenhermes chat --provider muse-code -m muse-spark-1.3 --base-url http://127.0.0.1:8920/v1→ answers with no fallback, session logged asmodel=muse-spark-1.3 provider=muse-code, andmuse usagewindow ticked 3% → 4% (billed to sub, not PAYG). Repeated, consistent.
What fails (verified, 4+ runs)
hermes chat --provider muse-code -m muse-spark-1.3 -q "Reply with exactly: relay-ok" (no --base-url) →
Model fallback: muse-spark-1.3 via muse-code unavailable (authentication failed); using muse-spark-1.3-contributor via meta-ai.
Evidence the adapter path is (nominally) active
- During a failing run,
ss -tlnshows the in-process adapter bound on127.0.0.1:43219and127.0.0.1:43220simultaneously (double-bind within a single CLI run). - Repro harness mimicking the Hermes loader (package import +
register(None)) then calling Hermes's ownresolve_api_key_provider_credentials('muse-code')returnsbase_url=http://127.0.0.1:<port>/v1,source=MUSE_CODE_SUB_TOKEN, key len 48. So resolution lands on the loopback adapter with the right key — yet the live call still 401/403s. hermes doctorreports the muse-code provider configured.
So: same key, same model, same adapter code — fails in-process, succeeds standalone. The failure is specific to the in-process wiring, not the credential.
Observations / pointers (not conclusions)
- Double-bind: two loopback ports (43219 + 43220) bound during one short-lived CLI run suggests
start_adapteris invoked twice with non-identical options (the idempotency key differs) or races with itself; the request may hit the superseded instance. - Orphaned listeners: a
LISTENon 43219 owned by ahermespid was observed after the CLI session had ended (gone on re-check), so short-lived CLI processes leave bound ports behind transiently; the next run then binds the next port while anything cached points at the old one. - Possible race between adapter readiness and the first inference request in a fresh process (standalone adapter is always warm when the request arrives).
- Worth ruling out: whether the failing request actually reaches the loopback socket at all vs. going direct (request dumps in v0.6.0: loopback wire adapter never receives traffic — runtime_base_url not overridden #12 showed direct URLs pre-fix; I could not capture per-request URLs on 0.22.5).
Workaround in use
Systemd user unit running start_adapter(ports=[8920]) persistently + explicit --base-url http://127.0.0.1:8920/v1 on every call. Functional but defeats the zero-config goal of the in-process adapter.
Ask
- Any way to surface the actual upstream status/body (or adapter-side log) for the failing in-process call? User-facing text only says
authentication failed. - Happy to test a debug build, capture traffic, or try a single-bind / ready-gate patch.
No secrets are included above (key lengths only). Logs/session IDs available on request.
Activity
TheStreamCode commentedon Nov 17, 2026
Thanks for the detail, and for the negative controls. I could not reproduce this here (no 0.22.5 install, no subscription key), but reading the adapter and reproducing the loader side turned up three states that match what you saw, and #21 makes them distinguishable.
What I verified in the code and with a harness that loads the plugin as a package, the way a directory plugin is loaded:
base_url=_adapter_base_url()in the profile constructor). Any process that only loads the plugin — the admission proberegister()mentions,hermes plugins validate, a linter — binds a port and exits. That is one source of theLISTENon 43219 you found after the run ended.start_adapter()is idempotent per process only (the state lives on asysattribute). With a listener already on 43219, the next process binds 43220: reproduced asA → 43219,B → 43220, which is exactly the pair you saw.classify_api_errorseparation either, so a 403client_attestation_deniedand a rejected key both surface as "authentication failed".#21 changes that: the adapter starts on first use (nothing binds at import or in
register()),GET /__muse_adapter/healthreportsadapter_version,pid,port,passthroughandupstream, every relayed response carriesx-muse-adapter, and each 4xx/5xx logs one line with method, path, status, adapter version, port, pid and the rules that fired.To pin down your case at the next failing run, cheapest first (items 1 and 3 need #21, or the branch
fix/adapter-health-and-lazy-start):python muse_adapter.py --health http://127.0.0.1:43219/v1and the same for43220.A pid you do not recognise means another process owns that port;
no Muse Code adapter answeredmeans that port is dead and whatever URL was cached pointed at it.muse-code adapter:. A 4xx line means the request reached an adapter and that status is Meta's. No line for the failing turn means the request never reached an adapter, which is the v0.6.0: loopback wire adapter never receives traffic — runtime_base_url not overridden #12 shape again.$HERMES_HOME/config.yamlfor amodel:block withbase_url:. The README lists it as a documented bypass — thehermes modelpicker writes it — and it is the same switch your standalone test used with--base-url. If it is there, remove it and restart the gateway.x-muse-adaptersays whether the hop happened at all.Two notes on the environment: the manifest floor is
>= 0.23.0, andresolve_api_key_secret(the 0.6.4 fix) only exists there. On 0.22.5 the key still resolves through the env export, but error classification is the older one, so 0.23.x is worth one run before chasing this further.Your double-bind observation is the most useful part of the report, and the single-bind/ready-gate question is now something the health endpoint can answer rather than infer. If you can run (1) and (2) I will take it from there.
fix(review): keep model discovery off the adapter and harden health
fix(adapter): start on first use and report which adapter answered (#21)
TheStreamCode commentedon Nov 17, 2026
Merged as 3c60c0d on
main(no release tag yet).mainis what contains the health endpoint, thex-muse-adapterheader and the 4xx/5xx warning line, so the two probes in my previous comment can run against it directly.One correction to that comment after reviewing my own change: model discovery (
GET /v1/models) stays off the adapter. It is a plain GET that the adapter relays untouched, Meta serves it directly with a subscription key, and routing it through a loopback port would only add a socket to short-lived processes.