Skip to content

0.6.4: in-process loopback adapter binds but inference still fails auth on Hermes 0.22.5; standalone adapter + --base-url works #20

Open

Description

@Code-Xer0

Summary

Follow-up to #12 (fixed in 0.6.1): on Hermes 0.22.5 with plugin 0.6.4 (main @ d684003), the in-process loopback adapter starts and binds, the resolved runtime base URL points at it, yet every inference through --provider muse-code fails with FailoverReason.auth (authentication failed) and falls back to meta-ai. Running the same adapter code standalone and pointing Hermes at it via --base-url succeeds end-to-end and bills to the subscription.

Environment

  • Hermes Agent v0.22.5 (linux x86_64), profile default
  • Plugin TheStreamCode/hermes-muse-code @ d684003 (v0.6.4), enabled
  • muse 1.4.14, logged in, tier High Usage (subscription active)
  • Credential cache ~/.hermes/muse-code-sub.json present (device-code login done); key also pinned as MUSE_CODE_SUB_TOKEN in ~/.hermes/.env (verified byte-identical to cache, len 48)
  • No META_BASE_URL set; no plugins.isolation override in config

What works (verified)

  1. Key is valid: GET https://api.meta.ai/v1/models with Bearer <key> → 200, 13 models incl. muse-spark-1.3.
  2. Raw Responses call with the key → 200.
  3. Standalone adapter: start_adapter(ports=[8920]) in its own process, then hermes chat --provider muse-code -m muse-spark-1.3 --base-url http://127.0.0.1:8920/v1 → answers with no fallback, session logged as model=muse-spark-1.3 provider=muse-code, and muse usage window ticked 3% → 4% (billed to sub, not PAYG). Repeated, consistent.

What fails (verified, 4+ runs)

hermes chat --provider muse-code -m muse-spark-1.3 -q "Reply with exactly: relay-ok" (no --base-url) →

Model fallback: muse-spark-1.3 via muse-code unavailable (authentication failed); using muse-spark-1.3-contributor via meta-ai.

Evidence the adapter path is (nominally) active

  • During a failing run, ss -tln shows the in-process adapter bound on 127.0.0.1:43219 and 127.0.0.1:43220 simultaneously (double-bind within a single CLI run).
  • Repro harness mimicking the Hermes loader (package import + register(None)) then calling Hermes's own resolve_api_key_provider_credentials('muse-code') returns base_url=http://127.0.0.1:<port>/v1, source=MUSE_CODE_SUB_TOKEN, key len 48. So resolution lands on the loopback adapter with the right key — yet the live call still 401/403s.
  • hermes doctor reports the muse-code provider configured.

So: same key, same model, same adapter code — fails in-process, succeeds standalone. The failure is specific to the in-process wiring, not the credential.

Observations / pointers (not conclusions)

  • Double-bind: two loopback ports (43219 + 43220) bound during one short-lived CLI run suggests start_adapter is invoked twice with non-identical options (the idempotency key differs) or races with itself; the request may hit the superseded instance.
  • Orphaned listeners: a LISTEN on 43219 owned by a hermes pid was observed after the CLI session had ended (gone on re-check), so short-lived CLI processes leave bound ports behind transiently; the next run then binds the next port while anything cached points at the old one.
  • Possible race between adapter readiness and the first inference request in a fresh process (standalone adapter is always warm when the request arrives).
  • Worth ruling out: whether the failing request actually reaches the loopback socket at all vs. going direct (request dumps in v0.6.0: loopback wire adapter never receives traffic — runtime_base_url not overridden #12 showed direct URLs pre-fix; I could not capture per-request URLs on 0.22.5).

Workaround in use

Systemd user unit running start_adapter(ports=[8920]) persistently + explicit --base-url http://127.0.0.1:8920/v1 on every call. Functional but defeats the zero-config goal of the in-process adapter.

Ask

  • Any way to surface the actual upstream status/body (or adapter-side log) for the failing in-process call? User-facing text only says authentication failed.
  • Happy to test a debug build, capture traffic, or try a single-bind / ready-gate patch.

No secrets are included above (key lengths only). Logs/session IDs available on request.

Activity

  1. TheStreamCode commented on Nov 17, 2026

    @TheStreamCode
    Owner

    Thanks for the detail, and for the negative controls. I could not reproduce this here (no 0.22.5 install, no subscription key), but reading the adapter and reproducing the loader side turned up three states that match what you saw, and #21 makes them distinguishable.

    What I verified in the code and with a harness that loads the plugin as a package, the way a directory plugin is loaded:

    • The adapter started at import (base_url=_adapter_base_url() in the profile constructor). Any process that only loads the plugin — the admission probe register() mentions, hermes plugins validate, a linter — binds a port and exits. That is one source of the LISTEN on 43219 you found after the run ended.
    • start_adapter() is idempotent per process only (the state lives on a sys attribute). With a listener already on 43219, the next process binds 43220: reproduced as A → 43219, B → 43220, which is exactly the pair you saw.
    • A refusal comes back from the adapter verbatim, but the adapter only logged at debug and only when a rule fired. On 0.22.5 there is no classify_api_error separation either, so a 403 client_attestation_denied and a rejected key both surface as "authentication failed".

    #21 changes that: the adapter starts on first use (nothing binds at import or in register()), GET /__muse_adapter/health reports adapter_version, pid, port, passthrough and upstream, every relayed response carries x-muse-adapter, and each 4xx/5xx logs one line with method, path, status, adapter version, port, pid and the rules that fired.

    To pin down your case at the next failing run, cheapest first (items 1 and 3 need #21, or the branch fix/adapter-health-and-lazy-start):

    1. While the failing session is alive:
      python muse_adapter.py --health http://127.0.0.1:43219/v1 and the same for 43220.
      A pid you do not recognise means another process owns that port; no Muse Code adapter answered means that port is dead and whatever URL was cached pointed at it.
    2. Grep the gateway log for muse-code adapter:. A 4xx line means the request reached an adapter and that status is Meta's. No line for the failing turn means the request never reached an adapter, which is the v0.6.0: loopback wire adapter never receives traffic — runtime_base_url not overridden #12 shape again.
    3. Check $HERMES_HOME/config.yaml for a model: block with base_url:. The README lists it as a documented bypass — the hermes model picker writes it — and it is the same switch your standalone test used with --base-url. If it is there, remove it and restart the gateway.
    4. If you can capture the failing response headers, x-muse-adapter says whether the hop happened at all.

    Two notes on the environment: the manifest floor is >= 0.23.0, and resolve_api_key_secret (the 0.6.4 fix) only exists there. On 0.22.5 the key still resolves through the env export, but error classification is the older one, so 0.23.x is worth one run before chasing this further.

    Your double-bind observation is the most useful part of the report, and the single-bind/ready-gate question is now something the health endpoint can answer rather than infer. If you can run (1) and (2) I will take it from there.

  2. TheStreamCode commented on Nov 17, 2026

    @TheStreamCode
    Owner

    Merged as 3c60c0d on main (no release tag yet). main is what contains the health endpoint, the x-muse-adapter header and the 4xx/5xx warning line, so the two probes in my previous comment can run against it directly.

    One correction to that comment after reviewing my own change: model discovery (GET /v1/models) stays off the adapter. It is a plain GET that the adapter relays untouched, Meta serves it directly with a subscription key, and routing it through a loopback port would only add a socket to short-lived processes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions