toolchain install llvm: payload patchelf'd against xim-x-glibc but the runtime dep is not installed — clang exec 127 on bare runners (gcc path provides it) #259

Activity

1. Sunrisepeak commented on Jul 21, 2026

   

   Sunrisepeak

   on Jul 21, 2026

   MemberAuthor

   More actions

   Root-cause investigation: the "missing dep declaration" hypothesis is wrong — the real chain is elsewhere

   Traced this against the mcpp tree at af25d18 (0.0.101), the local xim-pkgindex checkouts, and the xlings sources. Summary: the symptom is exactly as reported, but the attribution in the body ("missing runtime dependency in the llvm xim package") does not hold. Three separate findings, one of which is a previously-unknown mcpp bug.

   1. llvm.lua does declare glibc — hypothesis retracted

   xim-pkgindex pkgs/l/llvm.lua:25-31:

   xpm = { linux = {
       deps = { "xim:glibc@2.39", "xim:linux-headers@5.11.1",
                "xim:zlib@1.3.1", "xim:libxml2@2.13.5" },

   Present since 8e432d99 ("feat(llvm): add Linux x86_64 support with split packages (#182)"), identical in all three local checkouts, and structurally the same as pkgs/g/gcc.lua:34-35. xlings info llvm confirms it resolves: runtime deps xim:glibc@2.39 xim:linux-headers@5.11.1 xim:zlib@1.3.1 xim:libxml2@2.13.5.

   llvm.lua is also deliberately fail-loud when glibc is absent (:234-239, log.error("glibc payload not found …"); return false). And the PT_INTERP/RUNPATH rewrite is not done by llvm.lua at all — it is xlings' declarative elfpatch, driven by the declared dep nodes (glibc declares exports.runtime.loader).

   So "gcc declares it, llvm doesn't" is not the difference. Both declare it.

   2. mcpp does not bypass xlings' dep resolution either

   Checked because the natural follow-up hypothesis is "mcpp isn't running the full install chain":

   • XLINGS_HOME is always $MCPP_HOME/registry (src/config.cppm:82-84; every subprocess launch at src/xlings.cppm:734,743,752,1029,1186). The two homes do not diverge.
   • All three install paths — install_direct (xlings.cppm:1093-1105), install_with_progress (:955-1015), install → interface install_packages (:1029) — funnel into xlings' xim::cmd_install, which calls resolve(catalog, …) unconditionally. (noDeps is accepted by the interface schema and forwarded, but never read.)
   • Disk evidence on a working machine: in ~/.mcpp/registry/data/xpkgs, the dirs without a .mcpp_ok marker are exactly the transitive deps — xim-x-glibc/2.39, xim-x-linux-headers/5.11.1, xim-x-zlib/1.3.1, xim-x-libxml2/2.13.5, xim-x-binutils/2.42, … .mcpp_ok is written only by mcpp for packages it requested directly, so those dep payloads demonstrably arrived via xlings' own resolution.

   3. But mcpp's own sysroot safety-net is dead code — new bug

   This is the part that was not on anyone's radar. Two sites:

   // src/toolchain/lifecycle.cppm:562-567  and  src/build/prepare.cppm:1074-1076
   for (auto dep : {"xim:glibc", "xim:linux-headers"})
       fetcher.resolve_xpkg_path(dep, /*autoInstall=*/true, &progress);

   resolve_xpkg_path requires <name>@<version>: parse_target leaves version empty when there is no @ (src/pm/package_fetcher.cppm:770-790), and :805-810 returns

   unexpected("invalid xpkg target 'xim:glibc': expected `<name>@<version>`")
   

   immediately. These loops have never installed anything. And the failure was never visible, because one call site discards the result with (void) (prepare.cppm:1075) and the other only logs at debug level (lifecycle.cppm:566). No pinned glibc/linux-headers version constants exist (src/xlings.cppm:33-38 has only patchelf/ninja/xlings/nasm), and no test covers either loop.

   So the belt-and-braces mcpp advertises for the sysroot payloads is a no-op on every platform, every version.

   4. Most probable root cause: a silent per-dep skip on the xlings side

   xlings src/core/xim/installer.cppm:906-950 — when load_package or resolve_download_resource_ fails for a dep node, it is log::warn + continue, so the node never enters plannedDownloads. Phase 2's guard (:1101) only errors for nodes that were planned, so no InstallPhase::Failed is emitted, failedCount stays 0, and cmd_install returns 0 (commands.cppm:527). The arch gate at :1063-1076 has the same shape.

   Net effect: llvm installs, elfpatch bakes the declared glibc interpreter path into bin/clang*, glibc was silently dropped, exit code 0 — precisely the reported symptom, and it explains why it only shows up on bare runners (anywhere glibc is already parked, the skip is harmless).

   Filing this separately on the xlings side; it does not block the mcpp-side work.

   5. Two more mcpp-side amplifiers worth recording

   • post_install records a degraded state as success. src/toolchain/post_install.cppm:364-390: when find_sandbox_glibc_lib() returns empty the whole patchelf branch is skipped with no warning at all (the gcc path at least warns, :358-361); fixup_clang_cfg then rewrites every bin/*.cfg with the glibc -B/-L/--dynamic-linker/-isystem lines omitted — overwriting the cfg llvm.lua would have refused to write — and the idempotency marker (.mcpp-fixup.json, kFixupRev) records glibcLib: "" as a successful fixup, so a reinstall will not retry.
   • No post-install liveness check. lifecycle.cppm:578-585 only verifies the frontend file exists; nothing runs clang++ --version. mcpp doctor (src/doctor.cppm:239-313) readelf -ds installed toolchains and warns on missing RUNPATH entries, but never inspects PT_INTERP — which is the fatal axis here. Its warning text ("its providing xim package may have been removed") is also the wrong hint when the package was never installed in the first place.

   What this changes for the issue

   The title/body attribution ("missing runtime dependency in the llvm xim package") should be read as retracted. The accurate framing is: the install chain can drop a declared dependency silently and still report success, and mcpp then has no gate that would notice the resulting toolchain cannot exec.

   The mcpp toolchain install gcc workaround remains valid for now (it parks glibc by a different route). Planned mcpp-side work — pinned versions on the sysroot safety-net with the result actually checked, a post-install --version smoke probe, PT_INTERP verification in doctor, and making the post_install skip loud instead of silent — is captured in .agents/docs/2026-07-22-v0.0.102-batch-254-261-design.md §6.
2. Sunrisepeak

   added a commit that references this issue on Jul 21, 2026

   batch: #261 windows cmdline, #257 clang depfile, #258 conditional per…

   ...

   5807e36
3. speak-agent commented on Sep 20, 2026

   

   speak-agent

   on Sep 20, 2026

   Member

   More actions

   Fixed by #660, released as 2026.9.17.2 (commit 9bc00f8).

   This issue's mechanism was that the llvm payload is patched against xim-x-glibc, but nothing installed that payload — the gcc path only appeared to work because xlings installs glibc as a dependency of the gcc toolchain, so on a bare runner clang exec'd 127 and gcc did not.

   The half that was never visible: the two loops that were meant to install it passed xim:glibc without a version, which Fetcher::resolve_xpkg_path rejects outright, and the rejection was logged at debug level and discarded. So the declared runtime was never installed by mcpp on any path; it was present only as a side effect of a toolchain install. A toolchain restored from a CI cache is not reinstalled, so a cache holding another glibc revision left the declared payload absent — and a directory scan that accepted any 2.44.x for 2.44 concealed even that until two revisions were present at once.

   The fix:

   • mcpp::toolchain::ensure_declared_runtime (src/toolchain/lifecycle.cppm:868) installs xim:glibc@<declared> through xlings and re-resolves the binding. It runs before the first fixup that consumes a C runtime — gcc and llvm payloads alike (src/build/prepare.cppm:2530, lifecycle.cppm:1078), on Linux, for the glibc provider, and only when the binding locates no payload.
   • select_glibc_payload_lib and probe look the exact version up only; payload_dir_for_version and needs_linux_sysroot_payloads are gone. A missing payload is now reported with the coordinate that provides it, instead of a 127.
   • The xlings released beside the mcpp executable became an acquisition source ahead of PATH, which is what makes this work on a machine whose PATH xlings is older than the pin.

   Criteria: unit tests GlibcPayload.* / DeclaredRuntime.*, and tests/e2e/737_the_declared_runtime_payload_is_installed.sh (cache-shaped home, offline and online legs) plus 687 leg C. Both e2e are red on 2026.9.17.1 and green on 2026.9.17.2, so they measure the change rather than the state.

   Closing.
4. speak-agent

   closed this as completedon Sep 20, 2026