Skip to content

feat(release): publish a portable Linux build of openshell-gateway compatible with glibc 2.35 (Ubuntu 22.04, Debian 12, RHEL 9) #1456

Closed
#1490
Closed
feat(release): publish a portable Linux build of openshell-gateway compatible with glibc 2.35 (Ubuntu 22.04, Debian 12, RHEL 9)#1456
#1490

Description

@theFong
theFong
opened on May 19, 2026

Problem Statement

OpenShell's published Linux binaries (openshell, openshell-gateway, openshell-sandbox) are dynamically linked against glibc 2.39 because CI builds them on Fedora 44. That excludes every long-term-support Linux distro shipping today:

Distro glibc Runs openshell-gateway?
Ubuntu 24.04 Noble 2.39 ✅
Ubuntu 22.04 Jammy 2.35 ❌
Ubuntu 20.04 Focal 2.31 ❌
Debian 13 Trixie 2.41 ✅
Debian 12 Bookworm 2.36 ❌
RHEL 10 2.39 ✅
RHEL 9 2.34 ❌
Amazon Linux 2023 2.34 ❌

On any of the ❌ rows the binary fails at startup with:

/usr/bin/openshell-gateway: /lib/x86_64-linux-gnu/libc.so.6: version 'GLIBC_2.38' not found (required by /usr/bin/openshell-gateway)
/usr/bin/openshell-gateway: /lib/x86_64-linux-gnu/libc.so.6: version 'GLIBC_2.39' not found (required by /usr/bin/openshell-gateway)

NVIDIA Brev supports Ubuntu 22

NVIDIA Brev provisions Ubuntu 22.04 Jammy as its default base image for every Brev Launchable. That means any one-click deployment of OpenShell from Brev — including our own brevdev/dex-ui Launchable — has to override Brev's default image with a custom imageId pointing at ubuntu-2404-lts-amd64. We can't currently rely on the operator's preferred Brev image, can't piggyback on Brev's hardened defaults, and can't help operators who already have a 22.04 fleet they'd like to install OpenShell onto.

Same pain point applies to the much larger universe of operators running Ubuntu 22.04 LTS in their own VPCs (it's the most common Linux on AWS/GCP/Azure right now), RHEL 9 shops, Debian 12 servers, etc. They all hit the GLIBC error on first install and have no documented escape hatch other than "upgrade your OS."

Reproduction:

$ docker run -it ubuntu:22.04 bash -c "
    apt-get update -y && apt-get install -y curl
    curl -fsSL https://raw.githubusercontent.com/NVIDIA/OpenShell/main/install.sh | sh
    openshell-gateway --version
  "
…
/usr/bin/openshell-gateway: /lib/x86_64-linux-gnu/libc.so.6: version 'GLIBC_2.39' not found

Proposed Design

Publish a second binary build target per release that links against an older glibc. Two reasonable variants — either works, the first is preferred:

Option A (preferred): Build on Debian Bookworm in cargo-zigbuild

cargo-zigbuild lets us cross-compile against any libc version we choose. Targeting glibc 2.31 (Ubuntu 20.04 / Debian 11 baseline) covers every distro listed above plus everything older still in LTS. The Cargo.toml release profile and Cross.toml workflow would gain a second matrix dimension:

# .github/workflows/rust-native-build.yml — sketch
strategy:
  matrix:
    target:
      - x86_64-unknown-linux-gnu          # existing; glibc 2.39
      - x86_64-unknown-linux-gnu.2.31     # new; glibc 2.31 baseline
      - aarch64-unknown-linux-gnu
      - aarch64-unknown-linux-gnu.2.31

Release artifacts: ship both, with names that make the constraint discoverable:

  • openshell-gateway-x86_64-unknown-linux-gnu.tar.gz (current; glibc 2.39+)
  • openshell-gateway-x86_64-unknown-linux-gnu-glibc2.31.tar.gz (new; glibc 2.31+)

install.sh detects the host's glibc version (ldd --version parse) and picks the right artifact automatically. The current "latest glibc" build stays the default for distros that can handle it (smaller binary, more compiler optimizations, etc.); the portable build is only used when the host can't satisfy the modern variant.

Both .deb and .rpm packages would still ship — but .deb would gain a second variant (openshell_x.y.z-1_amd64_glibc2.31.deb) for Bookworm/Jammy etc. (.rpm already targets fc44; adding a .el9.x86_64.rpm from the portable build would cover RHEL 9.)

Option B (cheaper, less flexible): MUSL static binary

We already publish openshell-aarch64-unknown-linux-musl.tar.gz for the CLI on arm64. Extending the musl matrix to x86_64 and to the gateway + sandbox binaries would give us a statically-linked artifact that runs on any Linux regardless of glibc version. Cost: musl/glibc behavior differences on edge cases (DNS resolver, threading, mmap'd files), larger binary (~25% bigger), worse perf on certain workloads.

Option A is more work but ships a binary that looks/feels identical to today's just with a lower libc floor. Option B is a one-line workflow change but introduces a runtime that's less battle-tested for daemons doing heavy syscalls (which the gateway is).

Alternatives Considered

  1. Just install via Docker. OpenShell already publishes ghcr.io/nvidia/openshell/gateway with glibc baked in. Works, but:

    • Forces operators into a container topology they may not want (especially mTLS rotation, log shipping, host-namespace mounts).
    • Means the .deb / .rpm + install.sh paths are de facto broken on the most common LTS distros, with no upstream acknowledgement.
    • The container image itself still has to pull compute_drivers=["podman"|"docker"] config + the appropriate host socket, which moves the integration problem rather than removing it.

  2. Document "Ubuntu 24.04+ required" in install.sh and exit with a friendlier error. Necessary regardless — even with this issue resolved we should detect glibc-too-old at install time — but doesn't help operators who can't change their OS.

  3. Build openshell-gateway from source on the host. A ~15-25 min cold cargo build for any operator who can't run our binary. Hostile UX, also a non-trivial host requirement (Rust toolchain + dev headers).

  4. Backport newer glibc to old distros. Brittle, defeats the LTS contract, not a real option.

Agent Investigation

This came up while building a Brev Launchable that mirrors pnpm play for brevdev/dex-ui — a one-click "spin up an OpenShell + dex-ui stack" environment.

  • Brev's default GCP base image is Ubuntu 22.04.5 LTS (Jammy Jellyfish), glibc 2.35. Confirmed by cat /etc/os-release && ldd --version on a fresh n1-standard-2 instance.
  • install.sh from main installs the .deb cleanly; openshell-gateway --version immediately fails with the GLIBC_2.39 error above.
  • We currently work around this by setting createWorkspaceRequest.imageId = "projects/ubuntu-os-cloud/global/images/family/ubuntu-2404-lts-amd64" in the Launchable JSON, which forces a 24.04 base. That works but couples our Launchable to a single OS family, and doesn't help operators using OpenShell outside of Brev.
  • I inspected releases v0.0.36 → v0.0.43 with strings openshell-gateway | grep '^GLIBC_2\.' | sort -V. Every released binary in that window references GLIBC_2.34, GLIBC_2.38, GLIBC_2.39 — pinning to an older OpenShell version does not unblock Ubuntu 22.04.
  • The CI builder image / runner config is the single point where this changes. The cargo-zigbuild route preserves the existing GNU dynamic-linking ABI while pushing the minimum glibc down — no source-code changes needed in any crate.

Acceptance Criteria

  • CI produces a openshell-gateway-x86_64-unknown-linux-gnu-glibc<MIN>.tar.gz artifact for each tagged release.
  • install.sh picks the portable artifact automatically when the host's glibc is older than the modern build's minimum.
  • A .deb variant targeting Debian Bookworm / Ubuntu Jammy ships in the release (openshell_X.Y.Z-1_amd64_glibc2.31.deb or similar).
  • Manual smoke: docker run -it ubuntu:22.04 → install.sh → openshell-gateway --version exits 0 and prints the version.

Activity

  1. johntmyers commented on May 19, 2026

    @johntmyers
    johntmyers
    on May 19, 2026
    Collaborator

    related: #1117

  2. pimlock commented on May 21, 2026

    @pimlock
    pimlock
    on May 21, 2026
    Collaborator

    I think this issue should first focus on the host-executed Linux binaries. The Python wheel work in #1117 is related, but it has a different packaging shape and should probably be discussed separately.

    The current Linux binary compatibility picture looks roughly like this:

    Component Current libc variant / floor What to do for wider compatibility
    openshell-gateway GNU/glibc, host-executed gateway binary; Docker and Podman drivers are linked into this process Lower the glibc floor here. This covers Docker/Podman gateway deployments too.
    openshell-driver-vm GNU/glibc, host-executed subprocess used by the VM runtime Lower the glibc floor here as well.
    Standalone openshell CLI Static musl Already avoids host glibc.
    openshell-sandbox supervisor Runs inside sandbox containers/pods/VM guests; Docker/Podman delivery is controlled by supervisor image/bind mount Not the main host OS glibc blocker for this issue.

    There are two main implementation options for the GNU/glibc host-executed binaries:

    1. Publish one portable GNU/glibc artifact
      Build openshell-gateway and openshell-driver-vm with a lower glibc floor, for example glibc 2.31 or glibc 2.28, and make that the default Linux artifact.

    2. Publish multiple GNU/glibc tiers
      Keep the current builder-floor artifact and add a portable artifact, then update install.sh, package naming, docs, and release validation to choose the right one.

    My preference is to start with option 1 if validation passes. For Rust, the cost of targeting an older glibc baseline should usually be smaller than it would be for a C/C++ project, and a single Linux artifact keeps the install and support story much simpler.

    Some useful prior art:

    Project / tool type Linux strategy Relevance
    ripgrep and bat Publish both GNU and musl Linux assets Shows the dual-track model: GNU for normal Linux integration, musl as a portable fallback.
    rust-analyzer Publishes GNU Linux assets and also a musl Linux asset Good cautionary example that GNU builds need a deliberate floor. Their changelog explicitly notes that moving the build to Ubuntu 20.04 raised the minimum supported glibc to 2.31.
    Rust toolchain (rustc / cargo) Official Linux GNU targets document a glibc 2.17+ baseline Shows the controlled GNU baseline model for core tooling that needs broad enterprise compatibility.

    For OpenShell, I think the best first step for host-executed runtime binaries is the controlled GNU baseline model rather than a full musl switch. openshell-gateway is a daemon with more host integration surface than the CLI, so keeping GNU/glibc behavior close to today while lowering the required glibc version seems like the lower-risk compatibility fix. The standalone openshell CLI already follows the other common Rust distribution strategy by publishing a static musl artifact.

    Suggested acceptance criteria:

    • CI builds openshell-gateway and openshell-driver-vm with an explicit glibc floor.
    • CI verifies the generated binaries do not reference symbols above that floor.
    • Smoke tests run on representative older distros, at minimum Ubuntu 22.04 and ideally RHEL 9 / Amazon Linux 2023. If we choose glibc 2.28, also validate RHEL 8.
    • install.sh and package publishing consume the portable artifact by default, unless we decide to support multiple glibc tiers.
  3. pimlock
    closed this as completedin #1490on May 22, 2026

  4. pimlock commented on May 22, 2026

    @pimlock
    pimlock
    on May 22, 2026
    Collaborator

    @theFong Thank you for raising this issue! The fix for this was just merged. The new tagged release with this change will be released on Monday morning.

    We went with lowering of the overall floor, rather than publishing a separate version. This makes it easier to maintain and doesn't complicate our release process.

  5. pimlock commented on May 25, 2026

    @pimlock
    pimlock
    on May 25, 2026
    Collaborator

    Here's a breakdown of differences between 2.39 and 2.31. Note that none of this should result in any change in how our binary works.

    Differences:

    • Before, many pthread_* and dl* symbols were versioned as GLIBC_2.34.
    • After, those same kinds of symbols resolve to older versions like GLIBC_2.2.5 or GLIBC_2.12.
    • That is because glibc 2.34 merged libpthread, libdl, libutil, etc. into libc. Building on a newer glibc often records those symbols as GLIBC_2.34; building against the older bounded sysroot records older compatible symbol versions.

    Examples:

    Before: pthread_create@GLIBC_2.34
After:  pthread_create@GLIBC_2.2.5

Before: dlopen@GLIBC_2.34
After:  dlopen@GLIBC_2.2.5

Before: pthread_setname_np@GLIBC_2.34
After:  pthread_setname_np@GLIBC_2.12

    The stat64 symbols also changed in an expected compatibility-oriented way:

    Before: stat64@GLIBC_2.33
Before: fstat64@GLIBC_2.33
Before: lstat64@GLIBC_2.33

After:  __xstat64@GLIBC_2.2.5
After:  __fxstat64@GLIBC_2.2.5
After:  __lxstat64@GLIBC_2.2.5

    There was also a modern C library parsing symbol removed:

    Before: __isoc23_sscanf@GLIBC_2.38
After:  __isoc99_sscanf@GLIBC_2.7

    And two newer optional process-related symbols disappeared:

    Before: pidfd_getpid@GLIBC_2.39
Before: pidfd_spawnp@GLIBC_2.39
After:  absent

    Dynamic dependencies changed too:

    Before:
  libgcc_s.so.1
  libm.so.6
  libc.so.6
  ld-linux-x86-64.so.2

After:
  libm.so.6
  libpthread.so.0
  libc.so.6
  libdl.so.2
  ld-linux-x86-64.so.2

    That also tracks with the older glibc compatibility target: separate libpthread/libdl dependencies are normal for older glibc environments.

    Bottom line: the post-change binary did not lose “features” in an obvious way; it records older ABI-compatible symbol versions. The most important result is that the highest required glibc symbol dropped from GLIBC_2.39 to GLIBC_2.30, so it fits under the 2.31 floor.

  6. ACodingfreak commented on May 25, 2026

    @ACodingfreak
    ACodingfreak
    on May 25, 2026

    is this fix released ? I tried below command and it still fails

    docker run -it ubuntu:22.04 bash -c "
    apt-get update -y && apt-get install -y curl
    curl -fsSL https://raw.githubusercontent.com/NVIDIA/OpenShell/main/install.sh | sh
    openshell-gateway --version
  "

  7. pimlock commented on May 25, 2026

    @pimlock
    pimlock
    on May 25, 2026
    Collaborator

    @ACodingfreak today's release was delayed, so the fix is not out yet. I'm working on getting it fixed and it should be out in the next 30 mins.

  8. pimlock commented on May 25, 2026

    @pimlock
    pimlock
    on May 25, 2026
    Collaborator

    @ACodingfreak new version (0.0.48) is out now and the command you referenced works now (thanks for including it!).

  9. ACodingfreak commented on May 25, 2026

    @ACodingfreak
    ACodingfreak
    on May 25, 2026

    Thanks @pimlock

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions