Skip to content

xray: the version probe has no timeout, so a hung xray binary blocks Start and every other API call #127

Open

Description

@Free-Guy-IR

Found by code review and reproduced in a lab; never seen in production.

Summary

NewXRayCore runs xray version with exec.Command(...).Run(): no context, no timeout, no WaitDelay (backend/xray/core.go:94-99, called at :63, at dev 9ccd2638c7fe). If that process never exits, Start never returns. Because StartBackend holds the controller's c.mu for the whole build (controller/controller.go:281-283), and every REST request takes c.mu in middleware (controller/rest/middleware.go:20 ApiKey(), :70 NewRequest()), GetBaseInfo, GetSystemStats and Stop hang too, for as long as the probe hangs.

Reproduction (real node, stand-in binary)

Real node built from dev 9ccd2638c7fe, SERVICE_PROTOCOL=rest, on macOS arm64. XRAY_EXECUTABLE_PATH points to a shell script that stands in for a hung binary: on version it runs exec sleep 600; anything else goes to the real Xray 26.3.27. Then POST /start, and 2 s later GET /info, GET /stats/system and PUT /stop:

call result on dev 9ccd2638
Start no answer within the client's 60 s
GetBaseInfo, GetSystemStats, Stop sent during Start no answer within 45 s
the same three, sent after the Start client gave up no answer within 5 s each
SIGTERM to the node the process did not exit; about 60 s later I killed the stand-in's sleep and the node exited within 0.1 s

With the real Xray 26.3.27 as the executable, the same node starts in about 2 s (started=true, core_version=26.3.27).

Impact

  • While the probe hangs, the node answers nothing. The panel's Start call goes through node_bridge_py with the node's default_timeout (10 s unless an admin changed it; panel dev f92acaf51cec app/operation/node.py:427-439 sets no timeout for Xray, bridge c5632b65a10a rest.py:174 falls back to default_timeout), so the panel sees a timeout after 10 s; the node itself stays stuck after that. Found by reading the panel and bridge, not run against a real panel.
  • The process does not stop on SIGTERM while the probe hangs, because shutdown calls Disconnect, which also needs c.mu.
  • When the probe process exits, Start returns and the lock is released, so no manual restart is needed (found by reading the code; in the lab the pending SIGTERM made the node exit at that moment, so recovery on dev was not observed).

Only a binary that does not exit on version (or a wrapper that leaves a child holding stdout) triggers this. The real xray version (26.3.27) took 0.01 s in five runs here.

Affected versions

dev 9ccd2638c7fe and main v0.5.7 739eab94a934 (the probe code is the same on both).

Proposed fix

Run the probe under exec.CommandContext with a 30 s timeout (the same bound the MTProto backend uses for telemt --version, telemtCommandTimeout at backend/mtproto/process.go:29 on dev 9ccd2638) and set cmd.WaitDelay. PR #132.

Duplicate search

On 2026-11-04 a search of PasarGuard/node issues and PRs for version timeout and health restart found #50, #21, #117, #116 and #113, which are about restarts and start lifetimes, not this probe. The search was not repeated for this edit.

Activity

  1. changed the title [-]xray: the version probe has no timeout, so a hung xray binary blocks core creation until it exits[/-] [+]xray: the version probe has no timeout, so a hung xray binary blocks Start and every other API call[/+] on Nov 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions