Repository navigation
fix(rigs): one env-overridable device-lock budget for all 13 acquire sites - #215
Merged
Merged
Conversation
Every rig hardcoded its own DeviceLock acquire timeout with no way to
raise it from the environment. That is survivable only because the
harness's acquire is queue-aware: behind one live, progressing holder
the deadline is re-armed on every poll and `timeout` is never consulted.
Measured with no hardware (fake host, private lock dir):
acquire(timeout=2.0) behind an 8 s hold returned True after 8.0 s, and
False after 2.0 s with progress_window=None.
It stops being survivable when the harness stops extending. That happens
after the FOURTH change of holder identity -- not the third.
_MAX_HOLDER_HANDOFFS is 3, but acquire extends while
`handoffs <= _MAX_HOLDER_HANDOFFS` and increments on a change, so three
changes still extend for ever; the harness's own WARNING ("the holder
changed 4 times") agrees with its code against its own docstring. An
earlier draft of this commit inherited the docstring's three. Measured
by driving a scripted identity sequence, no hardware: 0/1/2/3 changes
then a stable holder still waited past the observation window, 4 and 5
returned False at `timeout`. A doc issue for the harness is drafted.
Several lanes cycling one U64E WOULD produce that within seconds, after
which the hardcoded number is the whole budget. The mechanism is
lab-measured; no such timeout has been captured from a real run here,
and the prose says so rather than implying field evidence.
So the helper gains the budget, in one place:
* C64_DEVICE_LOCK_TIMEOUT, default 1800 s (30 min). Justified against
both failure modes in the module docstring. It is also what the
harness itself recommends for long device holds
(backends/unified_manager.py:127-130) and what the CI path in this
file already used, so this unifies rather than inflates.
* A malformed value is fatal (LockTimeoutConfigError), never a silent
fall back to a budget nobody asked for. `VAR=` falls back with a
notice, because that is the shell's own spelling of "unset".
* progress_window is passed through explicitly, so the indefinite wait
behind one healthy long holder -- the part that already worked -- is
a decision this repo owns rather than a default it inherits.
* A stderr line every 30 s names the elapsed wait, the budget, the
holder PID, the LOCKFILE AGE (flagged STALE past the progress
window) and the queue depth. Age is the field that separates a
healthy long run from a wedged holder; without it that verdict
arrives only in the final DeviceLockTimeout, up to a whole budget
later, which is the time-to-diagnosis cost of the larger default.
Never stdout: rig stdout is parsed.
* DeviceLockTimeout propagates unchanged, so the rigs' existing
diagnostics (holder liveness, lockfile age, REST reachability) are
untouched.
The CI path now shares that budget and carries the harness's diagnostic
text in QueueSaturatedError.reason instead of "timeout exhausted", and
its stale TODO is retired: the harness grew queue_depth, so the probe
that returned None now returns a number.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Thirteen, not the two the defect report named:
bare-bool acquire() + `if not` (diagnostics discarded, bare False):
boot_check.py:272, phase2_check.py:119, phase3_tcp_echo.py:387,
rig_http_local.py:377, rig_http_live.py:256,
bench_ecdsa_u64e.py:422 (60 s), rig_https_banner.py:89 (120 s)
acquire_or_raise with a literal:
rig_https_local.py:1499, rig_https_live.py:398,
rig_https_wiki.py:599 (120 s), rig_https_bad_finished.py:532
(300 s), tests/rig_ip65_rrnet_hw.py:745 (LOCK_TIMEOUT_S = 120.0)
UnifiedManager keyword, one layer up:
tools/test_ecdsa_p384_kat.py:706 (lock_timeout=120.0)
The last one is neither in a rig directory nor a DeviceLock call, so the
first version of the AST guard could not see it and the claim "the repo
has one number" was false. _LockedU64Manager takes the same lock by the
same acquire_or_raise (backends/unified_manager.py:393), so it now gets
lock_timeout_s(). The P-384 lane is dormant, so this is a
correctness-of-claim fix more than an operational one.
The twelve direct sites call acquire_device_lock(lock), so the budget is
C64_DEVICE_LOCK_TIMEOUT everywhere and a wait says so on stderr. The
seven bare-bool sites gain the holder/liveness/age/reachability
diagnostics they never had; a malformed budget is a setup error (2)
raised before the device is touched, distinct from the timeout exit each
script already used.
tests/rig_ip65_rrnet_hw.py and tools/test_ecdsa_p384_kat.py reach the
helper by adding tools/uci to sys.path next to the insert each already
had, rather than growing a second copy of the budget.
Not touched, deliberately: rig_https_live.py:411 and rig_https_wiki.py:611
print lock.read_info() on kickoff, which names whoever held the lock LAST
(harness #194) -- after a completed run that is a dead PID. Both lines
are adjacent to this diff and both are pre-existing; fixing them belongs
in its own change.
Every modified module still imports cleanly, and the two offline
selfchecks that exist (rig_https_live --selfcheck, rig_https_wiki
--selfcheck) still pass. No hardware was touched: the U64E was in use by
another lane for the whole of this work.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
tools/test_device_lock_timeout.py, in the shape of test_reu_preflight.py and for the same reason: the thing being guarded protects a hardware run, and its only failure mode is to stop guarding, which no hardware run can observe. The DeviceLock is faked (the boundary case uses a real one with two internals overridden); no hardware, no VICE, no network, no build, ~2.6 s. Three independent groups. Behaviour: the env override reaches DeviceLock.acquire, its absence yields the documented default, an explicit caller argument beats the env, a malformed value raises before the device is touched, an empty one falls back with a notice, progress_window is never disabled, progress goes to stderr and never to stdout, the progress line carries the lockfile age and flags it STALE past the progress window, and DeviceLockTimeout propagates with the holder PID intact. Structure: an AST sweep over tools/uci/*.py and tests/rig_*.py -- nothing outside the helper may call acquire/acquire_or_raise, and a module that constructs a DeviceLock must import acquire_device_lock -- plus a third sweep over a wider glob forbidding a literal `lock_timeout=` anywhere, which is the only check that could have seen tools/test_ecdsa_p384_kat.py's thirteenth budget. The handoff boundary: four identity changes, not three. It had no coverage in either direction, here or upstream, which is how the wrong number (read off the harness's docstring) reached four files under a green suite. Real DeviceLock, scripted identity sequence, no hardware. Red-green. Against master's helper the module fails at import (LOCK_TIMEOUT_ENV does not exist). With the new helper but master's rigs, the structural cases fail and enumerate all 13 offending call sites with line numbers. Mutating the boundary assertion to claim three fails with "the boundary moved to three and every 'fourth' in the docs is now wrong". With everything in place, 18/18 pass. Registered in pytest.ini (test_pytest_boundary.py requires it: a pure-logic tools/test_*.py absent from testpaths fails that guard). It defines no module-level run_tests(), so run_all_tests.py's SUITE_ORDER does not apply and test_runner_coverage.py stays green -- same as test_reu_preflight.py. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both say the same four things, because getting any of them wrong is how the next lane misreads a queue: the budget and its default; that it is NOT how long you may queue behind a healthy run (the harness extends for that, and the budget only bounds the handoff-chain and wedged-holder cases); that the chain boundary is the FOURTH identity change, with the harness's contrary docstring named so nobody "corrects" it back; and that a malformed value is fatal rather than a silent default. The field claim is stated as what the mechanism WOULD produce, not as something observed: it was measured in a lab, and no DeviceLockTimeout from a real run here has been captured. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
acquire_with_queue_budget has zero callers in this repo and no test coverage, yet this branch changed its semantics: it now inherits the 1800 s interactive default, and LockTimeoutConfigError escapes it un-wrapped. Both are deliberate and both are invisible from here, so they are written down where an adopter will read them -- the budget must be under the runner's tick or the bot stacks its own waiters until max_queue_depth self-trips, and a malformed budget is a misconfiguration to fix rather than contention to retry. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both from the adversary's confirmation pass; test-only, no runtime code touched, and added on top rather than folded in because the previous range is now anchored -- the adversary's verdict names 761b753. That is the same criterion the PR body gives for having rebuilt the earlier commits: rewrite while nothing points at the SHAs, append once something does. * A *signature* change to DeviceLock._holder_progress / _try_acquire_once was caught only by the second boundary assertion, which would have reported "the boundary moved to three" -- a confident, specific and wrong diagnosis of a harness shape change. A cheap probe call now converts a TypeError into a message naming the real cause. (A rename was already caught by the hasattr checks.) * The still-extending arm leaked a daemon thread per call, polling at 10 Hz for the life of the process. It now grants shortly after the observation window and joins, with an assertion that the thread really ended -- a probe thread that outlives its acquire means the overrides no longer control it, so the result would be meaningless anyway. heartbeat_interval=None stops the grant from starting a heartbeat thread that sleeps a full interval before noticing the lockfile is not there. Measured: active_count 1 -> 1 across the test, was 1 -> 3. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
README was untouched by this branch and said nothing about queueing for the U64E beyond "set U64_HOST explicitly", so the one knob a contended run needs was documented in tools/uci/README.md and CLAUDE.md only. Adds, to the UCI hardware section: C64_DEVICE_LOCK_TIMEOUT (default 1800 s) as the single acquire budget every rig now shares through acquire_device_lock(), the malformed-value and stderr-progress behaviour, and — the part that actually surprises people — that the budget does not bound queueing behind a healthy holder, only the waits the harness refuses to extend. The handoff-chain boundary is stated as the FOURTH change of holder identity, matching _MAX_HOLDER_HANDOFFS = 3 with extension surviving `handoffs <= 3`, and pinned by test_the_handoff_boundary_is_four_changes_not_three in tools/test_device_lock_timeout.py rather than by the harness docstring, which is one out. Cites issue #212. Also adds the variable to rig_https_local.py's environment list. No claim here is a device measurement: the mechanism is lab-measured in the new suite, and no timeout from a real run has been captured. No hardware was touched for this commit. Documentation only; no PRG hash moves. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
032e22d said the harness's own docstring was "one out" on the handoff boundary. That was true when written and is not now: c64-test-harness#232 is CLOSED, and the installed editable checkout carries a deliberately relational, literal-free docstring (device_lock.py:466-475) — there is no number in it to be one out. The boundary claim itself is unchanged and still code-verified: four identity changes, from _MAX_HOLDER_HANDOFFS = 3 with extension surviving `handoffs <= 3` (device_lock.py:583). The README now points at tools/test_device_lock_timeout.py as where to read it, and cites #232 as fixed — including the reason there is no upstream literal to compare against any more. Documentation only; no PRG hash moves. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
JC-000
force-pushed
the
fix/uci-devicelock-timeout
branch
from
September 7, 2026 20:36
b06e25b to
3216e7d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thirteen call sites hardcode a
DeviceLockacquire budget (60 s ×6, 120 s ×5, 300 s ×1, plus oneUnifiedManager(lock_timeout=120.0)) with no environment override, so a rig cannot be told to queue behind a legitimate long hold. Found by a lane queued behind an 80-minute RR-Net run, which had to polldevice_lock_holder()in a side loop and race for the gap.The mechanism is not the obvious one, and the fix depends on that
DeviceLock.acquireis queue-aware: behind one live, progressing holder the deadline is re-armed every poll andtimeoutis never consulted. Measured, no hardware:So a 120 s literal does outlast an 80-minute run behind one healthy holder. It bites only where the harness refuses to extend: a wedged holder, or a handoff chain — after the fourth change of holder identity, extension stops for the rest of that acquire. With several lanes cycling one U64E that is reached in seconds.
Four, not three:
_MAX_HOLDER_HANDOFFSis 3 but extension surviveshandoffs <= 3. Measured — 0/1/2/3 changes still wait indefinitely; 4 and 5 return attimeout. The upstream docstring said "after_MAX_HOLDER_HANDOFFSidentity changes", one out, and that phrasing propagated into five files here before an adversarial pass caught it. Raised asJC-000/c64-test-harness#232and already fixed upstream; the boundary is now pinned locally bytest_the_handoff_boundary_is_four_changes_not_threerather than by any upstream literal.So this is not "we raised the timeout"
The literal was unreachable in the healthy case and load-bearing only in the failure case, so it became one env-overridable number chosen for the failure case. Measured:
Invisible to every run not already in a failure case.
The cost, stated rather than omitted: against a genuinely wedged holder a rig now waits up to 30 minutes before reporting, where it previously reported in 2. That is a real regression in time-to-diagnosis, accepted deliberately, and mitigated by the progress line below.
C64_DEVICE_LOCK_TIMEOUT=120restores the old behaviour.Three supports for 1800 s beyond preference: the harness itself recommends
lock_timeout=1800.0for long device holds (unified_manager.py:130,:464); this repo's CI path already used 1800, so the change unifies rather than inflates; and a dead holder drops its flock and the waiter proceeds, so the "wedged" case is nearly always alive-but-heartbeat-stale — a false wedge that 120 s fails on wrongly.Why the staleness threshold is the harness's own
The progress line flags lockfile age against
PROGRESS_WINDOW_S— the same 60 s windowacquireuses to decide whether to keep extending, against a 15 s heartbeat. So it cannot cry wedge at a holder the harness is still extending behind: the flag flips exactly when the harness itself stops treating that holder as progressing. Any other threshold reintroduces the disagreement.That turns the 30-minute wedge report back into a 30-second one.
Design decisions worth reviewing explicitly
30m,2 min,0,inf, negatives all raise. A typo silently reinstating 1800 s would fail a lane half an hour later against a budget nobody asked for.VAR=(empty) falls back with a notice — the shell's own spelling of "unset".progress_windowis passed explicitly and neverNone, pinned by a test — disabling it restores the legacy hard timeout and destroys the indefinite wait behind a healthy holder.DeviceLockTimeoutpropagates identity-intact, so every rig's existingexceptarm and its holder/liveness/REST diagnostics still work, and each rig keeps its old exit code.lock_timeout=sweep, not a widenedRIG_GLOBS. Widening would flag 33 call sites, 32 of them legitimatemgr.acquire()on VICE managers. This is a correction to the reviewer's own suggested fix, adopted at their request.Red-green
18 cases, hardware-free with a fake host and private lock dirs. Every one negative-tested: seven behavioural mutations of the helper (progress_window→None, malformed→silent default, accept 0/negative, progress→stdout, swallow
DeviceLockTimeout, budget resolved after acquire, empty env silent) each go red in the right case. The boundary test fails loudly if the harness renames either internal.18/18standalone and under pytest;41 passedacross the suite plus the three guards; all 13 modified modules compile and import; both rig selfchecks pass. Noc64-test-harnessfile modified. No device touched at any point.Related
JC-000/c64-https#212(this defect),JC-000/c64-test-harness#232(docstring off-by-one, fixed),#233(env default + progress callback upstream — we are not blocking on it).🤖 Generated with Claude Code