[rush] rushd: fix the bugs found by dogfooding the Rush daemon (stacked on #6046) by TheLarkInn · Pull Request #6102 · microsoft/rushstack

Sean Larkin (TheLarkInn) · 2026-09-30T05:18:16Z

Stacked on #6046, so the base is thelarkinn-dogfood-rush-daemon. It stays a draft until it's reviewed. The bugs that aren't fixed yet are listed below, with where each one stands.

Summary

This PR fixes the Rush daemon (rushd) bugs found by an overnight dogfooding run in a large monorepo. Agents built and tested each fix, and a second agent reproduced the result before it landed.

250 commits, one per integration step, in landing order. Each commit's tree equals that step's tree on the integration branch. The last commit's tree is identical to the integration branch's tip.
Each commit message lists the original commits folded into it. From step 7 on, it also says why the step exists and who confirmed it. In those messages, and in some test titles and comments, "board N" and "task N" are ids in the run's own tracker. They aren't GitHub references.
The first commit below the fixes, 530c72f, merges main into [rush] Dogfood the Rush daemon in rushstack: allow command-scoped plugins, add snapshot workflow and guide #6046's branch. Its 15 main commits appear in this PR's diff until [rush] Dogfood the Rush daemon in rushstack: allow command-scoped plugins, add snapshot workflow and guide #6046 takes the same merge. To see only this PR's changes, use 530c72f9ee...thelarkinn/rushd-swarm-bug-fixes.

Bugs (166 reported)

Check off each bug as you review it. Each link is a commit that fixes the bug.

	count
Fixed in this PR	130
Fixed, with a known residual	4
Not fixed yet	3
Covered by guidance, with no code change	1
Fixed outside rushstack, in the dogfood repo's own plugins, tools, scripts and config	22
In the run's own tooling and the dogfood repo's telemetry dashboard	6

Fixed in this PR (130)

Fixed, with a known residual (4)

LockFile can grant the repository lock to multiple processes. _{(board 302)} — 6a05b01dee
_{Residual: when old and new Rush versions share a repository, an old and a new process that create their lock files in the same millisecond can leave the lock with no holder. Every command in that tie can then fail with "Another Rush command is already running in this repository." although none holds the lock, and the next command gets it. That needs a release note, not code.}
LockFile can treat a held lock as stale when contenders use different TZ or locale settings. _{(board 395)} — 6a05b01dee
_{Residual: when old and new Rush versions share a repository and use different TZ or locale settings, an old process can still treat a lock that a new process holds as stale, delete it and take the lock. That needs a release note, not code.}
Sparse-checkout or assume-unchanged paths staged but missing on disk can make repo-state hashing fail. _{(board 3335, board 3728)} — eea0048cdc, 832fe1294d
_{Step 248 hashes the paths that Git lists but can't read from disk, and step 249 signs an input link that leads to no regular file by the link itself. So none of the three triggers fails any more: a sparse-checkout or assume-unchanged path that's missing on disk, an untracked link that dangles, and one that points at a folder. Residual: a file removed from disk that the index marks skip-worktree or assume-unchanged still costs the first build's cache write, as a row below says.}
The build cache can skip a project's first write, and the daemon then marks the result unverifiable so the next request rebuilds it, when an input that Git lists is missing on disk (an untracked dangling symlink, or a removed file that's still in the index) and its folder changed in the 2 s before the inputs snapshot. A project's first build always changes its own folder, because it creates rush-logs/. _{(board 9270)} — eea0048cdc, 832fe1294d
_{Introduced by an earlier commit in this PR, 9f5b48595f, so it isn't in the base branch. Step 249 fixes the link form: an input link that leads to no regular file is signed by the link itself, so it no longer blocks the write. Residual: a file removed from disk that the index still marks skip-worktree or assume-unchanged still blocks the first build's cache write, and a later build writes the entry. No wrong output, only a lost write and an extra rebuild.}

Not fixed yet (3): 1 in progress and 2 held back for a missing test

heft's copy-files watcher can take its watch baseline just after a write from the same pass, so the write counts as a change and watch mode loops. _{(board 2458)}
_{The dogfood repo works around it. A fix is written, and it passed every second-agent check but one: its tests don't catch a weaker version of the fix. A test fix is drafted but wasn't checked in time, so the fix isn't in this PR.}
In a linked Git worktree, rush install writes the wrappers for common/git-hooks into the worktree's private Git folder (.git/worktrees/<name>/hooks). Git never reads hooks from there. So a commit in a linked worktree runs no Rush hooks or, if the main worktree has run rush install, runs the main worktree's scripts. Git.getHooksFolder() uses the worktree's Git folder, while getIsHooksPathDefaultAsync() correctly compares against the shared one. _{(board 9425, board 9426)}
_{In the base since [rush] Properly handle Git worktrees #994, which fixed the crash in [rush] Git hooks feature causes Rush commands to fail in worktrees #990 by using the worktree's Git folder. Not a daemon bug; the run found it because its agents commit from linked worktrees. Reproduced with Git 2.55.0. The fix needs a design first: the shared hooks folder serves every worktree, and each wrapper finds its script by a path relative to the hooks folder, which would then point into whichever worktree ran the install. Not fixed yet.}
Sealing a build cache entry clones the operation's output files on Node's shared thread pool, with up to 16 clones in flight, so slow clones can hold every pool thread, and every other asynchronous file system call in rushd waits behind them. The 500 ms wait limits an operation's own wait for its seal, not this tail. _{(board 7532)}
_{Introduced by an earlier commit in this PR, 48930441c7, so it isn't in the base branch. Measured with 16 fragmented files: a small file's write, rename and stat, started 500 ms into the clones, took 1.2 s with 16 clones in flight and 1 ms with 2. A fix is written, but its check for output files that change while they're cloned isn't tested yet on the path rushd uses, so it isn't in this PR.}

Covered by guidance, with no code change (1)

Output-writing scripts can break concurrent builds. _{(board 237)}

Fixed outside rushstack: the dogfood repo's own plugins, tools, scripts and config (22)

The repo telemetry plugin can clear daemon timers. _{(board 140)}
The repo telemetry plugin can add an interval per daemon iteration and never clear it. _{(board 323 H9)}
Telemetry median calculations can sort values lexicographically. _{(board 28)}
The dogfood repo's agent-workflow validator can build only touched consumers and miss stale dependency outputs after a merge. _{(board 943)}
Telemetry upload without a timeout can hang a native build at the sending step. _{(board 1621)}
rush-client size-audit can fail when plugins access private command-line internals. _{(board 1654)}
The repo fstrace plugin can keep stale fstrace metadata in cache entries. _{(board 2084)}
The dogfood repo's tool proxy can ignore the telemetry opt-out and upload one event per tool call. _{(board 2223)}
With the repo fstrace plugin on, plain build or rebuild can skip build-cache reads and writes. _{(board 2278)}
A webpack plugin can emit DLL proxy chunks in nondeterministic order. _{(board 2583)}
The repo fstrace plugin can restore the same cache entry on every build when the entry has no usable fstrace state. _{(board 2544, board 2549)}
The repo fstrace plugin can drop dependency reads under node_modules and miss bundle inputs. _{(board 2905)}
The repo fstrace restored-state record can ignore build-cache salt and keep skipping after a salt change. _{(board 2904 F1)}
The repo fstrace plugin can lose skip state after daemon invalidation and rebuild unnecessarily. _{(board 2910)}
One CLI can receive verbose legacy Rush logs instead of concise agent-mode output. _{(board 3072)}
The repo fstrace LKG metadata can ignore build-cache version or salt and trust stale entries. _{(board 3000)}
A watch-mode localization plugin can fail to generate translations on the first pass or after recovery. _{(board 3173)}
The repo fstrace plugin can run operations with a stale copy of Rush's environment filtering, so RUSH_DAEMON* variables reach their child processes. _{(board 5686, task 289)}
A checked-in generated script can stay stale after a localization plugin change. _{(board 5756, task 293)}
The dogfood repo's agent-workflow format check can pass a file that needs formatting, because it reads counts but not stderr. _{(board 5158)}
The dogfood repo's agent-workflow validator can fail its format stage on docs-only or tooling-only changes. _{(board 5697 P14)}
The repo fstrace plugin can skip a consumer with stale inputs after its dependency is built alone. _{(board 2216)}

In the run's own tooling and the dogfood repo's telemetry dashboard (6)

The swarm install script can skip writing resolver-cache metadata. _{(board 251)}
The dogfood repo's build-telemetry dashboard can inflate agent-session counts by hundreds of times (the join matches a placeholder session id). _{(board 30)}
The run's shard files can split paths on whitespace, so guidance files with spaces in their names break across shards. _{(board 15)}
The run's install-slot lock cleanup can delete live Rush tool-folder locks, because it compares process start times in two different formats. _{(board 394)}
The swarm's rush-client wrapper can drop a caller's telemetry opt-out, because it runs each call under env -i. _{(board 1391, board 1382)}
On the dogfood repo's build-telemetry dashboard, the OKR tiles define "hot" two ways, leave Rush's boot out of cold time, and subtract install twice. _{(board 110)}

Notes for review

ODSP_TELEMETRY_TAG is added to the daemon's environment lists in 1b71530939. It's specific to the dogfood repo's telemetry. Generalize it or drop it before this merges.
Some test names and comments, and the messages of steps 6, 50, 69 and 75, name the dogfood repository.
The run's ids in code. Some test titles and comments cite the run's tracker as "task N". b55d11a350 drops the bare number-sign references to board posts from 13 lines in tests and one doc comment, because they read like rushstack issue or PR numbers. The comment that cites [rush] rushd: every rushx-client script takes exclusive workspace admission (serializing scripts and failing builds), and rushx-client hangs when a script leaves a background process holding its output #6085 is a real reference and stays.
The LockFile fixes need a release note for the two LockFile residuals above. While old and new Rush versions share a repository, an old process can take a lock that a new process holds if their TZ or locale settings differ, and lock files that tie can leave no holder, so every command in the tie can fail with "Another Rush command is already running in this repository." Both end once every Rush process that uses the repository has the fix.
Still to come. The bugs listed as not fixed yet have fixes or designs in progress. Any that land in this PR come as new commits on top of this branch, and no existing commit is rewritten.
A step index, listing each commit with its step number and subject, is in the first comment.

…ed under common/config (and 3 more) Swarm integration step 1; original commit 433c87f24f (merge of swarm/r05 at 190d77ea17). Commits folded into this step (4): - 40b5b24d72 [rush-lib] Workspace fingerprint: ignore build state of projects nested under common/config - 1880e7b2b7 [rush-daemon] Warm set: keep resource-free retained results under memory pressure and idle expiry - 07e908d32d [rush-daemon] Admission: don't spend a default wait budget behind a graph load or across an admitted request's execution - 190d77ea17 [rush-lib] Move the nested common/config fingerprint test to the end of its suite Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c

…ce per project (and 1 more) Swarm integration step 2; original commit 8553b2be55 (merge of swarm/r07 at d94054e04d). Commits folded into this step (2): - 57dd6a4b45 [rush-daemon] Warm set: read status once per maintenance pass, not once per project - d94054e04d [rush-lib] Share rig profile loads across projects in the uncached rush-project.json load Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c

…nments (and 3 more) Swarm integration step 3; original commit 3b2d24277d (merge of swarm/r06 at 6c10e241a4). Commits folded into this step (4): - 803345396b [rush-lib] Keep RUSH_DAEMON* variables out of lifecycle script environments - b49d8b10cb [rush-daemon] Keep one daemon across per-session environment differences - 36892176cb [rush-daemon-transport] Reap detached operation groups of a daemon that died uncleanly - 6c10e241a4 [rush-daemon] Run each operation with its requester's per-session environment Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c

…tible plugins (and 1 more) Swarm integration step 4; original commit 1e786a4252 (merge of swarm/r01 at 162ff05eaf). Commits folded into this step (2): - 5fe505bf1a [rush-lib] Let daemon engines serve builds with declared daemon-compatible plugins - 162ff05eaf [rush-lib] Walk each plugin package once in the daemon runtime fingerprint Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c

…ing client needs it Swarm integration step 5; original commit 02f83a99ae (merge of swarm/r05 at bd7ac88caa). Refs: board 207. Commits folded into this step (1): - bd7ac88caa [rush-daemon] Abort a shared iteration's leftover work once no remaining client needs it (board 207) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c

Swarm integration step 6; original commit 519ae33c8d. Refs: task 50, board 115. ShellOperationRunner runs a phase's `:incremental` script whenever the operation has a last state. Native `rush build` executes one iteration, so only watch mode ever had one, and watch mode turns cache writes off. rushd keeps one operation graph across requests, so from its second request on a daemon `build` ran `_phase:build:incremental` (in odsp-web, `heft run --only build --` without --clean) with cache writes on. Outputs of a deleted or renamed source file stayed in lib/ and were cached under the key native Rush computes for the clean build, so later native builds restored them (t04 board 115, board 450). ShellOperationRunnerPlugin now sets the incremental command only in watch mode. Every other command runs the initial script, exactly as a single `rush build` does, which is option (a) of task 50. Native `rush build` is unchanged (it never has a last state) and watch mode keeps `:incremental` with writes off. Tests: a rush-lib unit test runs a repeated operation with and without watch mode, and a daemon integration test runs three warm builds of a project with an `:incremental` script. Before the fix the second daemon build logged "Invoking (incremental): node build.cjs --incremental" followed by "Successfully set cache entry". (cherry picked from commit e382d2ac80f881451b7d527ea9a97e3037880923) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c

Swarm integration step 7; original commit f7ac88dc9d (merge of swarm/r07 at 55a16f14ef). Refs: F1 coalescer, board 555. GATE OK ch01 board 930 (tree 41b2227). CONFIRMED by ch03 board 681, t06 board 701/board 749, t01 board 861. Only 55a16f14ef; b6692d09a5 (task 74) is not gated. Commits folded into this step (1): - 55a16f14ef [rush-daemon] Share input captures between concurrent requests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c

… snapshot is taken Swarm integration step 8; original commit bc03023091 (merge of swarm/r06 at 29f86b181c). Refs: task 77. ch01 GATE OK board 1019 (tree 80c9659); claim board 698, CONFIRMED by o04 board 738, o03 board 742, o05 board 748. Commits folded into this step (2): - a644819daf [rush-lib] Don't write a build cache entry for inputs saved while the snapshot was taken - 29f86b181c [rush-lib] Make the parameter identity test independent of the host's RUSH_PARALLELISM Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c

… join that batch Swarm integration step 9; original commit d0ed461613 (merge of swarm/r05 at 199c69347c). Refs: board 371, task 72. ch01 GATE OK board 1152 (gated tree 5c2b6b6). Commits folded into this step (1): - 199c69347c [rush-daemon] Don't let a request received during a batch's reconcile join that batch (board 371) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c

…ait timeout Swarm integration step 10; original commit 908dcc1f27 (merge of swarm/r05 at 868e4b9299). Refs: task 7 part 3, board 891. ch01 GATE OK board 1196 (gated tree 55698be). Commits folded into this step (1): - 868e4b9299 [rush-daemon] Admission: another request's graph load doesn't spend any finite wait timeout Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c

…rify untrusted ones Swarm integration step 11; original commit 4456f17460 (merge of swarm/r07 at 53654af37c). Refs: board 401, task 74, board 982, board 1024. ch01 GATE OK board 1197 (gated tree 3014a03). Commits folded into this step (3): - b6692d09a5 [rush-lib] Keep cache trust for unselected retained results and re-verify untrusted ones - 492126a051 [rush-lib] Re-verify retained results built against unverified dependency outputs without the build cache - 53654af37c [rush-daemon] Test that input captures are shared only between equal fingerprint environments Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c

…ep the daemon alive when git hash-object fails Swarm integration step 12; original commit cc5c6aab7e (merge of swarm/r01 at 575cbf5698). Refs: board 601, task 80, board 899. ch01 GATE OK board 1272 and board 1273 (gated tree b691997). Commits folded into this step (3): - c11ee10320 [rush-cli-client] Show engine and plugin warnings in agent output - c77aa1dbef [package-deps-hash] Handle a git hash-object failure while ls-files and status still run - 575cbf5698 [package-deps-hash] Name the git command that failed in the error message Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c

…and own-start memo Swarm integration step 13; original commit 0347cd0dbb. Refs: tasks 67, D and 62. ch01 GATE OK board 1389; o05 board 1360 (task 62 on ch01-t62: 32/32 served in 3 of 3 bursts); t01 board 1106/board 1186. Commits folded into this step (6): - 3dfb75ec86 [node-core-library] Don't grant a LockFile twice when lockfile birthtimes tie - 074f207217 [node-core-library] Keep the LockFile of a running process with another time zone or locale - 4e43a1ef8f [node-core-library] Don't set dirtyWhenAcquired when the holder releases the lock during a check - ad9c1211b4 [node-core-library] Read the start times of other LockFile processes from /proc on Linux - 698af4ef92 [node-core-library] Run ps for the start time of the current LockFile process only once on Linux - 9249e85d63 [rush-client-core] Test clients that start the daemon at the same time Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c

… it was received Swarm integration step 14; original commit f2ec9ffc26. ch01 GATE OK board 1430 (tree ae49a66 with o07's stack); t06 CONFIRMED board 1365; o05 board 1412 (cold split gone in auto-start mode). Commits folded into this step (1): - a4de3fe5fe [rush-daemon] Share running input captures that started after a request was received Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c

…ation cannot load Swarm integration step 15; original commit 69f30e84ab (merge of swarm/r01-t70 at c1b5c95adb). Scope: task 70. ch01 GATE OK board 1522 (tree 3a564b4; WorkspaceRequestLifecycle.ts resolved with r07's board 1231 recipe: the Reuse capture stays tolerant and keeps task 94's receipt); o01 board 1291; r01 board 1250. Commits folded into this step (5): - 08aed7efd2 [package-deps-hash] Handle a git hash-object failure while ls-files and status still run - 3d786c1f78 [package-deps-hash] Name the git command that failed in the error message - a11aa6b774 [rush-lib] Name the project whose configuration an engine cannot load - 24cf595d6a [rush-lib] Report a missing project dependency file to an engine's host - c1b5c95adb [rush-daemon] Hand requests to in-process Rush when a project's configuration cannot load Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c

…g the run is not trusted as up to date Swarm integration step 16; original commit d6ac7c43f3 (merge of swarm/r06-t92 at 72c2f0e4f8). Scope: task 92. ch01 GATE OK board 1563 (tree 5c0b0ac); r06 board 1270; CONFIRMED by t03, t04, ch05 and o03. Commits folded into this step (1): - 72c2f0e4f8 [rush-lib] Re-run operations whose inputs changed during the iteration, also with cache writes off Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c

Swarm integration step 17; original commit 9a58fc2f85 (merge of swarm/r01-t103 at 93eacefb71). Scope: task 103. ch01 GATE OK board 1565 (tree e234c8d); o04 CONFIRMED board 1477; r01 board 1440. This is upstream row 16's commit. Commits folded into this step (1): - 93eacefb71 [rush-lib] Release the package manager lock when installing the package manager fails Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c

…ost-connection message and a private runtime folder Swarm integration step 18; original commit 261e4320dd (merge of swarm/r03-t87 at d9366621c8). Scope: tasks 4+5, 68, 86 and 87. ch01 GATE OK board 1675 (tree f85ce62). Commits: bb1d272738 (tasks 4+5), 959cbb84ee (task 68), a68e4d96a5, 696dba1d3e (task 86) and d9366621c8 (task 87). Commits folded into this step (5): - bb1d272738 [rush-lib] Hand off a by-name resolvable rush-lib and load built-in cache plugins from deploy outputs - 959cbb84ee [rush-client-core] Resolve a retained startup reservation next to a ready daemon - a68e4d96a5 [rush-client-core] Resolve a remaining startup reservation before a plain daemon stop - 696dba1d3e [rush-client-core] Explain a daemon that exited before delivering a result - d9366621c8 [rush-daemon] Meet one daemon per checkout whatever TMPDIR or XDG_RUNTIME_DIR says (task 87) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c

… instead of running Rush in-process Swarm integration step 19; original commit a81a00520d (merge of o07/startup-fallback at 668753ad4d). Scope: task 95. Brings 647a8bf902 [rush-client-core] and 668753ad4d [rush-cli-client]. Gate: ch01 GATE OK board 1731 (tree e4a0062); burst: t01 board 1716. Commits folded into this step (2): - 647a8bf902 [rush-client-core] Wait for a daemon that a live process is still starting before permitting in-process Rush - 668753ad4d [rush-cli-client] Don't run Rush in-process while daemon startup is still pending Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c

…hile their files are unchanged Swarm integration step 20; original commit 0f987a5b62 (merge of swarm/r07-t93-on-s10 at 18900fe380). Scope: task 93. Brings 143f21a891 and 18900fe380 [rush-lib]. Gate: ch01 GATE OK board 1869 (tree 8892a71); m01 CONFIRMED board 1738 and board 1818; t04 board 1794. Commits folded into this step (2): - 143f21a891 [rush-lib] Reuse workspace input digests and project configurations while their files are unchanged - 18900fe380 [rush-lib] Test that a project configuration that failed to load is loaded afresh once it is fixed Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c

… full in agent output, and rushd's startup wait as a pipe line Swarm integration step 21; original commit 478f4d3af0 (merge of swarm/r04-t83-a81 at 4d1a045fca). Scope: task 83. Brings task 83's chain as resolved onto a81a00520d: 244ed8b1f5 (merge of swarm/r04-t83-clip @9d71dbe9bb, the clip fix for board 1539) and 4d1a045fca, with board 711 (1886d80936), tasks 79, 71 and 50 and the restart-drain timeout. Gate: ch01 GATE OK board 1872 (lifts board 1778); combined with task 93, tree d0e8df0, full suite board 1888. Commits folded into this step (21): - 2e7d278d8f [rush-daemon] Report the full log path of failed and warning operations - 3b91eec856 [rush-cli-client] Keep spawned test clients on the default output when tests run in an agent shell - 1ae03fc555 [rush-cli-client] Agent output that holds at repository scale - 862ed5e676 [rush-cli-client] Do not report a daemon shutdown as a user cancellation - 39cae5f22b [rush-cli-client] Send no terminal width for a pseudo-terminal without a size - 5f3c36fe17 [rush-daemon] Describe a rejected request without the debug messages of the graph load (task 71) - 563e541fdf [rush-cli-client] Cap a multi-line error message in agent output (task 71) - 2377740ab9 [rush-daemon] Keep the loaded graph when a request only selects an unknown project (task 71) - e382d2ac80 [rush-lib] Run the initial script for every command outside watch mode (task 50, board 115) - b7bba00fb3 [rush-cli-client] Never leave agent output silent on a pipe (task 79) - feacc54bf7 [rush-daemon] Report a restart drain as a queue position (task 79) - 1886d80936 [rush-cli-client] Report Ctrl+C during graph preparation as cancelled (board 711) - ae31023aad [rush-cli-client] Report an empty selection in agent output (board 634) - bacfa3d5af [rush-daemon] Do not apply the client-default queue timeout to a restart drain (task 83) - 5f1c7b4f1b [rush-cli-client] Keep "daemon admission failed (<code>)" in the agent summary; word an empty selection like native Rush (board 785, board 634) - 3b5dc578d6 [rush-daemon] Document the restart drain and its client-default timeout (task 83) - cbf8a3a45e [rush-daemon] Apply the client-default timeout to a restart drain while a rushx script runs or for later requests (task 83) - 6cd768c6b3 [rush-daemon] Suggest stopping the rushx script when a restart drain times out behind one (task 83) - b2f9d78f1f [rush-daemon] Name the waived time when a restart drain times out (task 83) - 9d71dbe9bb Write the daemon's reason for an admission failure in full in agent output (task 83, board 1539) - 4d1a045fca Write rushd's startup wait as a pipe line in agent output (task 83 on task 95) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c

Swarm integration step 22; original commit 6bc6dd8f83 (merge of swarm/r04-t108-a81 at 14faa4e5be). Scope: task 108. Brings task 108 items 1 and 3 onto task 83: agent output prints a failed operation's first diagnostics as soon as that operation finishes, no `running` line before 10 s for quick requests, and each TypeScript error once (t07 board 1930: L4 17 of 41 on 4d1a045fca against 24 of 41 on s11; 38 of 41 with items 1 and 3 on their old base, board 1692). Gate: ch01 GATE OK board 1980 (tree 9079647) Commits folded into this step (1): - 14faa4e5be Report failed operations as they fail in agent output (task 108 items 1 and 3) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c

…operation graphs Swarm integration step 23; original commit e16ed01d3a (merge of swarm/r01-t85-a81 at 4066e8f5d7). Scope: task 85. Brings task 85 onto task 108 items 1+3: the daemon keeps a result that a plugin such as fstrace found up to date across requests, and checks a Skipped result again when it was marked unverifiable. m01 CONFIRMED the gated tip (board 2024: 72 of 72 hot requests "1066 no op") and measured 24 of 24 hot B6 requests as "no operations needed" (board 2073); ch04 CONFIRMED 6e39ea6527 (board 2003) and posted the product-default A/B (board 2132: -0.24 s [-0.57, +0.09]). ch01's combined suite on this tree passes (board 2150: rush-lib 1169/1, a timing flake in InstallRunScripts.test that task 85 doesn't touch, 5 of 5 on rerun; rush-daemon 515/0, rush-cli-client 367/0). Gate: ch01 GATE OK board 1955, combined suite board 2150 (tree 1c6a3bf) Commits folded into this step (2): - 95c8ece420 [rush-lib] Retain results that a plugin found up to date in long-lived operation graphs - 4066e8f5d7 [rush-lib] Check a Skipped result again if it was marked unverifiable Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c

… hooks (and 7 more) Swarm integration step 24; original commit 31a8db3e65 (merge of swarm/r01-t85-x106b at 4835a8700e). Scope: tasks 85 and 106. Brings task 106 (guarded :incremental builds in the daemon) and task 84 at de9a503a50 through r01's 85x106 resolution. This is step 1 of the path in board 2046 and board 2161; step 2 brings task 106's test-only tip b767c64dac. Merging b767c64dac straight onto task 85 conflicts in CacheableOperationPluginRetainedResults.test.ts. Gate: ch01 GATE OK board 2161 (tree 8250469) Commits folded into this step (8): - c7400b3f9c [rush-lib] Pass getOperationEnvironment to the before/after iteration hooks - e07159a38f [rush-lib] Let the daemon run :incremental scripts when a guard allows it - 90a76234e5 [rush-lib] Register RUSH_DAEMON_INCREMENTAL_BUILDS as a known environment variable - b1647a952a [rush-lib] Test that --only consumers of a retained incremental result skip cache writes - aed31544c2 [rush-lib] Test that a terminated command drops its incremental base - 69e33c03a8 [rush-lib] Format the incremental build changes with Prettier - c3afda3587 [rush-lib] Test that a run whose inputs changed while it ran leaves no incremental base - de9a503a50 [rush-lib] Check dependencies reached through same-project phases in the incremental guard Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c

…s it Swarm integration step 25; original commit 7d0ffb955b (merge of swarm/r06-t106 at b767c64dac). Scope: task 106. Brings task 106's test-only tip onto step 1 (31a8db3e65). With task 106 the daemon runs an operation's <phase>:incremental script, on top of the outputs of its last successful run in the daemon, when only files that the operation builds were edited since then and its output folders are unchanged; anything else runs the initial command. A result built on an incremental base is never written to the build cache. Kill switch: daemon.incrementalBuilds false in rush.json, or RUSH_DAEMON_INCREMENTAL_BUILDS=0. Task 84 passes getOperationEnvironment to the before/after iteration hooks. Second agents: t03 CONFIRMED (board 2067, 37 of 39; TAMPERI is the declared limitation) and t04 CONFIRMED (board 2100). ch01 on this tree (board 2161): rush-lib 1228/0, rush-daemon 516/0, rush-cli-client 367/0; 50 mutants, 42 killed; 33 of 33 end-to-end rows against a native cache-off oracle. Known costs, per ch01's rulings in board 2161: revisiting a state restores less from the cache (Rule 2), and an in-place rewrite of an output file by another writer is not detected (TAMPERI; task 139, r06's 641c39f120 in board 2171). Gate: ch01 GATE OK board 2161 (tree b1c0c2e) Commits folded into this step (1): - b767c64dac [rush-lib] Test that an incremental result taints cache writes through an operation without a script Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c

….json may still be changing Swarm integration step 26; original commit 8a1d1ee8dd (merge of swarm/r07-t93-nit at f5826544d1). Scope: task 93 NIT. Test-only: 7 lines in RushProjectConfiguration.test.ts, no source change. It kills ch01's task 93 mutant M13 (board 1869): with the mutant the new assertion fails, 25/1, and 18900fe380's test file lets it survive, 26/0. Full rush-lib on the cherry-pick onto b1c0c2e: 1228/0 (board 2178). The deployed daemon files don't change, so snapshot s14 (7d0ffb955b) is unaffected. Gate: ch01 GATE OK board 2178 (tree 09d9052) Commits folded into this step (1): - f5826544d1 [rush-lib] Test that a configuration is read again while only its rig.json may still be changing Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c

…tput file Swarm integration step 27; original commit 6eae8fadad (merge of swarm/r06-t106 at 641c39f120). Scope: task 139. Brings r06's fold-in on task 106 (board 2171), in place of b767c64dac. The output signature of an operation now records each output file's inode, size and mtime as well as the folder listing, so when another writer appends to or rewrites an output file in place, the next build runs the initial command ("its output folders changed since its last successful run") instead of an incremental run on the stale file. This closes t03's TAMPERI. A touch that changes only mtime now also costs one initial run. r06 measured about 4 µs per output file per read. ch01 on tree 1eb87abdd2 (board 2211): rush-lib 1231/0, rush-daemon 516/0, rush-cli-client 367/0; b767c64dac's manifest fails 4 of the new tests; 10 mutants, 6 killed (N1, N3 and N6 survive; task 138 adds their tests); tamper end-to-end rows 9/9 against 2/9 before, full rows 33/33, cache-off rows 7/7. On this branch the tree is 1eb87abdd2 plus r07's test-only 8a1d1ee8dd change, byte for byte. Gate: ch01 GATE OK board 2211 (tree 1eb87abdd2) Commits folded into this step (1): - 641c39f120 [rush-lib] Detect in-place rewrites of output files in the incremental guard Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c

…ntent-hashed outputs, invalidation and skip record Swarm integration step 28; original commit 0d30a5164c (merge of swarm/r06-t138 at dbd5d99147). Scope: task 138 and t106d. Brings r06's task 138 tests (board 2302) and t106d (4c9f23e650, board 2233). Only 3 test files change (+204/-18): ProductionDaemonRequestResolver.test.ts, IncrementalExecutionGuardPlugin.test.ts and OperationOutputManifest.test.ts. No product code changes, so no snapshot is needed. ch01 on tree 45c2c5e (board 2364): rush-lib 1241/0 (1 skipped); rush-daemon 516/0 with t106d. With the new tests, each of the 10 guard and manifest mutants (G9-G12, G19, G24, L1, N1, N3, N6) fails its intended test; all 10 pass the old tests. t106d's kill-switch test fails against a product mutant that drops the daemon.incrementalBuilds check at PhasedScriptAction.ts:666, and 6eae8fadad's version of it passed that mutant. Gate: ch01 GATE OK board 2364 (tree 45c2c5e) Commits folded into this step (2): - 4c9f23e650 [rush-daemon] Make the test of daemon.incrementalBuilds: false edit a source file - dbd5d99147 [rush-lib] Test the incremental guard's configuration inputs, content-hashed outputs, invalidation and skip record Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c

…it starts, so reloads don't wait for it (board 113) (and 5 more) Swarm integration step 29; original commit 3857bc9926 (merge of swarm/r05-stack-int at c35a77cc5c). Brings r05's stack (board 1931): - Task 55: a served rushx script releases the lifecycle gate once it starts, so reloads don't wait for it. - Task 7 part 3 text: admission timeouts name the client's wait timeout and the time that isn't counted. - Task 96: a rushx script that arrives while a restart is pending waits for the restart instead of postponing it, and admission failures in legacy output and rushx-client print the daemon's reason. - Task 98: a request spends its wait timeout only while it waits for other requests. - The race fix (c35a77cc5c, restacked from 9370662d5f): a client that follows a daemon restart connects to the successor that the restarting daemon launches instead of racing it. ch01 on tree b500e451d4 (board 2429): build rc 0 with 0 warnings; rush-daemon 544/0, rush-lib 1231/0, rush-cli-client 374/0, rush-client-core 129/0 on its own (2 known flakes in the full run), protocol 166/0, transport 73/0, reporter 519/0, wire-e2e 4/0. The red checks fail as intended without each change. 36 of 43 mutants are killed; the 7 survivors are test gaps and change no claim. The end-to-end rows R55, R96 and R96A pass on the stack and fail on s14b's files. Gate: ch01 GATE OK board 2429 (tree b500e451d4 on 6eae8fadad; 87e0e3a together with task 138's merge, in either order) Commits folded into this step (6): - c8cd36f574 [rush-daemon] A served rushx script releases the lifecycle gate once it starts, so reloads don't wait for it (board 113) - 4a5a13bddf [rush-daemon] Name the client's wait timeout and uncounted time in admission timeouts - cafa954eb4 [rush-daemon] A rushx script that arrives while a restart is pending waits for it instead of postponing it (task 96) - 7ba7021ed6 [rush-cli-client] Admission failures in legacy output and rushx-client print the daemon's reason (task 96) - 4ddb1fdbfe [rush-daemon] Spend a request's wait timeout only while it waits for other requests (task 98) - c35a77cc5c [rush-client-core] A client that follows a daemon restart connects to the successor the restarting daemon launches instead of racing it (task 96) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c

…has work Swarm integration step 30; original commit a4de651852 (merge of swarm/r07-t76 at 6b32aa340e). Scope: task 76. Brings r07's task 76 (board 1730; o05 CONFIRMED board 1904). A warm-set maintenance pass takes the native repository lease only when it has something to evict, a watcher change to apply, or a failed observation to retry. An idle daemon no longer takes common/temp/rush#<pid>.lock every 30 s, and a native command that holds the lock no longer puts the warm set in 'native-busy'. Two commits: 40a9f8f5eb (product) and 6b32aa340e (tests for the two lease exemptions a pass relies on). ch01 on tree fe0b6a4 (board 2511): rush-daemon 549/0 (544 + 5 new tests). The old WorkspaceWarmSet.ts fails exactly the 5 new tests. 15 of 16 WorkspaceWarmSet.ts mutants are killed; B6 (no protected check in #shouldEvict) survives and is a test NIT. E2E on ch01's own daemons: 0 lock holds in 150 s idle (5 before), 0 daemon lock attempts while native holds the lock 40 s (12 before, with 'native-busy'). Gate: ch01 GATE OK board 2511 (tree fe0b6a4) Commits folded into this step (2): - 40a9f8f5eb [rush-daemon] Warm set: take the repository lock only for a maintenance pass that has work - 6b32aa340e [rush-daemon] Warm set: test the two lease exemptions a pass relies on Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c

Swarm integration step 234; original commit 17a943c265 (merge of swarm/r07-t341 at 6bd98980ae). Scope: task 341, 113 follow-up: tests that kill two survivors. Brings 341 (branch swarm/r07-t341). Second agent: t06 board 9032. Evidence: ch01's GATE OK board 9622. Batch E2j, item 6 of 8. Gate: ch01 GATE OK board 9622 (tree fbdd18d) Commits folded into this step (1): - 6bd98980ae [rush-lib] Test two cases of deferred build cache writes Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c

…d or retest Swarm integration step 235; original commit bf1ac546df (merge of swarm/r04-t340 at 665d8ec1b5). Scope: task 340, 108 README NIT. Brings 340 (branch swarm/r04-t340). Second agent: t07 board 8441. Evidence: ch01's GATE OK board 9622. Batch E2j, item 7 of 8. Gate: ch01 GATE OK board 9622 (tree 1c0a571) Commits folded into this step (1): - 665d8ec1b5 [rush-cli-client] Document the silent stop by a served reload, rebuild or retest Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c

Swarm integration step 236; original commit ca8c052981 (merge of swarm/r04-t345 at 857fc03605). Scope: task 345, 340 follow-up: say which operations a served request stops. Brings 345 (branch swarm/r04-t345, 3 commits). Second agent: t07 board 9019 (m01 METRICS board 9124). GATE NIT: mutant M345-ANY-POSITION survives; the test-only follow-up is task 368 (D64 board 9574). Evidence: ch01's GATE OK board 9622. Batch E2j, item 8 of 8. Gate: ch01 GATE OK board 9622 (tree ed25ba3) Commits folded into this step (3): - f9ccf77215 [rush-daemon-protocol] Add a stopping flag to IDaemonContinuingOperations - 406b0d8325 [rush-daemon] Name the continuing operations that a queued request makes rushd stop - 857fc03605 [rush-cli-client] Say which operations rushd stops for a served command Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c

Swarm integration step 237; original commit 3e71185392 (merge of swarm/o08-d551-t2 at 66bf274454). Scope: task 350, D55.1 T2: rig-package's missing -rig suffix error names the wrong field. Brings 350 (branch swarm/o08-d551-t2). Second agent: o03 board 9087 (CLAIM board 8944). Evidence: ch01's GATE OK board 9991. Batch E2k part 1 (K1), item 1 of 5. Gate: ch01 GATE OK board 9991 (tree da36e22) Commits folded into this step (1): - 66bf274454 rig-package: name the package in the missing "-rig" suffix error Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c

…oject's script Swarm integration step 238; original commit dc83bf8af5 (merge of swarm/r03-d551-t1 at e89a36bdea). Scope: task 349, D55.1 T1: quote custom parameter values in shell commands. Brings 349 (branch swarm/r03-d551-t1). Second agent: o08 board 9188. Evidence: ch01's GATE OK board 9991. Batch E2k part 1 (K1), item 2 of 5. Gate: ch01 GATE OK board 9991 (tree 73b7b3b) Commits folded into this step (1): - e89a36bdea [rush-lib] Quote custom parameter values for the shell that runs a project's script Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c

Swarm integration step 239; original commit 83cc7cfc3d (merge of swarm/o07-d551-t5 at 5de59478c8). Scope: task 353, D55.1 T5: a warm TypeScript watch worker sees edits after a folder is recreated. Brings 353 (branch swarm/o07-d551-t5). Second agent: t03 board 9224. Evidence: ch01's GATE OK board 9991. Batch E2k part 1 (K1), item 3 of 5. Gate: ch01 GATE OK board 9991 (tree def670f) Commits folded into this step (1): - 5de59478c8 [rush-lib] Read a warm worker's input folders before its command starts Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c

…already waits first Swarm integration step 240; original commit 5f436fbe16 (merge of swarm/r04-t361 at 5156e5a0c6). Scope: task 361, 345 follow-up: a request queued before a failed build becomes preemptible. Brings 361 (branch swarm/r04-t361). Second agent: t07 board 9533 (RESULT board 9532). Evidence: ch01's GATE OK board 9991. Batch E2k part 1 (K1), item 4 of 5. Gate: ch01 GATE OK board 9991 (tree 964d518) Commits folded into this step (1): - 5156e5a0c6 [rush-daemon] Name the operations that rushd stops to a request that already waits first Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c

…d selects (and 1 more) Swarm integration step 241; original commit 0a7af6a3df (merge of swarm/r03-row23 at 4a746d0348). Scope: task 363, row 23: rush-bridge-cache-plugin acts only on the selected operations. Brings 363 (branch swarm/r03-row23, 2 commits). Second agent: ch05 board 9400. Evidence: ch01's GATE OK board 9991. Batch E2k part 1 (K1), item 5 of 5. Gate: ch01 GATE OK board 9991 (tree 228f843) Commits folded into this step (2): - 92b277d814 [rush-bridge-cache-plugin] Act only on the operations that the command selects - 4a746d0348 [rush-bridge-cache-plugin] Test a selected operation whose record is disabled Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c

Swarm integration step 242; original commit b20e9983fa (merge of swarm/r01-t117-p2 at ab4690c4b3). Refs: task 117. Scope: task 117 Part 2: prepare the next engine in the background. Brings R (branch swarm/r01-t117-p2). Second agent: r02 board 9649 (RESULT board 9647). Evidence: ch01's GATE OK board 10189. Batch E2k part 2 (K2), item 1 of 9. Gate: ch01 GATE OK board 10189 (tree 3c32e43) Commits folded into this step (1): - ab4690c4b3 rush-daemon: Prepare the next engine in the background (task 117) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c

Swarm integration step 243; original commit fb7eaa0bbf (merge of swarm/r01-t117-p2-count at 49b2a1b5c6). Refs: board 9212. Scope: task 117 Part 2 follow-up: count the capture's calls in BackgroundPrepare's tests. Brings F (branch swarm/r01-t117-p2-count). Second agent: r02 board 9865 (RESULT board 9863). Evidence: ch01's GATE OK board 10189. Batch E2k part 2 (K2), item 2 of 9. Gate: ch01 GATE OK board 10189 (tree c3f57b7) Commits folded into this step (1): - 49b2a1b5c6 rush-daemon: Count the capture's calls in BackgroundPrepare's tests (board 9212) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c

… preparing Swarm integration step 244; original commit 66229fc83a (merge of swarm/r01-t359 at 99625a4eb6). Refs: task 359. Scope: task 359: a preparation failure before the reload stops preparing. Brings T (branch swarm/r01-t359). Second agent: ch05 board 9916 (RESULT board 9915). Evidence: ch01's GATE OK board 10189. Batch E2k part 2 (K2), item 3 of 9. Gate: ch01 GATE OK board 10189 (tree 65d1ae3) Commits folded into this step (1): - 99625a4eb6 rush-daemon: Test that a preparation failure before the reload stops preparing (task 359) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c

Swarm integration step 245; original commit 1faa9f9d1f (merge of swarm/r02-d551-pr-hygiene at a82c9c190f). Scope: task 355, PR hygiene: drop internal references from test names and comments. Brings 355 (branch swarm/r02-d551-pr-hygiene). Second agent: ch05 board 9628 (RESULT board 9627). Evidence: ch01's GATE OK board 10189. Batch E2k part 2 (K2), item 4 of 9. Gate: ch01 GATE OK board 10189 (tree cfc98bc) Commits folded into this step (1): - a82c9c190f Drop internal references from test names and comments Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c

…MPDIR is Swarm integration step 246; original commit 5ab3a13dda (merge of swarm/r02-d551-t4 at d1b14ef3c3). Scope: task 352, D55.1 T4: rush-daemon test socket paths stay short however long TMPDIR is. Brings 352 (branch swarm/r02-d551-t4). Second agent: r01 board 9940 (RESULT board 9937). Evidence: ch01's GATE OK board 10189. Batch E2k part 2 (K2), item 5 of 9. Gate: ch01 GATE OK board 10189 (tree 77ba7e3) Commits folded into this step (1): - d1b14ef3c3 Keep rush-daemon test socket paths short however long TMPDIR is Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c

… responded Swarm integration step 247; original commit b41e167dbc (merge of swarm/r04-t364 at 455b9b1940). Scope: task 364: the silence line names `rush-client daemon status`. Brings 364 (branch swarm/r04-t364). Second agent: t07 board 9930 (RESULT board 9929). Evidence: ch01's GATE OK board 10189. Batch E2k part 2 (K2), item 6 of 9. Gate: ch01 GATE OK board 10189 (tree a9bde89) Commits folded into this step (1): - 455b9b1940 [rush-cli-client] Name `rush-client daemon status` when rushd has not responded Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c

Swarm integration step 248; original commit 86204d0550 (merge of swarm/r06-d551-t3 at 298b23b0d2). Scope: task 351, D55.1 T3: package-deps-hash hashes the paths that git hash-object can't read. Brings 351 (branch swarm/r06-d551-t3). Second agent: t04 board 10037. Evidence: ch01's GATE OK board 10189. Batch E2k part 2 (K2), item 7 of 9. Gate: ch01 GATE OK board 10189 (tree 772d755) Commits folded into this step (1): - 298b23b0d2 package-deps-hash: hash the paths that "git hash-object" can't read Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c

…r path against the working directory (and 1 more) Swarm integration step 249; original commit 5aecba33b6 (merge of swarm/r06-t366 at 03a65659c2). Scope: task 366, 351 follow-ups: sign an input that is a link to no regular file by the link itself. Brings 366 (branch swarm/r06-t366, 2 commits). Second agent: t04 board 10037. Evidence: ch01's GATE OK board 10189. Batch E2k part 2 (K2), item 8 of 9. Gate: ch01 GATE OK board 10189 (tree 506a5b0) Commits folded into this step (2): - 720874a782 [package-deps-hash] Resolve a relative RepoStateCache temporary folder path against the working directory - 03a65659c2 [rush-lib] Sign an input that is a symbolic link not leading to a regular file by the link itself Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c

… 1 more) Swarm integration step 250; original commit 051bfee3e4 (merge of swarm/o01-t365 at fb77adfde9). Scope: task 365, C1 and C2: rush-daemon tests print no environment in failed assertions, and delete secret-shaped variables first. Brings 365 (branch swarm/o01-t365, 2 commits). Second agent: r01 board 9962 (RESULT board 9901). Evidence: ch01's GATE OK board 10189. Batch E2k part 2 (K2), item 9 of 9. Gate: ch01 GATE OK board 10189 (tree 090484c) Commits folded into this step (2): - df1f14139a rush-daemon: keep the environment out of failed test assertions - fb77adfde9 rush-daemon: delete secret-shaped environment variables before each test Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c

Replace the 489 change files that this branch added with one change file for each of the 18 projects it changes, with shorter comments. Also delete 27 change files that came from main commits 1f51ac8..982b33b. Main has already published and deleted them, and the squash merge of #6102 dropped the merge of main that contained them, so merging this branch would add them back. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c

The branch tree already contains main up to 982b33b: the squash merge of #6102 was built on a merge of 982b33b, but the squash dropped that parent. This merge restores the ancestry without changing the tree, so the next merge of origin/main only brings the commits after 982b33b. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c

…gins, add snapshot workflow and guide (#6046) * rush-lib: allow daemon builds with plugins scoped to other commands; add daemon dogfooding workflow Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Add change files Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * docs: record Windows validation and daemon process check in dogfooding guide Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * rush-daemon: hide console windows of tools the detached Windows daemon starts Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * docs: describe hidden Windows tool consoles and silent up-to-date builds Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * rush-cli-client: start without loading the Rush engine; cheaper runtime fingerprint The client only needs to locate rush.json, read daemon settings, parse rushx arguments and build its daemon start command. Move those into small rush-lib and rush-daemon modules so the client no longer loads the whole engine for every invocation, and load version selection and rushx discovery only when needed. The runtime fingerprint now also covers the bundle chunks under dist, which contain the implementation, and recomputes a resolved path only when a file stat changes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * rush-client-core: let the startup helper finish a slow daemon start The detached helper waited for readiness only until the requesting client deadline (15 seconds). A slower first start, such as the first start from a freshly deployed snapshot on Windows, left its startup reservation behind. Every later client then refused to use the ready daemon and fell back to native Rush after 15 seconds. The helper now waits for a live launcher for at least 120 seconds; the reservation is still retained if the launcher exits or never becomes ready. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * rush-lib: resolve rig profiles asynchronously in the daemon project loader The heft-config-file rig fallback resolves the rig profile synchronously, which serializes the daemon's concurrent uncached project-configuration loads (two per request). Resolve it asynchronously first; RigConfig caches the same result, and a resolution failure still surfaces through a fresh instance exactly as before. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * docs: describe the first daemon start from a refreshed Windows snapshot Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * [rush] rushd: fix the bugs found by dogfooding the Rush daemon (stacked on #6046) (#6102) * [rush-lib] Workspace fingerprint: ignore build state of projects nested under common/config (and 3 more) Swarm integration step 1; original commit 433c87f24f (merge of swarm/r05 at 190d77ea17). Commits folded into this step (4): - 40b5b24d72 [rush-lib] Workspace fingerprint: ignore build state of projects nested under common/config - 1880e7b2b7 [rush-daemon] Warm set: keep resource-free retained results under memory pressure and idle expiry - 07e908d32d [rush-daemon] Admission: don't spend a default wait budget behind a graph load or across an admitted request's execution - 190d77ea17 [rush-lib] Move the nested common/config fingerprint test to the end of its suite Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [rush-daemon] Warm set: read status once per maintenance pass, not once per project (and 1 more) Swarm integration step 2; original commit 8553b2be55 (merge of swarm/r07 at d94054e04d). Commits folded into this step (2): - 57dd6a4b45 [rush-daemon] Warm set: read status once per maintenance pass, not once per project - d94054e04d [rush-lib] Share rig profile loads across projects in the uncached rush-project.json load Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [rush-lib] Keep RUSH_DAEMON* variables out of lifecycle script environments (and 3 more) Swarm integration step 3; original commit 3b2d24277d (merge of swarm/r06 at 6c10e241a4). Commits folded into this step (4): - 803345396b [rush-lib] Keep RUSH_DAEMON* variables out of lifecycle script environments - b49d8b10cb [rush-daemon] Keep one daemon across per-session environment differences - 36892176cb [rush-daemon-transport] Reap detached operation groups of a daemon that died uncleanly - 6c10e241a4 [rush-daemon] Run each operation with its requester's per-session environment Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [rush-lib] Let daemon engines serve builds with declared daemon-compatible plugins (and 1 more) Swarm integration step 4; original commit 1e786a4252 (merge of swarm/r01 at 162ff05eaf). Commits folded into this step (2): - 5fe505bf1a [rush-lib] Let daemon engines serve builds with declared daemon-compatible plugins - 162ff05eaf [rush-lib] Walk each plugin package once in the daemon runtime fingerprint Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [rush-daemon] Abort a shared iteration's leftover work once no remaining client needs it Swarm integration step 5; original commit 02f83a99ae (merge of swarm/r05 at bd7ac88caa). Refs: board 207. Commits folded into this step (1): - bd7ac88caa [rush-daemon] Abort a shared iteration's leftover work once no remaining client needs it (board 207) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [rush-lib] Run the initial script for every command outside watch mode Swarm integration step 6; original commit 519ae33c8d. Refs: task 50, board 115. ShellOperationRunner runs a phase's `:incremental` script whenever the operation has a last state. Native `rush build` executes one iteration, so only watch mode ever had one, and watch mode turns cache writes off. rushd keeps one operation graph across requests, so from its second request on a daemon `build` ran `_phase:build:incremental` (in odsp-web, `heft run --only build --` without --clean) with cache writes on. Outputs of a deleted or renamed source file stayed in lib/ and were cached under the key native Rush computes for the clean build, so later native builds restored them (t04 board 115, board 450). ShellOperationRunnerPlugin now sets the incremental command only in watch mode. Every other command runs the initial script, exactly as a single `rush build` does, which is option (a) of task 50. Native `rush build` is unchanged (it never has a last state) and watch mode keeps `:incremental` with writes off. Tests: a rush-lib unit test runs a repeated operation with and without watch mode, and a daemon integration test runs three warm builds of a project with an `:incremental` script. Before the fix the second daemon build logged "Invoking (incremental): node build.cjs --incremental" followed by "Successfully set cache entry". (cherry picked from commit e382d2ac80f881451b7d527ea9a97e3037880923) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [rush-daemon] Share input captures between concurrent requests Swarm integration step 7; original commit f7ac88dc9d (merge of swarm/r07 at 55a16f14ef). Refs: F1 coalescer, board 555. GATE OK ch01 board 930 (tree 41b2227d59). CONFIRMED by ch03 board 681, t06 board 701/board 749, t01 board 861. Only 55a16f14ef; b6692d09a5 (task 74) is not gated. Commits folded into this step (1): - 55a16f14ef [rush-daemon] Share input captures between concurrent requests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [rush-lib] Skip cache writes when input files change while the inputs snapshot is taken Swarm integration step 8; original commit bc03023091 (merge of swarm/r06 at 29f86b181c). Refs: task 77. ch01 GATE OK board 1019 (tree 80c9659bac); claim board 698, CONFIRMED by o04 board 738, o03 board 742, o05 board 748. Commits folded into this step (2): - a644819daf [rush-lib] Don't write a build cache entry for inputs saved while the snapshot was taken - 29f86b181c [rush-lib] Make the parameter identity test independent of the host's RUSH_PARALLELISM Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [rush-daemon] don't let a request received during a batch's reconcile join that batch Swarm integration step 9; original commit d0ed461613 (merge of swarm/r05 at 199c69347c). Refs: board 371, task 72. ch01 GATE OK board 1152 (gated tree 5c2b6b6cbf). Commits folded into this step (1): - 199c69347c [rush-daemon] Don't let a request received during a batch's reconcile join that batch (board 371) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [rush-daemon] Another request's graph load doesn't spend any finite wait timeout Swarm integration step 10; original commit 908dcc1f27 (merge of swarm/r05 at 868e4b9299). Refs: task 7 part 3, board 891. ch01 GATE OK board 1196 (gated tree 55698be825). Commits folded into this step (1): - 868e4b9299 [rush-daemon] Admission: another request's graph load doesn't spend any finite wait timeout Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [rush-lib] Keep cache trust for unselected retained results and re-verify untrusted ones Swarm integration step 11; original commit 4456f17460 (merge of swarm/r07 at 53654af37c). Refs: board 401, task 74, board 982, board 1024. ch01 GATE OK board 1197 (gated tree 3014a03bb8). Commits folded into this step (3): - b6692d09a5 [rush-lib] Keep cache trust for unselected retained results and re-verify untrusted ones - 492126a051 [rush-lib] Re-verify retained results built against unverified dependency outputs without the build cache - 53654af37c [rush-daemon] Test that input captures are shared only between equal fingerprint environments Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [rush-cli-client] Show engine and plugin warnings in agent output; keep the daemon alive when git hash-object fails Swarm integration step 12; original commit cc5c6aab7e (merge of swarm/r01 at 575cbf5698). Refs: board 601, task 80, board 899. ch01 GATE OK board 1272 and board 1273 (gated tree b691997c35). Commits folded into this step (3): - c11ee10320 [rush-cli-client] Show engine and plugin warnings in agent output - c77aa1dbef [package-deps-hash] Handle a git hash-object failure while ls-files and status still run - 575cbf5698 [package-deps-hash] Name the git command that failed in the error message Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [node-core-library] LockFile ties, TZ/locale rule, /proc start times and own-start memo Swarm integration step 13; original commit 0347cd0dbb. Refs: tasks 67, D and 62. ch01 GATE OK board 1389; o05 board 1360 (task 62 on ch01-t62: 32/32 served in 3 of 3 bursts); t01 board 1106/board 1186. Commits folded into this step (6): - 3dfb75ec86 [node-core-library] Don't grant a LockFile twice when lockfile birthtimes tie - 074f207217 [node-core-library] Keep the LockFile of a running process with another time zone or locale - 4e43a1ef8f [node-core-library] Don't set dirtyWhenAcquired when the holder releases the lock during a check - ad9c1211b4 [node-core-library] Read the start times of other LockFile processes from /proc on Linux - 698af4ef92 [node-core-library] Run ps for the start time of the current LockFile process only once on Linux - 9249e85d63 [rush-client-core] Test clients that start the daemon at the same time Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [rush-daemon] A request can share an input capture that started after it was received Swarm integration step 14; original commit f2ec9ffc26. ch01 GATE OK board 1430 (tree ae49a66e47 with o07's stack); t06 CONFIRMED board 1365; o05 board 1412 (cold split gone in auto-start mode). Commits folded into this step (1): - a4de3fe5fe [rush-daemon] Share running input captures that started after a request was received Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [rush-lib] Hand requests to in-process Rush when a project's configuration cannot load Swarm integration step 15; original commit 69f30e84ab (merge of swarm/r01-t70 at c1b5c95adb). Scope: task 70. ch01 GATE OK board 1522 (tree 3a564b4533; WorkspaceRequestLifecycle.ts resolved with r07's board 1231 recipe: the Reuse capture stays tolerant and keeps task 94's receipt); o01 board 1291; r01 board 1250. Commits folded into this step (5): - 08aed7efd2 [package-deps-hash] Handle a git hash-object failure while ls-files and status still run - 3d786c1f78 [package-deps-hash] Name the git command that failed in the error message - a11aa6b774 [rush-lib] Name the project whose configuration an engine cannot load - 24cf595d6a [rush-lib] Report a missing project dependency file to an engine's host - c1b5c95adb [rush-daemon] Hand requests to in-process Rush when a project's configuration cannot load Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [rush-lib] With cache writes off, a result whose inputs changed during the run is not trusted as up to date Swarm integration step 16; original commit d6ac7c43f3 (merge of swarm/r06-t92 at 72c2f0e4f8). Scope: task 92. ch01 GATE OK board 1563 (tree 5c0b0ac25d); r06 board 1270; CONFIRMED by t03, t04, ch05 and o03. Commits folded into this step (1): - 72c2f0e4f8 [rush-lib] Re-run operations whose inputs changed during the iteration, also with cache writes off Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [rush-lib] A failed pnpm install releases the package-manager lock Swarm integration step 17; original commit 9a58fc2f85 (merge of swarm/r01-t103 at 93eacefb71). Scope: task 103. ch01 GATE OK board 1565 (tree e234c8d86d); o04 CONFIRMED board 1477; r01 board 1440. This is upstream row 16's commit. Commits folded into this step (1): - 93eacefb71 [rush-lib] Release the package manager lock when installing the package manager fails Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [rush-daemon-transport] rush-lib path handoff, startup reservation, lost-connection message and a private runtime folder Swarm integration step 18; original commit 261e4320dd (merge of swarm/r03-t87 at d9366621c8). Scope: tasks 4+5, 68, 86 and 87. ch01 GATE OK board 1675 (tree f85ce62f88). Commits: bb1d272738 (tasks 4+5), 959cbb84ee (task 68), a68e4d96a5, 696dba1d3e (task 86) and d9366621c8 (task 87). Commits folded into this step (5): - bb1d272738 [rush-lib] Hand off a by-name resolvable rush-lib and load built-in cache plugins from deploy outputs - 959cbb84ee [rush-client-core] Resolve a retained startup reservation next to a ready daemon - a68e4d96a5 [rush-client-core] Resolve a remaining startup reservation before a plain daemon stop - 696dba1d3e [rush-client-core] Explain a daemon that exited before delivering a result - d9366621c8 [rush-daemon] Meet one daemon per checkout whatever TMPDIR or XDG_RUNTIME_DIR says (task 87) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [rush-client-core] Wait for a daemon a live process is still starting instead of running Rush in-process Swarm integration step 19; original commit a81a00520d (merge of o07/startup-fallback at 668753ad4d). Scope: task 95. Brings 647a8bf902 [rush-client-core] and 668753ad4d [rush-cli-client]. Gate: ch01 GATE OK board 1731 (tree e4a00622d4); burst: t01 board 1716. Commits folded into this step (2): - 647a8bf902 [rush-client-core] Wait for a daemon that a live process is still starting before permitting in-process Rush - 668753ad4d [rush-cli-client] Don't run Rush in-process while daemon startup is still pending Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [rush-lib] Reuse workspace input digests and project configurations while their files are unchanged Swarm integration step 20; original commit 0f987a5b62 (merge of swarm/r07-t93-on-s10 at 18900fe380). Scope: task 93. Brings 143f21a891 and 18900fe380 [rush-lib]. Gate: ch01 GATE OK board 1869 (tree 8892a71708); m01 CONFIRMED board 1738 and board 1818; t04 board 1794. Commits folded into this step (2): - 143f21a891 [rush-lib] Reuse workspace input digests and project configurations while their files are unchanged - 18900fe380 [rush-lib] Test that a project configuration that failed to load is loaded afresh once it is fixed Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [rush-cli-client] The daemon's admission and restart-drain reasons in full in agent output, and rushd's startup wait as a pipe line Swarm integration step 21; original commit 478f4d3af0 (merge of swarm/r04-t83-a81 at 4d1a045fca). Scope: task 83. Brings task 83's chain as resolved onto a81a00520d: 244ed8b1f5 (merge of swarm/r04-t83-clip @9d71dbe9bb, the clip fix for board 1539) and 4d1a045fca, with board 711 (1886d80936), tasks 79, 71 and 50 and the restart-drain timeout. Gate: ch01 GATE OK board 1872 (lifts board 1778); combined with task 93, tree d0e8df0156, full suite board 1888. Commits folded into this step (21): - 2e7d278d8f [rush-daemon] Report the full log path of failed and warning operations - 3b91eec856 [rush-cli-client] Keep spawned test clients on the default output when tests run in an agent shell - 1ae03fc555 [rush-cli-client] Agent output that holds at repository scale - 862ed5e676 [rush-cli-client] Do not report a daemon shutdown as a user cancellation - 39cae5f22b [rush-cli-client] Send no terminal width for a pseudo-terminal without a size - 5f3c36fe17 [rush-daemon] Describe a rejected request without the debug messages of the graph load (task 71) - 563e541fdf [rush-cli-client] Cap a multi-line error message in agent output (task 71) - 2377740ab9 [rush-daemon] Keep the loaded graph when a request only selects an unknown project (task 71) - e382d2ac80 [rush-lib] Run the initial script for every command outside watch mode (task 50, board 115) - b7bba00fb3 [rush-cli-client] Never leave agent output silent on a pipe (task 79) - feacc54bf7 [rush-daemon] Report a restart drain as a queue position (task 79) - 1886d80936 [rush-cli-client] Report Ctrl+C during graph preparation as cancelled (board 711) - ae31023aad [rush-cli-client] Report an empty selection in agent output (board 634) - bacfa3d5af [rush-daemon] Do not apply the client-default queue timeout to a restart drain (task 83) - 5f1c7b4f1b [rush-cli-client] Keep "daemon admission failed (<code>)" in the agent summary; word an empty selection like native Rush (board 785, board 634) - 3b5dc578d6 [rush-daemon] Document the restart drain and its client-default timeout (task 83) - cbf8a3a45e [rush-daemon] Apply the client-default timeout to a restart drain while a rushx script runs or for later requests (task 83) - 6cd768c6b3 [rush-daemon] Suggest stopping the rushx script when a restart drain times out behind one (task 83) - b2f9d78f1f [rush-daemon] Name the waived time when a restart drain times out (task 83) - 9d71dbe9bb Write the daemon's reason for an admission failure in full in agent output (task 83, board 1539) - 4d1a045fca Write rushd's startup wait as a pipe line in agent output (task 83 on task 95) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [rush-cli-client] Report failed operations as they fail in agent output Swarm integration step 22; original commit 6bc6dd8f83 (merge of swarm/r04-t108-a81 at 14faa4e5be). Scope: task 108. Brings task 108 items 1 and 3 onto task 83: agent output prints a failed operation's first diagnostics as soon as that operation finishes, no `running` line before 10 s for quick requests, and each TypeScript error once (t07 board 1930: L4 17 of 41 on 4d1a045fca against 24 of 41 on s11; 38 of 41 with items 1 and 3 on their old base, board 1692). Gate: ch01 GATE OK board 1980 (tree 9079647277) Commits folded into this step (1): - 14faa4e5be Report failed operations as they fail in agent output (task 108 items 1 and 3) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [rush-lib] Keep results that a plugin found up to date in long-lived operation graphs Swarm integration step 23; original commit e16ed01d3a (merge of swarm/r01-t85-a81 at 4066e8f5d7). Scope: task 85. Brings task 85 onto task 108 items 1+3: the daemon keeps a result that a plugin such as fstrace found up to date across requests, and checks a Skipped result again when it was marked unverifiable. m01 CONFIRMED the gated tip (board 2024: 72 of 72 hot requests "1066 no op") and measured 24 of 24 hot B6 requests as "no operations needed" (board 2073); ch04 CONFIRMED 6e39ea6527 (board 2003) and posted the product-default A/B (board 2132: -0.24 s [-0.57, +0.09]). ch01's combined suite on this tree passes (board 2150: rush-lib 1169/1, a timing flake in InstallRunScripts.test that task 85 doesn't touch, 5 of 5 on rerun; rush-daemon 515/0, rush-cli-client 367/0). Gate: ch01 GATE OK board 1955, combined suite board 2150 (tree 1c6a3bfa10) Commits folded into this step (2): - 95c8ece420 [rush-lib] Retain results that a plugin found up to date in long-lived operation graphs - 4066e8f5d7 [rush-lib] Check a Skipped result again if it was marked unverifiable Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [rush-lib] Pass getOperationEnvironment to the before/after iteration hooks (and 7 more) Swarm integration step 24; original commit 31a8db3e65 (merge of swarm/r01-t85-x106b at 4835a8700e). Scope: tasks 85 and 106. Brings task 106 (guarded :incremental builds in the daemon) and task 84 at de9a503a50 through r01's 85x106 resolution. This is step 1 of the path in board 2046 and board 2161; step 2 brings task 106's test-only tip b767c64dac. Merging b767c64dac straight onto task 85 conflicts in CacheableOperationPluginRetainedResults.test.ts. Gate: ch01 GATE OK board 2161 (tree 8250469c70) Commits folded into this step (8): - c7400b3f9c [rush-lib] Pass getOperationEnvironment to the before/after iteration hooks - e07159a38f [rush-lib] Let the daemon run :incremental scripts when a guard allows it - 90a76234e5 [rush-lib] Register RUSH_DAEMON_INCREMENTAL_BUILDS as a known environment variable - b1647a952a [rush-lib] Test that --only consumers of a retained incremental result skip cache writes - aed31544c2 [rush-lib] Test that a terminated command drops its incremental base - 69e33c03a8 [rush-lib] Format the incremental build changes with Prettier - c3afda3587 [rush-lib] Test that a run whose inputs changed while it ran leaves no incremental base - de9a503a50 [rush-lib] Check dependencies reached through same-project phases in the incremental guard Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [rush-lib] Let the daemon run :incremental scripts when a guard allows it Swarm integration step 25; original commit 7d0ffb955b (merge of swarm/r06-t106 at b767c64dac). Scope: task 106. Brings task 106's test-only tip onto step 1 (31a8db3e65). With task 106 the daemon runs an operation's <phase>:incremental script, on top of the outputs of its last successful run in the daemon, when only files that the operation builds were edited since then and its output folders are unchanged; anything else runs the initial command. A result built on an incremental base is never written to the build cache. Kill switch: daemon.incrementalBuilds false in rush.json, or RUSH_DAEMON_INCREMENTAL_BUILDS=0. Task 84 passes getOperationEnvironment to the before/after iteration hooks. Second agents: t03 CONFIRMED (board 2067, 37 of 39; TAMPERI is the declared limitation) and t04 CONFIRMED (board 2100). ch01 on this tree (board 2161): rush-lib 1228/0, rush-daemon 516/0, rush-cli-client 367/0; 50 mutants, 42 killed; 33 of 33 end-to-end rows against a native cache-off oracle. Known costs, per ch01's rulings in board 2161: revisiting a state restores less from the cache (Rule 2), and an in-place rewrite of an output file by another writer is not detected (TAMPERI; task 139, r06's 641c39f120 in board 2171). Gate: ch01 GATE OK board 2161 (tree b1c0c2e690) Commits folded into this step (1): - b767c64dac [rush-lib] Test that an incremental result taints cache writes through an operation without a script Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [rush-lib] Test that a configuration is read again while only its rig.json may still be changing Swarm integration step 26; original commit 8a1d1ee8dd (merge of swarm/r07-t93-nit at f5826544d1). Scope: task 93 NIT. Test-only: 7 lines in RushProjectConfiguration.test.ts, no source change. It kills ch01's task 93 mutant M13 (board 1869): with the mutant the new assertion fails, 25/1, and 18900fe380's test file lets it survive, 26/0. Full rush-lib on the cherry-pick onto b1c0c2e690: 1228/0 (board 2178). The deployed daemon files don't change, so snapshot s14 (7d0ffb955b) is unaffected. Gate: ch01 GATE OK board 2178 (tree 09d90522de) Commits folded into this step (1): - f5826544d1 [rush-lib] Test that a configuration is read again while only its rig.json may still be changing Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [rush-lib] The incremental guard detects an in-place rewrite of an output file Swarm integration step 27; original commit 6eae8fadad (merge of swarm/r06-t106 at 641c39f120). Scope: task 139. Brings r06's fold-in on task 106 (board 2171), in place of b767c64dac. The output signature of an operation now records each output file's inode, size and mtime as well as the folder listing, so when another writer appends to or rewrites an output file in place, the next build runs the initial command ("its output folders changed since its last successful run") instead of an incremental run on the stale file. This closes t03's TAMPERI. A touch that changes only mtime now also costs one initial run. r06 measured about 4 µs per output file per read. ch01 on tree 1eb87abdd2 (board 2211): rush-lib 1231/0, rush-daemon 516/0, rush-cli-client 367/0; b767c64dac's manifest fails 4 of the new tests; 10 mutants, 6 killed (N1, N3 and N6 survive; task 138 adds their tests); tamper end-to-end rows 9/9 against 2/9 before, full rows 33/33, cache-off rows 7/7. On this branch the tree is 1eb87abdd2 plus r07's test-only 8a1d1ee8dd change, byte for byte. Gate: ch01 GATE OK board 2211 (tree 1eb87abdd2) Commits folded into this step (1): - 641c39f120 [rush-lib] Detect in-place rewrites of output files in the incremental guard Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [rush-lib] Tests for the incremental guard's configuration inputs, content-hashed outputs, invalidation and skip record Swarm integration step 28; original commit 0d30a5164c (merge of swarm/r06-t138 at dbd5d99147). Scope: task 138 and t106d. Brings r06's task 138 tests (board 2302) and t106d (4c9f23e650, board 2233). Only 3 test files change (+204/-18): ProductionDaemonRequestResolver.test.ts, IncrementalExecutionGuardPlugin.test.ts and OperationOutputManifest.test.ts. No product code changes, so no snapshot is needed. ch01 on tree 45c2c5edd5 (board 2364): rush-lib 1241/0 (1 skipped); rush-daemon 516/0 with t106d. With the new tests, each of the 10 guard and manifest mutants (G9-G12, G19, G24, L1, N1, N3, N6) fails its intended test; all 10 pass the old tests. t106d's kill-switch test fails against a product mutant that drops the daemon.incrementalBuilds check at PhasedScriptAction.ts:666, and 6eae8fadad's version of it passed that mutant. Gate: ch01 GATE OK board 2364 (tree 45c2c5edd5) Commits folded into this step (2): - 4c9f23e650 [rush-daemon] Make the test of daemon.incrementalBuilds: false edit a source file - dbd5d99147 [rush-lib] Test the incremental guard's configuration inputs, content-hashed outputs, invalidation and skip record Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [rush-daemon] A served rushx script releases the lifecycle gate once it starts, so reloads don't wait for it (board 113) (and 5 more) Swarm integration step 29; original commit 3857bc9926 (merge of swarm/r05-stack-int at c35a77cc5c). Brings r05's stack (board 1931): - Task 55: a served rushx script releases the lifecycle gate once it starts, so reloads don't wait for it. - Task 7 part 3 text: admission timeouts name the client's wait timeout and the time that isn't counted. - Task 96: a rushx script that arrives while a restart is pending waits for the restart instead of postponing it, and admission failures in legacy output and rushx-client print the daemon's reason. - Task 98: a request spends its wait timeout only while it waits for other requests. - The race fix (c35a77cc5c, restacked from 9370662d5f): a client that follows a daemon restart connects to the successor that the restarting daemon launches instead of racing it. ch01 on tree b500e451d4 (board 2429): build rc 0 with 0 warnings; rush-daemon 544/0, rush-lib 1231/0, rush-cli-client 374/0, rush-client-core 129/0 on its own (2 known flakes in the full run), protocol 166/0, transport 73/0, reporter 519/0, wire-e2e 4/0. The red checks fail as intended without each change. 36 of 43 mutants are killed; the 7 survivors are test gaps and change no claim. The end-to-end rows R55, R96 and R96A pass on the stack and fail on s14b's files. Gate: ch01 GATE OK board 2429 (tree b500e451d4 on 6eae8fadad; 87e0e3a10f together with task 138's merge, in either order) Commits folded into this step (6): - c8cd36f574 [rush-daemon] A served rushx script releases the lifecycle gate once it starts, so reloads don't wait for it (board 113) - 4a5a13bddf [rush-daemon] Name the client's wait timeout and uncounted time in admission timeouts - cafa954eb4 [rush-daemon] A rushx script that arrives while a restart is pending waits for it instead of postponing it (task 96) - 7ba7021ed6 [rush-cli-client] Admission failures in legacy output and rushx-client print the daemon's reason (task 96) - 4ddb1fdbfe [rush-daemon] Spend a request's wait timeout only while it waits for other requests (task 98) - c35a77cc5c [rush-client-core] A client that follows a daemon restart connects to the successor the restarting daemon launches instead of racing it (task 96) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [rush-daemon] A warm-set pass takes the repository lock only when it has work Swarm integration step 30; original commit a4de651852 (merge of swarm/r07-t76 at 6b32aa340e). Scope: task 76. Brings r07's task 76 (board 1730; o05 CONFIRMED board 1904). A warm-set maintenance pass takes the native repository lease only when it has something to evict, a watcher change to apply, or a failed observation to retry. An idle daemon no longer takes common/temp/rush#<pid>.lock every 30 s, and a native command that holds the lock no longer puts the warm set in 'native-busy'. Two commits: 40a9f8f5eb (product) and 6b32aa340e (tests for the two lease exemptions a pass relies on). ch01 on tree fe0b6a452a (board 2511): rush-daemon 549/0 (544 + 5 new tests). The old WorkspaceWarmSet.ts fails exactly the 5 new tests. 15 of 16 WorkspaceWarmSet.ts mutants are killed; B6 (no protected check in #shouldEvict) survives and is a test NIT. E2E on ch01's own daemons: 0 lock holds in 150 s idle (5 before), 0 daemon lock attempts while native holds the lock 40 s (12 before, with 'native-busy'). Gate: ch01 GATE OK board 2511 (tree fe0b6a452a) Commits folded into this step (2): - 40a9f8f5eb [rush-daemon] Warm set: take the repository lock only for a maintenance pass that has work - 6b32aa340e [rush-daemon] Warm set: test the two lease exemptions a pass relies on Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [node-core-library] A leftover LockFile whose pid now belongs to an older process is deleted Swarm integration step 31; original commit 02087b3f11 (merge of o07/lockfile-tzrule at d5e74b1813). Scope: task 121. Brings o07's task 121 (board 1815, re-tipped in board 1949; t05 CONFIRMED board 1923 on faa7675fa6). A rush#<pid>.lock whose pid has been reused by a live process that started before the file was written no longer blocks every command until that process exits. The lock file keeps its holder only if the recorded start time matches the live process's start time in some real time zone, and on Linux that start time is read from /proc instead of spawning ps. Four commits: 595f8f63c7 (the time zone rule), faa7675fa6 (/proc start time for another time zone or locale), 315e1eae81 (ch04's test for a process that started after the file) and d5e74b1813 (t05's note that a clock change can make a start time fail the rule). ch01 on tree 73696a280a (board 2566): build rc 0 with 0 warnings, rush-daemon 549/0; on b10e8c53e0 node-core-library 329/0 (318 + 11 new tests) and rush-client-core 129/0. The new tests fail 7 times on the old LockFile.ts. 21 LockFile.ts mutants, 16 killed; the 5 survivors are 4 test NITs and one equivalent. E2E with 12 live holders in other time zones and locales: all kept, with 0 ps calls after the first attempt (1.2-7 ms per attempt, against 410-520 ms before); the stale files (pid 1 at +7 min, pid 1 with a 2024 start time, a predating process at +3 min) are now deleted. Gate: ch01 GATE OK board 2566 (tree 73696a280a) Commits folded into this step (4): - 595f8f63c7 [node-core-library] Keep a mismatched LockFile only if its start time is in some time zone - faa7675fa6 [node-core-library] Read the start time from /proc for a LockFile with another time zone or locale - 315e1eae81 [node-core-library] Test a LockFile whose PID started after the file, with a start time in another time zone - d5e74b1813 [node-core-library] Note that a clock change can make a LockFile's start time fail the time zone check Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [credential-cache] A credentials.json that isn't valid JSON no longer prints the stored SAS Swarm integration step 32; original commit 0bef326909 (merge of swarm/r03-t143 at 92cbe2336e). Scope: task 143. Brings r03's task 143 (board 2316; t07 CONFIRMED board 2443 and board 2445). CredentialCache no longer rethrows JsonFile's parse error, which quoted jju's excerpt of the failing line, and in credentials.json that line holds the credential. The error now gives only the line and column and says the contents aren't shown because the file stores credentials. A missing file still gives undefined, other read errors are still wrapped as before, and schema errors are unchanged. The branch carries 08e26894e7 (task 101 part 1: compile the credentials file schema once) through r03's merge eae2fc9ad3, so tasks 101 and 143 merge in either order. ch01 on 2c425fc4f5 (board 2576): build rc 0 with 0 warnings, credential-cache 34/0, azure plugin 10/0, http plugin 14/0, amazon-s3 plugin 52/0. The new tests fail exactly the three torn-file tests on eae2fc9ad3. 12 CredentialCache.ts mutants, 9 killed; the survivors are one equivalent and one test NIT (a read error other than ENOENT). E2E with a marker in the SAS: on the old engine a torn file printed the marker in the daemon, --no-daemon and update-cloud-credentials paths; now it is printed 0 times, and 11 of 20 outputs are byte-identical. Task 142 must land with or after this merge. Gate: ch01 GATE OK board 2576 (tree 692db7d548 after task 121) Commits folded into this step (2): - 08e26894e7 [credential-cache] Compile the credentials file schema once per process - 92cbe2336e [credential-cache] Don't quote the credentials file in its parse error (task 143) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [rush-daemon] A failed agent-output build returns as soon as nothing unfinished can change its result Swarm integration step 33; original commit 1b51e7af08 (merge of swarm/r04-t108-int at c76de10302). Scope: task 108 a. Brings r04's task 108 item 2, "108(a)": 129e04f8bb on the nits 4bad455570 and item 2 2e066a7758, re-merged onto 3857bc9926 as c76de10302 (board 2133, board 2533). With agent output, a failed `build` returns as soon as nothing unfinished can change its result. Its summary line ends `· <n> independent operations continue in rushd`, and those operations finish in the daemon. The next build waits for them (`queued behind another request`) and doesn't run them again; the daemon's own install and rebuild stop the leftover, and a request that falls back in-process stops it first. Until the leftover ends, a native `rush install` or `rush-client --no-daemon` fails with "Another Rush command is already running in this repository." (o01 board 2598, board 2612), and `daemon status` doesn't count it yet (r04's output follow-up, board 2133, board 2652). Legacy output keeps the old behavior. o02 CONFIRMED item 2 with one row refuted (board 2055), which 129e04f8bb addresses; CONFIRMED by t07 (board 2183) and t05 (board 2193). ch01 on 140b42ad50 (board 2678): build rc 0 with 0 warnings, rush-daemon 573/0, rush-cli-client 383/0, rush-client-core 129/0, rush-daemon-protocol 172/0. 31 new and changed tests fail on the old product. 27 mutants, 22 killed; F4 and Y4 are equivalent, and three are test NITs (the stop ignores the client leaving, the offer ignores a record that is already aborted, and silent operations count as continuing). E2E: a failed build returns in 0.3 s where the base takes 15.3 s, and its independent operation finishes 15.0 s later. Gate: ch01 GATE OK board 2678 (tree 140b42ad50) Commits folded into this step (3): - 2e066a7758 Return a failed build's result early in agent output (task 108 item 2) - 4bad455570 Name a failure without output sooner, and pluralize failures, in agent output (task 108 nits) - 129e04f8bb Stop continuing work before an in-process fallback (task 108, coord board 1981 a) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [rush-daemon] A request that meets a removed or changed installation waits for the old daemon, then restarts it and runs Swarm integration step 34; original commit c0e17d82ab (merge of swarm/r03-t91f at 503c9e6589). Scope: tasks 125, 91 and 116. Brings r03's task 125 (E1 and E2 of t05 board 1726) with task 91 (a note restarts the 25 s status timer) and task 116 (long socket paths, board 1476), re-tipped onto integration 1b51e7af08 with ch01's resolution of the 108(a) conflicts in 2 files and one test mock line (board 2697, board 2732). When the daemon's installation is removed or changes, a request waits for the running requests to drain, the daemon restarts and the request runs, and the wait line names the installation as the reason; `--wait-timeout` and `--no-wait` fail with that reason. CONFIRMED by t05 on r03-t91d (board 2154, board 2197: the daemon side of cold E2 is 19 of 19 rc 0 where s13 fails 3 of 3). r05 reviewed the drain part of the re-tip (board 2680). ch01 on d9f459d151 (board 2756): build rc 0 with 0 warnings, rush-daemon 590/0, rush-cli-client 398/0, rush-client-core 136/0, rush-daemon-protocol 200/0, rush-daemon-transport 81/0. 33 new tests fail on the old product and 4 suites can't load. 14 mutants, 8 killed; the 6 survivors are test NITs. E2E: with the installation renamed away mid-build, the next build is rc 0 in 21.4 s where the base fails at once with `Cannot find module '@microsoft/rush-lib'`. r03's own run (board 2760): rush-cli-client 398/0, rush-daemon 590/0. Gate: ch01 GATE OK board 2756 (tree d9f459d151) Commits folded into this step (4): - 29b5678225 [rush-daemon] Refuse a socket path too long to reach, and keep older daemons in their folder (task 116, board 1476) - 08ed06de32 [rush-daemon] Restart a daemon whose installation was removed or replaced (task 91) - 1393c18030 [rush-daemon] Answer requests with the installation restart once the running requests finish (task 125) - c2218d0792 [rush-daemon] Wait for an installation restart in the restart drain, like a restart for an environment (task 125, aligned with task 83) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [rush-daemon] A daemon-served build logs one native telemetry entry per request, and the daemon keeps serving when its own PATH repeats an entry Swarm integration step 35; original commit d87702c813 (merge of swarm/r02-int at c9d60eb5a4). Scope: task 49 and the PATH fix. Brings r02's task 49 (daemon-served builds emitted no Rush command telemetry, board 61): a long-lived engine logs one native telemetry entry per served build request, with the request id, and an iteration's requests are reported before the next iteration starts. Engine disposal waits at most 2 seconds for telemetry uploads (8eccf3a6f1, CONFIRMED by ch05 board 1987). The daemon keeps serving after a plugin adds names to process.env (80704de4e3), and keeps serving when its own startup PATH repeats an entry, where it used to turn away every build (202412e57b, board 2657, CONFIRMED by o05 board 2671; the blobs carry over byte for byte, board 2776). Re-tipped onto integration c0e17d82ab with ch01's order for the 125 conflict in WorkspaceRequestLifecycle.ts (board 2756, board 2774); 108(a)'s 'running' early result counts as early in the telemetry entry. ch01 on b8b1dc6e2f (board 2795): build rc 0 with 0 warnings, rush-daemon 607/0, rush-lib 1249/0, rush-cli-client 398/0, rush-client-core 136/0, rush-daemon-protocol 200/0, rush-daemon-transport 81/0. Reverting the PATH fix or the 2 s flush wait each fails its test; 3 of 5 mutants killed, and the 2 survivors are test NITs for s16. E2E: the early row logs earlyResult true, and a PATH that repeats an entry keeps the same daemon where the old product falls back. r02's own run on c9d60eb5a4 (board 2777) gives the same counts. ch01's finding that ODSP_TELEMETRY_TAG restarts the daemon is pre-existing and is task 168. Gate: ch01 GATE OK board 2795 (tree b8b1dc6e2f) Commits folded into this step (10): - 490b3d94f9 [rush-lib] Extract the phased telemetry entry builder from OperationGraph - 251d987b06 [rush-lib] Let a long-lived engine log one native telemetry entry per request - 6f62ff03f2 [rush-daemon] Log one native telemetry entry per served build request - eb0fd9090e Add change files for per-request daemon telemetry - 80704de4e3 [rush-daemon] Keep serving after a plugin adds names to process.env - 6a17f4bbdc [rush-daemon] Test that an iteration's requests are reported before the next iteration starts - e7b848d6a7 [rush-daemon] Add the request id to each served request's telemetry entry - 8eccf3a6f1 [rush-lib] Wait at most 2 seconds for telemetry uploads when an engine is disposed - ddff7a24ff [rush-daemon] Move the native engine test fixture into its own module - 202412e57b [rush-daemon] Keep serving when the daemon's own PATH repeats an entry Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [rush-daemon] A stuck request no longer keeps a stopped daemon alive Swarm integration step 36; original commit 372742afe7 (merge of swarm/r01-t147-int at a40a901822). Scope: tasks 115 and 147. Brings r01's task 115 (`daemon stop` and the first SIGTERM never ended a daemon whose request was stuck in an await that ignores cancellation) and task 147 (after a clean stop, a ref'd timer or handle kept the process alive after it released its socket and pid.json; t05 board 2199, o04 board 2269), re-tipped onto integration (board 2779, board 2793). The daemon now shuts down within a 10 s deadline, returns a typed result to the running request and releases its socket, lockfile and repository lock. CONFIRMED: 115 by ch05 board 2247 and o04 board 2266; 147 by t05 board 2608 and o04 board 2632. s16 batch A, item 1 of 5. ch01 on the batch (board 2867): build rc 0 with 0 warnings; rush-daemon 628/0, rush-lib 1252/0, rush-cli-client 398/0 on items 1-4; 45 of 48 mutants killed; in the e2e the daemon is gone 10.46 s after `daemon stop`, where s15's tree lingers 39.15 s. Gate: ch01 GATE OK board 2867 (tree 24fc1e6446) Commits folded into this step (2): - fe430fe236 rushd: a shutdown stuck on an await that ignores cancellation no longer hangs the daemon process (task 115) - 444107370d rushd: a daemon process that something else keeps running exits 2 s after the daemon stops (task 147) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [rush-daemon] A hot request reuses the latest input capture that started after it was received Swarm integration step 37; original commit 7ab68d689d (merge of r05 at a5e20e7c20). Scope: task 137. Brings r05's task 137: in a hot burst each request ran its own full-workspace project-config capture (about 0.09 s each), so n=32 ran 32 of them back to back and held the event loop for 3.1-3.3 s (board 2101). A request now reuses the latest capture that started after the request was received. CONFIRMED by t01 board 2303. s16 batch A, item 2 of 5 (ch01 board 2867; the suites and mutants are in item 1's message). Gate: ch01 GATE OK board 2867 (tree 3b55869396) Commits folded into this step (1): - a5e20e7c20 [rush-daemon] Reuse the latest input capture that started after a request was received Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [rush-azure-storage-build-cache-plugin] The Azure build cache picks up a credential written after the daemon's first cloud read Swarm integration step 38; original commit 0f5e259316 (merge of swarm/r03-t101 at f4a425ada3). Scope: task 101. Brings r03's task 101: the Azure storage build cache provider kept its first container client for the daemon's lifetime, so a credential written later was never used (ch05 board 1202, ch02 board 1229). It now uses the credential cached after the first cloud request. CONFIRMED by ch05 board 2290. s16 batch A, item 3 of 5 (ch01 board 2867). Gate: ch01 GATE OK board 2867 (tree 38dd18ec39) Commits folded into this step (1): - f4a425ada3 [rush-azure-storage-build-cache-plugin] Use a credential that is cached after the first cloud request (task 101) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [rush-lib] build-cache.json accepts azureBlobStorageConfiguration.storageEndpoint Swarm integration step 39; original commit ac2e51d2d2 (merge of swarm/r03-t144 at 04c7af3fbb). Scope: task 144. Brings r03's task 144: rush-lib's build-cache.schema.json rejected azureBlobStorageConfiguration.storageEndpoint, which the Azure plugin has read since microsoft/rushstack PR 5664. CONFIRMED by ch05 board 2396. s16 batch A, item 4 of 5 (ch01 board 2867). Gate: ch01 GATE OK board 2867 (tree 0fbdb27c44) Commits folded into this step (1): - 04c7af3fbb [rush-lib] Allow storageEndpoint in build-cache.json's azureBlobStorageConfiguration (task 144) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [rush-daemon] A lingering daemon logs its PID and stop time, without a stack Swarm integration step 40; original commit 86fc1ff37e (merge of swarm/r01-t147-nit at 7a76edd6ef). Scope: task 147 NITs. Brings r01's fix for the NITs in t05 board 2608 and o04 board 2632 (board 2814): the report is a timestamped log line through onLog, not an error with frames. t05 read the delta (board 2832), and r01's run through the real entry point is board 2851. ch01 on the final tree: build rc 0 with 0 warnings, rush-daemon 629/0, 7 of 7 mutants killed. s16 batch A, item 5 of 5 (ch01 board 2867). Gate: ch01 GATE OK board 2867 (tree d3a1eb9dbc) Commits folded into this step (1): - 7a76edd6ef rushd: a lingering daemon logs its PID and stop time, without a stack (task 147 NITs) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [rush-cli-client] The client prints a cancelling line at once, agent output shows an operation's error after its own output, and a cancelled request reports never-started operations as not run Swarm integration step 41; original commit cba3c1f17b (merge of swarm/r04-t132-153-int at 4b495d5f81). Scope: tasks 132, 142 and 153. Brings r04's tasks 132 (after Ctrl-C, SIGTERM or SIGHUP during a daemon request the client prints one line at once, plus a notice when the daemon doesn't confirm), 142 (agent output prints a failed operation's error even when the operation wrote output first) and 153 (a cancelled request no longer reports never-started operations with an earlier request's result), re-tipped on d87702c813. Second agents: t05 board 2323 (132), t07 board 2623 (142), t05 board 2524 (153). ch01's e2e on ch01-sB: the cancelling line and the unconfirmed notice appear, and c is reported aborted, not SUCCESS. s16 batch B, item 1 of 5 (ch01 board 2984). Gate: ch01 GATE OK board 2984 (tree 6453dd938e) Commits folded into this step (3): - 85ed03e693 Say at once that a cancelled request waits for rushd to stop it (task 132) - 892c1592eb Print a failed operation's error even after its output in agent output (task 142) - 52b9d3fe3b Report never-started operations of a cancelled request as aborted (task 153) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [rush-daemon] Tests for a cancel while continuing work, and for the early failure offer Swarm integration step 42; original commit c4693204c1 (merge of swarm/r04-t108-nits-int at 97e473c08e). Scope: task 108(a) NITs. Brings r04's tests for the three gate NITs of task 108(a) (ch01 board 2678). Tests only; they pass on the old product, and ch01's mutants show they're live. s16 batch B, item 2 of 5 (ch01 board 2984). Gate: ch01 GATE OK board 2984 (tree b21b6b1f0b) Commits folded into this step (1): - 97e473c08e Test the three gate NITs of task 108 (a): a cancel while continuing work stops, and the early failure offer's silent and aborted operations (ch01 board 2678) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [rush-client-core] The client relaunches the daemon after a startup helper exited, once its relaunch time passed Swarm integration step 43; original commit e073bd0f4b (merge of swarm/r03-t126 at da64013877). Scope: task 126. Brings r03's task 126: a client takes over a startup reservation whose helper is provably gone and whose daemon never became ready, instead of running in-process until `daemon stop --force` (t05 board 1574 and board 1652, row 13a). Second agent: t07 board 2751. ch01's e2e: the reservation is taken over after the relaunch time. s16 batch B, item 3 of 5 (ch01 board 2984). Gate: ch01 GATE OK board 2984 (tree a7707a430e) Commits folded into this step (1): - da64013877 rush-client-core: relaunch the daemon after a startup helper exited, once its relaunch time passed (task 126) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [rush-client-core] After a daemon crash the client reaps its orphaned operations before it returns Swarm integration step 44; original commit 80219e0892 (merge of swarm/r06-t154-int2 at 25c7f0135c). Scope: task 154. Brings r06's task 154 (board 2382), re-tipped on c0e17d82ab (board 2747): after a daemon crash, the client reaps the orphaned operations before returning, and before `--no-daemon` runs. Second agent: t05 board 2689 on 14baba4338. ch01's e2e: the orphan is reaped after a crash. s16 batch B, item 4 of 5 (ch01 board 2984). Gate: ch01 GATE OK board 2984 (tree 2453314813) Commits folded into this step (2): - e79486fa37 [rush-client-core] Stop a crashed daemon's operations before reporting the crash (task 154) - 14baba4338 [rush-cli-client] Reclaim a crashed daemon before Rush runs in-process (task 154) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [rush-daemon] Test that an expired protected project takes no warm-set lease Swarm integration step 45; original commit 1b18fcc3eb (merge of swarm/r07-t76-nit at 6c6f638a2e). Scope: task 76 NIT. Brings r07's test for the task 76 NIT. Test only. s16 batch B, item 5 of 5 (ch01 board 2984). Gate: ch01 GATE OK board 2984 (tree 104ca5f27d) Commits folded into this step (1): - 6c6f638a2e [rush-daemon] Warm set: test that an expired protected project takes no lease Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [package-deps-hash] A hot request with no file change since the previous one skips the repo-state work Swarm integration step 46; original commit a9e930338f (merge of swarm/r07-t114-int at e883c78766). Scope: task 114. Brings r07's task 114 (daemon L3b, D20.3): RepoStateCache keeps the last repo-state capture, and a hot request whose files haven't changed since reuses it instead of running git status, ls-tree, the inputs snapshot and each operation's own-state hash again. It also moves assertCompatibleInputs and isGraphDefinitionPath into WorkspaceInputsComparison.ts with the same path list. Second agents: m01 board 2842 and t04 board 2897 on d0f571a420 (t114verify: 0 STALE rows, board 2829); addenda t04 board 3042 and m01 board 3031. ch01's e2e on ch01-sC: 30 of 30 steps equal to --no-daemon, 0 STALE. s16 batch C, item 1 of 7 (ch01 board 3099). Gate: ch01 GATE OK board 3099 (tree 8299cd3e10) Commits folded into this step (4): - 5fbb206eb3 [package-deps-hash] Add RepoStateCache for processes that read the state of a repository repeatedly - b7904dd6d6 [rush-lib] Keep the state of the repository and reuse unchanged inputs snapshots in long-lived hosts - 7cb4c46d89 [rush-daemon] Skip comparing unchanged inputs snapshots on each request - d0f571a420 [package-deps-hash] Copy the index again when the sizes, attributes, configuration or filter change Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [rush-daemon] An invalid build command line fails with native Rush's error and exit code instead of running in-process Swarm integration step 47; original commit f84ce3a3cb (merge of swarm/r04-t78-int at cf5921e40f). Scope: task 78. Brings r04's task 78: when a warm daemon's parser rejects a build command line, the client prints the daemon's parse error once and exits 2, where it used to fall back in-process and print about 29 lines. Second agent: o04 board 2801. ch01's e2e on ch01-sC: a warm `build --nope` with agent output returns rc 2, 2 lines, in 0.16 s, answered by the daemon. s16 batch C, item 2 of 7 (ch01 board 3099). Gate: ch01 GATE OK board 3099 (tree d80d2a2ccb) Commits folded into this step (1): - cf5921e40f Fail an invalid build command line with native Rush's error and exit code instead of running it in-process (task 78) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [rush-daemon] Coalesced requests run with each requester's environment, and each gets its own telemetry key Swarm integration step 48; original commit b572eb50a0 (merge of swarm/r06-t90-int2 at 4b9a258212). Scope: tasks 81, 89 and 90. Brings r06's tasks 81 (coalesced requests no longer run once with the first requester's environment when an operation hashes an ignored variable they differ on), 89 (operations see process.env writes made by the same iteration's beforeExecuteIterationAsync, through a lazy per-participant environment) and 90 (a per-request telemetry key for operations in a coalesced batch), re-tipped on 1b51e7af08 (board 2747). Second agents: o04 board 783 and board 906, o05 board 1020, m03 board 1091 on adcb6f6e0f. It lands after r02's e7b848d6a7 (board 1166), which integration has. ch01's e2e on ch01-sC: two queued requests that differ in WT_SESSION run [ax, ay, c], as --no-daemon does. s16 batch C, item 3 of 7 (ch01 board 3099). Gate: ch01 GATE OK board 3099 (tree 8a64bfc100) Commits folded into this step (6): - 5565ff57ce [rush-daemon] Don't coalesce requests that disagree on a hashed per-session variable - 56ef144e1b [rush-lib] Pass getOperationEnvironment to the before/after iteration hooks - e4d4e90711 [rush-daemon] Let requests share a run when a hashed variable is unset in one and empty in the other - aec3ac8433 [rush-daemon] Start each operation from process.env as it is when the operation starts - 36d9a3a16f [rush-lib] Add getOperationRequestId to the operation graph's iteration options - 3099d5945f [rush-daemon] Give each operation the request id of the request whose environment it gets Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [rush-lib] A build-cache hit no longer restores an earlier failing run's stderr Swarm integration step 49; original commit c3f9fa2ac7 (merge of swarm/o03-t157 at 5b2a400f29). Scope: task 157. Brings o03's task 157: a cache hit no longer writes an older failing run's stderr into rush-logs/<name>.error.log, so a cache entry written after a failure isn't poisoned (ch02 board 2284, t05 board 2456). Second agent: t05 board 2704. ch01's e2e on ch01-sC: no error.log after either hit in the OK, ERR, OK sequence, and none in any cache entry. s16 batch C, item 4 of 7 (ch01 board 3099). Gate: ch01 GATE OK board 3099 (tree 09d5eabe82) Commits folded into this step (1): - 5b2a400f29 Keep cache hits from restoring an earlier run's stderr (task 157) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [rush-lib] An iteration checks an operation's build-cache inputs only when it needs them Swarm integration step 50; original commit 633640c81c (merge of swarm/r07-t152 at 7e98034976). Scope: task 152. Brings r07's task 152: an executed iteration no longer computes the build-cache disabled reason for all of the all-project graph's operations (about 0.7 s at odsp-web) or splices finished records out of the queue one at a time (about 0.1 s). Second agent: m01 board 2727. ch01's e2e on ch01-sC: the cache arm's hits and misses equal --no-daemon. s16 batch C, item 5 of 7 (ch01 board 3099). Gate: ch01 GATE OK board 3099 (tree 86cb10cdf4) Commits folded into this step (1): - 7e98034976 [rush-lib] Check the build cache inputs of an operation only when it's needed Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [rush-lib] A public beta API lets a plugin runner run :incremental under task 106's guard Swarm integration step 51; original commit e39062a035 (merge of swarm/r06-t131-int2 at 52c86b3174). Scope: task 131. Brings r06's task 131: rush-lib exposes a @beta API so that a plugin runner such as rush-fstrace-plugin can run an `:incremental` phase under task 106's L1a guard, re-tipped on 1b51e7af08 (board 2747). Second agent: o03 board 2360 on cb0891d79d. The test-file conflict in 7b8e8c665b keeps both sides. s16 batch C, item 6 of 7 (ch01 board 3099). Gate: ch01 GATE OK board 3099 (tree 07ac5f0139) Commits folded into this step (1): - cb0891d79d [rush-lib] Let operation runners from plugins use the incremental execution guard Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [rush-daemon] Answer a usage error only from a session that a build bound and checked Swarm integration step 52; original commit e12c8f3632 (merge of swarm/r04-t78-nit1 at 86a46e1d75). Scope: task 78 NIT 1. Brings r04's fix for o04's NIT 1 on task 78 (board 2801): the daemon answers a usage error only from a session that a build bound and checked. Second agent: o04 board 2948. s16 batch C, item 7 of 7 (ch01 board 3099). Gate: ch01 GATE OK board 3099 (tree 1af262606d) Commits folded into this step (1): - 86a46e1d75 Answer a usage error only from a session that a build bound and checked (task 78 follow-up, o04 board 2801 NIT 1) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [node-core-library] Test the edges of LockFile's time zone check Swarm integration step 53; original commit b122243826 (merge of o07/lockfile-tzrule-nits at 48fbc7ee88). Scope: task 121 NITs. Brings o07's fixes for ch01's 4 test NITs on task 121 (board 2972): tests, one doc sentence and a type-none change file. The branch is in <swarm path>, fetched read-only for this merge. s17 batch D1, item 1 of 10 (ch01 board 3310). Gate: ch01 GATE OK board 3104 and board 3310 (tree 20d42b41b6) Commits folded into this step (1): - 48fbc7ee88 [node-core-library] Test the edges of LockFile's time zone check Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [rush-daemon-protocol] Name the variables when a command's environment restarted the daemon Swarm integration step 54; original commit 898ff4dba1 (merge of swarm/r06-t88 at 04c6fb9d5a). Scope: task 88. Brings r06's fix for task 88 (board 2886): when a request restarts the daemon, or follows a restarted one, the client prints one line naming the differing variables (names only). Second agent: t07 board 2961. s17 batch D1, item 2 of 10. Gate: ch01 GATE OK board 3310 (tree ad9aa75f8c) Commits folded into this step (3): - ffbe43c152 [rush-daemon-protocol] Add the environmentChanged restart reason (task 88) - 8afd905246 [rush-daemon] Name the variables when the daemon restarts for an environment (task 88) - 04c6fb9d5a [rush-cli-client] Name the variables when a command's environment restarted the daemon (task 88) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [rush-daemon] A restart drain's waiter rechecks its tier Swarm integration step 55; original commit 075d2130b2 (merge of swarm/r05-t127-int at 236c184d94). Scope: task 127. Brings r05's task 127 follow-up (board 2436, re-tipped onto d87702c813 in board 2858): a Restart-tier change that is reverted under load no longer fails its waiter at 30 s. Second agent: o05 board 2552. s17 batch D1, item 3 of 10. Gate: ch01 GATE OK board 3310 (tree bdfad6af20) Commits folded into this step (2): - 961f007cae [rush-daemon] A request that waits for a restart drain rechecks whether it still needs the restart (task 127) - 0b1585667a [rush-daemon] Requests that wait for a restart drain share one recheck capture a second (task 127) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d3da49f-95f3-4291-9026-0b0f8e38675c * [rush-daemon] ODSP_TELEMETRY_TAG no longer restarts the daemon Swarm integration step 56; original commit 442af4ed14 (merge of swarm/r02-t168-int at b9b7f9dedd). Scope: task 168. Brings r02's task 168 (board 2939, re-tipped onto batch B in board 3030): setting, changing or unsetting ODSP_TELEMETRY_TAG keeps the warm daemon. Second agents: t07 board 3004 and m03 board 2975. s17 batch D1, item 4 of 10. Gate: ch01 GATE OK board 3310 (tree eadadd0318) Commits folded into this step (3): - 4a2af75d2b [rush-lib] [rush-daemon] Keep the warm daemon when a request's telemetry tag changes (task 168) - efb14d246a [rush-daemon] Log an early failed result's telemetry once its continuing work settles (task 168) - c888d079b3 [rush-lib] Test that th…

Sean Larkin (TheLarkInn) and others added 30 commits September 28, 2026 13:31

Sean Larkin (TheLarkInn) and others added 17 commits September 30, 2026 08:45

Sean Larkin (TheLarkInn) marked this pull request as ready for review September 30, 2026 11:53

Sean Larkin (TheLarkInn) requested review from Apostolis Haitalis (apostolisms), Bharat Middha (bmiddha), David Michon (dmichon-msft), Ian Clanton-Thuon (iclanton), Jonathon Anthony (jxanthony) and Pete Gonzalez (octogonz) as code owners September 30, 2026 11:53

Sean Larkin (TheLarkInn) merged commit 8c902ee into thelarkinn-dogfood-rush-daemon Sep 30, 2026
1 check passed

Sean Larkin (TheLarkInn) deleted the thelarkinn/rushd-swarm-bug-fixes branch September 30, 2026 11:54

Sean Larkin (TheLarkInn) mentioned this pull request Sep 30, 2026

[rush] Dogfood the Rush daemon in rushstack: allow command-scoped plugins, add snapshot workflow and guide #6046

Merged

Navigation Menu

[rush] rushd: fix the bugs found by dogfooding the Rush daemon (stacked on #6046) - #6102

[rush] rushd: fix the bugs found by dogfooding the Rush daemon (stacked on #6046)#6102
Sean Larkin (TheLarkInn) merged 266 commits into
thelarkinn-dogfood-rush-daemonfrom
thelarkinn/rushd-swarm-bug-fixes

Sean Larkin (TheLarkInn) commented Sep 30, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

Sean Larkin (TheLarkInn) commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Bugs (166 reported)

Fixed in this PR (130)

Fixed, with a known residual (4)

Not fixed yet (3): 1 in progress and 2 held back for a missing test

Covered by guidance, with no code change (1)

Notes for review

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Sean Larkin (TheLarkInn) commented Sep 30, 2026 •

edited

Loading