[Proposal] Add deterministic sparse worktree lifecycle commands #1829

Summary

Large repositories make one full working tree per issue expensive even though linked worktrees already share Git objects. This proposal adds a deterministic, CLI-owned lifecycle for linked worktrees that can materialize only the tracked directories needed for a task by using cone-mode sparse checkout.

The lifecycle should be owned by @bradygaster/squad-cli, not implemented as prompt-only Git commands. Prompts cannot reliably provide transactions, capability detection, state-backend coordination, structured errors, crash recovery, or safe cleanup. Coordinator prompts and skills should call the CLI and consume its result instead of creating worktrees or editing a ledger themselves.

This is a proposal for maintainer agreement on the public contract and ownership boundary before implementation begins.

Storage problem and scope

Linked worktrees share the repository's object database, refs, and other common Git data. They do not duplicate all Git objects. Sparse checkout primarily reduces the tracked files materialized in each working tree.

Sparse checkout does not by itself reduce:

• dependencies such as node_modules;
• generated output, build artifacts, or caches;
• Git LFS payload storage generally;
• objects already present in the shared Git object database; or
• blobs later fetched by a partial clone.

Those storage buckets need separate policy and measurement. The initial success metric should be checked-out tracked-file bytes, reported separately from dependencies/generated output, shared Git objects, and LFS objects.

Proposed ownership

@bradygaster/squad-cli

The CLI owns:

• command parsing and human/JSON output;
• repository, branch, base-ref, destination, and path validation;
• Git capability probes;
• linked-worktree creation and registration;
• sparse scope selection and expansion;
• explicit tracked-file materialization;
• full-checkout fallback;
• dependency/bootstrap policy dispatch;
• lifecycle reconciliation, diagnostics, recovery plans, and safe removal;
• ledger transitions through the configured state backend; and
• result/error normalization.

@bradygaster/squad-sdk

The SDK change should remain minimal:

1. Add one atomic read-modify-write primitive to StateBackend so a ledger transition cannot be implemented as an unsafe caller-side read() followed by write(). Local state should use an atomic file replacement/lock strategy; orphan and notes-backed state should retain their existing compare-and-swap behavior; two-layer state should expose one coherent transition result.
2. Expose requested and effective state information rather than only returning a backend after a silent fallback. The CLI needs the requested backend, effective backend, state location, verification result, and fallback reason so mutating worktree operations can fail closed instead of recording state in an unintended backend.
3. Register/export a typed worktrees state collection for the versioned ledger. Callers use the SDK/backend abstraction and do not depend on a local filesystem path.

No Git worktree lifecycle should move into the SDK.

Proposed commands

The public command group has five operations:

squad worktree create --repo <path-or-id> --issue <number> --branch <name> --base <ref>
  [--path <destination>] [--scope <repo-relative-dir> ...]
  [--mode auto|sparse|full] [--json]

squad worktree expand (--path <worktree> | --id <ledger-id>)
  (--scope <repo-relative-dir> ... | --full)
  [--reason <text>] [--json]

squad worktree status [--repo <path-or-id>] [--path <worktree>] [--json]

squad worktree doctor [--repo <path-or-id>] [--path <worktree>]
  [--repair-plan] [--json]

squad worktree remove (--path <worktree> | --id <ledger-id>)
  [--after-merge] [--delete-branch] [--json]

Contract details:

• create supports a new branch with git worktree add --no-checkout -b <branch> <path> <base> and a separately validated existing-branch flow.
• --scope accepts repository-relative directories. File inputs are normalized to their containing cone directory. Paths outside the canonical repository are rejected.
• expand --scope is monotonic and idempotent. It uses sparse-checkout add; it never narrows an active worktree.
• expand --full runs the supported sparse-to-full conversion and records why it happened.
• status is read-only and parses git worktree list --porcelain -z, never human-formatted output.
• doctor is read-only. --repair-plan returns explicit proposed actions; a later implementation may add an independently confirmed apply step, but diagnostics must not silently run git worktree repair, delete a branch, or remove a registration.
• remove refuses dirty, locked, active-session, unmerged, open-PR, unknown-PR, or path-unsafe worktrees. --delete-branch is still conditional on proof that deletion is safe.

Human and JSON behavior

Human output should be concise and action-oriented:

• one final state (ready, removed, repair required, and so on);
• repository, branch, path, and effective checkout mode;
• selected sparse cones or full-fallback reason;
• warnings; and
• an exact recovery command or next action when the operation cannot complete.

Human output may use color only on an interactive terminal.

--json should:

• write exactly one JSON document to stdout;
• omit color, progress text, prompts, and incidental Git output;
• put unexpected diagnostic text on stderr;
• return exit code 0 only when ok is true; and
• treat a policy-approved full fallback as success while reporting checkoutMode: "full" and a non-null fallbackReason.

All five commands use the same stable envelope:

{
  "schemaVersion": 1,
  "command": "worktree.create",
  "ok": true,
  "result": {
    "id": "stable-ledger-id",
    "status": "ready",
    "repositoryId": "repository-id",
    "path": "<worktree-path>",
    "branch": "squad/123-example",
    "checkoutMode": "sparse",
    "sparsePaths": [".github", ".squad", ".copilot", "packages/example"],
    "fallbackReason": null
  },
  "warnings": [],
  "error": null
}

On failure, result is null and error is:

{
  "code": "STATE_BACKEND_UNAVAILABLE",
  "category": "state",
  "message": "The configured state backend is unavailable.",
  "retryable": false,
  "recoveryActions": ["Restore the configured state backend, then rerun the command."]
}

Fields may be added compatibly within schema version 1. Existing fields, meanings, and enum values must not be removed or repurposed without a schema-version change.

Activation, configuration, and precedence

Automatic issue-worktree creation remains opt-in. Worktrees are enabled only by:

• SQUAD_WORKTREES=1, as a process-scoped override; or
• workflow.config.json with worktrees.enabled: true.

worktreePolicy: "branch-isolated" remains a routing policy and does not activate worktrees. A --mode argument selects behavior after activation; it does not silently enable automatic worktree use or rewrite configuration.

Proposed effective precedence, highest first:

1. explicit command arguments for that invocation;
2. SQUAD_WORKTREES=1 for activation only;
3. matching per-repository overrides under worktrees.repositories;
4. top-level workflow.config.json.worktrees;
5. built-in defaults, including enabled: false.

.squad/config.json remains responsible for team-root, state-location, and state-backend identity rather than repository checkout policy.

Initial configuration:

{
  "worktrees": {
    "enabled": false,
    "defaultMode": "sparse",
    "root": null,
    "fallback": "full",
    "minimumGitVersion": "2.36.0",
    "recommendedGitVersion": "2.42.0",
    "sparse": {
      "baselinePaths": [".github", ".squad", ".copilot"],
      "sparseIndex": false,
      "scopePolicy": "expand-on-demand"
    },
    "repositories": {}
  }
}

Unknown enum values, escaping paths, contradictory settings, and non-cone baseline entries are configuration errors.

Capability probes and checkout flow

Do not infer support from version text alone. Before sparse creation, the manager should probe or verify:

• Git is at least 2.36; 2.42 or newer is recommended;
• linked worktrees and worktree add --no-checkout are supported;
• sparse-checkout set --cone --no-sparse-index is supported;
• sparse configuration is worktree-specific and extensions.worktreeConfig is effective;
• git worktree list --porcelain -z is available;
• explicit materialization with git reset --hard HEAD produces the expected branch and files; and
• required hooks, filters, submodules, LFS behavior, and repository bootstrap policy are qualified for sparse mode.

The sparse create transaction is:

1. Resolve the canonical repository/common Git directory, effective team/state roots, repository profile, issue, branch, refs, and destination.
2. Validate activation, branch/path safety, state backend, task scope, dependencies, hooks, submodules, and LFS policy.
3. Atomically write a creating ledger record.
4. Add the linked worktree with --no-checkout, using explicit -b for a new branch.
5. Apply the baseline plus normalized task cones in one sparse-checkout set --cone --no-sparse-index call.
6. Materialize with git reset --hard HEAD.
7. Run approved bootstrap/dependency preparation.
8. Verify branch, HEAD, sparse settings, required tracked paths, and a clean initial status.
9. Atomically transition the record to ready.
10. On failure, remove only resources created by this transaction. If complete rollback cannot be proven, retain a non-success recovery record.

Sparse index remains off by default. Enabling it requires a separate compatibility gate for supported Git operations, IDEs, language tooling, and repository profiles.

Full fallback

Full linked worktrees remain a first-class, permanent mode.

auto mode selects full checkout when scope cannot be determined safely, the task requires repository-wide search/generation/validation, or the repository profile has not qualified sparse behavior. A configured sparse operation may fall back to full only for an approved compatibility condition. Policy, containment, dirty-state, backend, or destructive-safety failures remain failures rather than being hidden by fallback.

For an existing sparse worktree, the full conversion is git sparse-checkout disable, followed by verification that all tracked files are materialized. The ledger remains ready, changes checkoutMode from sparse to full, and records fallbackReason.

Backend-owned version 1 ledger

The configured state backend owns the logical ledger. A local backend may persist it as a file, but that physical path is not public API and must never be edited by prompts or agents.

Version 1 records should include:

schemaVersion, id, transactionId
repositoryId, repositoryUrl, commonGitDir, canonicalCheckoutPath
teamRoot, requestedStateBackend, effectiveStateBackend, stateLocationId
issue, branch, baseRef, targetRef, path
checkoutMode, sparsePaths, sparseIndex, scopeRevision
activationSource, fallbackReason
dependencyStrategy, dependencyState, temporaryLinks
partialClone, submodules, lfs, hooksPolicy
createdAt, updatedAt, lastActivityAt, lastVerifiedAt
status, sessionIds, prUrl
creationDurationMs, expansionCount
trackedWorkingTreeBytes, dependencyBytes, generatedBytes
sharedObjectBytes, lfsBytes
cleanupResult, recoveryAction, lastErrorCode

The ledger must not store credentials, file contents, hook bodies, excluded-file details, or raw command output that may contain sensitive data.

Lifecycle transitions:

absent -> creating -> ready
                  \-> failed              (rollback completed)
                  \-> repair-required     (rollback incomplete or ownership uncertain)

ready -> expanding -> ready               (scope revision or sparse-to-full fallback)
                   \-> repair-required

ready -> removing -> removed
                  \-> repair-required     (for example, retained directory or registration drift)

status and doctor may report derived classifications such as dirty, locked, missing-directory, unregistered-directory, retained-directory, and ledger-drift, but read-only commands do not change lifecycle state.

Error taxonomy

Stable error codes should be grouped into these categories:

Codes within these categories should be specific and stable, for example CONFIG_DISABLED, GIT_CAPABILITY_MISSING, BRANCH_ALREADY_CHECKED_OUT, PATH_OUTSIDE_ALLOWED_ROOT, STATE_CONFLICT, WORKTREE_DIRTY, and ROLLBACK_INCOMPLETE.

Partial clone is a separate non-goal

This proposal does not add or enable partial clone. git clone --filter=blob:none --sparse changes primary-clone and network/offline behavior; linked worktrees then share that clone's promisor object database. Fetched objects are not assumed to be evicted.

Any partial-clone work should be a separate opt-in pilot using a new sibling clone, with server-filter, offline, submodule, LFS, and rollback qualification. Existing clones must not be mutated or replaced by this work.

Upstream implementation and test surfaces

Source reconciliation against dev identifies these upstream surfaces.

CLI source

• packages/squad-cli/src/cli-entry.ts - register squad worktree and help.
• New packages/squad-cli/src/cli/commands/worktree.ts - parse the five subcommands and render the common result envelope.
• New packages/squad-cli/src/cli/core/worktree-manager.ts (and small Git/ledger helpers if needed) - own transactions, probes, reconciliation, fallback, and cleanup.
• packages/squad-cli/src/cli/core/detect-squad-dir.ts - reuse resolveWorktreeMainCheckout; extend context rather than reimplementing .git pointer parsing.
• packages/squad-cli/src/cli/core/effective-squad-dir.ts - preserve local versus external state resolution and expose the effective context consumed by the manager.
• packages/squad-cli/src/cli/core/templates.ts - add the worktree and spawn references to TEMPLATE_MANIFEST.
• packages/squad-cli/src/cli/core/init.ts and packages/squad-cli/src/cli/core/upgrade.ts - keep initialization/upgrade and generated configuration aligned with the manager contract.

SDK source

• packages/squad-sdk/src/state-backend.ts - atomic state transition primitive and requested/effective backend result.
• packages/squad-sdk/src/state/collection-map.ts - typed worktrees collection.
• packages/squad-sdk/src/state/schema.ts - backend-routed ledger mapping.
• packages/squad-sdk/src/state/index.ts and packages/squad-sdk/src/index.ts - public exports.
• packages/squad-sdk/src/resolution.ts only if the effective state-location result cannot be exposed without extending the existing resolution type.

Existing and new tests

• test/worktree.test.ts - preserve linked-worktree/main-checkout resolution regressions.
• test/effective-squad-dir.test.ts - requested/effective state location and externalized state cases.
• test/state-backend.test.ts - atomic transitions, conflicts, effective fallback metadata, and local/orphan/two-layer behavior.
• test/template-routing.test.ts - require both new manifest entries and their .squad/templates/ destinations.
• test/cli/upgrade.test.ts and test/cli/init-upgrade-parity.test.ts - clean install/upgrade convergence.
• New test/cli/worktree.test.ts and manager fixtures - command parsing, JSON envelope, capability failures, new/existing branch flows, sparse/full/auto modes, failure injection, expansion, doctor, and safe removal.
• Platform fixtures should cover Windows paths/spaces/locks/reparse points and at least one non-Windows CI environment.

Shipped templates

• packages/squad-cli/templates/squad.agent.md.template
• packages/squad-cli/templates/worktree-reference.md
• packages/squad-cli/templates/spawn-reference.md
• packages/squad-cli/templates/issue-lifecycle.md
• packages/squad-cli/templates/skills/git-workflow/SKILL.md
• packages/squad-cli/templates/skills/session-recovery/SKILL.md

These templates should invoke the manager and consume structured results; they should not contain independent lifecycle commands or direct ledger writes. If mirrored copies under packages/squad-sdk/templates/ remain part of packaging, they need parity enforcement or removal from the authority chain.

Pilot and integration

The integration tracker is cshoe_microsoft/squad-craig#9.

After upstream ownership and contract approval, implementation, and the mandatory compatibility/safety qualification, squad-craig will be the five-issue pilot. The pilot will include targeted scope, multi-directory scope, mid-task expansion, repository-wide full fallback, and cleanup/recovery. It will report all fallbacks and missing-scope incidents, with a target of at least 70% median reduction in materialized tracked-file bytes and zero silent missing-path failures.

Maintainer decision requested

Before implementation starts, please approve this ownership boundary and public contract, or identify the changes needed.

The following questions are intended to be answerable in one issue comment:

1. Should @bradygaster/squad-cli own all five lifecycle commands, with the SDK limited to atomic/effective state support?
2. Are the command names, key arguments, JSON envelope, and exit-code behavior acceptable?
3. Is the opt-in activation and precedence model acceptable, including branch-isolated not enabling worktrees?
4. Is a backend-owned schema-version 1 ledger with the proposed transitions and error categories acceptable?
5. Should sparse index remain off by default and full linked worktrees remain the permanent fallback?
6. Are the listed source, test, and template surfaces the right implementation boundary?
7. Is squad-craig acceptable as the five-issue pilot after qualification?