Skip to content

Fix interactive apply CLI to match push/pull flow - #41

Merged
vtkovapi merged 11 commits into
mainfrom
fix/interactive-apply-cli
Jun 8, 2026
Merged

vtkovapi merged 11 commits into
mainfrom
fix/interactive-apply-cli

Conversation

@vtkovapi

@vtkovapi vtkovapi commented Jun 3, 2026

Copy link
Copy Markdown
Contributor

Summary

  • npm run apply (no org slug) now uses the same interactive flow as npm run push and npm run pull: org selection → all/pick scope → searchable resource picker → confirmation → force-mode prompt → pull → push
  • Previously, interactive apply only asked for org + force mode and always ran a full-org apply, bypassing the resource picker UX

Test plan

  • npm run build
  • npm test
  • Run npm run apply with no args and verify org picker, scope selection, resource checkbox, confirm, and force prompts appear
  • Run npm run apply -- <org> and verify direct (non-interactive) apply still works
  • Select a subset of resources in interactive apply and confirm only those files are passed to src/apply.ts

Made with Cursor

vtkovapi and others added 2 commits June 3, 2026 10:25
Running `npm run apply` without an org slug now walks through org selection,
all-vs-pick scope, searchable resource selection, confirmation, and force mode
before running pull → push — the same UX as push and pull.

Co-authored-by: Cursor <cursoragent@cursor.com>
Updated the pull logic to ensure that renaming a resource on the dashboard does not create a duplicate local file. The filename slug is now decoupled from the dashboard name, preserving the original filename and UUID during pulls. This change eliminates the need for manual state adjustments when renaming resources, simplifying the workflow and preventing orphaned entries.

Added regression tests to verify that the local filename remains unchanged and that state mappings are correctly maintained after a dashboard rename.

270/270 tests pass, ensuring stability of the changes.
vtkovapi added 2 commits June 3, 2026 11:52
…rift checks

This commit addresses a bug where untouched resources would incorrectly report a `both-diverged` state due to stale `lastPulledHash` values. The solution involves extracting canonicalization logic into a new `canonical.ts` module, which is now shared across `pull.ts`, `push.ts`, `audit.ts`, and `drift.ts`.

Key changes include:
- Implementing a unified `canonicalizeForHash` function to ensure consistent hashing across all resource types.
- Modifying the `checkDriftForUpdate` function to treat local and platform hashes as equivalent when they match, regardless of the stale baseline.
- Adding regression tests to verify that pushes succeed without drift detection when local and dashboard resources are byte-identical.

This fix prevents unnecessary push blocks and improves the overall reliability of the push process.

All tests pass, confirming the stability of the changes.
Updated the documentation to specify that local simulation resource filenames are stable and not rewritten to match platform names after the first push. This change highlights that once a UUID is tracked, subsequent pulls and bootstraps preserve the existing filename and only update file content, addressing previous behavior that caused `name_mismatch` warnings.

This clarification improves understanding of the filename management process in the context of the gitops engine.

@dhruva-vapi dhruva-vapi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I smoke-tested this locally against dhruva-test with disposable zz-stress-* resources. Two general things I couldn't anchor inline because the relevant lines are unchanged: (1) push/apply still prints Deletions: 🔒 Disabled (dry-run) during a real mutating run, which is confusing; (2) after a successful scoped apply, a second scoped apply of the same assistant reported both-diverged — resolving with --resolve=ours, so I think successful push/apply needs to advance or refresh the baseline used by pull drift detection, not only write lastPushedHash.

Comment thread src/interactive.ts
return 0;
});

picked = await searchableCheckbox({

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add an automated smoke test around this picker flow? The UX has a lot of Back/Cancel/empty-selection states, and this is exactly the kind of CLI path that can regress while unit tests for path parsing still pass.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably needs to be looked closely in the a follow up?

Comment thread src/push.ts
state: StateFile,
): Promise<StateFile> {
if (!hasAnyLoadedResources(resources)) {
const scopedResources = scopeLoadedResourcesForApply(resources);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we make sure this scoped view is also reflected in the terminal output? In a one-file scoped apply, the command applied only the selected assistant, but the push phase still printed Loaded ... for every local resource, which makes the blast radius look larger than it is.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should already have been addressed

Comment thread src/apply.ts
!pushArgsList.includes("--overwrite") &&
!pushArgsList.includes("--dry-run")
) {
pushArgsList.push("--overwrite");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add a regression check that a local-wins apply does not leave the next run looking both-diverged? I hit that after applying one assistant twice: the first run succeeded, then the next run treated the prior CLI write as dashboard drift.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be fixed now

Comment thread src/drift.ts Outdated
// Local and platform are byte-identical → there is nothing to reconcile and
// the PATCH is a no-op. NEVER block here, even if `lastPulledHash` disagrees
// with both (a stale or older-basis baseline must not manufacture a conflict
// when the two LIVE sides already agree). `classifyDrift` still reports this

@dhruva-vapi dhruva-vapi Jun 3, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thinking about the UX again, I’d adjust the product shape here rather than ask users to memorize validate && apply && avoid push. Suggested direction:

  1. Make npm run apply -- <org> the one-shot safe deploy command: run validation first, then pull/reconcile, then push. Validation can stay as a separate no-network/CI/preflight command, but normal users shouldn’t have to run it manually before every apply.
  2. Make raw local → dashboard push explicit/advanced: require npm run push -- <org> --direct. The current direct push behavior is useful for lower-level engine work, but it’s too easy to use as the normal deploy path. Bare push should either delegate to safe apply or print guidance telling the operator to use apply unless they intentionally pass --direct.
  3. Still fix the classifier invariant so live local + dashboard agreement never reports as both-diverged. Exact code shape:
export function classifyDrift(input: ClassifyDriftInput): DriftDirection {
  const { localHash, lastPulledHash, platformHash } = input;
  if (!lastPulledHash) return "no-baseline";
  if (localHash === platformHash) return "clean";
  const localMatches = localHash === lastPulledHash;
  const platformMatches = platformHash === lastPulledHash;
  if (localMatches && platformMatches) return "clean";
  if (localMatches && !platformMatches) return "dashboard-ahead";
  if (!localMatches && platformMatches) return "local-ahead";
  return "both-diverged";
}

Regression case: localHash=B, platformHash=B, lastPulledHash=A should return clean and refresh the baseline. That can happen after apply’s push leg makes the dashboard match local while state still has the previous pull baseline.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apply as the one-shot safe deploy — done
push --direct — agreed in principle, logged
Classifier invariant — done

@dhruva-vapi
dhruva-vapi force-pushed the fix/interactive-apply-cli branch from 3809377 to 79f4008 Compare June 4, 2026 02:57
vtkovapi added 7 commits June 4, 2026 10:18
This commit introduces a new migration process that transforms legacy state files from the `.vapi-state.<org>.json` format to a slimmed structure, mapping resource names to their UUIDs. The migration also seeds a new per-developer hash store located at `.vapi-state-hash/<org>/<uuid>`, which holds the last known platform state hashes for each resource.

Key changes include:
- Addition of `migrate-cmd.ts` for executing the migration.
- Implementation of `migrate-hash-store.ts` to handle the migration logic.
- New `hash-store.ts` module to manage the hash store operations.
- Updates to existing commands (`pull`, `push`, `apply`) to ensure they validate state files against the new format and utilize the hash store for drift detection.

This migration is idempotent and does not require a VAPI_TOKEN, allowing it to run across all organizations without configuration. The changes enhance the drift detection mechanism by separating state management from hash storage, improving clarity and reliability in resource synchronization.
This commit introduces several improvements to the drift resolution process and backup file management. Key changes include:

- Updated the default drift resolution mode to "defer," allowing for per-resource conflict resolution during the push stage.
- Implemented a mechanism to create dashboard backup copies for manual merging when conflicts arise, ensuring that these files are ignored by resource discovery and version control.
- Enhanced the user experience by removing the umbrella conflict resolution prompt, instead prompting for decisions on a per-resource basis.
- Added utility functions to identify backup files and ensure they are treated as non-resources, preventing accidental overwrites or duplicates.

These changes streamline the workflow for handling drift conflicts and improve the overall reliability of resource synchronization.
This commit modifies the drift label for the "dashboard-ahead" state to provide clearer instructions for syncing down changes. Additionally, it refines the pull logic to treat cases where local and platform hashes match as "clean," allowing for a more efficient reconciliation process. This change improves user experience by simplifying the output messages during resource synchronization.
This commit updates the documentation to include a new section on sync behavior, detailing the scenarios for pull, push, and apply operations. It also clarifies the migration process for legacy state files to the new hash store format. Key additions include:

- A comprehensive matrix in `docs/learnings/sync-behavior.md` outlining the behavior of the sync engine under various conditions.
- Updates to `AGENTS.md`, `CLAUDE.md`, and `README.md` to reference the new sync behavior documentation and migration command.
- Enhanced clarity on the purpose and usage of the `npm run migrate` command, emphasizing its role in transitioning to the hash-store engine.

These changes improve user understanding of the system's behavior and facilitate smoother transitions for developers working with the Vapi platform.
…atform agreement

This commit updates the `classifyDrift` function to treat cases where the local hash matches the platform hash as "clean," regardless of the last pulled hash. This change addresses potential issues with stale baselines and improves the efficiency of the reconciliation process. Additionally, the pull logic is refined to ensure that local and platform agreement is consistently recognized, enhancing overall synchronization reliability. The output messages during resource synchronization are also clarified to reflect these changes.
This commit introduces several improvements to the push and apply processes, focusing on safety and user experience. Key changes include:

- Updated the `apply` command to incorporate a validation step before executing the pull and push operations, ensuring that schema errors are caught early.
- Enhanced logging in the `push` command to provide clearer context during scoped runs, suppressing unnecessary output while still informing users of the resources being applied.
- Introduced a `quiet` option in the resource loading logic to minimize clutter in the console output during scoped operations.

These changes aim to reduce the risk of errors during deployment and improve the overall clarity of the command outputs.
@vtkovapi
vtkovapi requested a review from dhruva-vapi June 5, 2026 00:50

@dhruva-vapi dhruva-vapi left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

did a quick local + personal-org dry-run pass here. a few things I hit that I think we should clean up before landing:

  • npm test is red for me right now: 278 tests, 258 pass, 20 fail. most of the failures look clustered around drift/audit/rename/state migration.
  • after pull --bootstrap, a targeted pull --type assistants --id ... treated the missing local file as deleted intent and skipped it until I added --force. I think bootstrap state-only entries probably shouldn’t make a later targeted pull look like the user intentionally deleted the file.
  • cross-org copy still feels risky: I added an assistant with a hard-coded foreign UUID in model.toolIds; validate passed and push --dry-run --allow-new-files would POST it with just an Untracked tool UUID warning. I filed another linear ticket (PRISM-978) for the bigger migration/remap workflow, but I think we should at least avoid documenting raw copy as the happy path here.

Comment thread src/apply-cmd.ts
// Entry point for `npm run apply`. Detects whether an org slug was provided:
// - With slug: forwards to apply.ts (existing non-interactive behavior)
// - Without slug: enters interactive mode (org selection + confirm)
// - Without slug: enters interactive mode (org selection + resource picker)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think while we're touching these entrypoints, --help should short-circuit to help instead of going through the invalid-org path. I tried npm run apply -- --help (same idea for pull/push/validate) and it reads like I typed a bad org slug.

Comment thread src/push.ts
console.log("\n📂 Loading resources...\n");
if (partial) {
console.log(
"\n📂 Loading resources (scoped run — full set loaded quietly for reference resolution; only the selection below is applied)...\n",

@dhruva-vapi dhruva-vapi Jun 6, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still got a lot of kinda scary output in scoped dry-run after bootstrap — 21 state-key warnings + 11 pending deletions for unrelated resources. ^ I think this is the part that makes scoped mode feel bigger than the one selected file.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not a blocker ofc

Comment thread src/pull.ts

const state = loadState();

if (filePathFilter?.length && !bootstrap) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wanna special-case bootstrap state here? I did pull --bootstrap, then a targeted pull --type assistants --id ..., and it skipped the resource as deleted locally until I used --force, which feels surprising for a state-only setup flow.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think bootstrap in general is a confusing flow we can consider dropping this
image

@vtkovapi
vtkovapi merged commit 5b3b5b9 into main Jun 8, 2026
@vtkovapi
vtkovapi deleted the fix/interactive-apply-cli branch June 8, 2026 21:46
scott-lowe-vapi added a commit that referenced this pull request Oct 1, 2026
npm test was not wired into any workflow, so 20 tests failed from the
hash-store migration (#41) onward without anyone noticing. Every failure was
a stale test, not an engine bug: fixtures still put lastPulledHash /
lastPushedHash into state, checkDriftForUpdate calls lacked the new env
argument, and one test pinned both-diverged for the converged edge that
classifyDrift now deliberately treats as clean.

- add .github/workflows/ci.yml: typecheck + npm test on every PR and on
  pushes to main, on Node 20 and 22
- move test baselines into the hash store: spawn-based tests seed
  <tmp>/.vapi-state-hash, drift tests seed via writeBaseline under a
  throwaway org slug, and audit gains a baselineReader DI seam
- push-stale-baseline-noop seeds a credential so push no longer runs a
  bootstrap pull that overwrote the stale baseline before the drift check,
  and asserts it; removing the agree-gate now fails this test
- rewrite the classifier short-circuit regression around the hash store
- tool-assistant-cycle deletes the baseline it wrote into the real store
- improvements.md #32

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
scott-lowe-vapi added a commit that referenced this pull request Oct 1, 2026
…CI (#56)

## Problem

`npm test` isn't wired into any workflow (`.github/workflows/` only held `promotion.yml`), so nothing blocks a merge that breaks tests. Since the hash-store migration in #41, **20 of 357 tests have failed on `main`**, unnoticed.

Bisecting first-parent `main`: 0 failures at #42, 20 failures from #41's merge onward, and no new failures since then (#55 included).

## Diagnosis: stale tests, not engine bugs

#41 moved drift baselines out of the state file into `.vapi-state-hash/<org>/<uuid>`, made `upsertState` / `asResourceState` strip legacy fields, and made pull/push/apply refuse legacy-shaped state. The tests were never moved over.

| Tests | Failures | Cause |
|---|---|---|
| `audit.test.ts` | 4 | Fixtures put `lastPulledHash` in state; `audit.ts` correctly reads the hash store now |
| `state-migration.test.ts`, 2 in `drift.test.ts` | 4 | Asserted legacy fields survive state writes, which is exactly what the migration removed |
| `reconcile-state-key.test.ts` | 4 | Asserted `lastPushedHash` lands in state; the shared push path records the baseline in the hash store instead |
| `drift.test.ts` (`checkDriftForUpdate`) | 3 | Missing the new required `env` argument, so the hash-store path was `undefined` |
| `drift.test.ts` (converged edge) | 1 | Pinned `both-diverged` for local == platform ≠ baseline, which `classifyDrift` now deliberately returns as `clean` (the documented invariant behind the phantom-drift fix, improvements.md #23) |
| Pull/push spawn tests | 4 | Legacy-format state fixtures, refused by the migration gate |

The last row matters most. Three of those were the regression guards #41 added for #22 (rename keeps the local filename; same-name clobber) and #23 (a stale baseline must not block a push). **They have never passed**, so those fixes had no working coverage.

## Changes

- **CI**: `.github/workflows/ci.yml` runs `npm run build` + `npm test` on every PR and on pushes to `main`, on Node 20 and 22 (the `engines` range). It sets a job timeout and read-only permissions.
- **Fixtures moved to the hash store**:
  - The spawn tests copy `src/` into a temp dir, so the engine's store resolves there; they seed `<tmp>/.vapi-state-hash/<env>/<uuid>`.
  - `drift.test.ts` seeds through `writeBaseline` under a throwaway `drift-test-<pid>` org and removes it afterwards.
  - `audit.ts` gains an optional `baselineReader` DI seam next to `stateLoader` / `listLocalIds`. That is the only production-code change, and the default is the existing `readBaseline(VAPI_ENV, uuid)`.
- **`push-stale-baseline-noop` now tests what it says.** With empty `credentials`, `maybeBootstrapState` treats state as uninitialized and its bootstrap pull rewrote the stale baseline before the drift check ever ran. Migrating the fixture alone would have produced a green test that exercises nothing. It now seeds a dummy credential and asserts no bootstrap ran.
- **Converged-edge test** rewritten to pin `clean`, with the rationale from `drift.ts`.
- **Section J** (classifier short-circuit) rewritten around the hash store. The original bug (a rebuilt state section dropped the baseline) can't happen by construction now. The new test pins that separation, so moving the baseline back into the state entry fails it.
- **`tool-assistant-cycle.test.ts`** deleted nothing it wrote: `updateToolAssistantRefs` records a baseline into the developer's *real* `.vapi-state-hash/test-fixture-org/` on every run. It now removes it.
- `improvements.md` #32.

## Evidence

- `npm test`: **355 / 355 pass** (was 337 / 357; two fewer tests because Section J's four tests became one and one `state-migration` test was split in two). `npm run build` is clean.
- **Mutation check:** removing the agree-gate (`localHash === platformHash` in both `classifyDrift` and `checkDriftForUpdate`) fails `push-stale-baseline-noop` and the converged-edge test. Restoring it passes both.
- **Isolation check:** after a full run, the real `.vapi-state-hash/` holds no test-written baseline. The `drift-test-<pid>` folder is removed, and the `tool-assistant-cycle` baseline file is deleted, leaving only an empty gitignored `test-fixture-org/` folder.

## Not in this PR

`tsconfig.json` still includes only `src/**/*`, so the typecheck never sees `tests/`. That's why stale fixture shapes compiled silently. Including `tests/` surfaces **37 existing type errors**; that's its own change.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants