Repository navigation
Fix interactive apply CLI to match push/pull flow - #41
Conversation
Running `npm run apply` without an org slug now walks through org selection, all-vs-pick scope, searchable resource selection, confirmation, and force mode before running pull → push — the same UX as push and pull. Co-authored-by: Cursor <cursoragent@cursor.com>
Updated the pull logic to ensure that renaming a resource on the dashboard does not create a duplicate local file. The filename slug is now decoupled from the dashboard name, preserving the original filename and UUID during pulls. This change eliminates the need for manual state adjustments when renaming resources, simplifying the workflow and preventing orphaned entries. Added regression tests to verify that the local filename remains unchanged and that state mappings are correctly maintained after a dashboard rename. 270/270 tests pass, ensuring stability of the changes.
…rift checks This commit addresses a bug where untouched resources would incorrectly report a `both-diverged` state due to stale `lastPulledHash` values. The solution involves extracting canonicalization logic into a new `canonical.ts` module, which is now shared across `pull.ts`, `push.ts`, `audit.ts`, and `drift.ts`. Key changes include: - Implementing a unified `canonicalizeForHash` function to ensure consistent hashing across all resource types. - Modifying the `checkDriftForUpdate` function to treat local and platform hashes as equivalent when they match, regardless of the stale baseline. - Adding regression tests to verify that pushes succeed without drift detection when local and dashboard resources are byte-identical. This fix prevents unnecessary push blocks and improves the overall reliability of the push process. All tests pass, confirming the stability of the changes.
Updated the documentation to specify that local simulation resource filenames are stable and not rewritten to match platform names after the first push. This change highlights that once a UUID is tracked, subsequent pulls and bootstraps preserve the existing filename and only update file content, addressing previous behavior that caused `name_mismatch` warnings. This clarification improves understanding of the filename management process in the context of the gitops engine.
dhruva-vapi
left a comment
There was a problem hiding this comment.
I smoke-tested this locally against dhruva-test with disposable zz-stress-* resources. Two general things I couldn't anchor inline because the relevant lines are unchanged: (1) push/apply still prints Deletions: 🔒 Disabled (dry-run) during a real mutating run, which is confusing; (2) after a successful scoped apply, a second scoped apply of the same assistant reported both-diverged — resolving with --resolve=ours, so I think successful push/apply needs to advance or refresh the baseline used by pull drift detection, not only write lastPushedHash.
| return 0; | ||
| }); | ||
|
|
||
| picked = await searchableCheckbox({ |
There was a problem hiding this comment.
Can we add an automated smoke test around this picker flow? The UX has a lot of Back/Cancel/empty-selection states, and this is exactly the kind of CLI path that can regress while unit tests for path parsing still pass.
There was a problem hiding this comment.
Probably needs to be looked closely in the a follow up?
| state: StateFile, | ||
| ): Promise<StateFile> { | ||
| if (!hasAnyLoadedResources(resources)) { | ||
| const scopedResources = scopeLoadedResourcesForApply(resources); |
There was a problem hiding this comment.
Can we make sure this scoped view is also reflected in the terminal output? In a one-file scoped apply, the command applied only the selected assistant, but the push phase still printed Loaded ... for every local resource, which makes the blast radius look larger than it is.
There was a problem hiding this comment.
Should already have been addressed
| !pushArgsList.includes("--overwrite") && | ||
| !pushArgsList.includes("--dry-run") | ||
| ) { | ||
| pushArgsList.push("--overwrite"); |
There was a problem hiding this comment.
Can we add a regression check that a local-wins apply does not leave the next run looking both-diverged? I hit that after applying one assistant twice: the first run succeeded, then the next run treated the prior CLI write as dashboard drift.
There was a problem hiding this comment.
Should be fixed now
| // Local and platform are byte-identical → there is nothing to reconcile and | ||
| // the PATCH is a no-op. NEVER block here, even if `lastPulledHash` disagrees | ||
| // with both (a stale or older-basis baseline must not manufacture a conflict | ||
| // when the two LIVE sides already agree). `classifyDrift` still reports this |
There was a problem hiding this comment.
Thinking about the UX again, I’d adjust the product shape here rather than ask users to memorize validate && apply && avoid push. Suggested direction:
- Make
npm run apply -- <org>the one-shot safe deploy command: run validation first, then pull/reconcile, then push. Validation can stay as a separate no-network/CI/preflight command, but normal users shouldn’t have to run it manually before every apply. - Make raw local → dashboard push explicit/advanced: require
npm run push -- <org> --direct. The current direct push behavior is useful for lower-level engine work, but it’s too easy to use as the normal deploy path. Barepushshould either delegate to safe apply or print guidance telling the operator to useapplyunless they intentionally pass--direct. - Still fix the classifier invariant so live local + dashboard agreement never reports as
both-diverged. Exact code shape:
export function classifyDrift(input: ClassifyDriftInput): DriftDirection {
const { localHash, lastPulledHash, platformHash } = input;
if (!lastPulledHash) return "no-baseline";
if (localHash === platformHash) return "clean";
const localMatches = localHash === lastPulledHash;
const platformMatches = platformHash === lastPulledHash;
if (localMatches && platformMatches) return "clean";
if (localMatches && !platformMatches) return "dashboard-ahead";
if (!localMatches && platformMatches) return "local-ahead";
return "both-diverged";
}Regression case: localHash=B, platformHash=B, lastPulledHash=A should return clean and refresh the baseline. That can happen after apply’s push leg makes the dashboard match local while state still has the previous pull baseline.
There was a problem hiding this comment.
Apply as the one-shot safe deploy — done
push --direct — agreed in principle, logged
Classifier invariant — done
3809377 to
79f4008
Compare
This commit introduces a new migration process that transforms legacy state files from the `.vapi-state.<org>.json` format to a slimmed structure, mapping resource names to their UUIDs. The migration also seeds a new per-developer hash store located at `.vapi-state-hash/<org>/<uuid>`, which holds the last known platform state hashes for each resource. Key changes include: - Addition of `migrate-cmd.ts` for executing the migration. - Implementation of `migrate-hash-store.ts` to handle the migration logic. - New `hash-store.ts` module to manage the hash store operations. - Updates to existing commands (`pull`, `push`, `apply`) to ensure they validate state files against the new format and utilize the hash store for drift detection. This migration is idempotent and does not require a VAPI_TOKEN, allowing it to run across all organizations without configuration. The changes enhance the drift detection mechanism by separating state management from hash storage, improving clarity and reliability in resource synchronization.
This commit introduces several improvements to the drift resolution process and backup file management. Key changes include: - Updated the default drift resolution mode to "defer," allowing for per-resource conflict resolution during the push stage. - Implemented a mechanism to create dashboard backup copies for manual merging when conflicts arise, ensuring that these files are ignored by resource discovery and version control. - Enhanced the user experience by removing the umbrella conflict resolution prompt, instead prompting for decisions on a per-resource basis. - Added utility functions to identify backup files and ensure they are treated as non-resources, preventing accidental overwrites or duplicates. These changes streamline the workflow for handling drift conflicts and improve the overall reliability of resource synchronization.
This commit modifies the drift label for the "dashboard-ahead" state to provide clearer instructions for syncing down changes. Additionally, it refines the pull logic to treat cases where local and platform hashes match as "clean," allowing for a more efficient reconciliation process. This change improves user experience by simplifying the output messages during resource synchronization.
This commit updates the documentation to include a new section on sync behavior, detailing the scenarios for pull, push, and apply operations. It also clarifies the migration process for legacy state files to the new hash store format. Key additions include: - A comprehensive matrix in `docs/learnings/sync-behavior.md` outlining the behavior of the sync engine under various conditions. - Updates to `AGENTS.md`, `CLAUDE.md`, and `README.md` to reference the new sync behavior documentation and migration command. - Enhanced clarity on the purpose and usage of the `npm run migrate` command, emphasizing its role in transitioning to the hash-store engine. These changes improve user understanding of the system's behavior and facilitate smoother transitions for developers working with the Vapi platform.
…atform agreement This commit updates the `classifyDrift` function to treat cases where the local hash matches the platform hash as "clean," regardless of the last pulled hash. This change addresses potential issues with stale baselines and improves the efficiency of the reconciliation process. Additionally, the pull logic is refined to ensure that local and platform agreement is consistently recognized, enhancing overall synchronization reliability. The output messages during resource synchronization are also clarified to reflect these changes.
This commit introduces several improvements to the push and apply processes, focusing on safety and user experience. Key changes include: - Updated the `apply` command to incorporate a validation step before executing the pull and push operations, ensuring that schema errors are caught early. - Enhanced logging in the `push` command to provide clearer context during scoped runs, suppressing unnecessary output while still informing users of the resources being applied. - Introduced a `quiet` option in the resource loading logic to minimize clutter in the console output during scoped operations. These changes aim to reduce the risk of errors during deployment and improve the overall clarity of the command outputs.
There was a problem hiding this comment.
did a quick local + personal-org dry-run pass here. a few things I hit that I think we should clean up before landing:
npm testis red for me right now: 278 tests, 258 pass, 20 fail. most of the failures look clustered around drift/audit/rename/state migration.- after
pull --bootstrap, a targetedpull --type assistants --id ...treated the missing local file as deleted intent and skipped it until I added--force. I think bootstrap state-only entries probably shouldn’t make a later targeted pull look like the user intentionally deleted the file. - cross-org copy still feels risky: I added an assistant with a hard-coded foreign UUID in
model.toolIds;validatepassed andpush --dry-run --allow-new-fileswould POST it with just anUntracked tool UUIDwarning. I filed another linear ticket (PRISM-978) for the bigger migration/remap workflow, but I think we should at least avoid documenting raw copy as the happy path here.
| // Entry point for `npm run apply`. Detects whether an org slug was provided: | ||
| // - With slug: forwards to apply.ts (existing non-interactive behavior) | ||
| // - Without slug: enters interactive mode (org selection + confirm) | ||
| // - Without slug: enters interactive mode (org selection + resource picker) |
There was a problem hiding this comment.
I think while we're touching these entrypoints, --help should short-circuit to help instead of going through the invalid-org path. I tried npm run apply -- --help (same idea for pull/push/validate) and it reads like I typed a bad org slug.
| console.log("\n📂 Loading resources...\n"); | ||
| if (partial) { | ||
| console.log( | ||
| "\n📂 Loading resources (scoped run — full set loaded quietly for reference resolution; only the selection below is applied)...\n", |
There was a problem hiding this comment.
I still got a lot of kinda scary output in scoped dry-run after bootstrap — 21 state-key warnings + 11 pending deletions for unrelated resources. ^ I think this is the part that makes scoped mode feel bigger than the one selected file.
|
|
||
| const state = loadState(); | ||
|
|
||
| if (filePathFilter?.length && !bootstrap) { |
There was a problem hiding this comment.
wanna special-case bootstrap state here? I did pull --bootstrap, then a targeted pull --type assistants --id ..., and it skipped the resource as deleted locally until I used --force, which feels surprising for a state-only setup flow.
npm test was not wired into any workflow, so 20 tests failed from the hash-store migration (#41) onward without anyone noticing. Every failure was a stale test, not an engine bug: fixtures still put lastPulledHash / lastPushedHash into state, checkDriftForUpdate calls lacked the new env argument, and one test pinned both-diverged for the converged edge that classifyDrift now deliberately treats as clean. - add .github/workflows/ci.yml: typecheck + npm test on every PR and on pushes to main, on Node 20 and 22 - move test baselines into the hash store: spawn-based tests seed <tmp>/.vapi-state-hash, drift tests seed via writeBaseline under a throwaway org slug, and audit gains a baselineReader DI seam - push-stale-baseline-noop seeds a credential so push no longer runs a bootstrap pull that overwrote the stale baseline before the drift check, and asserts it; removing the agree-gate now fails this test - rewrite the classifier short-circuit regression around the hash store - tool-assistant-cycle deletes the baseline it wrote into the real store - improvements.md #32 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…CI (#56) ## Problem `npm test` isn't wired into any workflow (`.github/workflows/` only held `promotion.yml`), so nothing blocks a merge that breaks tests. Since the hash-store migration in #41, **20 of 357 tests have failed on `main`**, unnoticed. Bisecting first-parent `main`: 0 failures at #42, 20 failures from #41's merge onward, and no new failures since then (#55 included). ## Diagnosis: stale tests, not engine bugs #41 moved drift baselines out of the state file into `.vapi-state-hash/<org>/<uuid>`, made `upsertState` / `asResourceState` strip legacy fields, and made pull/push/apply refuse legacy-shaped state. The tests were never moved over. | Tests | Failures | Cause | |---|---|---| | `audit.test.ts` | 4 | Fixtures put `lastPulledHash` in state; `audit.ts` correctly reads the hash store now | | `state-migration.test.ts`, 2 in `drift.test.ts` | 4 | Asserted legacy fields survive state writes, which is exactly what the migration removed | | `reconcile-state-key.test.ts` | 4 | Asserted `lastPushedHash` lands in state; the shared push path records the baseline in the hash store instead | | `drift.test.ts` (`checkDriftForUpdate`) | 3 | Missing the new required `env` argument, so the hash-store path was `undefined` | | `drift.test.ts` (converged edge) | 1 | Pinned `both-diverged` for local == platform ≠ baseline, which `classifyDrift` now deliberately returns as `clean` (the documented invariant behind the phantom-drift fix, improvements.md #23) | | Pull/push spawn tests | 4 | Legacy-format state fixtures, refused by the migration gate | The last row matters most. Three of those were the regression guards #41 added for #22 (rename keeps the local filename; same-name clobber) and #23 (a stale baseline must not block a push). **They have never passed**, so those fixes had no working coverage. ## Changes - **CI**: `.github/workflows/ci.yml` runs `npm run build` + `npm test` on every PR and on pushes to `main`, on Node 20 and 22 (the `engines` range). It sets a job timeout and read-only permissions. - **Fixtures moved to the hash store**: - The spawn tests copy `src/` into a temp dir, so the engine's store resolves there; they seed `<tmp>/.vapi-state-hash/<env>/<uuid>`. - `drift.test.ts` seeds through `writeBaseline` under a throwaway `drift-test-<pid>` org and removes it afterwards. - `audit.ts` gains an optional `baselineReader` DI seam next to `stateLoader` / `listLocalIds`. That is the only production-code change, and the default is the existing `readBaseline(VAPI_ENV, uuid)`. - **`push-stale-baseline-noop` now tests what it says.** With empty `credentials`, `maybeBootstrapState` treats state as uninitialized and its bootstrap pull rewrote the stale baseline before the drift check ever ran. Migrating the fixture alone would have produced a green test that exercises nothing. It now seeds a dummy credential and asserts no bootstrap ran. - **Converged-edge test** rewritten to pin `clean`, with the rationale from `drift.ts`. - **Section J** (classifier short-circuit) rewritten around the hash store. The original bug (a rebuilt state section dropped the baseline) can't happen by construction now. The new test pins that separation, so moving the baseline back into the state entry fails it. - **`tool-assistant-cycle.test.ts`** deleted nothing it wrote: `updateToolAssistantRefs` records a baseline into the developer's *real* `.vapi-state-hash/test-fixture-org/` on every run. It now removes it. - `improvements.md` #32. ## Evidence - `npm test`: **355 / 355 pass** (was 337 / 357; two fewer tests because Section J's four tests became one and one `state-migration` test was split in two). `npm run build` is clean. - **Mutation check:** removing the agree-gate (`localHash === platformHash` in both `classifyDrift` and `checkDriftForUpdate`) fails `push-stale-baseline-noop` and the converged-edge test. Restoring it passes both. - **Isolation check:** after a full run, the real `.vapi-state-hash/` holds no test-written baseline. The `drift-test-<pid>` folder is removed, and the `tool-assistant-cycle` baseline file is deleted, leaving only an empty gitignored `test-fixture-org/` folder. ## Not in this PR `tsconfig.json` still includes only `src/**/*`, so the typecheck never sees `tests/`. That's why stale fixture shapes compiled silently. Including `tests/` surfaces **37 existing type errors**; that's its own change. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

Summary
npm run apply(no org slug) now uses the same interactive flow asnpm run pushandnpm run pull: org selection → all/pick scope → searchable resource picker → confirmation → force-mode prompt → pull → pushTest plan
npm run buildnpm testnpm run applywith no args and verify org picker, scope selection, resource checkbox, confirm, and force prompts appearnpm run apply -- <org>and verify direct (non-interactive) apply still workssrc/apply.tsMade with Cursor