ci: validate every org's resources on every pull request

[scott-lowe-vapi]

Value

V.A.L.U.E. tier: small — touches .github/workflows/ (a blast-radius path), and changes what customer forks see on their pull requests.

• Problem: nothing ran npm run validate before merge. apply, and so promotion, refuses to deploy a config with validation errors: a name over 40 characters, structured-output lockstep, duplicated prompts, the maxTokens floor, or the voice schema. That refusal came after merge: main held a config that wouldn't deploy, and promotion stopped until someone opened a fix PR. Plain push only warns, then can fail partway with an API 400. Each rule in the validator comes from a real mid-push failure (improvements.md membersOverrides.artifactPlan.structuredOutputIds is requiring UUID #8, Specifying handoff tools in a squad requires UUID to function correctly #9, fix(call): clear wrapped partial transcripts cleanly in npm run call #11, feat: simulation suite runner (npm run sim) #18, refactor: state schema with per-resource content hashes #19).
• Who it affects: every repository on this template, and especially multi-org repos that promote from main.
• What changes:
  • A new Validate resources job in ci.yml runs validate for every folder under resources/ on every pull request. It reports every failing org rather than stopping at the first.
  • No engine change. validate makes no network call; loading the engine's config only requires a key to be set, so the step sets a placeholder that is never sent. The job has no secrets, so it runs the same on forks and Dependabot PRs.
  • It's in ci.yml, not the PR check workflow, so every fork gets it without turning on PR checks. On this template, which has no org folders, it does nothing.
  • Docs:
    • the README quick start;
    • the AGENTS.md change loop: if the check fails, fix the errors and don't weaken the check;
    • the workflows guide: make it a required check;
    • the PR checks and troubleshooting guides;
    • improvements.md Add GitOps Support for Pronunciation Dictionaries with Versioned Updates #37.

Heads-up for forks: plain push only warned about these errors, so a repo may already carry some. The first PR after this lands will show them, whatever it changes. The troubleshooting guide covers it. apply already refused those configs, so this moves an existing failure earlier rather than adding a new one.

Evidence of value

tests/ci-validate-workflow.test.ts runs the job's real step, read from ci.yml, against copies of the starter example:

Case	Result
No org folders	passes, "nothing to validate"
Two valid orgs	passes, both validated
One org with a 41+ character assistant name, one valid	fails; both validated, the error names only the bad org and the reason ("Vapi caps at 40")
A folder that isn't a valid org name	fails, naming it
The job's secrets	none; checkout doesn't persist credentials; the only key is the placeholder

Mutation: making the loop ignore validate's exit code fails the two failure-case tests.

Testing plan

• npm test (514 tests) and npx tsc --noEmit pass.
• actionlint, from ci: lint workflows and test the PR check's key-sharing decision #75, lints the new job in this PR's CI.
• Not tested: a customer repo with real resources. The validator itself is unchanged, and apply already runs it on every deploy.

Refs TEST-141

🤖 Generated with Claude Code

[scott-lowe-vapi]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• feat: identify every gitops API request with a User-Agent #78
• feat(validate): catch broken references and show findings on the PR #77
• ci: validate every org's resources on every pull request #76 👈 (View in Graphite)
• ci: lint workflows and test the PR check's key-sharing decision #75
• test(docs): check that doc links, commands and flags exist #74
• fix(cleanup): never delete resources matched by .vapi-ignore #73
• docs(agents): one compact guide for Claude, Codex and Cursor #72
• docs: edit the guides for new readers and fix inaccuracies #71
• docs: restructure the README into a short landing page and topic guides #70
• docs: prepare for public release: scrub internal names, fix examples, add community files #69
• feat(promotion): gate promotion out of an org on a passing check #66
• fix(promotion): commit the files of transitions that applied when a later one fails #65
• test(promotion): pin the exact files promoting an assistant writes to the target #68
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[chris-garber-vapi]

I think running validate in CI makes sense

[chris-garber-vapi]

Review of the Validate resources job. Nothing here blocks merge (no 🔴 or 🟠). The main themes: the docs overstate what fails (3 of the 5 named rules are warnings, and they stay hidden), the local repro path needs a real org key, env-dependent .ts resources validate differently in CI than under apply, and every org being validated on every PR widens the blast radius. Suggestion blocks were run locally against the PR head unless noted.

[chris-garber-vapi]

🟢 npm ci --ignore-scripts is enough here, and it skips the native builds of the optional audio deps (mic, speaker) that validate never loads.

I checked this locally. After npm ci --ignore-scripts, tsx and esbuild run fine and all 5 tests in tests/ci-validate-workflow.test.ts pass. The job also stops depending on the runner having a C toolchain or ALSA headers. Don't use --omit=optional: esbuild's platform binary is an optional dependency, so tsx would break.

Suggested change

	- run: npm ci
	# Install scripts only build the optional audio deps for `npm run call`.
	- run: npm ci --ignore-scripts

[chris-garber-vapi]

🟡 This isn't quite "the same checks apply runs": .ts resources execute on load, and here they can't see the .env.<org> values they get under apply.

config.ts merges .env.<org> into process.env before any resource loads. resourceDirLoad then import()s every .ts resource, and docs/guides/file-formats.md pitches those files as "useful for generating config". This job has no .env.<org>, so a resource that reads the environment validates differently here. Reproduced:

// resources/clinic/assistants/overflow.ts
const site = process.env.CLINIC_SITE;
if (!site) throw new Error("CLINIC_SITE is not set");
export default { name: `${site} Overflow` };

== local, with .env.clinic (what apply runs):
✅ Validation passed.
== CI-like, placeholder key only:
❌ Validation failed: Failed to import TypeScript resource "overflow.ts": Error: CLINIC_SITE is not set

So this check fails a config that apply accepts. Once the check is required, that blocks every PR in the repo, and AGENTS.md now says not to weaken the check. It can go wrong the other way too: name: `${process.env.SITE ?? ""} …` can pass here and still break the 40-character cap under apply.

The cheapest fix is to document the constraint instead of changing behavior:

• In docs/guides/file-formats.md, under TypeScript resources, add something like: "CI validates .ts resources without your .env.<org>. Build them from files in the repo, not from process.env."
• In docs/guides/troubleshooting.md, add a line saying that a Failed to import TypeScript resource … is not set error from this check means the resource reads .env.<org>.
• Soften this comment, and the README and workflows.md wording, to "the same validator apply runs".

[chris-garber-vapi]

🟡 Validating every org on every PR means that one broken org, or one validator false positive, fails every PR in the repo, including PRs from teams that never touch that org.

The heads-up in the description covers errors that already exist. The ongoing cost is the blast radius. When apply refuses a config, only that org's deploys stop. When this job fails, every PR in the repo is blocked. In a multi-team repo, team A's PRs go red because of team B's org, and the docs tell A to make this a required check. With AGENTS.md's new "Don't weaken the check", an agent working for team A can only get unblocked by editing team B's resources.

If a PR touches neither the engine nor org B, org B's result can't change. Re-validating B adds no signal; it only adds ways to block. Suggest:

• Pushes to main, and PRs that touch src/**, package*.json or ci.yml: validate every org. That catches validator and engine regressions, and breakage that was already there turns main red, which is where it belongs.
• All other PRs: validate only the orgs whose resources/<org>/ changed.

Sketch. Checkout needs fetch-depth: 2: on pull_request, HEAD is the merge commit and HEAD^1 is the base.

if [[ "$GITHUB_EVENT_NAME" == "pull_request" ]]; then
  changed=$(git diff --name-only HEAD^1 HEAD)
  if ! grep -qE '^(src/|package(-lock)?\.json$|\.github/workflows/ci\.yml$)' <<<"$changed"; then
    mapfile -t touched < <(sed -nE 's#^resources/([^/]+)/.*#\1#p' <<<"$changed" | sort -u)
    # keep only entries of $orgs that appear in $touched (a deleted org has no folder)
  fi
fi

Keeping every org is a defensible choice too. In that case, workflows.md should say plainly that making this check required ties every team's PRs to the health of every org.

[chris-garber-vapi]

🟢 Nothing enforces "never sent". Pinning VAPI_BASE_URL to an unroutable address would make any accidental API call fail on the runner instead.

Right now the claim holds only because nobody has added a network call. Suppose validate-cmd, or something it imports, starts calling the API, for example to check references against the platform. This step would then send Bearer validate-only-never-sent to api.vapi.ai and fail with a confusing 401. In config.ts an exported env var beats .env* files, so the pin sticks:

Suggested change

	VAPI_PRIVATE_API_KEY: validate-only-never-sent
	VAPI_PRIVATE_API_KEY: validate-only-never-sent
	# Unroutable, so an accidental API call fails here instead of
	# reaching api.vapi.ai.
	VAPI_BASE_URL: http://127.0.0.1:9

The key: expectation in tests/ci-validate-workflow.test.ts compares the whole env with deepEqual, so it needs VAPI_BASE_URL added too.

The root cause is worth a stacked PR. config.ts exits at import when no key is set, even for offline commands. Checking for the key lazily, in the API client or in a requireApiKey() that only network commands call, would remove the need for this placeholder. It would also drop it from the troubleshooting guide and AGENTS.md (see my comments there).

[chris-garber-vapi]

🟡 Three of the five rules this PR says it catches are warnings, so they pass this check and stay hidden inside a collapsed log group.

In src/validate.ts, so-assistant-lockstep, prompt-duplicate-* and max-tokens-floor have severity: "warn". Only name-length and voice-provider-schema are errors. validate-cmd exits 0 on warnings, so on a green run nobody sees them unless they open the job and expand the group. For example, the starter example that the "every org is valid" test copies already emits one, and the test passes without anyone noticing:

⚠️  [so-assistant-lockstep] assistants/receptionist (artifactPlan.structuredOutputIds): assistant "receptionist" lists SO "call-summary" … but SO "call-summary" does NOT list this assistant in assistant_ids

Turning findings into annotations shows them on the PR's Checks summary without changing what fails, and needs no engine change. I ran this against the starter fixture, and all 5 tests still pass:

Suggested change

	for org in "${orgs[@]}"; do
	echo "::group::Validate ${org}"
	if ! node --import tsx src/validate-cmd.ts "$org"; then
	failed+=("$org")
	fi
	echo "::endgroup::"
	done
	for org in "${orgs[@]}"; do
	echo "::group::Validate ${org}"
	output=$(node --import tsx src/validate-cmd.ts "$org" 2>&1) || failed+=("$org")
	printf '%s\n' "$output"
	echo "::endgroup::"
	# Lift each finding into an annotation, warnings included, so it
	# shows on the PR's Checks summary and not only inside the
	# collapsed group above.
	sed -nE \
	-e "s/^ ❌ \[([^]]+)\] (.*)$/::error title=\1 (${org})::\2/p" \
	-e "s/^ ⚠️ +\[([^]]+)\] (.*)$/::warning title=\1 (${org})::\2/p" \
	<<<"$output"
	done

On the starter, it prints ::warning title=so-assistant-lockstep (clinic)::assistants/receptionist (artifactPlan.structuredOutputIds): …

Also:

• Pin this in the "every org is valid" test by asserting that run.output includes ::warning title=so-assistant-lockstep (clinic)::. That also records that warnings don't fail the job.
• The PR description and commit message list lockstep, duplicated prompts and the maxTokens floor among the errors apply refuses. Please fix both. (The improvements.md wording has its own comment.)
• Possible stacked PR: a validate-cmd --format=github that emits ::error file=<ResourceFile.filePath>,… would pin each annotation to the offending file in the diff, which beats scraping the emoji output.

[chris-garber-vapi]

🟢 "Make it a required check" needs a caveat for merge queues: ci.yml has no merge_group trigger, so inside a GitHub merge queue the required check never reports and the queue stalls.

This applies to the existing test job too. This PR is the first place the docs tell people to require a check from ci.yml, so it's the natural spot for the caveat:

Suggested change

	It needs no secrets, so it runs on forks too. Make it a required check in
	branch protection, so a config that `apply` would refuse can't reach
	`main`, where it would block deploys and promotion.
	It needs no secrets, so it runs on forks too. Make it a required check in
	branch protection, so a config that `apply` would refuse can't reach
	`main`, where it would block deploys and promotion. If you merge through a
	GitHub merge queue, first add `merge_group:` to `ci.yml`'s `on:` triggers, or
	the required check never reports and the queue stalls.

[chris-garber-vapi]

🟢 CLAUDE.md asks for the PR number on resolved entries ([RESOLVED YYYY-MM-DD] (#<PR-number>)), and #37 doesn't have one.

#33–#36, earlier in this stack, also leave it out, but #29, #30 and #32 include it. Please add (#76) here and on the **[RESOLVED 2026-10-03]** line under ## 37..

Suggested change

	| 37 | Resource validation ran only at deploy time, after merge | A config `apply` refuses could merge and block deploys and promotion | #32 | RESOLVED 2026-10-03 |
	| 37 | Resource validation ran only at deploy time, after merge | A config `apply` refuses could merge and block deploys and promotion | #32 | RESOLVED 2026-10-03 (#76) |

[chris-garber-vapi]

🟡 The entry lists lockstep, duplicated prompts and the maxTokens floor as shapes the API rejects, but they're warnings: they fail neither validate, apply, nor this check.

In src/validate.ts, name-length and voice-provider-schema have severity: "error", while so-assistant-lockstep, prompt-duplicate-h1/-block and max-tokens-floor have "warn". Lockstep problems and maxTokens: 1 aren't API rejections either; they're silent inconsistencies. This file is the durable record ("the history is the point"), so a wrong claim here will outlive the PR description.

Suggested change

	`npm run validate` catches the shapes the API rejects (name length,
	structured-output lockstep, duplicated prompts, the `maxTokens` floor,
	per-provider voice schema), but nothing ran it before merge. A config that
	`apply` refuses could land on `main`, and was found only when someone
	deployed or promoted it.
	`npm run validate` fails on shapes the API rejects mid-push (a name over 40
	characters, per-provider voice schema) and warns on silent inconsistencies
	(structured-output lockstep, duplicated prompts, the `maxTokens` floor), but
	nothing ran it before merge. A config that `apply` refuses could land on
	`main`, and was found only when someone deployed or promoted it.

[chris-garber-vapi]

🟢 The ! hides a renamed step: every test would fail with Cannot read properties of undefined (reading 'run') instead of naming the cause.

Suggested change

	const STEP = JOB.steps.find((s) => s.name === "Validate every org")!;
	const STEP = JOB.steps.find((s) => s.name === "Validate every org");
	assert.ok(STEP, 'ci.yml has no "Validate every org" step; update this test if it was renamed');

[chris-garber-vapi]

🟢 The no-secrets assertion searches all of ci.yml for the literal secrets.. That's too broad, because it breaks on unrelated jobs, and too narrow, because it misses the ways this job could actually gain privileges.

• Too broad: if another ci.yml job ever needs a secret (a coverage upload token, say), this test fails even though the validate job didn't change.
• Too narrow: it misses secrets['X'], toJSON(secrets), secrets: inherit, a job-level permissions: escalation, and a switch to pull_request_target. That last one is what would actually hand fork code a write token and secrets.

I tested the version below. It passes on this PR as-is, and it fails when I add permissions: { contents: write } to the job:

Suggested change

	test("validate job gets no secrets and keeps no credentials", () => {
	assert.deepEqual(
	{
	secrets: WORKFLOW_TEXT.includes("secrets."),
	key: STEP.env,
	checkout: JOB.steps.find((s) => s.uses?.startsWith("actions/checkout"))
	?.with,
	},
	{
	secrets: false,
	key: { VAPI_PRIVATE_API_KEY: "validate-only-never-sent" },
	checkout: { "persist-credentials": false },
	},
	);
	});
	test("validate job gets no secrets and keeps no credentials", () => {
	const workflow = parseYaml(WORKFLOW_TEXT) as {
	on: Record<string, unknown>;
	jobs: { validate: Record<string, unknown> };
	};
	assert.deepEqual(
	{
	// Fork code runs in this job, so it must never get the privileged trigger.
	privilegedTrigger: "pull_request_target" in workflow.on,
	// Scoped to this job, and catches secrets.X, secrets['X'],
	// toJSON(secrets) and `secrets: inherit`.
	secrets: /\bsecrets\b/.test(JSON.stringify(workflow.jobs.validate)),
	permissions: workflow.jobs.validate.permissions,
	key: STEP.env,
	checkout: JOB.steps.find((s) => s.uses?.startsWith("actions/checkout"))
	?.with,
	},
	{
	privilegedTrigger: false,
	secrets: false,
	permissions: undefined,
	key: { VAPI_PRIVATE_API_KEY: "validate-only-never-sent" },
	checkout: { "persist-credentials": false },
	},
	);
	});