Skip to content

fix(cli): measure and diff report why they failed, and reverse reports its runs - #6997

Merged
WaterrrForever merged 2 commits into
mainfrom
miao/measure-telemetry-and-skill-tag
Nov 2, 2026
Merged

WaterrrForever merged 2 commits into
mainfrom
miao/measure-telemetry-and-skill-tag

Conversation

@WaterrrForever

Copy link
Copy Markdown
Collaborator

What

  • measure and diff keep the real error when they fail, so cli_error says why instead of "Command failed".
  • hyperframes measure clip.mp4 (a file where an op belongs) now prints which commands to put first.
  • reverse-any-video reports its own start and finish with hyperframes events, from references/start.md.

Why

PostHog since the reverse launch (project Hyperframes, CI excluded):

  • 97 people hit measure failures recorded only as "Command failed". runMeasure and diff printed the cause to the agent, then called failCommand() without it, so trackCommandFailure had only the wrapper.
  • 59 agents ran measure <file> with no op and got "Unknown command" with no suggestion.
  • Cloud renders carry no skill, and reverse rarely passes --skill, so 5 people show up as reverse renders. Nobody can count starts or finishes.

How

failCommand(1, err) in both places: trackCommandFailure already unwraps a CliRuntimeError to its cause. The file hint lives in fileHint next to unknownCommandError, and only fires when the word exists on disk and the group has subcommands, after the did-you-mean match. The beacon is the same events --skill pattern figma uses. It sits in start.md because the agent opens that file on every start and resume, and SKILL.md has 18 bytes left in its budget.

Test plan

fails without the fix: measure/command.test.ts "keeps the real error as the failure's cause, so telemetry reports it"

  • src/measure and src/commands/diff tests: 87 passed. Typecheck clean.
  • measure package.json prints the hint; measure prob still suggests probe; a missing file prints the old message.
  • events --skill=reverse-any-video --event=skill_completed --outcome=success exits 0.
  • The beacon depends on the agent running it; check the event counts a few days after release.

…s its runs

Since launch, 97 people saw a bare "Command failed" from measure: runMeasure
and diff called failCommand() without the error, so telemetry kept only the
wrapper. 59 agents ran `measure <file>` with no op and got no hint. Reverse
runs had no skill tag, so nobody could count how many finished.

- runMeasure and diff pass the error to failCommand.
- An unknown word that is an existing file names the commands to put first.
- references/start.md sends skill_invoked and skill_completed beacons.

@jrusso1020 jrusso1020 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at d706ba401485a5aefbe8e119eb335a612cf1b717.

No blocker found. measure/command.ts:28 and commands/diff.ts:258 preserve the original failure through the existing failCommand/trackCommandFailure contract, including its deduplication and redaction. cli.ts:549 keeps typo suggestions ahead of the file hint. The reverse instructions reuse the existing consent-gated, best-effort events command.

Validation: 217 focused tests pass (src/measure, telemetry events, command-failure tracking, events command and CLI lifecycle). The new measure regression fails with the cause forwarding removed. Independent checks pass for seven real CLI cases (including spaces/apostrophes, typo precedence and missing files) and six cause/event controls, including diff's actual rejection path and reverse invoked/success/error payloads. Skill lint and generated manifest verification pass.

Nit: cli.ts:549 and skills/reverse-any-video/references/start.md:7: add committed positive-pin regressions for the new hint and beacon instructions. The added test currently pins only the measure cause; a later wording change could silently remove either recovery advice or skill attribution. Pin the command-before-file concept, typo precedence, and invoked/completed skill slugs/outcomes.

The reported PostHog counts were not independently queried. No live PostHog delivery or full reverse session was exercised; instruction-based beacons still depend on the agent executing them, as the PR notes.

Need: yes · Size: ok · Reuse: existing error boundary and events pipeline · Simpler: already minimal · Structure: fine
Risk: low — localized diagnostic and instruction changes; no rendering or paid-generation behavior changes.

Verdict: APPROVE
Reasoning: Both failure paths retain the cause and the hint and event contracts work in behavioral checks. Remaining test pins are a nonblocking follow-up.

— hf-review

@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Edit accuracy: accurate 271 (base branch 271), smooth 110 of those

The gate passes.
Smoothness is reported in the artifact, not gated. A case fails only if it fails 2 of 3 runs.

Quarantined, measured but not gated (0)

@jerrai-bot-heygen jerrai-bot-heygen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the exact-head failure and beacon paths. runMeasure and diff now give failCommand the real cause, which the existing telemetry unwraps; the measure regression checks identity of that cause. The reverse beacon command shape matches the existing events contract, although agent-instruction delivery and PostHog reception were not exercised. One small follow-up in packages/cli/src/cli.ts:562–565: fileHint also runs for root-level unknown commands. An existing file passed as hyperframes index.html can produce a huge suggestion listing every top-level command, not just the intended measure <op> <file> hint. Please scope it to command groups (or cap the list) and test the root case. This is not blocking; Rames has already approved this exact head, so I am not adding a redundant stamp. CLI test shards and several Windows/studio checks were still pending at my last check; no local tests or live telemetry run. — Jerrai

Jerrai: `hyperframes index.html` would list every top-level command. fileHint moves to
didYouMean.ts, answers only inside a command group, and has tests for the root, a missing
file and a group with no commands.
@WaterrrForever

Copy link
Copy Markdown
Collaborator Author

Thanks Jerrai. Fixed in f73bd95: fileHint moved to utils/didYouMean.ts and answers only inside a command group, so hyperframes index.html prints the plain unknown-command message. New tests cover the group hint, the root, a missing file and a group with no commands (9 pass). Checked by hand: measure package.json still hints, hyperframes package.json does not.

@jrusso1020 jrusso1020 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at f73bd95f72319771dcb1b1de57081b3f4e72b464, delta from my approved d706ba401485a5aefbe8e119eb335a612cf1b717.

Jerrai's root-hint follow-up is fixed: cli.ts passes the resolved command path to the extracted fileHint, which refuses an empty root path. Group advice and typo/synonym precedence remain intact. The two new tests pin the group wording and root/missing-file/empty-group cases.

Validation: 8 utility tests pass. Removing only the root guard in copied scratch code fails the root regression (1 failed, 7 passed); two independent controls pass for nested groups, paths with spaces/apostrophes, root suppression and existing matching. The earlier 217-test result belongs to the previous head, not a full rerun at this head. No live PostHog delivery or full reverse run.

Need: yes · Size: ok · Reuse: existing matching utility · Simpler: already minimal · Structure: fine
Risk: low — scopes an error hint without changing command execution.

Verdict: APPROVE
Reasoning: The root case is now guarded and covered, with no new issue in the delta. The earlier beacon-instruction test note remains nonblocking.

— hf-review

@WaterrrForever
WaterrrForever added this pull request to the merge queue Nov 2, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 2, 2026
@WaterrrForever
WaterrrForever added this pull request to the merge queue Nov 2, 2026
Merged via the queue into main with commit 618c486 Nov 2, 2026
85 checks passed
@WaterrrForever
WaterrrForever deleted the miao/measure-telemetry-and-skill-tag branch November 2, 2026 10:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants