Repository navigation
fix(cli): measure and diff report why they failed, and reverse reports its runs - #6997
Conversation
…s its runs Since launch, 97 people saw a bare "Command failed" from measure: runMeasure and diff called failCommand() without the error, so telemetry kept only the wrapper. 59 agents ran `measure <file>` with no op and got no hint. Reverse runs had no skill tag, so nobody could count how many finished. - runMeasure and diff pass the error to failCommand. - An unknown word that is an existing file names the commands to put first. - references/start.md sends skill_invoked and skill_completed beacons.
jrusso1020
left a comment
There was a problem hiding this comment.
Reviewed at d706ba401485a5aefbe8e119eb335a612cf1b717.
No blocker found. measure/command.ts:28 and commands/diff.ts:258 preserve the original failure through the existing failCommand/trackCommandFailure contract, including its deduplication and redaction. cli.ts:549 keeps typo suggestions ahead of the file hint. The reverse instructions reuse the existing consent-gated, best-effort events command.
Validation: 217 focused tests pass (src/measure, telemetry events, command-failure tracking, events command and CLI lifecycle). The new measure regression fails with the cause forwarding removed. Independent checks pass for seven real CLI cases (including spaces/apostrophes, typo precedence and missing files) and six cause/event controls, including diff's actual rejection path and reverse invoked/success/error payloads. Skill lint and generated manifest verification pass.
Nit: cli.ts:549 and skills/reverse-any-video/references/start.md:7: add committed positive-pin regressions for the new hint and beacon instructions. The added test currently pins only the measure cause; a later wording change could silently remove either recovery advice or skill attribution. Pin the command-before-file concept, typo precedence, and invoked/completed skill slugs/outcomes.
The reported PostHog counts were not independently queried. No live PostHog delivery or full reverse session was exercised; instruction-based beacons still depend on the agent executing them, as the PR notes.
Need: yes · Size: ok · Reuse: existing error boundary and events pipeline · Simpler: already minimal · Structure: fine
Risk: low — localized diagnostic and instruction changes; no rendering or paid-generation behavior changes.
Verdict: APPROVE
Reasoning: Both failure paths retain the cause and the hint and event contracts work in behavioral checks. Remaining test pins are a nonblocking follow-up.
— hf-review
Edit accuracy: accurate 271 (base branch 271), smooth 110 of thoseThe gate passes. Quarantined, measured but not gated (0) |
jerrai-bot-heygen
left a comment
There was a problem hiding this comment.
Reviewed the exact-head failure and beacon paths. runMeasure and diff now give failCommand the real cause, which the existing telemetry unwraps; the measure regression checks identity of that cause. The reverse beacon command shape matches the existing events contract, although agent-instruction delivery and PostHog reception were not exercised. One small follow-up in packages/cli/src/cli.ts:562–565: fileHint also runs for root-level unknown commands. An existing file passed as hyperframes index.html can produce a huge suggestion listing every top-level command, not just the intended measure <op> <file> hint. Please scope it to command groups (or cap the list) and test the root case. This is not blocking; Rames has already approved this exact head, so I am not adding a redundant stamp. CLI test shards and several Windows/studio checks were still pending at my last check; no local tests or live telemetry run. — Jerrai
Jerrai: `hyperframes index.html` would list every top-level command. fileHint moves to didYouMean.ts, answers only inside a command group, and has tests for the root, a missing file and a group with no commands.
|
Thanks Jerrai. Fixed in f73bd95: |
jrusso1020
left a comment
There was a problem hiding this comment.
Reviewed at f73bd95f72319771dcb1b1de57081b3f4e72b464, delta from my approved d706ba401485a5aefbe8e119eb335a612cf1b717.
Jerrai's root-hint follow-up is fixed: cli.ts passes the resolved command path to the extracted fileHint, which refuses an empty root path. Group advice and typo/synonym precedence remain intact. The two new tests pin the group wording and root/missing-file/empty-group cases.
Validation: 8 utility tests pass. Removing only the root guard in copied scratch code fails the root regression (1 failed, 7 passed); two independent controls pass for nested groups, paths with spaces/apostrophes, root suppression and existing matching. The earlier 217-test result belongs to the previous head, not a full rerun at this head. No live PostHog delivery or full reverse run.
Need: yes · Size: ok · Reuse: existing matching utility · Simpler: already minimal · Structure: fine
Risk: low — scopes an error hint without changing command execution.
Verdict: APPROVE
Reasoning: The root case is now guarded and covered, with no new issue in the delta. The earlier beacon-instruction test note remains nonblocking.
— hf-review
What
measureanddiffkeep the real error when they fail, socli_errorsays why instead of "Command failed".hyperframes measure clip.mp4(a file where an op belongs) now prints which commands to put first.reverse-any-videoreports its own start and finish withhyperframes events, fromreferences/start.md.Why
PostHog since the reverse launch (project Hyperframes, CI excluded):
measurefailures recorded only as "Command failed".runMeasureanddiffprinted the cause to the agent, then calledfailCommand()without it, sotrackCommandFailurehad only the wrapper.measure <file>with no op and got "Unknown command" with no suggestion.--skill, so 5 people show up as reverse renders. Nobody can count starts or finishes.How
failCommand(1, err)in both places:trackCommandFailurealready unwraps aCliRuntimeErrorto its cause. The file hint lives infileHintnext tounknownCommandError, and only fires when the word exists on disk and the group has subcommands, after the did-you-mean match. The beacon is the sameevents --skillpatternfigmauses. It sits instart.mdbecause the agent opens that file on every start and resume, andSKILL.mdhas 18 bytes left in its budget.Test plan
fails without the fix:
measure/command.test.ts"keeps the real error as the failure's cause, so telemetry reports it"src/measureandsrc/commands/difftests: 87 passed. Typecheck clean.measure package.jsonprints the hint;measure probstill suggestsprobe; a missing file prints the old message.events --skill=reverse-any-video --event=skill_completed --outcome=successexits 0.