[WaterrrForever]

Why

Skill telemetry for /reverse-any-video and /style-sampling leaves four questions open:

• Did the work run? The feat(skills): every published skill reports its own run #7176 beacon fires only when the agent runs it, and agents often skip it.
• Did a style-sampling run end in a video? Its renders carry no authoring_skill.
• Was the skill there to trigger? cli_command records no arguments, so an on-demand skill that never triggered looks the same as one that was never installed.
• Did sign-in fail? auth status reports a signed-out user by exiting 1, as skills expect, but cli_error shows it as a bare "Command failed" that reads like a login failure.

A related gap: a reverse started signed out often goes on without HeyGen and never tries to sign in. The sign-in step lives only in the skill's references/start.md.

What changes

• skill_command_ran: a new event, sent by the commands only one skill runs. skill_invoked and the beacon are unchanged from feat(skills): every published skill reports its own run #7176, so a skill run is still one skill_invoked observation.
  • reverse start sends it after the probe succeeds, with signed_in and, when recorded, the reverse session_id. A retry after sign-in or upgrade resumes the same session, so count distinct session_id. A video it cannot probe sends nothing.
  • templates pull sends it only when it actually downloads. A re-pull of a template already there, or a refused pull, sends nothing.
• templates pull sets authoringSkill: style-sampling on the folder the episode is built in, only if that folder is already a project. A pull never creates one.
• cli_command carries skills_installed and skills_outdated: the skill names from the cached background skills check, comma-joined. Both are absent until a check has run, and a later check that finds no install clears them. The telemetry disclosure in docs/packages/cli.mdx says so.
• auth status exits 1 as before, with a cause that carries only a code: not_signed_in, or the auth error's code. No free text.
• A signed-out reverse start adds login_command and a next line to its JSON, so the sign-in step is in the output the agent reads.

Checked

vitest passes for commands/auth, telemetry, skillsUpdateCheck, commands/events, commands/reverse and commands/templates. They include:

• a beacon → signed-out reverse start → signed-in retry flow: one skill_invoked, two skill_command_ran, and only the second carries the session;
• no event for an unprobeable video, a re-pull, or a refused pull;
• check → uninstall → check, which clears the installed names;
• the sign-in fields appearing only when signed out.

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated
hyperframes	🟢 Ready	View Preview	Nov 9, 2026, 9:40 AM

💡 Tip: Enable Automations to automatically generate PRs for you.

[github-actions]

Edit accuracy: accurate 267 (base branch 267), smooth 95 of those

The gate passes.
Smoothness is reported in the artifact, not gated. A case fails only if it fails 2 of 3 runs.

Quarantined, measured but not gated (8)

• seqplainkeys-none-center-r0-nested-z100 (fixed by studio: undo/redo on a keyframed plain element sometimes leaves the box 230-400px off (edit-accuracy seqplainkeys-none-*) #6989): pass, passes
• seqplainkeys-none-center-r0-root-z100 (fixed by studio: undo/redo on a keyframed plain element sometimes leaves the box 230-400px off (edit-accuracy seqplainkeys-none-*) #6989): pass, passes
• seqplainkeys-none-center-r30-root-z100 (fixed by studio: undo/redo on a keyframed plain element sometimes leaves the box 230-400px off (edit-accuracy seqplainkeys-none-*) #6989): not run
• seqplainkeys-none-pct-r0-root-z100 (fixed by studio: undo/redo on a keyframed plain element sometimes leaves the box 230-400px off (edit-accuracy seqplainkeys-none-*) #6989): pass, passes
• seqplainkeys-none-pct-r30-root-z100 (fixed by studio: undo/redo on a keyframed plain element sometimes leaves the box 230-400px off (edit-accuracy seqplainkeys-none-*) #6989): not run
• seqplainkeys-none-px-r0-nested-z100 (fixed by studio: undo/redo on a keyframed plain element sometimes leaves the box 230-400px off (edit-accuracy seqplainkeys-none-*) #6989): pass, passes
• seqplainkeys-none-px-r30-nested-z100 (fixed by studio: undo/redo on a keyframed plain element sometimes leaves the box 230-400px off (edit-accuracy seqplainkeys-none-*) #6989): not run
• seqplainkeys-none-px-r30-root-z100 (fixed by studio: undo/redo on a keyframed plain element sometimes leaves the box 230-400px off (edit-accuracy seqplainkeys-none-*) #6989): not run

Unstable (1)

• seqnudge-none-px-r0-root-z100: tracking 0.03, pressJump 0, drop 0, reload 0, render 0.02, renderKey -, undo false, teleport true / tracking 0.03, pressJump 0, drop 0, reload 0, render 0.02, renderKey -, undo true, teleport true / tracking 0.03, pressJump 0, drop 0, reload 0, render 0.02, renderKey -, undo true, teleport true

[jrusso1020]

Reviewed at 1f3912073fc252bcc6f8eb45a7f425be8bffcd9c against #7176's existing usage-beacon contract.

The installed-skill cache is populated from canonical manifest skill names rather than local paths or arbitrary installed-folder names; invalidation clears the added fields, and the disclosure correctly says “last background skills check.” The template attribution reuses the seed-once writer and does not create a project, and the login instruction is restricted to not_signed_in.

Important — preserve the invocation event's counting contract. packages/cli/src/telemetry/events.ts:1030–1031 emits skill_invoked from each command while the shipped skill still instructs an initial beacon (skills/reverse-any-video/SKILL.md:8, skills/style-sampling/SKILL.md:8), which commands/events.ts:50–55 sends under that same event name. A normal obedient run now emits two skill_invoked events, one source=beacon and one source=command. There is no common skill-run identity; transport generates a fresh random dedup UUID per event. Thus #7176's previously single invocation observation becomes two, and consumers counting raw skill_invoked events will double-count that run. Unique-person counting avoids that particular inflation but cannot establish the PR's “each run counts once” claim or recover repeated runs per person.

Concrete fix: emit the new command observation under a distinct event name (for example skill_command_started) while preserving the existing beacon event, or introduce a shared logical run ID and deduplication across both paths. Add a regression exercising the beacon plus command together and proving that the existing skill_invoked metric receives one observation. A source property alone distinguishes two records; it does not make them one invocation. Michael's consumer review remains useful, but this finding is the demonstrated event-contract change, not a hold for a human review.

Evidence: #7176 explicitly moved the reverse start beacon out of start.md to avoid counting it twice. An independent gated Vitest probe calls the actual events command and trackSkillRunByCommand and captures [skill_invoked/{source:beacon}, skill_invoked/{source:command}] for the same skill. Both still use the usual consent pipeline. Fresh source adversary confirmed the two observations and absence of a shared run ID, while considering source-aware unique-user analysis a valid alternative metric.

Validation limits: 24 existing targeted tests passed, but four other suites failed collection against borrowed dependencies missing current core/engine subpath exports; no full-build/suite or live PostHog/dashboard validation claimed. The duplicate-event probe passed locally with telemetry capture mocked, so it made no network send.

Need: yes · Size: ok · Reuse: existing beacon and consent pipeline, canonical skill cache and project attribution writer · Simpler: a separate command event preserves existing metrics · Structure: fine (checked existing trackEvent transport, skill manifest and seed-once writer; no split)
Risk: medium — changes existing telemetry event semantics across CLI invocations.

Verdict: REQUEST_CHANGES
Reasoning: Keep the deterministic command observation, but avoid counting the same obedient skill run twice under the existing invocation event before merging.

— Rames / hf-review

[jerrai-bot-heygen]

Reviewed the exact head. The signed-out auth status preserves exit 1 with a typed NotSignedIn cause, and reverse prints a login command only for not_signed_in. Template tagging seeds only an existing project and does not overwrite a valid authoring skill. Two telemetry defects remain:

1. Both skills still instruct an invocation beacon at start (skills/reverse-any-video/SKILL.md:8 and skills/style-sampling/SKILL.md:8). The new reverse start and templates pull also emit skill_invoked with the same authoring_skill. The only distinction is source=beacon versus source=command; neither carries a shared run identity. A normal skill run emits two events, and a signed-out reverse followed by the instructed retry emits a third. Existing event counts cannot distinguish runs from duplicate observations; count-unique-persons is not a substitute for run counts. Rames independently reproduced this same-head finding. Use a separate command event or a shared run identity/dedup contract, and test a complete beacon→command→retry flow.
2. recordSkillsCheck at packages/cli/src/utils/skillsUpdateCheck.ts:51-55 returns without clearing prior skill names when a successful check finds no skills directory (location=null). trackCommand at telemetry/events.ts:286-292 then continues reporting the old skillsInstalled/skillsOutdated arrays indefinitely after a user removes the install, even if later refreshes succeed. Clear the inventory on a meaningful no-location check, or stop emitting when cache freshness/location cannot be established, and add a check→uninstall→check regression.

Hosted checks are green at this head, but telemetry consumers' dedup behavior and Michael's requested beacon acceptance are not evidenced in this PR. No reviewer merge/enqueue.

— Jerrai

[miguel-heygen]

Reviewed at 1f3912073fc252bcc6f8eb45a7f425be8bffcd9c, against the #7176 beacon contract.

I confirm the double-count finding Rames and Jerrai raised, from the code, and don't repeat it. Jerrai's stale installed-skills cache point isn't repeated either. #7176's description says the start beacon moved out of references/start.md "to avoid counting the start twice", and this PR brings that back. Below are gaps that remain even if the beacon/command split is fixed as proposed.

Important: the command event counts invocations, not runs. trackSkillRunByCommand(SKILL) is the first line of run() (commands/reverse/start.ts:121), before the probe and before any server answer. Two probes against the real command (with the PR's own mocks) show:

• A failed start counts as a run. A video that can't be probed still records skill_invoked with source=command, though the command did nothing.
• A signed-out run counts twice. A signed-out reverse start followed by a signed-in one records two events. The PR's own SIGN_IN.next (start.ts:45) and the Pro refusal (start.ts:100) both tell the agent to "run reverse start again". The skill also re-runs reverse start on resume and after upgrading.
• A re-pull counts again. templates pull of a template already pulled returns at templates.ts:220 and still reports a run (templates.ts:278).

So "each run counts once" fails even within source=command. Suggested fixes:

• fire after the probe succeeds;
• key the event on the reverse session (start.ts already has the rvs_ id once recorded) or on project + skill;
• or name it for what it is (reverse_start_ran) and count distinct sessions.

Please add a test for the signed-out → signed-in sequence.

Nit: cli_error now carries the auth error's own message. failCommand(1, err) (auth/status.ts:171) means trackCommandFailure unwraps the AuthError, so error_message becomes, for example, Failed to refresh OAuth tokens: <response body>. The body comes from safeText(res) (auth/oauth.ts:403). Redaction (events.ts:937 → redactTelemetryString) strips paths and URL query strings only, not tokens or emails. HeyGen's token endpoint is unlikely to echo either, but an allowlisted code (REFRESH_FAILED etc.) would separate auth failures from other failures without sending free text.

Checked, no issue:

• skills_installed/skills_outdated are skill names from the manifest diff (diffSkills), so they carry no paths or folder names.
• Invalidation clears both fields.
• The disclosure in cli.mdx matches the new fields.
• A signed-out auth status still exits 1.
• The sign-in fields appear only for not_signed_in.

Not verified: whether any PostHog insight counts skill_invoked without a source filter.

Verdict: REQUEST CHANGES
Reasoning: The run events double-count against #7176 (Rames). Separately, they count retries and failed starts, so "each run counts once" doesn't hold even for the new source.

[WaterrrForever]

Addressed in 08a9d4c (description updated):

• Double count (Rames, Jerrai, Miguel): the commands now send skill_command_ran, and skill_invoked/the beacon are back to feat(skills): every published skill reports its own run #7176's shape. A test runs beacon → signed-out start → signed-in retry: one skill_invoked, two skill_command_ran.
• Invocations, not runs (Miguel): reverse start reports after the probe, keyed by the reverse session_id that a retry resumes. An unprobeable video reports nothing. templates pull reports only on a real download, never on a re-pull or a refused pull.
• Stale installed names (Jerrai): a check that finds no install clears them, with a check → uninstall → check test.
• Auth free text (Miguel nit): auth status failures carry only a code (not_signed_in or the AuthError code).

🤖 Addressed by Claude Code

[jrusso1020]

Reviewed at 08a9d4c15401a950d23fc0131d353b3bd352e82f. The invocation-counting blocker and the adjacent retry/cache/auth findings are addressed.

telemetry/events.ts:1030 now sends command observations as skill_command_ran; the beacon remains the #7176 skill_invoked shape. Reverse reports after probing, and templates report after actual successful download/unpack, excluding reused/refused pulls. A no-install check clears cached skill names, and auth status passes only an error code into failure telemetry. Attribution still reuses the existing seed-once project writer and consent pipeline.

Signed-out and signed-in attempts intentionally remain two command events: only the recorded attempt carries session_id. Counting distinct nonempty session IDs measures recorded sessions; this is not transport deduplication of both attempts. Michael’s event-consumer review remains useful, without holding this code verdict.

Validation: Gated focused cache/auth/beacon-command suites passed 38 tests. The new combined beacon/command/retry event assertion separately passed (1 test; 84 unrelated tests skipped). A fresh independent source adversary found no additional blocker. Broader selected suites encountered missing current borrowed engine/core/parser exports and render telemetry constants; no full-suite/build, live PostHog or dashboard validation claimed.

Need: yes · Size: ok · Reuse: beacon consent transport, canonical skills inventory and project attribution · Simpler: already minimal · Structure: fine (checked trackEvent, recordSkillsCheck/invalidateSkillsCache and seedProjectAuthoringSkill; no split)
Risk: medium — adds telemetry observations across command and session boundaries.

Verdict: APPROVE
Reasoning: The existing invocation event contract is preserved; command observations now describe their actual timing and identity.

— Rames / hf-review

[jerrai-bot-heygen]

Re-reviewed exact head 08a9d4c15401a950d23fc0131d353b3bd352e82f. Both prior blockers are fixed: command observations now emit distinct skill_command_ran, leaving the #7176 skill_invoked beacon's event contract intact; a no-location skills check clears cached installed/outdated names. Reverse reports only after probing and carries the persisted session when recorded; templates report only completed downloads, and auth failure telemetry carries a typed code rather than response prose. The new event measures command observations, not universally unique skill runs; a signed-out attempt has no session ID and needs consumer-aware interpretation. Exact-head hosted CLI test shards executed green, including new event/cache/retry regressions; later skip-only workflow jobs were not counted. No concrete code blocker found. Michael's consumer analysis is independent, not a claimed live dashboard verification. No merge or enqueue by reviewer.

— Jerrai

[miguel-heygen]

Re-review at 08a9d4c15401a950d23fc0131d353b3bd352e82f, covering the delta since our CHANGES at 1f391207 (08a9d4c).

This addresses the double count that Rames and Jerrai flagged and we confirmed:

• #7176 contract. skill_invoked and the beacon are back to how #7176 left them. git diff main... over skills/, scripts/check-core-skills.mjs, skills-manifest.json and commands/events.ts is empty. The commands now send a separate skill_command_ran (events.ts:1030-1042). The beacon → signed-out start → signed-in retry test pins one skill_invoked and two skill_command_ran.
• reverse start reports only after probeReport succeeds (start.ts:122-134). The no-probe test pins that.
• templates pull reports only after a real download (templates.ts:222-224). A re-pull returns before it, and a refused pull throws before it.
• Stale skill names (Jerrai's point). A no-install check now clears them (skillsUpdateCheck.ts:53-56). The skills commands' own invalidateSkillsCache also clears them.
• auth status failures now carry only a code (not_signed_in or the AuthError code), never the server text.

Local: 269/269 targeted tests pass. CI's full run at this head is green (Typecheck, Lint, Build, Test cli 1/2 and 2/2, CLI smoke).

Important (non-blocking):

• docs/packages/cli.mdx:2309: the telemetry disclosure now mentions installed skill names. It doesn't mention what skill_command_ran adds:

  • which skill a command ran for;
  • the reverse session_id, a server record id. No event on main sends a session id today, so this is new to telemetry, and the paragraph two lines down says "Nothing else personal is added".

  One clause covers it, e.g. "…which skill a reverse start or templates pull ran for and, when a reverse run is recorded, its session id…".

Notes:

• Dedup scope. As the body says, only the recorded run carries session_id. The signed-out run sends none, and start.test.ts asserts sessionId: undefined on it. So "count distinct session_id" counts recorded runs, and signed-out rows are attempts that have no key. That is fine as designed. Whoever builds the dashboard needs to know raw skill_command_ran rows double-count a sign-in retry.
• Nit. skillsUpdateCheck.ts:84-94: forgetInstalledSkills was inserted between invalidateSkillsCache's doc block and the function it documents. The "after a successful install/update… Best-effort" block now sits on the wrong function. Move the new helper above line 76.

Verdict: APPROVE
Reasoning: The #7176 counting contract is untouched, and command runs report once per real piece of work on their own event. The only gap is a one-line telemetry disclosure.