Skip to content

feat(studio): timeline select and selection refresh read an opaque preview over its channel - #7613

Merged
jrusso1020 merged 2 commits into
mainfrom
hardening/studio-selection-channel
Nov 12, 2026
Merged

jrusso1020 merged 2 commits into
mainfrom
hardening/studio-selection-channel

Conversation

@jrusso1020

Copy link
Copy Markdown
Collaborator

On a preview frame Studio cannot script, selecting a clip on the timeline and refreshing a selection now build the selection from the runtime's element descriptions, over the editing channel. Which element gets selected comes from one set of rules that both kinds of frame use, so an opaque frame selects what a same-origin frame selects.

Same-origin paths are unchanged. Every preview today is same-origin. Studio still reads those frames directly; their selection code now calls the shared rules, which are the old loops moved out. Nothing flips sandbox-origin. Clicking and hovering on the canvas stay same-origin only; they move in the next PR.

What

  • Runtime editing port (packages/core/src/runtime/editingPort.ts, editingAgent.ts, editingHandles.ts):

    • computed-styles also takes { handle, properties } in place of { selector, properties }. The result shape is unchanged: { styles }, null for a removed element or a handle from another load.
    • New ancestors { handles } returns { ancestors: Array<handle[] | null> }: each handle's ancestors, parent first, up to the document element. A chain stops at 256 entries, so one describe takes it. A removed or unknown handle gives null.
    • New query { selector } returns { matches: Array<{ handle, source }>, truncated }: the selector's matches in document order, each with the source facts the parent resolves a file from, at most 2,048. An invalid selector gets a typed failed reply.
    • Params are checked on the wire the same way as the existing kinds: exact keys, a well-formed handle, a selector of 1 to 2,000 characters, 1 to 256 handles.
  • Shared selection rules (packages/studio/src/components/editor/domEditSelectionRules.ts, new). These are pure over a target's ancestor chain, through a small reader (attr, selector, isDocumentRoot):

    • group capture, with drill-in;
    • the clip ancestor;
    • the walk up to an addressable element;
    • timeline-locked;
    • the lookups findElementForSelection tries;
    • the timeline-element lookup with its master-view root fallback.

    resolveDomEditSelection, findElementForSelection and findElementForTimelineElement use them on live elements. resolveGroupCapture and getSelectionCandidate are gone, since the rules replace them.

  • Opaque selections (packages/studio/src/components/editor/previewSelection.ts, new):

    • The frame's element is found with query and describe, filtered by source file as before.
    • The ancestor chain comes from ancestors plus one describe.
    • The rules pick the element. Its fields come from its description, plus computed-styles by handle for the curated computed styles.
    • The composition-root check runs only for a child of body. Its geometry part is now a pure coversCompositionRoot, which both paths call.
    • Label, data attributes and curated inline styles are read with the same helpers as a live element. They run on a throwaway element in a window-less document, built from the description. So a shorthand such as border-radius serializes as the browser does.
    • The selection carries the element's handle. Its element is a stand-in: an element with only the target's tag, in a document with no window, one per handle for the frame load. Nothing from the frame is copied into it.
  • useDomSelection: timeline select, single refresh and group refresh take the opaque path when previewFrameOpaque says so. buildDomSelectionFromTarget also accepts a description and the frame it came from.

  • The source-existence probe (existsInSourceFile) is shared by both paths.

Why

Before the main preview frame can go opaque, Studio has to select elements without the frame's DOM. This PR covers the two selection flows that find an element by its identity rather than by a point.

Related work

Refs #7603 (element handles), #7602 (picker styles and composition size), #7561 (playback and timeline readers) and #7556 (the runtime's editing port).

How

Claude Opus 5.5

Why shared pure rules, not a second walk over descriptions. Studio could walk the described ancestors with its own copy of each rule, but then group capture, the clip ancestor and the walk to an addressable element would live in two places. The rules now take a chain and a reader. The live path gives them elements and getAttribute. The opaque path gives them descriptions, whose selector is the runtime's buildStableSelector, the same function since #7603. No rule needs layout or computed styles, so the runtime needs no rule code.

Runtime surface. The rules need the whole ancestor chain. describe gives only the parent, so walking up would be one round trip per level. ancestors returns every handle at once, and one describe takes them all.

Finding an element by identity needs a document query, which query answers. It returns each match's source facts, so Studio picks the right file's match before describing one.

Computed styles reuse computed-styles, which now names its element by handle too. A selector cannot name one element of many.

A single resolution makes these requests, in order:

  1. query and one describe;
  2. ancestors and one describe;
  3. one computed-styles;
  4. for a child of body only, query and describe for the composition root.

Same answer, checked. A parity test runs the real runtime agent from core against the same document. For each case, it compares the selection a same-origin frame builds with the one an opaque frame builds. Everything matches except element, handle and text fields. In real Chrome, against the real runtime and the studio-open fixture, eight selections matched field for field, computed styles and rects included: the shape, the title, the scene host and the root, each with and without the clip-ancestor preference.

Stale handles. A selection is re-found by identity on refresh, as on a same-origin frame, not by its handle, so a reloaded frame works. A description whose element is gone, a handle from another load, or a chain whose parents do not link up builds nothing.

"Cannot tell" is kept apart from "not there": a frame with no editing agent, or a query cut off before the wanted index. The finder returns undefined for it, and a refresh keeps the selection, as it does today when the frame has no document. A real miss drops the selection, as before.

The stand-in. About 600 places across Studio read selection.element, so a selection still needs an HTMLElement. A frame Studio cannot script has none to give. The stand-in has no attributes, no text and no window, and it is never connected. Readers that need the frame's element move to handle in later PRs. It is the same object for a handle all load long, so identity checks (knownSourceAnswer, drill-in) keep working.

Not migrated here, still direct reads (same-origin only):

  • Canvas click and hover, resolveDomSelectionFromPreviewPoint and resolveAllDomSelectionsFromPreviewPoint. Their hit test reads computed styles across the stack, its ancestors and children. That is the next PR.
  • Text fields are left empty on an opaque frame. Text edits write through the frame's document, and they move with the text commits.
  • Overlay geometry, marquee, snapping, Layers, the motion path and picker writes read selection.element. On an opaque frame they get the stand-in and draw nothing, as on main.
  • The scene hold and position-edit replay before a timeline select, which need the frame's window.
  • The timeline's own DOM enrichment, which fills a clip's domId, selector and compositionSrc from the frame. On an opaque frame those stay empty today, so a real timeline click there has nothing to look up yet. The After capture fills them in by hand; see below.

Size. About half of the diff is tests (471 lines). In domEditingLayers.ts, about 60 lines only change indentation, because the resolver lost its loop.

Before

Before: on main, Studio with its preview frame forced opaque and Scene0 clicked on the timeline. The inspector says Nothing selected, because timeline select reads the frame's document and gets none

After

After: the same opaque frame on this branch. Clicking Scene0 selects it: the inspector shows Scene0, #s0, div, X 0, Y 0, W 1920, H 1080, the same values a same-origin frame shows. The canvas has no selection box yet, since overlay geometry still reads the frame's element

Both are real Studio sessions: the Vite dev server, the studio-open fixture and headless Chrome 153. A page patch, not committed, sets every <hyperframes-player> to sandbox-origin="opaque". The frame's sandbox was allow-scripts, and Studio could not read its document.

Before the click, both sessions put each clip's domId, selector, sourceFile and compositionSrc into the timeline store, copied from a same-origin session of the same fixture. That stands in for the timeline enrichment, which reads the frame and so skips an opaque one. The click then runs the real handleTimelineElementSelect. On this branch the selection came from the real runtime over the real port.

Test plan

  • Unit tests added/updated. Each new test fails without the change it covers:
    • previewSelection.test.ts uses detached fake frames. The opaque frame is served by core's real installEditingAgent over this document, and the same-origin frame reads the same document.
      • Parity cases: a group member at the top level, and in each step of a drill-in (nested group, child, and a drilled group that is elsewhere); a clip ancestor; a host clip; the walk up from an element with no selector; timeline-locked; a class match indexed within its own file; a body child on the root (composition root). Each also checks the rule actually decided, not only that the two paths agree.
      • Finders: selection lookups by selector and per-file index, id, hf id, no match and an invalid selector; timeline lookups by selector, the root fallback outside the master view, and none in the master view.
      • Stand-in: tag only, no window, the same object for a handle.
      • Stale and unknown cases: a removed element, a handle from another load and an unlinked chain build nothing. No editing agent, and a capped query short of the index, each give "cannot tell".
      • The hook on an opaque frame: timeline select matches the same-origin hook. A refresh picks up a style change. A refresh drops a removed element. A group refresh keeps the members that remain.
    • editingHandles.test.ts: ancestors parent first to html, and null once removed; the 256 cap; query with sources in document order, failed for an invalid selector, and the 2,048 cap; computed-styles by handle, and null once removed.
    • editingAgent.test.ts: 14 more malformed requests get invalid-params: handle-form styles (5), ancestors (4) and query (5). hello lists the two new kinds, here and in editingPort.test.ts.
  • Mutation probes. Each change below was reverted on its own, and the named tests failed (23 probes, all caught):
    • the hook's opaque paths: timeline lookup, refresh lookup, build from a description, and passing the refresh frame;
    • the drilled-group stand-in lookup;
    • locked, the composition-root check, the walk's stop at body, and inline styles through the probe element;
    • the per-file filter, the capped-query and invalid-selector cases, the chain-link check, the stand-in cache, and computed styles by handle;
    • the nested-group and host-clip rules;
    • in core: the ancestor cap, the query cap, styles by handle, the handle and selector param checks, and null for a dead handle.
  • Full Studio suite, core test:hyperframe-runtime-ci (typecheck, preview-guard lint, runtime contract, behavior, seek, parity, security, coverage and linter), Studio and core tsc --noEmit, oxlint, oxfmt, fallow audit, the comment ratchet, check:package-cycles, check:package-subpaths and check:studio-copy-dashes: all passed.
  • Same-origin UX parity in a real Studio session. The same scripted journey ran against the merge base and against this branch. It clicked Scene0 on the timeline, hovered the scene's shape, clicked to select it, dragged and dropped it, then drew a marquee around the title. Every recorded box was identical: the timeline selection (409.3, 129.5, 741.3 × 417), hover and select (911.3, 226, 84.9 × 84.9), the drag and drop (441.8), the marquee band and its selection (440.2, 450, 138.1 × 44.8). Neither run had a console error besides the same four 404s.
  • Opaque-frame run in a real Studio session: see Before and After. With no page errors, the inspector showed the same Scene0 values as the same-origin session.
  • Not exercised: a real timeline click on an opaque frame without the hand-filled clip targets (it finds nothing until the timeline enrichment moves), and Desktop.
  • Comments follow CONTRIBUTING.md "Comments".

— Rames Jusso

🤖 Generated with Claude Code

jrusso1020 and others added 2 commits November 12, 2026 16:33
…eview over its channel

On a preview frame Studio cannot script, timeline select, a selection
refresh and a group refresh build the selection from the runtime's element
descriptions. The rules that pick the element (group capture and drill-in,
the clip ancestor, the walk to an addressable element, timeline-locked, the
lookups and the master-view root fallback) move to a pure module over an
ancestor chain, used by both paths, so both select the same element.
Same-origin frames keep their direct reads.

The editing port gains `ancestors` (each handle's chain, parent first) and
`query` (a selector's matches with their source facts), and
`computed-styles` also names its element by handle. An opaque selection
carries the element's handle; its `element` is a tag-only stand-in in a
document with no window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

Edit accuracy: accurate 267 (base branch 267), smooth 113 of those

The gate passes.
Smoothness is reported in the artifact, not gated. A case fails only if it fails 2 of 3 runs.

Quarantined, measured but not gated (8)

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: APPROVE · 0 blocker · 2 important · 2 nits · cb62a3a

Same-origin paths are unchanged: I checked each moved loop against main and found no DOM where they disagree. The core wire checks are tight. Both important findings are on the opaque path, which nothing turns on yet, so they gate flipping sandbox-origin, not this merge. ↓ inline

Scope

Audited: editingPort.ts, editingHandles.ts, editingAgent.ts, domEditSelectionRules.ts, previewSelection.ts, domEditingLayers.ts/domEditingElement.ts/domEditingRootLayer.ts (compared with main), useDomSelection.ts, previewElements.ts/previewChannel.ts. Trusting: test files (read, not run locally; CI green at head). There was a separate adversarial pass. No prior reviews.

— Genesis reviewer

const index = chain && chainIndex(chain, chainReader(chain), options);
if (!chain || index === null) return null;
const current = chain[index]!;
const element = standInFor(iframe, current);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

important — Some panel readers still read attributes off selection.element, and a tag-only stand-in gives them wrong answers, not empty ones:

  • domEditSelectionToFacts → isAudibleVideoNode(selection.element) (domEditingLayers.ts:249). For a <video muted> or data-has-audio="false" clip that is timeline-selected in an opaque frame, hasAudio comes out true, so Audio FX shows for a silent clip. Same-origin hides it.
  • propertyPanelFill.tsx:104 reads element.element.getAttribute("src"), so an <img> fill shows an empty URL.

The parity test drops element, so it can't see either one. Fix: put hasAudio/src on the selection and read them there, or copy a small inert allow-list (muted, src, data-*) onto the stand-in.

tagName: current.tag,
boundingBox: { x, y, width, height },
textContent: current.text.trim() || null,
dataAttributes: getDataAttributes(probe),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

important — describeElement cuts each attribute and inline-style value to 2,000 chars (and data: values to their header). This path ignores current.truncated and uses the cut values as if they were complete. Example: a clip whose data-audio-fx or data-audio-automation JSON is over 2,000 chars. propertyPanelAudioFxGroup.tsx:115-128 fails to parse it and shows an empty chain, and the next edit writes a chain built from that empty one back to source, so the existing nodes and lanes are lost. A long inline background-image:url(data:…) is also dropped by setProperty. Fix: when current.truncated is set, treat it as "cannot tell" (build nothing), or fetch the full data-* values Studio reads.

const probe = readable(current);
const inlineStyles = getInlineStyles(probe);
const computedStyles =
(await previewComputedStyles(iframe, current, [...CURATED_STYLE_PROPERTIES])) ?? {};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit — Here ?? {} turns "cannot tell" into empty styles. Example: the element is parked or the frame reloads between describe and computed-styles, and the selection is still built, with its capabilities and root check run on {}. That goes against the PR's "a description whose element is gone builds nothing". compositionRoot (L189) has the same issue: a failed query falls back to the document element's rect. Returning null on either failure would keep the contract.

const element = await findSelectionElement(iframe, selection, activeCompPath);
if (element === undefined) return;
if (!element) continue;
const nextSelection = await buildDomSelectionFromTarget(element, { previous: selection });

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit — The single refresh threads iframe through to the build; this one doesn't. If the frame reloads mid-loop, members found in the old load resolve against the new frame, their handles are stale, and they drop out silently. The group then shrinks rather than being kept. The opaque path also makes this loop about 7 round trips per member, with no stillSelected-style guard before it writes domEditSelectionRef. Pass iframe and check before committing.

@jrusso1020
jrusso1020 added this pull request to the merge queue Nov 12, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 12, 2026
@jrusso1020
jrusso1020 added this pull request to the merge queue Nov 12, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 12, 2026
@jrusso1020
jrusso1020 added this pull request to the merge queue Nov 12, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 12, 2026
@jrusso1020
jrusso1020 added this pull request to the merge queue Nov 12, 2026
@jrusso1020 jrusso1020 mentioned this pull request Nov 12, 2026
1 of 4 tasks
Merged via the queue into main with commit aed4c18 Nov 12, 2026
97 checks passed
@jrusso1020
jrusso1020 deleted the hardening/studio-selection-channel branch November 12, 2026 22:59
@miguel-heygen miguel-heygen mentioned this pull request Nov 13, 2026
3 of 5 tasks
pattoneirc pushed a commit to pattoneirc/hyperframes that referenced this pull request Nov 13, 2026
…hannel (heygen-com#7628)

* feat(studio): read overlay geometry from an opaque preview over its channel

The runtime's editing channel gains a `geometry` request: per element
handle its client rect, the corners it paints at when rotated, whether it
renders, its clip-path, its source boundary's size, a group wrapper's
member rects, and optionally one element's descendant boxes. On a preview
frame Studio cannot script, the selection, child, group and hover boxes,
the root scale (from the mirrored stage size) and a press held across a
preview change read that geometry, with one read in flight at a time.
Same-origin frames keep reading the DOM directly; the measuring code they
use now lives in core so both paths share one rule.

Also from the review of heygen-com#7613: opaque selections carry `hasAudio` and
`src` read off the description, a description with a cut attribute or
inline style builds no selection, a failed computed-styles or root read
builds none either, and the group refresh threads its frame and leaves a
group the user changed meanwhile. `findClosestByAttribute`, left without
a caller by heygen-com#7613, is removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(producer): raise the packed size budget for the runtime's geometry read

The producer ships the preview runtime, which now carries the editing
channel's `geometry` request and the measuring code it shares with Studio.
The packed tarball measured 888,810 bytes in CI against an 880,000 budget.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant