Repository navigation
feat(studio): timeline select and selection refresh read an opaque preview over its channel - #7613
Conversation
…eview over its channel On a preview frame Studio cannot script, timeline select, a selection refresh and a group refresh build the selection from the runtime's element descriptions. The rules that pick the element (group capture and drill-in, the clip ancestor, the walk to an addressable element, timeline-locked, the lookups and the master-view root fallback) move to a pure module over an ancestor chain, used by both paths, so both select the same element. Same-origin frames keep their direct reads. The editing port gains `ancestors` (each handle's chain, parent first) and `query` (a selector's matches with their source facts), and `computed-styles` also names its element by handle. An opaque selection carries the element's handle; its `element` is a tag-only stand-in in a document with no window. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Verdict: APPROVE · 0 blocker · 2 important · 2 nits · cb62a3a
Same-origin paths are unchanged: I checked each moved loop against main and found no DOM where they disagree. The core wire checks are tight. Both important findings are on the opaque path, which nothing turns on yet, so they gate flipping sandbox-origin, not this merge. ↓ inline
Scope
Audited: editingPort.ts, editingHandles.ts, editingAgent.ts, domEditSelectionRules.ts, previewSelection.ts, domEditingLayers.ts/domEditingElement.ts/domEditingRootLayer.ts (compared with main), useDomSelection.ts, previewElements.ts/previewChannel.ts. Trusting: test files (read, not run locally; CI green at head). There was a separate adversarial pass. No prior reviews.
— Genesis reviewer
| const index = chain && chainIndex(chain, chainReader(chain), options); | ||
| if (!chain || index === null) return null; | ||
| const current = chain[index]!; | ||
| const element = standInFor(iframe, current); |
There was a problem hiding this comment.
important — Some panel readers still read attributes off selection.element, and a tag-only stand-in gives them wrong answers, not empty ones:
domEditSelectionToFacts→isAudibleVideoNode(selection.element)(domEditingLayers.ts:249). For a<video muted>ordata-has-audio="false"clip that is timeline-selected in an opaque frame,hasAudiocomes out true, so Audio FX shows for a silent clip. Same-origin hides it.propertyPanelFill.tsx:104readselement.element.getAttribute("src"), so an<img>fill shows an empty URL.
The parity test drops element, so it can't see either one. Fix: put hasAudio/src on the selection and read them there, or copy a small inert allow-list (muted, src, data-*) onto the stand-in.
| tagName: current.tag, | ||
| boundingBox: { x, y, width, height }, | ||
| textContent: current.text.trim() || null, | ||
| dataAttributes: getDataAttributes(probe), |
There was a problem hiding this comment.
important — describeElement cuts each attribute and inline-style value to 2,000 chars (and data: values to their header). This path ignores current.truncated and uses the cut values as if they were complete. Example: a clip whose data-audio-fx or data-audio-automation JSON is over 2,000 chars. propertyPanelAudioFxGroup.tsx:115-128 fails to parse it and shows an empty chain, and the next edit writes a chain built from that empty one back to source, so the existing nodes and lanes are lost. A long inline background-image:url(data:…) is also dropped by setProperty. Fix: when current.truncated is set, treat it as "cannot tell" (build nothing), or fetch the full data-* values Studio reads.
| const probe = readable(current); | ||
| const inlineStyles = getInlineStyles(probe); | ||
| const computedStyles = | ||
| (await previewComputedStyles(iframe, current, [...CURATED_STYLE_PROPERTIES])) ?? {}; |
There was a problem hiding this comment.
nit — Here ?? {} turns "cannot tell" into empty styles. Example: the element is parked or the frame reloads between describe and computed-styles, and the selection is still built, with its capabilities and root check run on {}. That goes against the PR's "a description whose element is gone builds nothing". compositionRoot (L189) has the same issue: a failed query falls back to the document element's rect. Returning null on either failure would keep the contract.
| const element = await findSelectionElement(iframe, selection, activeCompPath); | ||
| if (element === undefined) return; | ||
| if (!element) continue; | ||
| const nextSelection = await buildDomSelectionFromTarget(element, { previous: selection }); |
There was a problem hiding this comment.
nit — The single refresh threads iframe through to the build; this one doesn't. If the frame reloads mid-loop, members found in the old load resolve against the new frame, their handles are stale, and they drop out silently. The group then shrinks rather than being kept. The opaque path also makes this loop about 7 round trips per member, with no stillSelected-style guard before it writes domEditSelectionRef. Pass iframe and check before committing.
…hannel (heygen-com#7628) * feat(studio): read overlay geometry from an opaque preview over its channel The runtime's editing channel gains a `geometry` request: per element handle its client rect, the corners it paints at when rotated, whether it renders, its clip-path, its source boundary's size, a group wrapper's member rects, and optionally one element's descendant boxes. On a preview frame Studio cannot script, the selection, child, group and hover boxes, the root scale (from the mirrored stage size) and a press held across a preview change read that geometry, with one read in flight at a time. Same-origin frames keep reading the DOM directly; the measuring code they use now lives in core so both paths share one rule. Also from the review of heygen-com#7613: opaque selections carry `hasAudio` and `src` read off the description, a description with a cut attribute or inline style builds no selection, a failed computed-styles or root read builds none either, and the group refresh threads its frame and leaves a group the user changed meanwhile. `findClosestByAttribute`, left without a caller by heygen-com#7613, is removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore(producer): raise the packed size budget for the runtime's geometry read The producer ships the preview runtime, which now carries the editing channel's `geometry` request and the measuring code it shares with Studio. The packed tarball measured 888,810 bytes in CI against an 880,000 budget. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
On a preview frame Studio cannot script, selecting a clip on the timeline and refreshing a selection now build the selection from the runtime's element descriptions, over the editing channel. Which element gets selected comes from one set of rules that both kinds of frame use, so an opaque frame selects what a same-origin frame selects.
Same-origin paths are unchanged. Every preview today is same-origin. Studio still reads those frames directly; their selection code now calls the shared rules, which are the old loops moved out. Nothing flips
sandbox-origin. Clicking and hovering on the canvas stay same-origin only; they move in the next PR.What
Runtime editing port (
packages/core/src/runtime/editingPort.ts,editingAgent.ts,editingHandles.ts):computed-stylesalso takes{ handle, properties }in place of{ selector, properties }. The result shape is unchanged:{ styles },nullfor a removed element or a handle from another load.ancestors { handles }returns{ ancestors: Array<handle[] | null> }: each handle's ancestors, parent first, up to the document element. A chain stops at 256 entries, so onedescribetakes it. A removed or unknown handle givesnull.query { selector }returns{ matches: Array<{ handle, source }>, truncated }: the selector's matches in document order, each with the source facts the parent resolves a file from, at most 2,048. An invalid selector gets a typedfailedreply.Shared selection rules (
packages/studio/src/components/editor/domEditSelectionRules.ts, new). These are pure over a target's ancestor chain, through a small reader (attr,selector,isDocumentRoot):findElementForSelectiontries;resolveDomEditSelection,findElementForSelectionandfindElementForTimelineElementuse them on live elements.resolveGroupCaptureandgetSelectionCandidateare gone, since the rules replace them.Opaque selections (
packages/studio/src/components/editor/previewSelection.ts, new):queryanddescribe, filtered by source file as before.ancestorsplus onedescribe.computed-stylesby handle for the curated computed styles.body. Its geometry part is now a purecoversCompositionRoot, which both paths call.border-radiusserializes as the browser does.handle. Itselementis a stand-in: an element with only the target's tag, in a document with no window, one per handle for the frame load. Nothing from the frame is copied into it.useDomSelection: timeline select, single refresh and group refresh take the opaque path whenpreviewFrameOpaquesays so.buildDomSelectionFromTargetalso accepts a description and the frame it came from.The source-existence probe (
existsInSourceFile) is shared by both paths.Why
Before the main preview frame can go opaque, Studio has to select elements without the frame's DOM. This PR covers the two selection flows that find an element by its identity rather than by a point.
Related work
Refs #7603 (element handles), #7602 (picker styles and composition size), #7561 (playback and timeline readers) and #7556 (the runtime's editing port).
How
Claude Opus 5.5
Why shared pure rules, not a second walk over descriptions. Studio could walk the described ancestors with its own copy of each rule, but then group capture, the clip ancestor and the walk to an addressable element would live in two places. The rules now take a chain and a reader. The live path gives them elements and
getAttribute. The opaque path gives them descriptions, whoseselectoris the runtime'sbuildStableSelector, the same function since #7603. No rule needs layout or computed styles, so the runtime needs no rule code.Runtime surface. The rules need the whole ancestor chain.
describegives only the parent, so walking up would be one round trip per level.ancestorsreturns every handle at once, and onedescribetakes them all.Finding an element by identity needs a document query, which
queryanswers. It returns each match's source facts, so Studio picks the right file's match before describing one.Computed styles reuse
computed-styles, which now names its element by handle too. A selector cannot name one element of many.A single resolution makes these requests, in order:
queryand onedescribe;ancestorsand onedescribe;computed-styles;bodyonly,queryanddescribefor the composition root.Same answer, checked. A parity test runs the real runtime agent from core against the same document. For each case, it compares the selection a same-origin frame builds with the one an opaque frame builds. Everything matches except
element,handleand text fields. In real Chrome, against the real runtime and thestudio-openfixture, eight selections matched field for field, computed styles and rects included: the shape, the title, the scene host and the root, each with and without the clip-ancestor preference.Stale handles. A selection is re-found by identity on refresh, as on a same-origin frame, not by its handle, so a reloaded frame works. A description whose element is gone, a handle from another load, or a chain whose parents do not link up builds nothing.
"Cannot tell" is kept apart from "not there": a frame with no editing agent, or a query cut off before the wanted index. The finder returns
undefinedfor it, and a refresh keeps the selection, as it does today when the frame has no document. A real miss drops the selection, as before.The stand-in. About 600 places across Studio read
selection.element, so a selection still needs anHTMLElement. A frame Studio cannot script has none to give. The stand-in has no attributes, no text and no window, and it is never connected. Readers that need the frame's element move tohandlein later PRs. It is the same object for a handle all load long, so identity checks (knownSourceAnswer, drill-in) keep working.Not migrated here, still direct reads (same-origin only):
resolveDomSelectionFromPreviewPointandresolveAllDomSelectionsFromPreviewPoint. Their hit test reads computed styles across the stack, its ancestors and children. That is the next PR.selection.element. On an opaque frame they get the stand-in and draw nothing, as onmain.domId,selectorandcompositionSrcfrom the frame. On an opaque frame those stay empty today, so a real timeline click there has nothing to look up yet. The After capture fills them in by hand; see below.Size. About half of the diff is tests (471 lines). In
domEditingLayers.ts, about 60 lines only change indentation, because the resolver lost its loop.Before
After
Both are real Studio sessions: the Vite dev server, the
studio-openfixture and headless Chrome 153. A page patch, not committed, sets every<hyperframes-player>tosandbox-origin="opaque". The frame'ssandboxwasallow-scripts, and Studio could not read its document.Before the click, both sessions put each clip's
domId,selector,sourceFileandcompositionSrcinto the timeline store, copied from a same-origin session of the same fixture. That stands in for the timeline enrichment, which reads the frame and so skips an opaque one. The click then runs the realhandleTimelineElementSelect. On this branch the selection came from the real runtime over the real port.Test plan
previewSelection.test.tsuses detached fake frames. The opaque frame is served by core's realinstallEditingAgentover this document, and the same-origin frame reads the same document.editingHandles.test.ts:ancestorsparent first tohtml, and null once removed; the 256 cap;querywith sources in document order,failedfor an invalid selector, and the 2,048 cap;computed-stylesby handle, and null once removed.editingAgent.test.ts: 14 more malformed requests getinvalid-params: handle-form styles (5),ancestors(4) andquery(5).hellolists the two new kinds, here and ineditingPort.test.ts.body, and inline styles through the probe element;test:hyperframe-runtime-ci(typecheck, preview-guard lint, runtime contract, behavior, seek, parity, security, coverage and linter), Studio and coretsc --noEmit, oxlint, oxfmt,fallow audit, the comment ratchet,check:package-cycles,check:package-subpathsandcheck:studio-copy-dashes: all passed.— Rames Jusso
🤖 Generated with Claude Code