[valeriangalliat]

🤖 AI agent: Arcade fork patch. Do not merge.

This draft shows patch/vp9-alpha-probe against main, the upstream mirror, so the patch has one page for its diff, its revisions and its reason. Arcade's /hyperframes-update rebases the branch onto main and cherry-picks it onto arcade; this PR is not part of that flow. When a rebuild changes the patch, its git range-diff against the previous sync/<date>/vp9-alpha-probe tag is posted as a comment.

Why

Upstream (#6284) extracts VP8 and VP9 as JPEG when the stream has no alpha tag and a frame decoded with the alpha-aware decoder has no alpha plane. A stream tagged alpha is not probed and stays PNG. Chrome's MediaRecorder writes ALPHA_MODE=1 on every VP9 recording, opaque or not, so those recordings still paid the PNG cost.

The patch probes tagged VP8/VP9 streams too and lets the decoded plane pick the frame format for these codecs. With no alpha plane it extracts JPEG; an explicit png is still honored. An alpha plane, a failed probe or a missing field keeps PNG. Other codecs still go by the tag.

Before sync/2026-11-10 the patch carried its own probe. Upstream's replaced it, and only the tagged case is left here.

Staging A/B of the original patch: export about 28% faster at 1080p and about 40% faster at 4K retina, extracted frames 3–4× smaller, worst-frame VMAF ≥ 98.4 against PNG.

Upstream

No issue or PR filed. Kept local.

Drop when

Upstream probes the decoded alpha plane of VP8/VP9 streams that carry an alpha tag, or the renders stop receiving Chrome-recorded VP9.

Arcade

ARC-15481 · ArcadeHQ/arcade#11146

[valeriangalliat]

🤖 AI agent: sync/2026-10-24 (tags moved the same day): rebased onto upstream 2e397dba5 (v0.8.194). Replayed clean.

First pass on this date, onto 722481214 (0.8.193): ProRes dropped from the patch: upstream heygen-com#6130 extracts opaque ProRes as JPEG from its pixel format. The probe now runs for VP8/VP9 only; pixelFormatHasAlpha comes from @hyperframes/parsers/media-alpha (heygen-com#6025). Upstream heygen-com#3771 probeInputAlphaPlane is a warning only and samples rgba (which invents an opaque channel), so it cannot pick the frame format.

The range-diff below is against sync/2026-10-05.

git range-diff sync/2026-10-05/base..sync/2026-10-05/vp9-alpha-probe sync/2026-10-24/base..sync/2026-10-24/vp9-alpha-probe

1:  966c3442f ! 1:  b534efd07 feat(engine): extract opaque VP8/VP9/ProRes frames as JPEG after an alpha-plane probe
    @@ Metadata
     Author: Val <val@codejam.info>
     
      ## Commit message ##
    -    feat(engine): extract opaque VP8/VP9/ProRes frames as JPEG after an alpha-plane probe
    +    feat(engine): extract opaque VP8/VP9 frames as JPEG after an alpha-plane probe
     
         Alpha-capable codecs were always extracted as PNG because tags cannot be
         trusted: a missed WebM BlockAdditional sidecar reads as opaque, and Chrome
    @@ Commit message
         picks JPEG (explicit png is still honored); a plane, a decode failure or a
         timeout keeps PNG.
     
    +    ProRes is left to upstream (#6130): its alpha shows in the pixel format,
    +    so the tag already picks jpg or png and no probe runs for it.
    +
      ## packages/engine/src/services/videoFrameExtractor.test.ts ##
     @@ packages/engine/src/services/videoFrameExtractor.test.ts: import {
        FrameLookupTable,
    @@ packages/engine/src/services/videoFrameExtractor.test.ts: import {
        decoderForCodec,
        resolveVideoExtractionWindow,
     @@ packages/engine/src/services/videoFrameExtractor.test.ts: describe("resolveFrameFormat", () => {
    -     expect(resolveFrameFormat(metadata({ hasAlpha: true }), "jpg")).toBe("png");
    -     expect(resolveFrameFormat(metadata({ videoCodec: "vp9" }), "jpg")).toBe("png");
    +       "png",
    +     );
        });
     +
     +  it("picks jpg for an alpha-capable codec only when the probe finds no plane", () => {
    @@ packages/engine/src/services/videoFrameExtractor.test.ts: describe("resolveFrame
      describe("parseVideoElements", () => {
     
      ## packages/engine/src/services/videoFrameExtractor.ts ##
    -@@ packages/engine/src/services/videoFrameExtractor.ts: import { isKnownInactiveTimelineWindow } from "./mediaTimelineWindow.js";
    - import {
    -   extractFinalVideoFrameTimestamp,
    -   extractMediaMetadata,
    -+  pixelFormatHasAlpha,
    -   type VideoMetadata,
    - } from "../utils/ffprobe.js";
    +@@ packages/engine/src/services/videoFrameExtractor.ts: import { createConcurrencyLimit } from "../utils/concurrencyLimit.js";
    + import { parseHTML } from "linkedom";
    + import { isPathInside } from "@hyperframes/parsers/asset-paths";
    + import { resolveProjectRelativeSrc } from "@hyperframes/parsers/asset-resolution";
    +-import { alphaDecoderFor } from "@hyperframes/parsers/media-alpha";
    ++import { alphaDecoderFor, pixelFormatHasAlpha } from "@hyperframes/parsers/media-alpha";
      import {
    -@@ packages/engine/src/services/videoFrameExtractor.ts: export async function extractVideoFramesRange(
    +   MEDIA_RENDER_ID_ATTR,
    +   isAudibleVideoElement,
    +@@ packages/engine/src/services/videoFrameExtractor.ts: async function extractVideoFramesRangeUnredacted(
            `Video media start ${startTime}s is outside playable video duration ${playableDuration}s`,
          );
        }
    @@ packages/engine/src/services/videoFrameExtractor.ts: export async function extra
     +    await probeAlphaPlane(videoPath, metadata.videoCodec, signal),
     +  );
        const framePattern = `${FRAME_FILENAME_PREFIX}%05d.${format}`;
    -   const outputPattern = join(videoOutputDir, framePattern);
    +   const outputPattern = frameSequencePath(videoOutputDir, framePattern);
    + 
    +@@ packages/engine/src/services/videoFrameExtractor.ts: export function resolveVideoExtractionDuration(
    + 
    + /** Codecs whose bitstream can carry alpha: decoded with the alpha-aware decoder whether or not a tag says so. */
    + const ALPHA_CAPABLE_CODECS = new Set(["vp9", "vp8", "prores"]);
    +-/** Their alpha rides a side channel ffprobe can miss, so frames stay PNG; ProRes alpha shows in its pixel format. */
    ++/** Their alpha rides a side channel ffprobe can miss, so frames stay PNG unless a decoded frame has no plane; ProRes alpha shows in its pixel format. */
    + const SIDECAR_ALPHA_CODECS = new Set(["vp9", "vp8"]);
      
    + export function codecMayHaveAlpha(codec: string | undefined): boolean {
     @@ packages/engine/src/services/videoFrameExtractor.ts: export function decoderForCodec(codec: string | undefined): string {
    -   return c;
    +   return alphaDecoderFor(codec) ?? (codec ?? "").toLowerCase();
      }
      
     +/**
    @@ packages/engine/src/services/videoFrameExtractor.ts: export function decoderForC
     +  codec: string | undefined,
     +  signal?: AbortSignal,
     +): Promise<AlphaPlaneProbe> {
    -+  if (!codecMayHaveAlpha(codec)) return Promise.resolve("absent");
    ++  // Other codecs show alpha in their pixel format, so the tag already answers.
    ++  if (!SIDECAR_ALPHA_CODECS.has((codec ?? "").toLowerCase())) return Promise.resolve("unknown");
     +  const key = `${videoPath}\0${codec}`;
     +  const cached = alphaPlaneProbes.get(key);
     +  if (cached) return cached;
    @@ packages/engine/src/services/videoFrameExtractor.ts: export function decoderForC
        requested?: VideoFrameFormat,
     +  alphaPlane: AlphaPlaneProbe = "unknown",
      ): CacheFrameFormat {
    --  if (metadata.hasAlpha || codecMayHaveAlpha(metadata.videoCodec)) return "png";
    -+  // For alpha-capable codecs the decoded plane is the answer; the tag is wrong
    -+  // both ways (missed sidecar, or `ALPHA_MODE=1` on opaque Chrome recordings).
    -+  if (codecMayHaveAlpha(metadata.videoCodec)) {
    +-  if (metadata.hasAlpha || SIDECAR_ALPHA_CODECS.has((metadata.videoCodec ?? "").toLowerCase()))
    ++  // VP8/VP9: the decoded plane is the answer; the tag is wrong both ways (missed
    ++  // sidecar, or `ALPHA_MODE=1` on opaque Chrome recordings).
    ++  if (SIDECAR_ALPHA_CODECS.has((metadata.videoCodec ?? "").toLowerCase())) {
     +    if (alphaPlane !== "absent") return "png";
     +  } else if (metadata.hasAlpha) {
    -+    return "png";
    +     return "png";
     +  }
        if (requested === "png" || requested === "jpg") return requested;
        return "jpg";

[github-actions]

Edit accuracy: accurate 271 (base branch 271), smooth 151 of those

The gate passes.
Smoothness is reported in the artifact, not gated. A case fails only if it fails 2 of 3 runs.

Quarantined, measured but not gated (0)

[valeriangalliat]

🤖 AI agent: sync/2026-11-10: rebased onto upstream cda84222a (just past v0.9.30). Rewritten. Upstream #6284 now probes the decoded alpha plane itself, but only for streams with no alpha tag. The patch is now a small change on top of it: tagged VP8/VP9 streams are probed too, and the decoded plane picks the frame format. Chrome's MediaRecorder tags every VP9 recording as alpha, opaque or not.

1:  b534efd07 < -:  --------- feat(engine): extract opaque VP8/VP9 frames as JPEG after an alpha-plane probe
-:  --------- > 1:  6b1b8bd41 feat(engine): probe the alpha plane of VP8/VP9 streams tagged alpha too