Handoff · Live demo · All projects
For a keyless local demo, current checks and known limits, read the developer/user handoff first. The handoff distinguishes current local proof from optional private-media, durable and provider workflows.
NodeVideo is a local-first, artifact-driven editor for short-form creator productions. Dance is the first deeply instrumented profile, while the creator-taste and audit contracts also cover tutorials, talking-head videos, comedy, montage, and other formats. Its primary workflow is:
authorized prior productions -> reusable creator profile -> new source/reference media + brief -> source-only interpretation -> global edit and editorial plan -> fixed render -> freeze -> evaluator-only conjunctive fidelity gates
The product value is the completed editorial outcome: understanding the choreography, choosing the right performance for each musical phrase, cutting on intentional beats, keeping text off the body, muting unusable camera audio, and handing the creator a usable export or licensed-platform music handoff. Agents choose and explain; versioned primitives measure; typed artifacts control fixed render code.
NodeVideo does not ask a language model to guess frame math from raw media. Versioned workers turn
the files into time-indexed evidence; the agent chooses among typed options and explains them; a
fixed renderer executes only the accepted EditPlan.
music -> BPM / beats / downbeats / onsets / lyrics -------\
> candidate cut moments
video -> normalized pose / motion / holds / framing ------/ |
v
global boundary + take search
|
v
typed EditPlan
|
+-------------------------+------------------+
v v
agent patch / approval fixed render/export
The machine-readable signals correspond to the cues a human editor uses:
| A human editor notices | NodeVideo records |
|---|---|
| "That is the beat or musical lift." | BPM, beats, downbeats, onset strength, lyric boundaries, and signed offsets |
| "The move lands or changes direction here." | Pose velocity, gesture apexes, holds/completions, and multi-take direction consensus |
| "Take B shows this phrase better." | Choreography agreement, landmark completeness, framing, and relative motion energy |
| "Cut here, but do not make the sequence frantic." | Multimodal boundary evidence plus a global DP/beam sequence score |
The boundary search does not force a fixed beat grammar. It can retain an anticipatory or delayed
cut when motion evidence is stronger, while recording the exact offset from the nearest musical
event. For example, the integrated fixture places a boundary at 6.633 s, 108 ms after its
nearest music event, then selects Take B with fill framing for the following phrase. Both takes
were scored; the complete sequence, not one greedy per-shot choice, determined the result.
raw files
-> measured evidence
-> candidate boundaries and source ranges
-> globally coherent sequence
-> reviewable EditPlan
-> preview, agent patches, undo, and export
- Media workers own decoding, beat/onset detection, pose normalization, temporal alignment, candidate scoring, and rendering.
- The agent owns intent, tool selection, bounded plan changes, and explanations. It does not calculate pose distance, invent an FFmpeg graph, or mutate pixels directly.
- The creator approves every proposed mutation. The accepted plan is the shared state used by the player, undo history, studio renderer, and local browser export.
Open the production edit studio. Its current loop is:
ask -> inspect tool results -> review patch -> Apply patch -> preview -> undo or export
The browser exporter snapshots only the accepted plan and renders a silent H.264 MP4 locally with pinned FFmpeg WASM. The private song master is intentionally omitted from that download.
/creator.html is the generalized, local-first orchestration surface. Drop an authorized video,
optionally paste a transcript, choose cleanup, golden-quote variants, or founder launch, and compile
one shared MediaIndex into reviewable output variants. Each variant exposes its source lineage,
semantic EditPlan v2 operations, approval requirements, selected executors, and estimated cost
before local export.
npm run creator:dev
npm run test:creator
npm run proof:creatorThe agent owns intent and routing; replaceable executors own transcription, VAD, shot detection, semantic selection, reframing, transitions, and rendering. The browser proof uses metadata, an optional low-confidence untimed transcript index, and a video-only H.264 renderer. The production local path now adds FFprobe/FFmpeg, optional Whisper, PySceneDetect, OpenCV sampling, approval-gated audio-preserving renders, and proof receipts. A pinned Higgsfield CLI adapter supplies optional specialist generation behind login, egress, cost, and rights gates; it is never reported as live until a provider job receipt exists.
npm run media:index:doctor
npm run executors:doctor
npm run creator:local -- path/to/owned-source.mp4 --output .qa/evidence/my-run --preset variantsSee the generalized architecture, executor integration contract, Higgsfield runbook, and copy-ready end-to-end prompt suite.
/creatorbench is the evidence surface for measured generalization. It is separate from the creator
workspace and the fixture-bound Artifact Atlas. Every benchmark request uses the same
nodevideo.creator-request/v1 contract, and every outcome is classified as automatic usable,
assisted usable, review required, safe abstention, unsupported, technical failure, or silent
failure. A rendered file is never counted as usable without a blinded human judgment.
npm run creatorbench:acquire
npm run creatorbench:manifests
npm run creatorbench:dedupe
npm run creatorbench:validate
npm run creatorbench:evaluate
npm run creatorbench:freeze
$env:NODEVIDEO_CREATORBENCH_EVALUATOR_TOKEN='<post-freeze evaluator credential>'
npm run creatorbench:evaluate:sealed
npm run creatorbench:report
npm run proof:creatorbenchAcquisition is fail-honest: the target remains 250 rights-cleared clips, 75 creator-disjoint sources,
15 domains, 8 workflows, and 2,000 instances. If a source host or rights review prevents that target,
the receipt publishes the achieved population and gap; it does not prefill the claim. See the
governance contract, sealed evaluation
boundary, and end-to-end benchmark prompts.
The report command also emits the standalone machine-readable
benchmarks/creatorbench-v1/results/public-claim.json; it is derived from the frozen held-out
results and is scanned alongside the public report for private locators and hidden targets.
| Music input | NodeVideo behavior |
|---|---|
| Owned or otherwise authorized audio | Bind its rights/provenance, analyze the declared excerpt, and route it through the studio render. |
| Instagram or another licensed-platform catalog | Export no commercial bytes; provide track identity, search text, phrase anchors, and alignment instructions for in-platform confirmation. |
| Disclosed calibration oracle | Use the supplied timing/audio only for that calibration and label it as non-blind evidence, never as proof of song selection or generalized taste. |
Open the NodeVideo production entry point. When this branch is deployed, it presents the song-conditioned replay on laptop and phone with no sign-in. Vercel serves hash-bound replay artifacts; heavy media analysis and rendering remain local or in a media worker.
The animated 19-stage pipeline and live LocateAnything comparison are also served by the production deployment.
The foreground demo is a deterministic six-second public fixture with:
- one generated original choreography reference;
- two generated creator takes;
- a public-domain generated song segment;
- source-only beat/phrase and pose evidence;
- three body-safe timed text cues;
- muted camera audio; and
- canonical analysis, plan, render, read-log, and freeze artifacts.
The public replay route and its manifest route are published with this branch. This validates mechanics and isolation, not generalized creative taste or arbitrary human-pose accuracy. A separate research-only live receipt exercises NVIDIA LocateAnything on the supplied case; one successful frame is not a grounding-accuracy benchmark.
The branch also publishes the supplied real-media case as a silent picture-only calibration via its 44.5-second plan route and score-manifest route. No commercial soundtrack or source container is included in that new public preview.
nodevideo.creator-taste-audit converts authorized production evidence into a versioned
CreatorTasteProfile. Each learned value retains supporting production counts, confidence, and
evidence references across editorial attention, creator voice, spatial grammar, visual treatment,
identity, CTA, and end-card behavior. One-production profiles are explicitly provisional.
The same pack audits candidate productions and validates the evaluator's target interpretation.
If the claimed spec cannot explain visible OCR roles, persistent branding, layout zones, grade, or
delivery, creative scoring is invalid. Once that prerequisite passes, provenance, structure,
semantic overlays, layout, visual treatment, creator identity, and delivery are conjunctive gates:
every required gate must pass. NodeAgent proposes profile/audit candidates; NodeVideo retains
schema, hash, project-boundary, persistence, and owner-review authority.
npm run taste:audit -- `
--input .qa/evidence/private/style-gap-audit/style-gap-report.json `
--out .qa/evidence/private/creator-taste/run.json `
--profile-id creator-taste.owner-v1 `
--content-kind dance `
--derive-target-spec--derive-target-spec is for an owner-authorized reference-learning run. Omit it when auditing an
existing interpretation so the consistency gate can expose missing semantics instead of silently
repairing them.
For each production, npm run production:audit produces content-neutral frame, OCR, edit-plan, and
seven-gate evidence. --reuse-observations-from makes evaluator changes cheap to replay while
preserving the original observations and disclosure. This is how the same NodeAgent workflow can
evaluate dance, tutorials, talking-head clips, comedy, and montage work without introducing a new
custom renderer or scoring script for each format.
Kinetic text is represented as discrete overlay events rather than one lyric-length caption. The
fixed renderer width-fits each cue, emits an inspectable glyph box, and npm run overlay:clearance
checks that box against Pose Landmarker evidence from the rendered timeline. More than five percent
body overlap blocks approval, including collisions that occur only near a cut boundary.
src/lib/frame-evidence.ts decides whether a frame may be presented as the running product. It
does not produce frames. For a product walkthrough they come from FeatureClipStudio
(feature-walkthrough-gif/), and the two halves only work together:
npm run capture # Playwright drives the live flow, one frame per UI state
npm run render # Remotion adds cursor, ripple, captions, zoom-to-focus
node find-references.mjs "<query>" # build/extend the reference corpus (cached, cumulative)
node judge-video.mjs out/demo.mp4 # Gemini scores the render against that corpusA frame captured from a live deployment is kind: "live-product", and this module then requires it
to carry deploymentRevision, browserTraceId, journeyState and screenshotSha256 — a claim
about the running application must name the exact deployment it came from. A Remotion-composited
frame is motion-graphic and owes its sourceRef instead. Presenting either as the other is the
failure presentedAs exists to make checkable, and it is the likely one: the composited frame
usually looks better.
The reference corpus is observations, never footage. find-references.mjs uses yt-dlp to SEARCH
only and Gemini watches the URL, so references are cited by URL plus timestamp and nothing is
re-hosted. Its ledger reuses anything already observed under the same prompt and model, and records
what it rejected and why, so a second run costs nothing and still discovers.
For each proposed phrase, the source-only dance analyzer currently weights choreography agreement
at 40%, landmark completeness at 20%, framing at 15%, and relative movement energy at 25%.
These are observable source-quality signals, not claims about artistry, confidence, or talent. A
quality gate allows visual contrast between takes without silently selecting a materially worse
performance.
Optional timed lyrics are represented as discrete overlay events and placed in normalized safe zones outside the grounded body and face. Missing or ambiguous grounding stops for manual review instead of inventing a valid result.
Generation produces four canonical boundaries:
ChoreographyAnalysis(nodevideo.choreography-analysis.v1);SongConditionedPlan(nodevideo.song-conditioned-plan.v1);EditPlan(nodevideo.edit-plan.v1); andFreeze(nodevideo.choreography-freeze.v1plus a wrapper receipt).
The EditPlan routes the chosen song and explicitly mutes audio from every take. It drives the fixed
renderer in scripts/workers/edit-plan-renderer.mjs; an
agent cannot inject JSX, CSS, shell commands, or a bespoke FFmpeg graph. The evaluator verifies all
frozen hashes before it can open a held-out target plan, keeping generation evidence separate from
evaluation-only ground truth.
Read the full workflow, artifact contracts, rights boundary, proof ledger, and exact evaluator
command in docs/song-conditioned-pipeline.md.
NodeVideo now includes a real /health + /locate sidecar backed by NVIDIA's official free queued
Hugging Face Space. Copy .env.example to .env, review the NVIDIA code and model licenses, and set
NODEVIDEO_LOCATEANYTHING_LICENSE_ACCEPTED=true only for permitted research/non-commercial use.
The included registry points at the public proof frame; replace it with a private ignored registry
for local assets.
npm run grounding:sidecar
node --env-file=.env scripts/quality/grounding-doctor.mjs --jsonThe free endpoint can sleep, queue, or rate-limit. HF_TOKEN is optional for the public Space but
recommended for an identifiable Hugging Face quota. The same typed contract can target an
operator-managed local worker without changing choreography or caption-planning callers.
The real-media generation interface never accepted the final target picture or target plan. It used Source A as a disclosed creator-selected choreography fallback, aligned Source B by normalized pose, applied the beat template, selected A/B/A/B/A over five phrases, rendered, and froze. Only then did the evaluator open the target plan. This is an audited CLI/read-log boundary, not an OS sandbox.
| Post-freeze measure | Result |
|---|---|
Cut F1 at 0.75 s tolerance |
0.909091 |
| Mean nearest-cut error | 0.366667 s |
| Maximum nearest-cut error | 0.633333 s |
| Neutral source agreement | 5 / 5 phrases |
| Duration | 44.5 s generated / 44.5 s target |
| Taste status | not-evaluated |
These are legacy proximity diagnostics, not an editorial pass. The 0.75 s tolerance is too loose
for dance editing, and the nearest-cut aggregate does not replace signed, frame-level accounting.
The supplied-case reconstruction verdict is therefore failed under the current two-frame gate.
This is a target-picture-isolated, target-audio-oracle calibration. The exact authorized audio excerpt was supplied as timing input, so it does not prove song identity/excerpt selection or blind creative taste. The current supplied case also lacks an independent original choreography video, an independent creator-supplied song master/segment, and timed lyrics. Source A is therefore a clearly labeled fallback reference, and no lyric overlay is invented.
The earlier owner-authorized V2 target-guided reconstruction remains useful forensic/render
calibration, not source-only planning proof. V1 remains immutable failure evidence, including its
wrong 16.067-19.633 s phrase and missing music/text. See
docs/authorized-case-v2-forensics.md.
Requirements: Node.js 22.12+, npm 10+, Chromium for browser tests, and FFmpeg/FFprobe for regeneration or independent media verification.
npm ci
cp .env.example .env
npm run devIn PowerShell, use Copy-Item .env.example .env. Open http://localhost:4173 from the laptop, or
use the laptop's LAN address from a phone on the same network; Vite is configured to listen on the
host interface.
When the private integrated render exists at
.qa/evidence/private/integrated-inspector-v1/frozen-generation-v5/source-only-song-preview.mp4,
npm run dev exposes it through a byte-range-capable development-only route. The generated-edit
player then shows Local soundtrack enabled, plays the AAC soundtrack, and drives the shared
frame inspector while it plays. Set NODEVIDEO_LOCAL_PREVIEW to another private MP4 path to
override the default. npm run build, npm run preview, Git, and Vercel continue to use the silent
hash-bound public preview; outside the Vite development server the private URL never returns video
bytes (static hosts may route the unknown URL to the HTML application shell).
Run the complete capability/grounding/worker health check:
npm run doctorRegenerate and verify the public-domain song-conditioned replay:
npm run proof:song:publicVerify the published real-case calibration without private media:
npm run proof:song:real:verifyRun the post-freeze evaluator explicitly:
npm run eval:song -- `
--freeze .qa/evidence/private/song-conditioned-source-only-v1/freeze-receipt.json `
--generated-plan .qa/evidence/private/song-conditioned-source-only-v1/edit-plan.json `
--target-plan .qa/evidence/private/reference-analysis-v2/edit-plan-reviewed.json `
--output .qa/evidence/private/song-conditioned-source-only-v1/post-freeze-evaluation.json `
--allow-target-audio-oracleDo not run the evaluator until the generation freeze exists. Omit
--allow-target-audio-oracle for a normal source-only case.
The same provider-neutral LocateRequest, LocateResult, and GroundingHealth contract supports:
- deterministic
replaygeometry for CI; - creator-reviewed
manualgeometry; - an explicit fail-closed
disabledprofile; and - an optional operator-managed LocateAnything HTTP sidecar.
The default demo needs no model or GPU. npm run grounding:doctor does not contact a model service,
download weights, or accept a license. Durable artifacts contain trace/asset IDs and normalized
geometry, not raw frames, local paths, credentials, or vendor payloads.
LocateAnything code and weights have separate terms: the upstream Eagle code license is Apache-2.0, while the LocateAnything model license limits noncommercial use to research/evaluation. NodeVideo requires explicit operator license references and acceptance before an HTTP model profile can activate. It makes no visual-prompt claim: the released checkpoint is documented as not supporting visual-prompt inference out of the box.
Owned or otherwise authorized audio can drive preview/export when its rights attestation is bound to the plan. For Instagram catalog music, NodeVideo instead supplies the track/search identity, intended segment, beat/phrase anchors, and visual instructions so the creator can add and confirm the recording inside Instagram for their account and region. NodeVideo does not redistribute the commercial file or claim an exact Instagram waveform offset until it is confirmed in-platform.
npm run lint
npm run typecheck
npm run test
npm run check:ui
npm run capability:validate
npm run grounding:doctor
npm run worker:verify
npm run worker:song:verify
npm run proof:song:real:verify
npm run build
npx playwright install chromium
npm run test:e2enpm run check runs the static, unit, schema, grounding, receipt, UI-policy, worker, real-case proof,
historical-proof, and build gates. Browser QA covers desktop, tablet, and phone widths,
accessibility, keyboard operation, media decoding, and horizontal overflow. The stable selector and
consent contract is in .qa/profile.md.
NodeVideo keeps custom maintenance small by composing existing primitives:
- generated shadcn/ui and AI Elements for generic interaction and artifacts;
- react-o11y for recorded trace presentation;
- NodeAgent-style typed tools, deterministic receipts, and validation;
- Convex workflow/event patterns for durable orchestration;
- MediaPipe pose evidence, librosa beat/onset analysis, OpenTimelineIO interchange, and FFmpeg fixed rendering; and
- provider-neutral grounding rather than vendor-specific plan fields.
npm run check:ui rejects raw generic controls, direct Radix imports, inline layout styles,
arbitrary Tailwind dimensions, extra authored stylesheets, and custom media queries. See
docs/ui-primitives.md.
packs/song-conditioned-auto-edit/- source-only schemas, tools, claims, and replay gates.packs/embodied-grounding/- provider-neutral grounding contract and license boundary.fixtures/media/song-conditioned-auto-edit-v1/- public-domain reference/takes/song plus canonical artifacts and preview.fixtures/media/song-conditioned-real-calibration-v1/- target-isolated real-case artifacts, silent preview, and post-freeze score.scripts/analysis/song_choreography_analyzer.py- source-only pose/song interpretation.scripts/quality/evaluate-song-conditioned.mjs- freeze-first target evaluator.docs/song-conditioned-pipeline.md- detailed workflow and proof ledger.docs/architecture.md- system responsibility and evidence boundaries.fixtures/media/blind-source-only-pilot-01/- earlier blind exploration retained as secondary evidence, not the primary product contract.
The checked-in synthetic replay proves deterministic mechanics. The supplied real case is a
target-picture-isolated development calibration under a disclosed target-audio oracle; it does not
pass strict editorial timing. The newer integrated case proves independent-reference ingestion,
target isolation, artifact integrity, and 5 / 5 source choices, but its signed boundary errors are
-3, -8, +5, +5, -2 frames, with one additional unmatched target boundary. Its strict verdict is
also failed. Neither case proves general creative taste. See
docs/strict-editorial-audit.md.
