[eval] Daily generation-quality sample set (fixed prompts × providers) #215

Goal

See whether generation quality improves day over day—not only whether CI/perf is green.

Room 2026-11-07: Hongsheng asked how we judge generation quality. Separate from #213 (perf trends) and #214 (human UX smoke). This issue owns fixed sample-set quality comparison.

v1 scope (proposed)

1. Fixed sample set: a few fixed prompts (image + video), fixed model/provider pairs (needs auth + budget)
2. Comparison: today’s v0.1.x / nightly SHA vs yesterday’s same samples
3. Judging (human-first): usable output / obvious defects / consistency (character/style drift); optional semi-auto hard checks only (decodes, duration, audio present)—aligned with [RFC] First release scope and readiness checklist #7 G4 / [release] Validate real-model video creation and MP4 export #19 subset
4. Artifacts: store or link samples + one-line room conclusion; does not gate release unless we later promote it

Interim

Until this lands, #214 carries 1–2 “glance at generation” checklist lines.

Out of scope (v1)

• Replacing Release checks / [ci] Daily comparable benchmark (trend after nightly) #213 perf trends
• Full G0–G7 acceptance
• Unauthorized paid spam

Open questions

• Who owns the prompt/provider list and budget approval
• Where artifacts live (avoid huge binaries in the public repo)
• Who judges daily; same person as [process] Daily human UX smoke (after nightly / v0.1.x) #214?

Owner

Tracking / nudge: OpenDirector Coordinator. Sample harness / semi-auto hooks: OpenDirector Dev (after a clear brief). Human judge: named by Hongsheng.