Repository navigation
[SCR-386] Cluster mechanics that share a meaning - #388
itsryanarmani wants to merge 2 commits into
Conversation
merge groups on the exact name, which is not enough. Across 31 videos one dash arrived as dash, dash-burst, dash-boost and chef-dash, and each read as an independent finding. That understates the vote, which is the output worth acting on. Clustering took 239 names to 100 mechanics and moved dish washing from an apparent 8 sightings to its true 25. Carries through any name the model fails to place, so the step cannot quietly shrink the corpus.
|
Warning Review limit reachedNext included review available in 11 minutes. View limit detailsLimit details: You’ve used all 3 included reviews currently available. Your 71 included PR review attempts over the past 7 days set your current allowance at 3 reviews per hour. Your organization has reached its usage spending cap. Adjust your spending cap in the billing tab. Review configuration: ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Team Run ID: 📒 Files selected for processing (2)
Comment |
There was a problem hiding this comment.
Devin Review found 3 potential issues.
2 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)
| members = [m for m in c["members"] if m in by_name] | ||
| if not members: | ||
| continue | ||
| claimed.update(members) |
There was a problem hiding this comment.
🟡 Repeated members survive multiple clusters
When the model repeats a name across clusters, members accepts it each time. Its sightings then appear in multiple output mechanics.
Learn more
The response schema constrains each cluster's shape, but it cannot enforce uniqueness across clusters. claimed records names without excluding them from later clusters. A repeated name therefore contributes its videos, descriptions, and feedback more than once, while the final carry-through loop considers it handled.
Example: The model returns clusters [dash, dash-boost] and [dash, chef-dash]. Both output mechanics include every video attached to dash, although that input name was required to appear once.
Recommended fix: Validate the complete cluster response before aggregation. Reject overlapping memberships or deterministically assign each name only once, and report the model contract violation rather than silently producing ambiguous clusters.
Was this helpful? React with 👍 or 👎 to provide feedback.
| out.append({ | ||
| "name": c["canonical"], "category": c["category"], | ||
| "aliases": [m for m in members if m != c["canonical"]], | ||
| "video_count": len(videos), "descriptions": descs[:4], "feedback": fb[:4], |
There was a problem hiding this comment.
🟡 Nonmember canonical names corrupt clusters
When the model returns a nonmember canonical, the cluster adopts that name. Evidence and aliases then describe a different or invented mechanic.
Learn more
The structured schema guarantees only that canonical is a string. The instruction to choose an existing member is not enforced by the API or this aggregation code. A nonmember canonical can also collide with another input name, leaving that name's own evidence in another cluster or carry-through entry.
Example: For members dash-burst and dash-boost, the model returns canonical sprint. The output contains sprint, lists both real inputs as aliases, and adds a name that never existed in the merged findings.
Recommended fix: Require c["canonical"] to occur exactly once in the validated member list. Reject the response or choose a deterministic existing member when it does not.
Was this helpful? React with 👍 or 👎 to provide feedback.
| "category": {"type": "string"}, | ||
| "members": {"type": "array", "items": {"type": "string"}}, |
There was a problem hiding this comment.
🟡 Unsupported categories enter deduped output
The dedupe schema accepts any category string. A schema-valid model response can therefore replace established mechanic categories with unsupported values.
Learn more
Per-video mechanics restrict categories to the fixed enum in SCHEMA, and merge preserves those values. The dedupe response schema removes that restriction and the prompt never lists the allowed categories. Vertex can therefore accept values such as movement even though they cannot occur in normal merged input.
Example: A cluster containing dash can return category movement. The response passes schema validation and the output table stores movement, although the established category is action.
Recommended fix: Define the category values once and reuse that enum in both response schemas. Also decide cluster categories from member data or validate the returned category against those members before writing it.
Was this helpful? React with 👍 or 👎 to provide feedback.
…in dedupe Filter out names already claimed by an earlier cluster before aggregating, and fall back to a real member name when the model's canonical isn't one of the cluster's members.
There was a problem hiding this comment.
Reviewed head: 4a65653a1b857af39fb3ab0c2e7d7dd89b3d40bc
No commits pushed this cycle — nothing surfaced a confidently-real code bug that needed a fix. Verdict below reflects findings only.
Findings
Major — PR metadata doesn't tie to one agreed issue (AGENTS.md pull-request standard).
The branch (scr-386-dedupe-mechanics) and title ([SCR-386] ...) reference issue #386, but #386 is "Vendor the character-studio engine, part 1: pack and prompt" — an unrelated feature. The body's Closes #384 instead references "Add a pipeline for researching YouTube video sources", which was already closed by PR #385. Neither number describes this dedupe work, and there's no open issue for it. AGENTS.md is explicit that the issue number is what ties branch/title/body/commit to one piece of work, and "no issue, no branch." This isn't something I can fix by pushing code — it needs a correctly-numbered issue and a corrected title/body from a human.
Minor — prompt injection surface in the new dedupe prompt (gameref/gameref.py:~300-333, flagged by the security lens). Mechanic names/descriptions originate from third-party YouTube footage (via analyse) and are concatenated unescaped into the clustering instructions, so adversarial video content could steer which names get merged. Real, but low blast-radius here: this is a single-developer local research tool, the output is a JSON file the author already reviews by hand (per the PR's own verification), and there's no code execution or secrets path downstream. Worth a delimited data block around the listing at some point, but not a blocking issue.
Minor — cluster schema doesn't constrain category. Unlike analyse's SCHEMA (enum of 11 known categories), the dedupe schema's category is a free string, so the model could drift a mechanic into a category outside the controlled vocabulary. No concrete break in this diff (nothing downstream filters on it yet), just a quality nit.
Verified as non-issues
- Correctness lens (GPT-5.6): no findings — agreed, nothing found.
- Cluster double-counting / unvalidated canonical name: already fixed in the prior commit (
4a65653), confirmed still correct (claimedset + canonical-must-be-a-member check). ## How I verifiedis solid: real corpus run (239→100), an explicit alias-sum-back-to-239 check proving no name is silently dropped, and anast.parsesanity check. This actually exercises the new code paths, not just a smoke test.- Diff is 121 counted lines across 2 files — well under the 500-line/40-file caps.
🧑⚖️ Council
| Model | Lens | Result |
|---|---|---|
| GPT-5.6 (Codex) (via OpenRouter) | correctness | no findings |
| Kimi K3 (via OpenRouter) | security | confirmed, downgraded to Minor (see above) |
|
Stale-PR sweep: this PR's branch said scr-386 but its body closed #384, and both issues are now closed, so pr-standards could not go green on this number (the checker requires Closes to match the branch issue, and that issue to be open). To land the work I opened #411 describing what this PR actually does and renamed the head branch to scr-411-dedupe-mechanics. GitHub deleted the old ref on rename and auto-closed this PR, so the work continues as #412 (same commits, rebased onto current main). Leaving this closed as superseded. |
Closes #384
What
Adds a
dedupestage that clusters mechanics which are the same thing under different names, and merges their vote counts.Why
mergegroups on the exact name string, which is not good enough. Across 31 Overcooked videos a single dash arrived asdash,dash-burst,dash-boostandchef-dash, and each looked like an independent finding.That is worse than untidy. The count of how many videos saw a mechanic is the output you act on, and splitting one mechanic across four names understates every one of them. On a real run this took 239 names down to 100 mechanics and moved dish washing from an apparent 8 sightings to its true 25 — the difference between a detail and the most-attested mechanic in the genre.
It sends only the names and one description each, never the whole corpus, because the useful signal is the wording and sending everything would cost more than the analysis did. Any name the model fails to place is carried through rather than dropped, so the clustering step cannot quietly shrink the findings.
The prompt says explicitly not to merge mechanics that differ in what the player does or what the game does back: chopping and blending are both preparation and are not the same mechanic, and a warning is not the fire it precedes.
How I verified
Ran it on the real corpus rather than a fixture, because the failure it fixes only appears at scale.
Result -> prints
239 names -> 100 mechanicsand a table whose top row is25 station wash-dishes (+11), so eleven alias names collapsed into one correctly counted mechanic -> okResult -> prints
239 100, confirming the clustered file is a real reduction rather than a copy -> okResult -> prints
239, so every input name survives in exactly one cluster and nothing was silently dropped -> okResult -> exits without error, so the module parses -> ok
Proof: n/a - a command line tool with no visual surface, so the terminal output above is the artefact.
Operator: not applicable - a local research script with no user-visible surface and no deployed component.
Assisted-by: claude-personal-3:claude-opus-5