Skip to content

[SCR-386] Cluster mechanics that share a meaning - #388

Closed
itsryanarmani wants to merge 2 commits into
mainfrom
scr-386-dedupe-mechanics
Closed

itsryanarmani wants to merge 2 commits into
mainfrom
scr-386-dedupe-mechanics

Conversation

@itsryanarmani

@itsryanarmani itsryanarmani commented Sep 13, 2026 •

Copy link
Copy Markdown
Owner

Closes #384

What

Adds a dedupe stage that clusters mechanics which are the same thing under different names, and merges their vote counts.

Why

merge groups on the exact name string, which is not good enough. Across 31 Overcooked videos a single dash arrived as dash, dash-burst, dash-boost and chef-dash, and each looked like an independent finding.

That is worse than untidy. The count of how many videos saw a mechanic is the output you act on, and splitting one mechanic across four names understates every one of them. On a real run this took 239 names down to 100 mechanics and moved dish washing from an apparent 8 sightings to its true 25 — the difference between a detail and the most-attested mechanic in the genre.

It sends only the names and one description each, never the whole corpus, because the useful signal is the wording and sending everything would cost more than the analysis did. Any name the model fails to place is carried through rather than dropped, so the clustering step cannot quietly shrink the findings.

The prompt says explicitly not to merge mechanics that differ in what the player does or what the game does back: chopping and blending are both preparation and are not the same mechanic, and a warning is not the fire it precedes.

How I verified

Ran it on the real corpus rather than a fixture, because the failure it fixes only appears at scale.

python3 gameref/gameref.py dedupe mechanics.json --out mechanics-deduped.json --show 40

Result -> prints 239 names -> 100 mechanics and a table whose top row is 25 station wash-dishes (+11), so eleven alias names collapsed into one correctly counted mechanic -> ok

bash -c "python3 -c \"import json;print(len(json.load(open('mechanics.json'))), len(json.load(open('mechanics-deduped.json'))))\""

Result -> prints 239 100, confirming the clustered file is a real reduction rather than a copy -> ok

bash -c "python3 -c \"import json;d=json.load(open('mechanics-deduped.json'));print(sum(len(e['aliases']) for e in d) + len(d))\""

Result -> prints 239, so every input name survives in exactly one cluster and nothing was silently dropped -> ok

python3 -c "import ast;ast.parse(open('gameref/gameref.py').read())"

Result -> exits without error, so the module parses -> ok

Proof: n/a - a command line tool with no visual surface, so the terminal output above is the artefact.

Operator: not applicable - a local research script with no user-visible surface and no deployed component.

Assisted-by: claude-personal-3:claude-opus-5


Devin Review

merge groups on the exact name, which is not enough. Across 31 videos one
dash arrived as dash, dash-burst, dash-boost and chef-dash, and each read as
an independent finding.

That understates the vote, which is the output worth acting on. Clustering
took 239 names to 100 mechanics and moved dish washing from an apparent 8
sightings to its true 25.

Carries through any name the model fails to place, so the step cannot quietly
shrink the corpus.
@coderabbitai

coderabbitai Bot commented Sep 13, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

Next included review available in 11 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 3 included reviews currently available. Your 71 included PR review attempts over the past 7 days set your current allowance at 3 reviews per hour.

Your organization has reached its usage spending cap. Adjust your spending cap in the billing tab.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Team

Run ID: e964b65e-b0fa-4b56-b81c-805efa68dde3

📥 Commits

Reviewing files that changed from the base of the PR and between ddc3d9b and 4a65653.

📒 Files selected for processing (2)
  • gameref/README.md
  • gameref/gameref.py

Comment @coderabbitai help to get the list of available commands.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 3 potential issues.

2 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)

Devin Review

Comment thread gameref/gameref.py Outdated
Comment on lines +342 to +345
members = [m for m in c["members"] if m in by_name]
if not members:
continue
claimed.update(members)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Repeated members survive multiple clusters

When the model repeats a name across clusters, members accepts it each time. Its sightings then appear in multiple output mechanics.

Learn more

The response schema constrains each cluster's shape, but it cannot enforce uniqueness across clusters. claimed records names without excluding them from later clusters. A repeated name therefore contributes its videos, descriptions, and feedback more than once, while the final carry-through loop considers it handled.

Example: The model returns clusters [dash, dash-boost] and [dash, chef-dash]. Both output mechanics include every video attached to dash, although that input name was required to appear once.

Recommended fix: Validate the complete cluster response before aggregation. Reject overlapping memberships or deterministically assign each name only once, and report the model contract violation rather than silently producing ambiguous clusters.

Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Comment thread gameref/gameref.py
Comment on lines +352 to +355
out.append({
"name": c["canonical"], "category": c["category"],
"aliases": [m for m in members if m != c["canonical"]],
"video_count": len(videos), "descriptions": descs[:4], "feedback": fb[:4],

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Nonmember canonical names corrupt clusters

When the model returns a nonmember canonical, the cluster adopts that name. Evidence and aliases then describe a different or invented mechanic.

Learn more

The structured schema guarantees only that canonical is a string. The instruction to choose an existing member is not enforced by the API or this aggregation code. A nonmember canonical can also collide with another input name, leaving that name's own evidence in another cluster or carry-through entry.

Example: For members dash-burst and dash-boost, the model returns canonical sprint. The output contains sprint, lists both real inputs as aliases, and adds a name that never existed in the merged findings.

Recommended fix: Require c["canonical"] to occur exactly once in the validated member list. Reject the response or choose a deterministic existing member when it does not.

Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Comment thread gameref/gameref.py
Comment on lines +313 to +314
"category": {"type": "string"},
"members": {"type": "array", "items": {"type": "string"}},

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Unsupported categories enter deduped output

The dedupe schema accepts any category string. A schema-valid model response can therefore replace established mechanic categories with unsupported values.

Learn more

Per-video mechanics restrict categories to the fixed enum in SCHEMA, and merge preserves those values. The dedupe response schema removes that restriction and the prompt never lists the allowed categories. Vertex can therefore accept values such as movement even though they cannot occur in normal merged input.

Example: A cluster containing dash can return category movement. The response passes schema validation and the output table stores movement, although the established category is action.

Recommended fix: Define the category values once and reuse that enum in both response schemas. Also decide cluster categories from member data or validate the returned category against those members before writing it.

Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

…in dedupe

Filter out names already claimed by an earlier cluster before aggregating, and fall back to a real member name when the model's canonical isn't one of the cluster's members.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed head: 4a65653a1b857af39fb3ab0c2e7d7dd89b3d40bc

No commits pushed this cycle — nothing surfaced a confidently-real code bug that needed a fix. Verdict below reflects findings only.

Findings

Major — PR metadata doesn't tie to one agreed issue (AGENTS.md pull-request standard).
The branch (scr-386-dedupe-mechanics) and title ([SCR-386] ...) reference issue #386, but #386 is "Vendor the character-studio engine, part 1: pack and prompt" — an unrelated feature. The body's Closes #384 instead references "Add a pipeline for researching YouTube video sources", which was already closed by PR #385. Neither number describes this dedupe work, and there's no open issue for it. AGENTS.md is explicit that the issue number is what ties branch/title/body/commit to one piece of work, and "no issue, no branch." This isn't something I can fix by pushing code — it needs a correctly-numbered issue and a corrected title/body from a human.

Minor — prompt injection surface in the new dedupe prompt (gameref/gameref.py:~300-333, flagged by the security lens). Mechanic names/descriptions originate from third-party YouTube footage (via analyse) and are concatenated unescaped into the clustering instructions, so adversarial video content could steer which names get merged. Real, but low blast-radius here: this is a single-developer local research tool, the output is a JSON file the author already reviews by hand (per the PR's own verification), and there's no code execution or secrets path downstream. Worth a delimited data block around the listing at some point, but not a blocking issue.

Minor — cluster schema doesn't constrain category. Unlike analyse's SCHEMA (enum of 11 known categories), the dedupe schema's category is a free string, so the model could drift a mechanic into a category outside the controlled vocabulary. No concrete break in this diff (nothing downstream filters on it yet), just a quality nit.

Verified as non-issues

  • Correctness lens (GPT-5.6): no findings — agreed, nothing found.
  • Cluster double-counting / unvalidated canonical name: already fixed in the prior commit (4a65653), confirmed still correct (claimed set + canonical-must-be-a-member check).
  • ## How I verified is solid: real corpus run (239→100), an explicit alias-sum-back-to-239 check proving no name is silently dropped, and an ast.parse sanity check. This actually exercises the new code paths, not just a smoke test.
  • Diff is 121 counted lines across 2 files — well under the 500-line/40-file caps.
🧑‍⚖️ Council
Model Lens Result
GPT-5.6 (Codex) (via OpenRouter) correctness no findings
Kimi K3 (via OpenRouter) security confirmed, downgraded to Minor (see above)

@itsryanarmani

Copy link
Copy Markdown
Owner Author

Stale-PR sweep: this PR's branch said scr-386 but its body closed #384, and both issues are now closed, so pr-standards could not go green on this number (the checker requires Closes to match the branch issue, and that issue to be open).

To land the work I opened #411 describing what this PR actually does and renamed the head branch to scr-411-dedupe-mechanics. GitHub deleted the old ref on rename and auto-closed this PR, so the work continues as #412 (same commits, rebased onto current main). Leaving this closed as superseded.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a pipeline for researching YouTube video sources

1 participant