Skip to content

Validate frozen GitHub fixture integrity (issue #147) - #150

Merged
PhysShell merged 2 commits into
mainfrom
claude/fixture-integrity-validator
Aug 16, 2026
Merged

PhysShell merged 2 commits into
mainfrom
claude/fixture-integrity-validator

Conversation

@PhysShell

@PhysShell PhysShell commented Aug 16, 2026 •

Copy link
Copy Markdown
Owner

This is the first consumer of the Step 0B fixtures after they were frozen on main. It validates fixture integrity only. It does not map any observed fixture condition to a closure outcome.

Two commits, one file: tests/github_fixture_integrity.rs. No classifier code, and none to be added to this branch.

89057aa  test: validate frozen GitHub fixture integrity
fc6d4d9  test: drop an integrity relation the frozen README does not state

Why the ordering matters

0B fixtures frozen on main        eb885924 -> merged b0cc6447
        v
first consumer = integrity only   THIS PR
        v
semantic consumer = classifier    later, separate

With the corpus already immutable, a failure here is a discovery about the frozen corpus, not licence to move fixture and checker toward each other until they agree. That property only exists because 0B merged with no consumer, so this boundary should not later be squashed into the classifier commit.

A defect found in review, and what it teaches

89057aa asserted /review_comments/0/commit_id == /reviews/0/commit_id for the stale specimen. The frozen README states no such relation. It gives one observed condition — /reviews/0/commit_id != /pull_request/head/sha — plus the prose "the check runs are bound to the head", which justifies the check-run equality. /review_comments/0/commit_id appears only among the decisive observations, meaning it must resolve; nothing is claimed about its value.

The validator had therefore invented an integrity invariant the evidence never provided. The test was green purely because these bytes happen to satisfy the stronger condition — a green test making a claim it had not earned.

readme_documents_every_asserted_pointer could not catch it: both pointers do appear in the README, and two pointers appearing separately says nothing about a relation between them. That guard checks pointer provenance, not relation provenance. fc6d4d9 removes the relation and records that limit.

The README and fixtures were not touched. Adjusting frozen evidence to justify the checker would have destroyed the ordering property that made the finding meaningful in the first place.

Checks implemented

  • the three expected fixture files exist;
  • each parses as JSON;
  • each exposes only the documented API surfaces — pull_request, reviews, review_comments, issue_comments, check_runs;
  • every JSON Pointer the README lists under Decisive observations resolves;
  • the one array the README documents as empty (falsification-in-comment.json → /review_comments) is empty;
  • the equality/inequality relations the README states as observed conditions hold:
stale-review-wrong-sha        /reviews/0/commit_id             != /pull_request/head/sha
                              /check_runs/check_runs/0/head_sha == /pull_request/head/sha
falsification-in-comment      /reviews/0/commit_id             == /pull_request/head/sha
conflicting-review-surfaces   /reviews/0/commit_id             == /pull_request/head/sha
                              /reviews/0/user/login            == /issue_comments/0/user/login
                              /review_comments/0/commit_id     == /pull_request/head/sha

All fail closed: a missing file, malformed JSON, an unresolved pointer, an extra top-level surface, a broken relation, and a documented-empty array gaining an element are violations, never skips.

Negative tests

Every check has a witness proving it can fire — otherwise it would be advertised coverage with no witness, per #147's fixture rules:

test mutation
missing_fixture_is_a_violation delete a fixture from the copy
malformed_json_is_a_violation overwrite one copy with non-JSON
unresolved_decisive_pointer_is_a_violation remove submitted_at from reviews[0]
undocumented_top_level_surface_is_a_violation insert a labels surface
violated_inequality_relation_is_a_violation set the stale review's commit_id to the head SHA
violated_equality_relation_is_a_violation change the issue comment's author
documented_empty_array_becoming_non_empty_is_a_violation push an element into /review_comments

Mutations are applied to temporary copies; the committed corpus is never written to.

The README boundary

The README stays authoritative for what the fixtures mean. The assertions mirror its decisive observations rather than scraping English prose — a Markdown parser would be brittle in precisely the way this project keeps finding expensive.

readme_documents_every_asserted_pointer guards the mirror in one direction only: it proves this file invents no pointer the README does not mention. It cannot prove the converse, and — as the finding above showed — it says nothing about invented relations between documented pointers.

Semantic-leakage self-review

question answer
mapping from fixture name/condition → closure outcome? no
classifier verdict vocabulary (PASS/FINDING/OWED/CANNOT_CHECK/STALE)? no — appears only in the doc comment that forbids it
assertion that a fixture "should be classified as X"? no
network or GitHub API dependency? no
any change to frozen fixture payloads or README? no — git diff origin/main -- tests/fixtures/github is empty

Local verification

cargo fmt --all --check                                       clean
cargo clippy --workspace --all-targets --locked -D warnings   clean
cargo test --test github_fixture_integrity --locked            9 passed, 0 failed

No new dependencies: serde_json is already a dependency and tempfile already a dev-dependency, so Cargo.toml and Cargo.lock are untouched.

Findings against the frozen corpus

None. Every integrity assertion the README documents holds. The one defect found in this work was in the validator, not the corpus, and is fixed by fc6d4d9.

Refs #147


🤖 Generated with Claude Code

https://claude.ai/code/session_013K3sUCBMGGjf5v4AqsW7Qk

First consumer of the Step 0B fixtures, introduced after they were frozen on
main (fixture commit eb88592, merged as b0cc644). Ordering is the point: with
the corpus already immutable, a failure here is a discovery about the frozen
corpus rather than licence to move fixture and checker toward each other until
they agree.

Integrity only. Checks:

  - the three expected fixture files exist;
  - each parses as JSON;
  - each exposes only the documented API surfaces (pull_request, reviews,
    review_comments, issue_comments, check_runs);
  - every JSON Pointer the README lists as a decisive observation resolves;
  - the one array the README documents as empty is empty;
  - the equality/inequality relations the README states as observed conditions
    actually hold.

Fails closed on every one of those: a missing file, malformed JSON, an
unresolved pointer, an extra surface, a broken relation and a
documented-empty array gaining an element are all violations, never skips.
Each has a negative test that mutates a temporary COPY, so every assertion is
shown able to fire; the committed corpus is never written to.

HARD BOUNDARY: no mapping from any fixture condition to a closure outcome. No
verdict vocabulary, no accept/reject notion, no network or GitHub API access. A
stale review SHA is recorded as "these two pointers differ" and nothing more.
Deciding what that means belongs to the classifier, which does not exist yet
and is deliberately not anticipated from inside a hygiene check.

The README stays authoritative; the assertions mirror it rather than scraping
English prose. readme_documents_every_asserted_pointer guards the mirror in one
direction only — it proves this file invents no pointer the README omits, and
cannot prove the converse.

Frozen Step 0B fixtures and README are unmodified.

Refs #147
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013K3sUCBMGGjf5v4AqsW7Qk
@coderabbitai

coderabbitai Bot commented Aug 16, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: QUIET

Plan: Pro Plus

Run ID: 5ddd581f-9cf4-46a2-b56b-7777a1ef6105

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Review finding against the validator, not against the corpus.

The stale-review-wrong-sha spec asserted

    Relation::Equal("/review_comments/0/commit_id", "/reviews/0/commit_id")

The frozen README states no such relation. It gives one observed condition for
that fixture — /reviews/0/commit_id != /pull_request/head/sha — plus the prose
"the check runs are bound to the head", which is what justifies the check-run
equality. /review_comments/0/commit_id appears only in the list of decisive
observations, i.e. it must RESOLVE; nothing is claimed about its value.

So the validator had quietly invented an integrity invariant the evidence never
provided. The test was green only because these particular bytes happen to
satisfy the stronger condition — a green test making a claim it had not earned,
which is the failure mode this whole ordering exists to expose.

readme_documents_every_asserted_pointer could not catch it: both pointers do
appear in the README, and two pointers appearing separately says nothing about a
relation between them. That guard checks pointer provenance, not relation
provenance, and this records its limit.

Requiring the pointer to resolve is the whole of what the README supports, so
the relation is removed and nothing replaces it. The README and fixtures are NOT
touched: adjusting frozen evidence to justify the checker would destroy the
ordering property that made this finding meaningful.

Refs #147
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013K3sUCBMGGjf5v4AqsW7Qk
@PhysShell
PhysShell merged commit a4ed489 into main Aug 16, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants