Repository navigation
Validate frozen GitHub fixture integrity (issue #147) - #150
Merged
Merged
Conversation
First consumer of the Step 0B fixtures, introduced after they were frozen on main (fixture commit eb88592, merged as b0cc644). Ordering is the point: with the corpus already immutable, a failure here is a discovery about the frozen corpus rather than licence to move fixture and checker toward each other until they agree. Integrity only. Checks: - the three expected fixture files exist; - each parses as JSON; - each exposes only the documented API surfaces (pull_request, reviews, review_comments, issue_comments, check_runs); - every JSON Pointer the README lists as a decisive observation resolves; - the one array the README documents as empty is empty; - the equality/inequality relations the README states as observed conditions actually hold. Fails closed on every one of those: a missing file, malformed JSON, an unresolved pointer, an extra surface, a broken relation and a documented-empty array gaining an element are all violations, never skips. Each has a negative test that mutates a temporary COPY, so every assertion is shown able to fire; the committed corpus is never written to. HARD BOUNDARY: no mapping from any fixture condition to a closure outcome. No verdict vocabulary, no accept/reject notion, no network or GitHub API access. A stale review SHA is recorded as "these two pointers differ" and nothing more. Deciding what that means belongs to the classifier, which does not exist yet and is deliberately not anticipated from inside a hygiene check. The README stays authoritative; the assertions mirror it rather than scraping English prose. readme_documents_every_asserted_pointer guards the mirror in one direction only — it proves this file invents no pointer the README omits, and cannot prove the converse. Frozen Step 0B fixtures and README are unmodified. Refs #147 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013K3sUCBMGGjf5v4AqsW7Qk
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: QUIET Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Review finding against the validator, not against the corpus.
The stale-review-wrong-sha spec asserted
Relation::Equal("/review_comments/0/commit_id", "/reviews/0/commit_id")
The frozen README states no such relation. It gives one observed condition for
that fixture — /reviews/0/commit_id != /pull_request/head/sha — plus the prose
"the check runs are bound to the head", which is what justifies the check-run
equality. /review_comments/0/commit_id appears only in the list of decisive
observations, i.e. it must RESOLVE; nothing is claimed about its value.
So the validator had quietly invented an integrity invariant the evidence never
provided. The test was green only because these particular bytes happen to
satisfy the stronger condition — a green test making a claim it had not earned,
which is the failure mode this whole ordering exists to expose.
readme_documents_every_asserted_pointer could not catch it: both pointers do
appear in the README, and two pointers appearing separately says nothing about a
relation between them. That guard checks pointer provenance, not relation
provenance, and this records its limit.
Requiring the pointer to resolve is the whole of what the README supports, so
the relation is removed and nothing replaces it. The README and fixtures are NOT
touched: adjusting frozen evidence to justify the checker would destroy the
ordering property that made this finding meaningful.
Refs #147
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013K3sUCBMGGjf5v4AqsW7Qk
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is the first consumer of the Step 0B fixtures after they were frozen on main. It validates fixture integrity only. It does not map any observed fixture condition to a closure outcome.
Two commits, one file:
tests/github_fixture_integrity.rs. No classifier code, and none to be added to this branch.Why the ordering matters
With the corpus already immutable, a failure here is a discovery about the frozen corpus, not licence to move fixture and checker toward each other until they agree. That property only exists because 0B merged with no consumer, so this boundary should not later be squashed into the classifier commit.
A defect found in review, and what it teaches
89057aaasserted/review_comments/0/commit_id == /reviews/0/commit_idfor the stale specimen. The frozen README states no such relation. It gives one observed condition —/reviews/0/commit_id != /pull_request/head/sha— plus the prose "the check runs are bound to the head", which justifies the check-run equality./review_comments/0/commit_idappears only among the decisive observations, meaning it must resolve; nothing is claimed about its value.The validator had therefore invented an integrity invariant the evidence never provided. The test was green purely because these bytes happen to satisfy the stronger condition — a green test making a claim it had not earned.
readme_documents_every_asserted_pointercould not catch it: both pointers do appear in the README, and two pointers appearing separately says nothing about a relation between them. That guard checks pointer provenance, not relation provenance.fc6d4d9removes the relation and records that limit.The README and fixtures were not touched. Adjusting frozen evidence to justify the checker would have destroyed the ordering property that made the finding meaningful in the first place.
Checks implemented
pull_request,reviews,review_comments,issue_comments,check_runs;falsification-in-comment.json→/review_comments) is empty;All fail closed: a missing file, malformed JSON, an unresolved pointer, an extra top-level surface, a broken relation, and a documented-empty array gaining an element are violations, never skips.
Negative tests
Every check has a witness proving it can fire — otherwise it would be advertised coverage with no witness, per #147's fixture rules:
missing_fixture_is_a_violationmalformed_json_is_a_violationunresolved_decisive_pointer_is_a_violationsubmitted_atfromreviews[0]undocumented_top_level_surface_is_a_violationlabelssurfaceviolated_inequality_relation_is_a_violationcommit_idto the head SHAviolated_equality_relation_is_a_violationdocumented_empty_array_becoming_non_empty_is_a_violation/review_commentsMutations are applied to temporary copies; the committed corpus is never written to.
The README boundary
The README stays authoritative for what the fixtures mean. The assertions mirror its decisive observations rather than scraping English prose — a Markdown parser would be brittle in precisely the way this project keeps finding expensive.
readme_documents_every_asserted_pointerguards the mirror in one direction only: it proves this file invents no pointer the README does not mention. It cannot prove the converse, and — as the finding above showed — it says nothing about invented relations between documented pointers.Semantic-leakage self-review
PASS/FINDING/OWED/CANNOT_CHECK/STALE)?git diff origin/main -- tests/fixtures/githubis emptyLocal verification
No new dependencies:
serde_jsonis already a dependency andtempfilealready a dev-dependency, soCargo.tomlandCargo.lockare untouched.Findings against the frozen corpus
None. Every integrity assertion the README documents holds. The one defect found in this work was in the validator, not the corpus, and is fixed by
fc6d4d9.Refs #147
🤖 Generated with Claude Code
https://claude.ai/code/session_013K3sUCBMGGjf5v4AqsW7Qk