Repository navigation
Assert what a test's own setup guarantees, instead of skipping on it - #393
Merged
Merged
Conversation
#321's self-skips: tests that skipped when a condition their fixed pathway and window guarantee did not hold, which reports green having checked nothing. Each is now an assertion, shown failing where the old test skipped: - back-button and content-page-urls: the Molecule tab. With the tab renamed, the old tests skip; the new ones fail by name. The fixed pauses after choosing a tab are now waits on the URL. - hierarchy-scroll: that the tree scrolls, and that rows sit above the fold. With a 6000px-tall window the old tests skip; the new fail on 'the hierarchy scrolls'. - interactor-threshold: the entity offers interactions; the pathway opens zoomed out past the badge threshold (in a 4000x3000 window the old test skips, the new fails, 'Received: 0.91'); and the tree click leads elsewhere, now waited for instead of read after 8s. One stays a skip, because third-party servers decide it: whether a second interaction resource holds anything here. It now waits up to a minute for their counts instead of reading them once, which raced them. Refs #321 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review of #393: the comment said finding no second resource is a skip, not a failure. True against a live site; in CI the counts are replayed and an unlisted skip fails the shard. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #321 (self-skips that hide the behaviour under test).
Some tests skipped when a condition didn't hold that their own fixed pathway and window guarantee. A skip like that reports green having checked nothing. Each one is now an assertion. In CI's replay none of these ever skipped, and
expected-skips.jsonalready fails a new skip there, so this matters most for runs against a live site.Each was shown failing where the old test skipped:
back-button,content-page-urlshierarchy-scroll(×2)interactor-threshold, says so, and offers the way to itinteractor-threshold, raising it removes…interactor-threshold, comes back with the Back buttonThe fixed pauses after choosing a tab are now waits on the URL.
One skip stays, because a third party decides it: whether a second interaction resource holds anything for this pathway. The PSICQUIC servers answer in 6 to 17 s each, and the old test read their counts once, so it raced them. It now waits up to a minute for a count before it skips.
Tests: back-button and content-page-urls (6), hierarchy-scroll (2) and interactor-threshold (12) all pass.
🤖 Generated with Claude Code