Skip to content

ci: run one UI test class ten times per OS on request - #156

Open
BartvHelvert wants to merge 1 commit into
masterfrom
ci/repeat-one-ui-test
Open

BartvHelvert wants to merge 1 commit into
masterfrom
ci/repeat-one-ui-test

Conversation

@BartvHelvert

Copy link
Copy Markdown
Owner

What this changes

workflow_dispatch takes the UI test class to repeat, as a --tests filter. Given one, the run
is twenty UI jobs — ten per OS — and nothing else, and it keeps every run's IDE logs, the passing
ones too, so a failure can be read against the runs either side of it. Given nothing, which is
every push and pull request, the workflow is exactly what it was.

Reading a flake takes repeats, and until now each one cost a whole matrix. The Escape
investigation (#113) did this twice: ten throwaway ci-flake/escape-* branches, ten dispatches,
and each run carried the unit tests, the plugin verifier and the other UI shard along with it —
for one test class. Twenty full matrices to get five failures.

Where it comes from

#149 and #150 each grew their own copy of this, differing only in the test class written into the
job name and the Gradle arguments:

-        description: Run the full snippet test class ten times per OS
+        description: Run the full runtime test class ten times per OS
-            test_args=(--tests '*SplitModeSnippetsTest')
+            test_args=(--tests '*SplitModeRuntimeTest')

They would have conflicted had both landed, and a third investigation would have needed a third
copy. This is that mechanism, extracted once, with the class as an input.

Two changes from those copies:

  • The input is the filter, not a boolean. An empty string is falsy in GitHub expressions, so
    inputs.repeat_ui_test && … || … keeps working unchanged while the value also serves as the
    --tests argument. One input instead of two, and no value can select a class the dispatcher
    did not name.
  • The filter reaches the script through env:, rather than an expression spliced into it. It
    is text whoever dispatched the run chose, and it should not be able to become part of the
    command.

How it was tested

  • actionlint passes. Its one remaining complaint, SC2064 on trap "kill $!" EXIT, is
    pre-existing on master — and expanding $! at trap time is what that line wants.
  • Not dispatched yet: workflow_dispatch inputs only take effect once the workflow file is on the
    default branch, so this cannot be exercised from a PR branch. The expressions are the ones
    already proven by test(completion): trace a missing snippet selection #149's and test(debugger): capture startup subscription failures #150's batches
    (42728115535,
    42728125991), which ran
    twenty jobs each and uploaded per-repeat artifacts; what differs here is the input's type and
    how it reaches the shell.
  • The unchanged path — no input — is the same matrix, the same arguments, and the same artifact
    names as before.

Checklist

  • Nothing is special-cased for one extension.
  • A new feature has a UI test. CI only.
  • No internal, override-only, obsolete or deprecated platform API.
  • docs/ is updated where behaviour changed, docs/status.md especially. No product change.
  • Third-party material keeps its notice and is listed in NOTICE.md. None added.

Reading a flake takes repeats, and until now each one cost a whole
matrix: ten throwaway branches, ten dispatches, and the unit tests,
the plugin verifier and the other shard run along with it, for one
class. The Escape investigation did that twice.

`workflow_dispatch` now takes the class to repeat, as a `--tests`
filter. Given one, the run is twenty UI jobs -- ten per OS -- and
nothing else, and it keeps every run's IDE logs, the passing ones too,
so a failure can be read against the runs either side of it. Given
nothing, which is every push and pull request, the workflow is what it
was.

The filter reaches the script through the environment rather than an
expression in it: it is text whoever dispatched the run chose.

Taken from the two investigations that each grew their own copy with
the class written into it (#146, #147), so neither has to.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant