diff --git a/.changeset/honest-lemons-travel.md b/.changeset/honest-lemons-travel.md new file mode 100644 index 00000000000..a845151cc84 --- /dev/null +++ b/.changeset/honest-lemons-travel.md @@ -0,0 +1,2 @@ +--- +--- diff --git a/.claude/skills/verify-clerk-expo/SKILL.md b/.claude/skills/verify-clerk-expo/SKILL.md index 5eb741c9064..e03293718b9 100644 --- a/.claude/skills/verify-clerk-expo/SKILL.md +++ b/.claude/skills/verify-clerk-expo/SKILL.md @@ -319,7 +319,7 @@ For a remote device, `down` ends the runner job and waits for it to finish, and - `src/host.ts`, `src/fixture.ts`, `src/native-build.ts`, and `src/freshness.ts` are this repository's own: the build of the test app, the native build that CI keeps between runs, the Metro ports, and the check that Metro serves current JS. `specs/app.ts` names the test app and its entry for a dev client, and `specs/native.ts` holds the per-platform locators for the native views and the locators of the home's links. Both are this repository's own too. - `npm test --prefix integration/expo-native` runs the CLI's unit tests, with no network, key, or device. `npm run typecheck --prefix integration/expo-native` runs `tsc`. The `Expo Native Runner Tests` job in `.github/workflows/ci.yml` runs both on Linux when a pull request changes the package, the test app, or a package the test app links. - `run --github-report` hands the results of the run to `@e2e-dev/github` as one report. The reporter writes the report to the job summary. With a `GITHUB_TOKEN` that may write pull request comments, it also posts one comment on the pull request and updates that comment on later runs. The reporter never changes the exit code, and nothing is reported for a run with a tainted file. -- `.github/workflows/verify-e2e.yml`, the `Expo E2E` workflow, runs `up --backend local`, `run --all --retries 1 --max-failures 5 --github-report`, and `down` on a runner for each platform, with a device and a Clerk application for each. It runs on the Version packages pull request (`changeset-release/main`), so each release is tested before it publishes, and the `Expo E2E` check fails when a spec fails. On other pull requests the device tests skip and the `Expo E2E` check passes. Start it by hand with `gh workflow run verify-e2e.yml --ref `. A test that fails and then passes on its one retry is `flaky` and does not fail the job. The runners are Blacksmith's `blacksmith-6vcpu-macos-26` and `blacksmith-8vcpu-ubuntu-2204`, and the repository variables `VERIFY_CI_RUNNER_IOS` and `VERIFY_CI_RUNNER_ANDROID` name other labels. The Platform API key comes from the `MOBILE_VERIFICATION_PLATFORM_API_KEY` repository secret. The workflow sets `VERIFY_LOCAL_BUILD=standalone` and `VERIFY_NATIVE_CACHE` ([freshness.md](references/freshness.md)). It uploads each run's `run.json`, app log, driver logs, video, screenshots, and e2e's `report.json`, `junit.xml`, summary, and failure pages for three days, and only when no secret is found in the run. `bin/configure-ios-simulators.sh` speeds up animations and turns off autocorrect and the keyboard tutorials on each simulator in `CLERK_TEST_DEVICES`, and this workflow and `.github/workflows/verify-remote.yml` call it. +- `.github/workflows/verify-e2e.yml`, the `Expo E2E` workflow, runs `up --backend local`, `run --all --retries 1 --max-failures 5 --github-report`, and `down` on a runner for each platform, with a device and a Clerk application for each. It runs on the Version packages pull request (`changeset-release/main`), so each release is tested before it publishes, and the `Expo E2E` check fails when a spec fails. On other pull requests the device tests skip and the `Expo E2E` check passes. Start it by hand with `gh workflow run verify-e2e.yml --ref `. A test that fails and then passes on its one retry is `flaky` and does not fail the job. The runners are Blacksmith's `blacksmith-6vcpu-macos-26` and `blacksmith-8vcpu-ubuntu-2204`, and the repository variables `VERIFY_CI_RUNNER_IOS` and `VERIFY_CI_RUNNER_ANDROID` name other labels. The Platform API key comes from the `MOBILE_VERIFICATION_PLATFORM_API_KEY` repository secret. The workflow sets `VERIFY_LOCAL_BUILD=standalone` and `VERIFY_NATIVE_CACHE` ([freshness.md](references/freshness.md)). It uploads each run's `run.json`, app log, driver logs, video, screenshots, and e2e's `report.json`, `junit.xml`, summary, and failure pages for three days, and only when no secret is found in the run. After a run on the release pull request, the `Post the flaky tests to Slack` job posts the tests that passed only on their retry to #team-mobile-alerts. `bin/configure-ios-simulators.sh` speeds up animations and turns off autocorrect and the keyboard tutorials on each simulator in `CLERK_TEST_DEVICES`, and this workflow and `.github/workflows/verify-remote.yml` call it. - The `Expo` workflow, `.github/workflows/expo-native-build.yml`, runs nothing on a device. It builds the test app as a Release app on Expo SDK 55 and 58 for each platform, and runs the Android unit tests of `@clerk/expo-biometrics`. - `.github/workflows/verify-attach.yml` publishes the evidence that `attach` hands to a session's runner, with `scripts/verify-attach.mjs`. It reads the manifest that `src/core/remote/handoff.ts` of `@clerk/internal-mobile-e2e` writes. It and the package's `src/core/publish.ts` each replace the block the other wrote, each applies the same rules to the comments and to the start of the line, and each lays the files out in the same tables. The script is checked out alone, so it imports nothing from the package. So a change to the manifest, the comments, those rules, or the tables is a change in both places. `scripts/verify-attach.test.mjs` has its tests. - A remote session runs in `.github/workflows/verify-remote.yml`. `src/platform/session-device.ts` holds the steps that the session runs on its device, and `src/core/remote/` of `@clerk/internal-mobile-e2e` is the code on both ends of it. The runner installs the package with `npm ci`. A session keeps the core it installed when it started, so after a change to the commit pin of `@clerk/internal-mobile-e2e`, commit, push, `down`, then `up`. diff --git a/.github/workflows/actions.lock b/.github/workflows/actions.lock index 30b77fea432..710a02f8d5d 100644 --- a/.github/workflows/actions.lock +++ b/.github/workflows/actions.lock @@ -98,10 +98,12 @@ workflows: '.github/workflows/verify-e2e.yml': - 'actions/cache@v4.3.0' - 'actions/checkout@v4.4.0' + - 'actions/download-artifact@v4.3.0' - 'actions/setup-java@v4.9.1' - 'actions/setup-node@v4.4.0' - 'actions/upload-artifact@v4.6.2' - 'pnpm/action-setup@v4.3.0' + - 'slackapi/slack-github-action@v1.27.1' '.github/workflows/verify-remote.yml': - 'actions/checkout@v4.4.0' - 'actions/setup-java@v4.9.1' diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 9e33e2f87dc..854be349382 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -379,9 +379,9 @@ jobs: env: NODE_VERSION: ${{ matrix.node-version }} - - name: Run verify-attach script tests + - name: Run verify script tests if: matrix.node-version == '24.15.0' - run: pnpm vitest run --root scripts verify-attach + run: pnpm vitest run --root scripts verify-attach verify-flaky - name: Upload Turbo Summary uses: actions/upload-artifact@v4.6.2 diff --git a/.github/workflows/verify-e2e.yml b/.github/workflows/verify-e2e.yml index d3cae0c03b9..81437d712db 100644 --- a/.github/workflows/verify-e2e.yml +++ b/.github/workflows/verify-e2e.yml @@ -221,6 +221,74 @@ jobs: retention-days: 3 if-no-files-found: warn + - name: Record the flaky tests + if: ${{ !cancelled() && steps.evidence.outputs.upload == 'true' }} + continue-on-error: true + run: node scripts/verify-flaky.mjs record "$PACKAGE/.verify/runs" "$RUNNER_TEMP/verify-flaky.json" + + - name: Keep the flaky tests for the Slack post + if: ${{ !cancelled() && steps.evidence.outputs.upload == 'true' }} + continue-on-error: true + uses: actions/upload-artifact@v4.6.2 + with: + name: verify-flaky-${{ matrix.platform }} + path: ${{ runner.temp }}/verify-flaky.json + retention-days: 1 + overwrite: true + if-no-files-found: ignore + + report-flaky: + name: Post the flaky tests to Slack + needs: e2e-tests + if: ${{ !cancelled() && needs.e2e-tests.result != 'skipped' && github.head_ref == 'changeset-release/main' }} + runs-on: ubuntu-latest + timeout-minutes: 5 + permissions: + contents: read + steps: + - name: Checkout repo + uses: actions/checkout@v4.4.0 + with: + persist-credentials: false + sparse-checkout: scripts/verify-flaky.mjs + sparse-checkout-cone-mode: false + + - uses: actions/setup-node@v4.4.0 + with: + node-version: 24.15.0 + + - name: Download the flaky tests of both platforms + continue-on-error: true + uses: actions/download-artifact@v4.3.0 + with: + pattern: verify-flaky-* + path: ${{ runner.temp }}/flaky + + - name: Generate notification payload + id: notification + run: | + mkdir -p "$RUNNER_TEMP/flaky" + payload="$(node scripts/verify-flaky.mjs payload "$RUNNER_TEMP/flaky")" + if [ -z "$payload" ]; then + echo "No flaky tests in this run." + exit 0 + fi + echo "$payload" + { + echo 'payload<<__NOTIFY_EOF__' + echo "$payload" + echo '__NOTIFY_EOF__' + } >> "$GITHUB_OUTPUT" + + - name: Send the flaky tests to Slack + if: ${{ steps.notification.outputs.payload != '' }} + uses: slackapi/slack-github-action@v1.27.1 + with: + payload: ${{ steps.notification.outputs.payload }} + env: + SLACK_WEBHOOK_URL: ${{ secrets.SLACK_MOBILE_TEAM_ALERTS_WEBHOOK_URL }} + SLACK_WEBHOOK_TYPE: INCOMING_WEBHOOK + verify-e2e: name: Expo E2E needs: e2e-tests diff --git a/scripts/verify-flaky.mjs b/scripts/verify-flaky.mjs new file mode 100644 index 00000000000..55322c6d727 --- /dev/null +++ b/scripts/verify-flaky.mjs @@ -0,0 +1,133 @@ +import fs from 'node:fs'; +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; + +const ERROR_CHARS = 300; + +function flakesFromReport(report) { + return report.run.results + .filter(result => result.status === 'flaky') + .map(result => ({ + platform: result.platform, + file: result.file, + title: result.titlePath.join(' › '), + error: result.attempts.find(attempt => attempt.status === 'failed')?.error?.message ?? '', + })); +} + +function normalizeError(message) { + const normalized = message + .replace(/[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}/g, '') + .replace(/\b(user|sess|ins|app|org)_[A-Za-z0-9]+/g, '$1_') + .replace(/after launch [0-9a-f]+/g, 'after launch ') + .replace(/; passes: \d+;.*$/s, '') + .replace(/\s+/g, ' ') + .trim(); + return normalized.length > ERROR_CHARS ? `${normalized.slice(0, ERROR_CHARS)}…` : normalized; +} + +const slackText = text => text.replaceAll('&', '&').replaceAll('<', '<').replaceAll('>', '>'); + +const mostCommon = counts => [...counts.entries()].sort((a, b) => b[1] - a[1])[0][0]; + +function flakyPayload(records) { + const tests = new Map(); + for (const record of records) { + for (const flake of record.flakes) { + const key = `${flake.file}\n${flake.title}`; + const test = tests.get(key) ?? { file: flake.file, title: flake.title, platforms: new Set(), errors: new Map() }; + const error = normalizeError(flake.error); + test.platforms.add(flake.platform); + test.errors.set(error, (test.errors.get(error) ?? 0) + 1); + tests.set(key, test); + } + } + if (tests.size === 0) { + return null; + } + + const markdown = text => ({ type: 'section', text: { type: 'mrkdwn', text } }); + const runUrl = records.find(record => record.runUrl)?.runUrl; + const count = tests.size === 1 ? '1 test' : `${tests.size} tests`; + const header = [ + '*:warning: Expo E2E flaky tests*', + '*Pull request:* Version packages', + `*Flaky:* ${count} passed only on a retry`, + ...(runUrl ? [`*Run:* <${runUrl}|View logs>`] : []), + ]; + const sorted = [...tests.values()].sort((a, b) => a.file.localeCompare(b.file) || a.title.localeCompare(b.title)); + const blocks = [markdown(header.join('\n'))]; + for (const test of sorted) { + const lines = [ + `\`${test.file.replace(/^specs\/features\//, '')}\` ${slackText(test.title)}`, + `*Platforms:* ${[...test.platforms].sort().join(', ')}`, + ]; + const error = mostCommon(test.errors); + if (error !== '') { + lines.push(`*First attempt failed:* ${slackText(error)}`); + } + blocks.push(markdown(lines.join('\n'))); + } + return { blocks, unfurl_links: false, unfurl_media: false }; +} + +const readJson = file => JSON.parse(fs.readFileSync(file, 'utf8')); + +const byDirectory = (a, b) => path.dirname(a).localeCompare(path.dirname(b), 'en', { numeric: true }); + +function recordStep(runsDir, outFile, env) { + const reports = fs.existsSync(runsDir) + ? fs + .globSync('*/e2e*/report.json', { cwd: runsDir }) + .sort(byDirectory) + .map(file => readJson(path.join(runsDir, file))) + : []; + const runUrl = + env.GITHUB_SERVER_URL && env.GITHUB_REPOSITORY && env.GITHUB_RUN_ID + ? `${env.GITHUB_SERVER_URL}/${env.GITHUB_REPOSITORY}/actions/runs/${env.GITHUB_RUN_ID}` + : null; + const record = { + runUrl, + commit: env.GITHUB_SHA || (reports[0]?.run.vcs?.commit ?? null), + flakes: reports.flatMap(flakesFromReport), + }; + fs.writeFileSync(outFile, `${JSON.stringify(record, null, 2)}\n`); + return record; +} + +function readRecords(dir) { + return fs + .globSync('**/*.json', { cwd: dir }) + .sort(byDirectory) + .map(file => { + const record = readJson(path.join(dir, file)); + if (!Array.isArray(record?.flakes)) { + throw new Error(`${file} is not a record of flaky tests`); + } + return record; + }); +} + +function main(argv, env, log = console.log) { + const [step, ...rest] = argv; + if (step === 'record' && rest.length === 2) { + const record = recordStep(rest[0], rest[1], env); + log(`Recorded ${record.flakes.length} flaky tests in ${rest[1]}.`); + return 0; + } + if (step === 'payload' && rest.length === 1) { + const payload = flakyPayload(readRecords(rest[0])); + if (payload !== null) { + log(JSON.stringify(payload)); + } + return 0; + } + log('usage: verify-flaky.mjs record | payload '); + return 2; +} + +export { flakesFromReport, flakyPayload, main, normalizeError }; + +if (process.argv[1] !== undefined && fs.realpathSync(process.argv[1]) === fileURLToPath(import.meta.url)) { + process.exit(main(process.argv.slice(2), process.env)); +} diff --git a/scripts/verify-flaky.test.mjs b/scripts/verify-flaky.test.mjs new file mode 100644 index 00000000000..373b1ed95ab --- /dev/null +++ b/scripts/verify-flaky.test.mjs @@ -0,0 +1,216 @@ +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; + +import { describe, expect, it } from 'vitest'; + +import { flakesFromReport, flakyPayload, main, normalizeError } from './verify-flaky.mjs'; + +const signedInError = (name, user, launch, session, passes = 2) => + `the app did not show "Signed in as ${name}+clerk_test@example.com" with user ID ${user} and a session ID on the home after launch ${launch}; it shows its error screen: Something went wrong No session was found with id ${session}; passes: ${passes}; reads: ${passes * 2}, the slowest ${passes * 70}ms (the heading)`; + +const result = (status, attempts, change = {}) => ({ + status, + platform: 'android', + file: 'specs/features/native-js-sync/sign-out-from-native.e2e.ts', + titlePath: ['signing out in the native profile signs out the JS hooks'], + attempts, + ...change, +}); + +const attempt = (status, message) => ({ status, error: message === undefined ? null : { message } }); + +const report = results => ({ run: { vcs: { commit: 'a'.repeat(40) }, results } }); + +const flake = (change = {}) => ({ + platform: 'android', + file: 'specs/features/native-js-sync/sign-out-from-native.e2e.ts', + title: 'signing out in the native profile signs out the JS hooks', + error: 'No session was found with id sess_3L6DwJjKc2IDyokGGeluncGnY69', + ...change, +}); + +const runUrl = id => `https://github.com/clerk/javascript/actions/runs/${id}`; + +describe('flakesFromReport', () => { + it('keeps only the flaky results, with the error of the first failed attempt', () => { + const flakes = flakesFromReport( + report([ + result('passed', [attempt('passed')]), + result('failed', [attempt('failed', 'failed twice'), attempt('failed', 'failed twice')]), + result('skipped', []), + result( + 'flaky', + [attempt('failed', 'the first failure'), attempt('failed', 'the second failure'), attempt('passed')], + { platform: 'ios', titlePath: ['profile', 'signs out'] }, + ), + ]), + ); + + expect(flakes).toEqual([ + { + platform: 'ios', + file: 'specs/features/native-js-sync/sign-out-from-native.e2e.ts', + title: 'profile › signs out', + error: 'the first failure', + }, + ]); + }); +}); + +describe('normalizeError', () => { + it('gives the same text for the same failure in different runs', () => { + expect( + normalizeError( + signedInError( + 'ben.a30f', + 'user_3L6FBztORCO03ZuOC5J50LJzABj', + '2bbc58c2e52b4448', + 'sess_3L6FCP4vbLeA13m6zfP70AEVLx7', + ), + ), + ).toBe( + normalizeError( + signedInError( + 'eli.b71c', + 'user_3L6FGxY6BKLgtHLELbgYi9pPd79', + '0ebe1bf6eff0cba0', + 'sess_3L6FHLmSGr9hI2LbSBm2E4BHlHl', + 3, + ), + ), + ); + expect( + normalizeError( + signedInError( + 'ben.a30f', + 'user_3L6FBztORCO03ZuOC5J50LJzABj', + '2bbc58c2e52b4448', + 'sess_3L6FCP4vbLeA13m6zfP70AEVLx7', + ), + ), + ).toBe( + 'the app did not show "Signed in as " with user ID user_ and a session ID on the home after launch ; it shows its error screen: Something went wrong No session was found with id sess_', + ); + expect(normalizeError('No session was found with id sess_3L6DwJjKc2IDyokGGeluncGnY69')).toBe( + normalizeError('No session was found\nwith id sess_3L6FHLmSGr9hI2LbSBm2E4BHlHl'), + ); + expect(normalizeError('ins_2a app_2b org_2c')).toBe('ins_ app_ org_'); + }); + + it('cuts a long message', () => { + expect(normalizeError('x'.repeat(500))).toBe(`${'x'.repeat(300)}…`); + }); +}); + +const section = text => ({ type: 'section', text: { type: 'mrkdwn', text } }); + +describe('flakyPayload', () => { + it('is null when no test was flaky', () => { + expect(flakyPayload([])).toBeNull(); + expect(flakyPayload([{ runUrl: runUrl(1), commit: null, flakes: [] }])).toBeNull(); + }); + + it('gives each flaky test its own block with its platforms and first error, after a header that links the run', () => { + const profile = flake({ + file: 'specs/features/user-button-and-profile/profile.e2e.ts', + title: 'the home sign-out button ends the session', + error: 'a & c', + }); + const payload = flakyPayload([ + { runUrl: runUrl(1), commit: null, flakes: [profile, flake()] }, + { runUrl: runUrl(1), commit: null, flakes: [flake({ platform: 'ios' })] }, + ]); + + expect(payload).toEqual({ + blocks: [ + section( + [ + '*:warning: Expo E2E flaky tests*', + '*Pull request:* Version packages', + '*Flaky:* 2 tests passed only on a retry', + `*Run:* <${runUrl(1)}|View logs>`, + ].join('\n'), + ), + section( + [ + '`native-js-sync/sign-out-from-native.e2e.ts` signing out in the native profile signs out the JS hooks', + '*Platforms:* android, ios', + '*First attempt failed:* No session was found with id sess_<id>', + ].join('\n'), + ), + section( + [ + '`user-button-and-profile/profile.e2e.ts` the home sign-out button ends the session', + '*Platforms:* android', + '*First attempt failed:* a <b> & c', + ].join('\n'), + ), + ], + unfurl_links: false, + unfurl_media: false, + }); + }); + + it('names one flaky test in the singular and leaves out the run without a link', () => { + expect(flakyPayload([{ runUrl: null, commit: null, flakes: [flake()] }]).blocks[0]).toEqual( + section( + [ + '*:warning: Expo E2E flaky tests*', + '*Pull request:* Version packages', + '*Flaky:* 1 test passed only on a retry', + ].join('\n'), + ), + ); + }); + + it('keeps every block under the Slack limit of 3000 characters', () => { + const payload = flakyPayload([{ runUrl: runUrl(1), commit: null, flakes: [flake({ error: 'x'.repeat(5000) })] }]); + expect(Math.max(...payload.blocks.map(block => block.text.text.length))).toBeLessThan(3000); + }); +}); + +describe('the command line', () => { + it('records the flaky tests of every e2e report, then writes the Slack payload', () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'verify-flaky-')); + const runs = path.join(dir, 'runs'); + const write = (file, value) => { + fs.mkdirSync(path.dirname(file), { recursive: true }); + fs.writeFileSync(file, JSON.stringify(value)); + }; + write( + path.join(runs, 'r20261023-143803-a30f', 'e2e', 'report.json'), + report([result('flaky', [attempt('failed', 'first'), attempt('passed')]), result('passed', [])]), + ); + write( + path.join(runs, 'r20261023-143803-a30f', 'e2e-2', 'report.json'), + report([result('flaky', [attempt('failed', 'second'), attempt('passed')], { platform: 'ios' })]), + ); + write(path.join(runs, 'r20261023-150000-b00c', 'e2e', 'report.json'), report([])); + const env = { + GITHUB_SERVER_URL: 'https://github.com', + GITHUB_REPOSITORY: 'clerk/javascript', + GITHUB_RUN_ID: '42', + GITHUB_SHA: 'b'.repeat(40), + }; + const out = path.join(dir, 'records', '42', 'verify-flaky-android', 'verify-flaky.json'); + fs.mkdirSync(path.dirname(out), { recursive: true }); + const log = []; + + expect(main(['record', runs, out], env, line => log.push(line))).toBe(0); + expect(JSON.parse(fs.readFileSync(out, 'utf8'))).toEqual({ + runUrl: runUrl(42), + commit: 'b'.repeat(40), + flakes: [flake({ error: 'first' }), flake({ platform: 'ios', error: 'second' })], + }); + + const empty = path.join(dir, 'empty.json'); + expect(main(['record', path.join(dir, 'no-runs'), empty], {}, () => {})).toBe(0); + expect(JSON.parse(fs.readFileSync(empty, 'utf8'))).toEqual({ runUrl: null, commit: null, flakes: [] }); + + const printed = []; + expect(main(['payload', path.join(dir, 'records')], {}, line => printed.push(line))).toBe(0); + expect(JSON.parse(printed.join('\n')).blocks[1].text.text.split('\n')[1]).toBe('*Platforms:* android, ios'); + expect(main(['payload'], {}, () => {})).toBe(2); + }); +});