Skip to content

docs(ops): close the 2026-08-16 triage rows with production evidence #30

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
harlan-zw merged 1 commit into from
Aug 16, 2026
Merged

docs(ops): close the 2026-08-16 triage rows with production evidence #30

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 43 additions & 4 deletions docs/ops/checkins/2026-08-16.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# 2026-08-16

**RED.** Repository sync could not finish any repository over 200 SKILL.md files from 2026-08-12 onward, dead-lettering the same six repositories every hour for 548 failures. This pass root-caused it and fixed three defects in the working tree, all undeployed.
**RED at probe time, resolved the same day.** Repository sync could not finish any repository over 200 SKILL.md files from 2026-08-12 onward, dead-lettering the same six repositories every hour for 548 failures. This pass root-caused it, fixed three defects, and shipped them in `23aecea`. See Resolution at the end for the production evidence.

Window: 2026-08-12T01:52:17Z to 2026-08-16T04:09:13Z (4.1 days). Archive: `docs/ops/checkins/2026-08-16.json`.

Expand Down Expand Up @@ -79,7 +79,7 @@ deliveries, a genuinely throwing job reached the DLQ before it reached its fifth
attempt, so `failed_jobs.exception` recorded the DLQ headline instead of the real
error.

Fixed in this pass, in three parts. All undeployed.
Fixed in this pass, in three parts. All shipped in `23aecea`.

**The delivery budget.** `max_retries` raised from 3 to 100 on
`skilld-repo-sync` and `skilld-repo-review-sync`, with the invariant written into
Expand Down Expand Up @@ -124,7 +124,7 @@ Fixed in this pass: the stale window is derived from the cron's actual period
(one full period plus half a period of slack, floored at 20 minutes). The two
shapes the list did cover land on the same 20 minutes and 36 hours as before.
Covered by `cronPeriodSeconds` unit cases and by a health-check case that fails
RED without the fix. Undeployed.
RED without the fix. Shipped in `23aecea`; the exit condition needs the next nightly run.

### 3. Rejected submissions are recorded as job failures

Expand All @@ -145,7 +145,7 @@ parameters, plus the `skill_parse_rejected:` prefix that carries a path.
`failedJobs24h` counts faults only; `rejectedJobs24h` carries the decisions and
appears in both rendered reports, so the number stays visible rather than
disappearing. `ctx.fail` stays as it is, because the submission UI reads its
reason back out of `failed_jobs`. Undeployed.
reason back out of `failed_jobs`. Shipped in `23aecea`; the exit condition needs the next nightly run.

### 4. Known, low volume

Expand Down Expand Up @@ -218,3 +218,42 @@ token inheritance, and a stable job id for `repository-index.ts` submission
tracking, and the failure has to propagate from a successor back to the tracked
root. The named `repo_too_large_to_index` rejection makes the remaining ceiling
visible in the meantime, which is what made the old one dangerous.

## Resolution

Merged as `23aecea` and deployed 2026-08-16. Verified against production rather
than against a green pipeline.

| Check | Before | After |
| --- | ---: | ---: |
| Repositories parked at `next_offset = 200` | 6 | 0 |
| Registry batch failures | 6 and 6 | 0 and 0 |
| Dead-letter rows | 596, climbing about 12 an hour | 598, frozen |
| Failed jobs in the trailing hour | about 12 | 0 |

Live queue consumers report `max_retries: 100` on both queues, read back from the
Cloudflare API rather than inferred from the deploy. `github/awesome-copilot`
settled at 430 indexed skills and `posthog/posthog` at 276, both past the wall
that had held since 2026-08-12; `awesome-copilot` had been serving 200 of its 425
skills on a public page for four days. The two largest repositories were seen
mid-chain at offsets the old ceiling made unreachable, `sickn33` at 2,250 of 6,341
and `jeremylongshore` at 1,500 of 5,598, then both completed and released their
progress rows. The newest dead-letter row predates the merge by over an hour.

The cleanest single number is the hourly batch: 6 failed jobs before the deploy,
0 after, on the same workload.

Two defects surfaced during delivery that this check-in had not caught. Required
CI typecheck failed on five `TS2345`s in `cronPeriodSeconds`, because
`noUncheckedIndexedAccess` makes a five-field destructure `string | undefined` and
the length guard does not narrow it; the local run had exited 0 through a pipe.
Required CI test then failed on `Hook timed out in 10000ms` across three arbitrary
files with zero assertion failures, and a different three locally on the same
commit. That is the flake class `vitest.config.ts` already documents: every worker
boots a full Nuxt environment, and capping the workers fixed the spawn failures
but left the per-file boot on vitest's 10-second default. `hookTimeout` raised to
30 seconds rather than re-running, which would only have moved the failure.

Still open from the actions above: the successor-job design (3), the
`x_ledger_stalled` investigation (5), the X projection decision (6), and the
usage token (7). Actions 1 and 2 are done. Action 4 needs the next nightly report.
Loading
Loading