diff --git a/docs/ops/checkins/2026-08-16.md b/docs/ops/checkins/2026-08-16.md index 7262ce57..74d06449 100644 --- a/docs/ops/checkins/2026-08-16.md +++ b/docs/ops/checkins/2026-08-16.md @@ -1,6 +1,6 @@ # 2026-08-16 -**RED.** Repository sync could not finish any repository over 200 SKILL.md files from 2026-08-12 onward, dead-lettering the same six repositories every hour for 548 failures. This pass root-caused it and fixed three defects in the working tree, all undeployed. +**RED at probe time, resolved the same day.** Repository sync could not finish any repository over 200 SKILL.md files from 2026-08-12 onward, dead-lettering the same six repositories every hour for 548 failures. This pass root-caused it, fixed three defects, and shipped them in `23aecea`. See Resolution at the end for the production evidence. Window: 2026-08-12T01:52:17Z to 2026-08-16T04:09:13Z (4.1 days). Archive: `docs/ops/checkins/2026-08-16.json`. @@ -79,7 +79,7 @@ deliveries, a genuinely throwing job reached the DLQ before it reached its fifth attempt, so `failed_jobs.exception` recorded the DLQ headline instead of the real error. -Fixed in this pass, in three parts. All undeployed. +Fixed in this pass, in three parts. All shipped in `23aecea`. **The delivery budget.** `max_retries` raised from 3 to 100 on `skilld-repo-sync` and `skilld-repo-review-sync`, with the invariant written into @@ -124,7 +124,7 @@ Fixed in this pass: the stale window is derived from the cron's actual period (one full period plus half a period of slack, floored at 20 minutes). The two shapes the list did cover land on the same 20 minutes and 36 hours as before. Covered by `cronPeriodSeconds` unit cases and by a health-check case that fails -RED without the fix. Undeployed. +RED without the fix. Shipped in `23aecea`; the exit condition needs the next nightly run. ### 3. Rejected submissions are recorded as job failures @@ -145,7 +145,7 @@ parameters, plus the `skill_parse_rejected:` prefix that carries a path. `failedJobs24h` counts faults only; `rejectedJobs24h` carries the decisions and appears in both rendered reports, so the number stays visible rather than disappearing. `ctx.fail` stays as it is, because the submission UI reads its -reason back out of `failed_jobs`. Undeployed. +reason back out of `failed_jobs`. Shipped in `23aecea`; the exit condition needs the next nightly run. ### 4. Known, low volume @@ -218,3 +218,42 @@ token inheritance, and a stable job id for `repository-index.ts` submission tracking, and the failure has to propagate from a successor back to the tracked root. The named `repo_too_large_to_index` rejection makes the remaining ceiling visible in the meantime, which is what made the old one dangerous. + +## Resolution + +Merged as `23aecea` and deployed 2026-08-16. Verified against production rather +than against a green pipeline. + +| Check | Before | After | +| --- | ---: | ---: | +| Repositories parked at `next_offset = 200` | 6 | 0 | +| Registry batch failures | 6 and 6 | 0 and 0 | +| Dead-letter rows | 596, climbing about 12 an hour | 598, frozen | +| Failed jobs in the trailing hour | about 12 | 0 | + +Live queue consumers report `max_retries: 100` on both queues, read back from the +Cloudflare API rather than inferred from the deploy. `github/awesome-copilot` +settled at 430 indexed skills and `posthog/posthog` at 276, both past the wall +that had held since 2026-08-12; `awesome-copilot` had been serving 200 of its 425 +skills on a public page for four days. The two largest repositories were seen +mid-chain at offsets the old ceiling made unreachable, `sickn33` at 2,250 of 6,341 +and `jeremylongshore` at 1,500 of 5,598, then both completed and released their +progress rows. The newest dead-letter row predates the merge by over an hour. + +The cleanest single number is the hourly batch: 6 failed jobs before the deploy, +0 after, on the same workload. + +Two defects surfaced during delivery that this check-in had not caught. Required +CI typecheck failed on five `TS2345`s in `cronPeriodSeconds`, because +`noUncheckedIndexedAccess` makes a five-field destructure `string | undefined` and +the length guard does not narrow it; the local run had exited 0 through a pipe. +Required CI test then failed on `Hook timed out in 10000ms` across three arbitrary +files with zero assertion failures, and a different three locally on the same +commit. That is the flake class `vitest.config.ts` already documents: every worker +boots a full Nuxt environment, and capping the workers fixed the spawn failures +but left the per-file boot on vitest's 10-second default. `hookTimeout` raised to +30 seconds rather than re-running, which would only have moved the failure. + +Still open from the actions above: the successor-job design (3), the +`x_ledger_stalled` investigation (5), the X projection decision (6), and the +usage token (7). Actions 1 and 2 are done. Action 4 needs the next nightly report. diff --git a/docs/ops/triage-ledger.md b/docs/ops/triage-ledger.md index 7620cfc4..cc598887 100644 --- a/docs/ops/triage-ledger.md +++ b/docs/ops/triage-ledger.md @@ -43,7 +43,7 @@ Persistent fingerprints found by the daily check-in. Add rows only after evidenc | `Cannot find module './scripts/lib/static-asset-fallthrough'` | GitHub Actions Test + Deploy | resolved 2026-08-06 | 2026-08-05 | 0 new | `nuxt.config.ts:4` imports `./scripts/lib/static-asset-fallthrough`, and `git ls-files scripts/lib/static-asset-fallthrough.ts` returns nothing, so the module exists only in the working tree. CI dies in `postinstall` before any test runs, on `d1c156c` (13:54:09Z), `33d7254` (13:59:50Z) and `fda2756` ([run 30918582393](https://github.com/skilld-dev/skilld.dev/actions/runs/30918582393), 14:22:14Z). `Deploy to Cloudflare` is gated on `Test` and records `skipped` on all three, so production has been frozen at `e885251` (version `ef31aea8`, 10:24:14Z) while HEAD moved three commits ahead. This is a gate failure and a deploy freeze from one uncommitted file. Exit condition: the file is committed and both workflows pass on the resulting SHA. Closed 2026-08-06: `512e96d` committed the helper, `Test` ([run 30977361385](https://github.com/skilld-dev/skilld.dev/actions/runs/30977361385)) and `Deploy to Cloudflare` ([run 30977528760](https://github.com/skilld-dev/skilld.dev/actions/runs/30977528760)) both pass on `5fd9dbc`, and production version `bb598b64` reports `approxDeployedSha` equal to HEAD, so the freeze is over and the 18-commit backlog shipped. | | `SKILLD-F` `no such table: activity` | GET /api/feed/recent-updates | resolved 2026-08-06 | 2026-08-05 | 27 events, 2 users | [Permalink](https://harlan-zw.sentry.io/issues/7650905358/), events 2026-08-04T02:24:04Z to 07:44:39Z, `environment = production`. `server/api/feed/recent-updates.get.ts:77` reads `FROM activity a`. Production still holds the table: `d1.tables` contains `activity`, `d1.missingExpectedTables` is empty, and `migrations/0007_activity.sql` created it. All 27 events are SSR-internal calls from one page render across four handlers that read `activity`: `/api/feed/recent-publishes` (20), `/api/feed/recent-updates` (3), `/api/collections/featured` (2), `/api/clusters` (1), in two clusters (02:24:04Z to 02:50:31Z, then 07:44:39Z). Four explanations are ruled out. Not a preview database: `wrangler.jsonc:15` declares one `DB` binding to `skilld-db` with no `preview_database_id`. Not local development: contexts report `runtime: cloudflare`, `cloud.provider: cloudflare`, SDK `sentry.javascript.cloudflare`. Not deploy churn: version `fb18dac9` was live across the whole window, with the next deploy at 10:24Z. Not read replication: no `withSession` or `read_replication` anywhere in `server`, `shared`, `layers` or `wrangler.jsonc`. Also not a different database (`wrangler d1 list` returns exactly one skilld database) and not a table rebuild (migrations `0087`, `0088` and `0089` were applied to production during the window at 01:15:28, 02:53:44 and 04:43:02, none referencing `activity`). The failing request is `HEAD http://skilld.dev/`; nothing in `server`, `shared` or `layers` issues a HEAD request, so this is external crawler traffic, not the `SELF` self-fetch, whose only caller is the 22:00Z front-door probe. One unexplained correlation, possibly coincidence: the first cluster ends three minutes before migration `0088` was applied; the second burst correlates with nothing. Cause not established: a whole-binding failure should also have broken the `skills` and `repos` reads on the same page render and did not. Exit condition: D1 observability attributes the fault, or no further events appear. Triage cost is itself fixed: every event carried `release: None`, so "which version served this" could not be answered from the event and was reconstructed from deployment history instead. `shared/sentry.ts` now resolves a release from `GITHUB_SHA`, the runtime reports it, and the bundler plugin is pinned to the same name, which also lets uploaded sourcemaps match an event for the first time. Corrections: the first pass claimed `/api/clusters` was absent from HEAD on the strength of a shell glob that missed `layers/registry/server/api/clusters/` being a directory. All four routes exist; the route inventory carried no signal at all. The second pass then read the `HEAD http://skilld.dev/` request as the `SELF` self-fetch on shape alone, without checking whether anything in the codebase issues a HEAD request. Nothing does. Closed 2026-08-06: the `activity` table is present in the archived table list, Sentry reports the issue `resolved` with no event since 2026-08-04T07:44:39Z, and the issue carries no release, so every event predates the current version. | | D1 overload burst on `skill-related` and `orgs` reads | Sentry `SKILLD-G/H/J/K/M/N/P/Q` | watch | 2026-08-05 | 686 post-fix events | Eight new issues, all `D1_ERROR: D1 DB is overloaded. Requests queued for too long.`, confined to 2026-08-04T10:34:56Z to 11:35:20Z. [`SKILLD-H`](https://harlan-zw.sentry.io/issues/7651521804/) carries 839 of them on `GET /api/skill-related/kotlin/kotlin-agent-skills/kotlin-backend-jpa-entity-mapping` alone; the rest are [`SKILLD-M`](https://harlan-zw.sentry.io/issues/7651521831/) 51, [`SKILLD-J`](https://harlan-zw.sentry.io/issues/7651521810/) 37, [`SKILLD-G`](https://harlan-zw.sentry.io/issues/7651521803/) 36, [`SKILLD-N`](https://harlan-zw.sentry.io/issues/7651521837/) 21, [`SKILLD-Q`](https://harlan-zw.sentry.io/issues/7651521866/) and [`SKILLD-K`](https://harlan-zw.sentry.io/issues/7651521818/) 6 each on `GET /api/orgs/webflow`, and [`SKILLD-P`](https://harlan-zw.sentry.io/issues/7651521838/) 1. The window opens ten minutes after the `ef31aea8` deploy and closes an hour later, overlapping the 879-skill ingest. Root-caused 2026-08-05 with `wrangler d1 insights skilld-db`, which reaches the historical query statistics the observability telemetry API refuses for the wrangler OAuth token, so the attribution gap noted in the `exceededMemory` row above is worked around for D1 at least. Two causes, both fixed. The hot path: `/api/skill-related` ran roughly six D1 reads per request with nothing cached, now cached whole on a one-hour TTL through `writeCache`, measured in production at 2.97s cold against 0.10s warm. The background pressure: the `ai-ready:cron` status counters averaged 430,099 and 143,319 rows read per run, at 421ms and 128ms, against a table of 143,470 rows, on a five-minute cron. Correction: those totals were first written as covering seven days. `wrangler d1 insights` ignores `--time-period` (1d, 3d and 7d return byte-identical output), so the span cannot be stated; per-run averages, the ranking, and the direct before-and-after measurements are unaffected. The cause was index selectivity, not volume. `ai_ready_pages` carries single-column indexes on `indexed` and on `is_error`, and the planner chose `idx_ai_ready_pages_is_error` for `WHERE indexed = ? AND is_error = 0`; every row but 80 is `is_error = 0`, so it matched nearly everything and `indexed` was filtered row by row. Production holds 143,390 rows at `indexed = 1, is_error = 0`, 80 at `indexed = 1, is_error = 1`, and none at `indexed = 0`, so the pending counter read the whole table every five minutes to return zero. Migration `0091` adds the composite index and was applied to production; the pending counter now plans as `SEARCH ... USING COVERING INDEX` reading 0 rows in 0.22ms, and the combined status query fell from 430,099 rows to 286,861. Remaining: the `indexnow_pending` subquery still scans despite a partial index matching its predicate exactly and holding 171 entries, because the planner has no size estimate for it; `ANALYZE` is the likely fix and is left for its own pass. A third source was found in the same insights run and fixed by migration `0092`: `querySkills` matches a batch of skills as `owner || '/' || repo || '/' || name` (`skills-registry.ts:172`), which no index can serve, so [SQLite](https://sqlite.org) scanned the whole `skills` table on every lookup. That query was the largest single consumer of D1 time on the database at 102.7ms average and 6,793 rows read per run, and 6,793 against 6,553 rows in the table is exactly one full scan however few identities the caller asked for. An index over the same expression was verified on a copy of the local database, then applied to production: the plan moved from `SCAN s` to `SEARCH s USING INDEX idx_skills_identity_path`, and a single-identity lookup fell from ~6,793 rows at 102.7ms to 5 rows at 0.57ms. `test/unit/migrations-bootstrap.test.ts` asserts the stored expression verbatim, because SQLite only uses an expression index when both sides match after normalisation and editing one alone would silently restore the scan. Still unaddressed: the `skills_fts MATCH` lane (51.8ms and 11,394 rows read per run against 6,553 indexed rows), where `bm25` ordering scores every match before `LIMIT` truncates, so no index and no smaller `LEXICAL_TOP_K` changes the cost. Attributed 2026-08-05 rather than optimised. Only three callers reach `querySkills` and only two pass a `search`, so only two can reach the lexical lane: the search API (`skills/index.get.ts`) and the related-skills fallback (`skill-related/[...slug].get.ts:102`). The org page passes `owner` with no `search` and never touches FTS, which rules out the crawlable surface the overload burst appeared on. The fallback fires only when a skill has no embedding neighbours, so its volume is a function of the embedding backlog, and both levers are already pulled: `/api/skill-related` is cached for an hour, and the adaptive cap drains the 479 missing embeddings that were forcing the fallback. Expect the lane to decline without further work. Next step is to re-read insights once those have taken effect, not to add a third mechanism speculatively. Exit condition: a later ingest spike produces no burst. Update 2026-08-06: the fix reduced the burst but did not stop it, and the check-in archive could not show that. The archive recorded zero *new* Sentry issues, which was true and irrelevant, because recurrence lands on the existing ids. Direct reads: `SKILLD-H` 839 → 1,410 events, `SKILLD-M` 51 → 80, `SKILLD-J` 37 → 71, `SKILLD-G` 36 → 65, `SKILLD-N` 21 → 36, all carrying `lastRelease = 5fd9dbc`, the fixed HEAD, with the newest event at 2026-08-05T16:24:40Z. Hourly stats place 686 events after the 05:18:25Z deploy, in three bursts: 501 at 07Z, 139 at 12Z, 46 at 16Z, then nine quiet hours to the window close. The three `orgs` issues (`SKILLD-K/P/Q`) did not recur and carry no release, so they died with `ef31aea8`. The surviving culprits moved off `orgs` and onto `GET /api/skill-related/*`, `GET /api/users/*/skills` and `GET /api/skills-raw/*`, and `userCount` stays at 1 per issue, which points at a single crawler traversal rather than organic traffic. Reading: the three indexes and the one-hour `writeCache` cut the blast radius by roughly a third per burst without removing the trigger, and cold keys still queue D1 faster than the cache fills. The 07Z burst is also the peak traffic hour (9,304 successes) and the hour holding 6 of the window's `exceededMemory` events, so the memory row below is the same incident seen through a second lens. Exit condition tightened: no burst on a crawler sweep, verified against `lastSeen` rather than against new-issue counts. Insights re-read 2026-08-06, post-deploy, which the previous entry left as the next step. The batch identity lookup now plans as `SEARCH s USING INDEX idx_skills_identity_path`, verified with `EXPLAIN QUERY PLAN` against production, so its 84ms/1,203-row insights average is pre-`0092` history rather than current cost; the single-identity lookup measures 0.73ms at 0 rows across 146,926 runs. The `/api/skills` detail path is therefore not the load source, which rules out the skill-detail caching that was proposed on the strength of the failing Sentry transaction alone: that request was queued behind the saturation, not causing it. The remaining lane is `skills_fts MATCH` at 133,760,656 rows read over 11,201 runs, efficiency 0.017, unfixable by indexing because `bm25` scores every match before `LIMIT`. Its two levers are already deployed (hourly `writeCache` on `/api/skill-related`, and an embedding backlog now closed at 5,816 embeddings against 5,442 indexable skills), and `wrangler d1 insights` ignores `--time-period` so the totals cannot be sliced to prove the decline. No third mechanism added, deliberately. Correction 2026-08-06: the fixed probe surfaced a sixth recurring id in this family, `SKILLD-2`, same `D1_ERROR` message on `GET /api/skill-files/...`, first seen 2026-07-23 and last seen 2026-08-05T12:17:19Z, so the burst predates the ids opened on 08-04 and reaches `skill-files` as well. | -| `registry/repo-maintenance` dead-letter | repo-sync | watch | 2026-08-05 | 548 | `Exhausted retries (dead-letter queue)`{lang="ts"} first appeared once on 2026-08-04 and was absent from the 2026-08-06 archive. Update 2026-08-10: it recurred 12 times from 2026-08-06T03:34:48Z through 2026-08-09T14:13:32Z. All nine archived registry sync batches failed, so this is now an active incident. It coincides with the 32 MiB serialized RPC fingerprint below. Exit condition: the cause is fixed, a deployed hourly registry batch finishes clean, and the fingerprint stops. Update 2026-08-11: 4 more dead-letter exhaustions after the deploy, latest 2026-08-11T05:13:45Z. This is the downstream half of the 32 MiB row: `garrytan/gstack` fails its retries and dead-letters. It closes when that row closes, so no separate work. Update 2026-08-12: it did not close with the 32 MiB row, so it is now its own fingerprint. The 2026-08-12 archive records 16 dead-letter exhaustions from 2026-08-11T12:22:37Z through 2026-08-12T01:01:42Z, and seven `registry-sync` batches after the `d2616c5` deploy at 2026-08-11T17:35:36Z each finished with exactly `failed_jobs: 1`, latest `registry-sync:1786496438`. The 32 MiB fix `865619c` is an ancestor of that deployed SHA and its raw `D1_ERROR` stopped at 2026-08-11T10:18:42Z, before the commit, so these dead-letters have no archived explanation. Blocked on observability: the dead-letter fingerprint replaces the original exception, so the archive alone cannot name the failure. Exit condition, revised: the last real error is recorded alongside the dead-letter row, the cause is named, and one hourly registry batch finishes with zero failed jobs. Root-caused 2026-08-16 from production, and the earlier "blocked on observability" reading was itself a symptom of the same setting. `skilld-repo-sync` set `max_retries: 3`, so a queue message got four deliveries. The sync handler releases the message back after each 50-file slice, and `releaseDurableJob` is followed by `message.retry()`{lang="ts"}, so pagination progress spent the delivery budget. Every dead-lettering repository sat at `repo_sync_progress.next_offset = 200`, exactly four slices, whatever its size: `posthog/posthog` 236 skills, `shipshitdev/library` 375, `github/awesome-copilot` 425, `affaan-m/everything-claude-code` 897, `jeremylongshore/claude-code-plugins-plus` 5,598, `sickn33/antigravity-awesome-skills` 6,341. `jobs.retry_reasons` carries only `controlled release` entries at `delaySeconds: 1` with `last_error` null, then `dlq-arrival`. Nothing failed. The hourly tick re-enqueued the same doomed job, producing 548 rows from 2026-08-12T02:02Z to 2026-08-16T04:04Z against 18 failed jobs in the whole prior window. The observability gap has the same cause: the failure budget is the job definition's `tries: 5`, checked against `jobs.attempts` in the runtime catch path, so at four deliveries a throwing job reached the DLQ before its fifth attempt and the DLQ headline replaced the real exception. Fixed 2026-08-16, undeployed: `max_retries` 3 -> 100 on `skilld-repo-sync` and `skilld-repo-review-sync`, with the invariant written into `wrangler.jsonc`. `SKILL_SLICE_SIZE` is deliberately untouched, being pinned at the GraphQL batch ceiling that bounds isolate memory. Also fixed 2026-08-16: `SKILL_PATHS_PER_INVOCATION` was pinned to `SKILL_SLICE_SIZE`, which spent one delivery per 50 files. The two constants bound different things. `syncRepo` walks its chunk in `SKILL_SLICE_SIZE` slices and releases each slice's content before the next, so the memory bound belongs to the inner slice; the repo-wide tree listing and existing-skill map are paid once per invocation whatever the outer cap is. Five slices per invocation now cost one delivery, moving the reachable ceiling from 200 files to 25,250. `SKILL_SLICE_SIZE` stays at 50, unchanged reasoning. A chain is still finite, so a repository past `MAX_INDEXABLE_SKILL_FILES` (20,000) now fails with the named reason `repo_too_large_to_index` instead of dead-lettering silently on every tick. That reason joins `TERMINAL_DISCOVERY_REJECTION_REASONS` so discovery stops retrying it, and `repositoryIndexFailureMessage` renders it for a submitter. Every known repository is under the limit; the largest holds 6,341 skills. Covered by `test/unit/repo-maintenance-continuation.test.ts`, which drives the handler and fails on both guards when they are reverted. Deferred, deliberately: replacing the continuation release with a successor job would remove the ceiling rather than raise it, but it needs `repo_sync_progress` ownership handover, discovery claim token inheritance, a stable job id for `repository-index.ts` submission tracking, and failure propagation from a successor back to the tracked root. Exit condition, revised: the six named repositories finish a sync, `repo_sync_progress` stops parking at 200, and one hourly registry batch finishes with zero failed jobs. | +| `registry/repo-maintenance` dead-letter | repo-sync | resolved 2026-08-16 | 2026-08-05 | 598 | `Exhausted retries (dead-letter queue)`{lang="ts"} first appeared once on 2026-08-04 and was absent from the 2026-08-06 archive. Update 2026-08-10: it recurred 12 times from 2026-08-06T03:34:48Z through 2026-08-09T14:13:32Z. All nine archived registry sync batches failed, so this is now an active incident. It coincides with the 32 MiB serialized RPC fingerprint below. Exit condition: the cause is fixed, a deployed hourly registry batch finishes clean, and the fingerprint stops. Update 2026-08-11: 4 more dead-letter exhaustions after the deploy, latest 2026-08-11T05:13:45Z. This is the downstream half of the 32 MiB row: `garrytan/gstack` fails its retries and dead-letters. It closes when that row closes, so no separate work. Update 2026-08-12: it did not close with the 32 MiB row, so it is now its own fingerprint. The 2026-08-12 archive records 16 dead-letter exhaustions from 2026-08-11T12:22:37Z through 2026-08-12T01:01:42Z, and seven `registry-sync` batches after the `d2616c5` deploy at 2026-08-11T17:35:36Z each finished with exactly `failed_jobs: 1`, latest `registry-sync:1786496438`. The 32 MiB fix `865619c` is an ancestor of that deployed SHA and its raw `D1_ERROR` stopped at 2026-08-11T10:18:42Z, before the commit, so these dead-letters have no archived explanation. Blocked on observability: the dead-letter fingerprint replaces the original exception, so the archive alone cannot name the failure. Exit condition, revised: the last real error is recorded alongside the dead-letter row, the cause is named, and one hourly registry batch finishes with zero failed jobs. Root-caused 2026-08-16 from production, and the earlier "blocked on observability" reading was itself a symptom of the same setting. `skilld-repo-sync` set `max_retries: 3`, so a queue message got four deliveries. The sync handler releases the message back after each 50-file slice, and `releaseDurableJob` is followed by `message.retry()`{lang="ts"}, so pagination progress spent the delivery budget. Every dead-lettering repository sat at `repo_sync_progress.next_offset = 200`, exactly four slices, whatever its size: `posthog/posthog` 236 skills, `shipshitdev/library` 375, `github/awesome-copilot` 425, `affaan-m/everything-claude-code` 897, `jeremylongshore/claude-code-plugins-plus` 5,598, `sickn33/antigravity-awesome-skills` 6,341. `jobs.retry_reasons` carries only `controlled release` entries at `delaySeconds: 1` with `last_error` null, then `dlq-arrival`. Nothing failed. The hourly tick re-enqueued the same doomed job, producing 548 rows from 2026-08-12T02:02Z to 2026-08-16T04:04Z against 18 failed jobs in the whole prior window. The observability gap has the same cause: the failure budget is the job definition's `tries: 5`, checked against `jobs.attempts` in the runtime catch path, so at four deliveries a throwing job reached the DLQ before its fifth attempt and the DLQ headline replaced the real exception. Fixed 2026-08-16, undeployed: `max_retries` 3 -> 100 on `skilld-repo-sync` and `skilld-repo-review-sync`, with the invariant written into `wrangler.jsonc`. `SKILL_SLICE_SIZE` is deliberately untouched, being pinned at the GraphQL batch ceiling that bounds isolate memory. Also fixed 2026-08-16: `SKILL_PATHS_PER_INVOCATION` was pinned to `SKILL_SLICE_SIZE`, which spent one delivery per 50 files. The two constants bound different things. `syncRepo` walks its chunk in `SKILL_SLICE_SIZE` slices and releases each slice's content before the next, so the memory bound belongs to the inner slice; the repo-wide tree listing and existing-skill map are paid once per invocation whatever the outer cap is. Five slices per invocation now cost one delivery, moving the reachable ceiling from 200 files to 25,250. `SKILL_SLICE_SIZE` stays at 50, unchanged reasoning. A chain is still finite, so a repository past `MAX_INDEXABLE_SKILL_FILES` (20,000) now fails with the named reason `repo_too_large_to_index` instead of dead-lettering silently on every tick. That reason joins `TERMINAL_DISCOVERY_REJECTION_REASONS` so discovery stops retrying it, and `repositoryIndexFailureMessage` renders it for a submitter. Every known repository is under the limit; the largest holds 6,341 skills. Covered by `test/unit/repo-maintenance-continuation.test.ts`, which drives the handler and fails on both guards when they are reverted. Deferred, deliberately: replacing the continuation release with a successor job would remove the ceiling rather than raise it, but it needs `repo_sync_progress` ownership handover, discovery claim token inheritance, a stable job id for `repository-index.ts` submission tracking, and failure propagation from a successor back to the tracked root. Exit condition, revised: the six named repositories finish a sync, `repo_sync_progress` stops parking at 200, and one hourly registry batch finishes with zero failed jobs. Deployed and verified 2026-08-16 on `23aecea`. Live queue consumers report `max_retries: 100` on both `skilld-repo-sync` and `skilld-repo-review-sync`, read back from the Cloudflare API rather than inferred from a green deploy. All three exit conditions are met. `repo_sync_progress` holds no row at `next_offset = 200` and no row above 500 skills, so every one of the six named repositories finished its chain: `github/awesome-copilot` settled at 430 indexed skills and `posthog/posthog` at 276, both past the 200-file wall that had held since 2026-08-12. The two largest reached offsets the old ceiling made unreachable, `sickn33/antigravity-awesome-skills` at 2,250 of 6,341 and `jeremylongshore/claude-code-plugins-plus` at 1,500 of 5,598, then both completed and released their rows. `registry-sync:1786863625` finished 19 jobs with zero failures and `render-repair:1786861225` 48 with zero, against 6 failures in each of the two batches immediately before the deploy. The dead-letter total froze at 598 with its newest row predating the merge by over an hour. Resolved. | | D1 serialized RPC exceeds 32 MiB | `registry/repo-maintenance` | watch | 2026-08-09 | 19 | The 2026-08-10 archive records a 48,516,427 byte RPC value against a 32 MiB limit, recurring through 2026-08-10T03:17:41Z. All nine archived registry sync batches failed, including the newest. Fixed locally 2026-08-10: the repository summary query excludes assets, while asset reads target only the current slice in chunks of 15. A regression test models the archived failure size and enforces the bound. Exit condition: deploy, one hourly registry batch finishes clean, and no new failure appears. Update 2026-08-11: the fix deployed and did not work. `EXISTING_SKILL_ASSET_CHUNK` (`layers/registry/server/utils/sync-repo.ts:106`) landed in `b2cb46f`, an ancestor of the deployed SHA `9a92d8f` live from 2026-08-10T16:28:57Z. A read-only count of `failed_jobs` after that timestamp returns 16 further `Serialized RPC` failures, newest 2026-08-11T08:17:41Z, so none of the three exit conditions passed. The window also narrowed the target: every post-deploy payload is the same repository, `{"_task":"registry/repo-maintenance","operation":"sync","owner":"garrytan","repo":"gstack"}`, and the size is byte-identical at 48,516,427 across all 16, with a second variant at 48,518,315 appearing twice. A constant byte count across 16 attempts is a deterministic whole-repository read, so the chunked asset path is not the path that overflows and some other query in the maintenance job still returns `gstack` in full. Blast radius is one poisoned job per hourly tick rather than a broken queue: `registry-sync:1786435221` ran 36 jobs and failed 1. Exit condition unchanged, plus: the failing query is named rather than assumed. Root-caused 2026-08-11: it is not a read at all, which is why a read-side chunk fixed nothing. The overflow is `db.batch(writes)`{lang="ts"} at the end of each slice in `syncRepo`, whose bound arguments carry `rendered_raw` and `rendered_html` for every admitted skill in the slice. `SKILL_SLICE_SIZE` bounds that batch at 50 *skills*, and skill content is unbounded, so the batch size tracks content rather than count. Measured in production: `garrytan/gstack` holds 58 skills with 11,140,589 characters of `rendered_html` and 3,298,415 of `rendered_raw`, averaging 192 KB of HTML each and peaking at 428,755 (`spec`). One 50-skill slice binds roughly 12.4M characters, which serialized to 48,516,427 bytes, near 3.9 bytes per character once escaping is counted. The count is byte-identical across all 16 post-deploy attempts because upstream is unchanged and the payload is deterministic. Fixed locally 2026-08-11, not yet deployed: `weighWrite` binds and measures a statement in one step so an unmeasured write cannot enter the queue, `planWriteBatches` splits the queue into contiguous groups under `D1_BATCH_CHAR_BUDGET` (8,000,000 characters, about 31 MB at the measured ratio), and `runBoundedBatch` merges results back in input order so `revisionWriteIndexes` and `activityWriteIndexes` still address the right rows. Splitting is safe for the reason the existing comment already gives: every write in the slice is an upsert, an `INSERT OR IGNORE`, or `NOT EXISTS`-guarded, so the batch never relied on atomicity. A single write larger than the whole budget still gets its own batch, because refusing it would strand the repository rather than fail one oversized skill. Covered by five cases in `test/unit/sync-repo-d1-bounds.test.ts`, including a reconstruction of the archived gstack slice against a 32 MiB stub ceiling. Exit condition: deploy, then one hourly registry batch finishes with zero failed jobs and the fingerprint stops. Update 2026-08-12: half the exit condition passed. `865619c` (`fix(registry): bound the slice batch by payload, not by skill count`) was committed 2026-08-11T10:46:36+10:00 and is an ancestor of the deployed SHA `d2616c5`, live from 2026-08-11T17:35:36Z. The 2026-08-12 archive records only 2 `Serialized RPC` failures, both at 2026-08-11T09:18:43Z and 10:18:42Z, which predate the commit, so nothing has hit the 32 MiB limit since the fix shipped. Held at `watch` rather than resolved because the other half did not pass: no post-deploy registry batch has finished with zero failed jobs. Those remaining failures now carry no archived exception and belong to the dead-letter row above, not here. Resolve this row once a clean hourly batch appears. | | Uncommitted file breaks the CI gate, second occurrence | GitHub Actions Test | resolved 2026-08-11 | 2026-08-11 | 1 run | Same failure mode as the `static-asset-fallthrough` row above, which was closed 2026-08-06. `Test` [run 31472932474](https://github.com/skilld-dev/skilld.dev/actions/runs/31472932474) failed on `d4f1a82` in the `typecheck` job: `layers/registry/app/pages/gh/[owner]/[repo]/[name]/-/[...file].vue:36` imports `#shared/skill-asset-path`, and `shared/skill-asset-path.ts` existed only in the working tree, so the Nuxt build died with `No such file or directory (os error 2)`{lang="ts"} and the gated deploy was skipped. Recovered without intervention: `39144c2` committed the helper, `Test` passed, and `Deploy to Cloudflare` succeeded at 2026-08-11T08:26:32Z, so production moved to `39144c2`. Recorded because the mechanism recurred rather than because the incident is open: a page can be committed while the module it imports is not, and nothing catches that before push. The 2026-08-06 closure fixed one instance, not the class. Worth a pre-push guard if it happens a third time. | | `SKILLD-S` KV GET 500 | `GET /api/skills/runwayml/skills/rw-generate-image` | watch | 2026-08-09 | 1 event | [Permalink](https://harlan-zw.sentry.io/issues/7662446055/). A retired empty endorsement map created the failing KV dependency. Removed locally 2026-08-10, with a route dependency regression test. Exit condition: deploy and no recurrence. | @@ -53,9 +53,9 @@ Persistent fingerprints found by the daily check-in. Add rows only after evidenc | Workers `internalError` (36) + `loadShed` (9) | Workers runtime | resolved 2026-08-11 | 2026-08-11 | 0 current | Attributed and closed the same day. Correction first: this row and the check-in both said URL-level attribution was blocked by a missing scoped `CLOUDFLARE_USAGE_TOKEN`. That is wrong, and it repeats the mistake the `exceededMemory` row made. The 403 applies to the observability *telemetry* API only; `workersInvocationsAdaptive` on the GraphQL analytics API accepts the wrangler OAuth token the check-in already uses, and it carries a `datetimeHour` dimension. No new token was needed for any of this. The hourly breakdown over 48 hours, scoped to `skilld-dev`, shows neither signal is a trend. All 36 `internalError` land in one four-hour block, 2026-08-10 08:00Z to 11:00Z (2, 17, 12, 5), with zero in every other hour. Cross-referencing `wrangler deployments list`: version `8fcae6fc` went live at 06:07:08Z and `42d35de9` replaced it at 11:21:21Z, so the burst opens and closes inside one version's life. Traffic in those hours was depressed too (733 to 1,032 successes against a 1,438 to 1,620 baseline), which reads as an unhealthy Worker rather than one overwhelmed by load. That version came out of the dependency incident the same morning, alongside `34b6680 fix(ci): restore dependency-compatible build` and `c6a84f8 fix(deploy): pin production-safe Nuxt runtime`, and `feee1ca6` before it was rolled back within 74 seconds. All 9 `loadShed` land in the single hour 2026-08-10T14:00Z, under `42d35de9`, in that window's lowest-traffic hour. Both are bounded bursts on versions no longer running: the current version has recorded zero of either across roughly 16 hours. Closed rather than watched, since a step change was the wrong reading. Prior text kept below for the correction record. The 2026-08-11 archive records 36 `internalError` and 9 `loadShed` against 33,835 successes over 28.9 hours. The prior archive recorded 5 `internalError` and zero `loadShed` over 97.7 hours, so `internalError` moved from about 0.05 to about 1.25 per hour and `loadShed` is new. `loadShed` means the runtime dropped requests under pressure, which fits the same capacity story as the D1 overload row above, but the dataset carries no request path and URL-level attribution is still blocked by the 403 on the observability telemetry API for the wrangler OAuth token. Recorded as a step change, not a diagnosis. Successful requests also fell to about 1,171 per hour from about 2,752, which is an observation about traffic or probe scope rather than a fault. Exit condition: an hourly breakdown scoped to `skilld-dev` shows either a bounded burst or a sustained rise with a named cause. | | Skills deleted upstream presented as live | GET /gh/:owner/:name | resolved 2026-08-05 | 2026-08-05 | 129 skills | 127 skills carry `source_resolved = 0`, the sync's verdict that their SKILL.md is gone from the source repository, and every one served 200 while its API response reported `resolved: true`. `microsoft/skills/entra-app-registration` is one: its upstream SKILL.md is 404 while the page presented a live skill. Cause: resolution was computed from the cached render alone (`const sourceResolved = Boolean(rendered.status === 'ok' && ...)`{lang="ts"}), and `rendered_status` describes the copy we stored, which survives an upstream deletion and stays `ok` forever. The detail query never selected `source_resolved` at all. Fixed 2026-08-05: the column is read and reported as `sourceGone`, verified in production (`sourceGone: true`, `source.resolved: false`), and the existing source-unavailable banner now fires on it, previously unreachable for these skills because their render status is `ok`. The copy says installing will fail, which is what a reader on an activation surface needs. The status code took two attempts. A `setResponseStatus(event, 410)`{lang="ts"} in `SkillDetail.vue` was written, deployed, and measured as having no effect: skill detail is fetched client-side, so the component never runs during SSR and `sourceGone` never reached the SSR payload. That code was removed rather than left in place looking correct. The working fix is `server/middleware/skill-source-gone.ts`, which sees the request early enough. It sets the status rather than throwing, so the page still renders the last indexed copy with its banner; 410 is valid with a body, and a thrown error would replace a useful tombstone with an error page. Both shapes a skill answers on are covered, the flat `/`{lang="html"} slug and the full `//`{lang="html"}, and a flat slug is dropped when a repository already owns that path so a live repository page can never be 410'd (measured at 0 such collisions across the 129 gone skills). The key set is cached for an hour, because the middleware runs on every `/gh` request and an uncached per-request D1 read on a hot path is what produced the overload burst above. Verified in production 2026-08-05: `/gh/microsoft/entra-app-registration`, its three-segment form, and `/gh/microsoft/azure-diagnostics` all answer 410 while still serving a 90KB page; `/gh/aahl/edge-tts`, `/gh/aahl/skills`, `/gh/microsoft`, `/gh/0xbigboss/claude-code` and the front door all still answer 200. Exit condition: the count of `source_resolved = 0` skills serving 200 stays at zero as new ones appear. | | `SKILLD-W` / `SKILLD-V` curator OG image crashes on an all-digit login | `/__nuxt_island/OgImageCuratorTakumi` and `/_og/d/…Curator.takumi….png` | watch | 2026-08-12 | 2 events, 1 user | [`SKILLD-W`](https://harlan-zw.sentry.io/issues/7666848478/) is `Error: e58.split is not a function`, culprit `GET /__nuxt_island/OgImageCuratorTakumi_….json?props={"handle":2114632142,"collectionCount":0,"skillCount":0}`. [`SKILLD-V`](https://harlan-zw.sentry.io/issues/7666848469/) is the same failure one layer out, the `/_og/d/…Curator.takumi,handle_2114632142….png` fetch surfacing the island error. Both first seen 2026-08-11T23:17:36Z, last seen 2026-08-12T01:32:13Z, 2 events and 1 user each. Cause read directly from the deployed source rather than inferred: `app/components/OgImage/Curator.takumi.vue` at `d2616c5` declares `handle: string` and calls `getInitials(displayName || handle)`{lang="ts"}, which calls `.split(' ')`{lang="ts"}. A GitHub login made only of digits deserializes from the island query string as a `number`, so `.split` is absent. The prop type asserts a wire format the wire does not honour, which makes this a boundary-parsing defect rather than a one-off. Fixed in the working tree, uncommitted and undeployed: props are typed `string | number | null` and coerced through new `app/utils/og-props.ts` helpers (`ogText`, `ogInitials`, `ogCount`, `ogTextList`). `Collection.takumi.vue`, `Page.takumi.vue`, and `Skill.takumi.vue` are modified in the same change, so the hazard is believed shared, but only the curator variant has production evidence. Exit condition: commit and deploy with a case rendering `handle: 2114632142`, then no further events on either id. | -| Daily task reported stale on every run | daily-health-check | fixed 2026-08-16 | 2026-08-16 | 2 reports | The nightly email named `detect-star-surges` stale on 2026-08-15 and 2026-08-16 while `sync_jobs` recorded `last_status = 'ok'` at 2026-08-15T04:30Z, exactly on its `30 4 * * *` schedule. `staleAfterSeconds` was a hand-kept list of cron prefixes with a 3-hour default, and a daily cron that was not one of the two literals it named fell to that default, so the check called it stale three hours after running. Any task added later with an unlisted shape inherits the same false positive, which is the actual defect. Fixed 2026-08-16, undeployed: the window is derived from the cron's own period, one full period plus half a period of slack, floored at 20 minutes. The two shapes the list did cover land on the same 20 minutes and 36 hours. Covered by `cronPeriodSeconds` cases and a health-check case that returns RED without the fix. Exit condition: one nightly report that names no stale task while every task is on schedule. | -| Skipped deploy guard read as a broken gate | daily-checkin observability | fixed 2026-08-16 | 2026-08-16 | 1 archive | The 2026-08-16 archive recorded `Deploy to Cloudflare` in state `failure` with `consecutiveFailures: 1`. The run it read had conclusion `skipped`: a `workflow_run` event from every branch triggers the deploy workflow, and its guard skips itself off `main`, which is the guard working. `completedState` treated any non-`success` conclusion as a failure. Fixed 2026-08-16: a `skipped` run carries no verdict, so it is read through to the last run that reached one, and a workflow that only ever skipped reports `missing` rather than `failure`. Covered by two cases in `test/unit/daily-checkin-observability.test.ts`. Exit condition: one archive where a skipped guard run does not move the workflow state. | -| Rejected submission counted as a job failure | repo-review-sync | watch | 2026-08-16 | 73 | 61 `no_supported_skill_paths` and 12 `root_skill_unsupported` rows in `failed_jobs` over the 2026-08-16 window. These are terminal decisions, not faults: `handleRegistryRepoJob` calls `ctx.fail(reason)`{lang="ts"} when a submitted repository has no supported SKILL.md. Because they land in `failed_jobs` they count toward the "N jobs failed in 24 hours" reason that drives the nightly email RED, so a healthy day of correctly rejecting unsuitable submissions reads as an incident. Fixed 2026-08-16, undeployed: the health check splits `failed_jobs` into faults and decisions, applying the principle its own discovery-candidate alarm states three lines below in the same query. The reason list is the existing `TERMINAL_DISCOVERY_REJECTION_REASONS`, bound as JSON so the values stay parameters, plus the `skill_parse_rejected:` prefix that carries a path. `failedJobs24h` counts faults only, `rejectedJobs24h` carries the decisions and appears in both rendered reports so the number stays visible. `ctx.fail` is unchanged, because the submission UI reads its reason back out of `failed_jobs`. Exit condition: one nightly report whose job line shows a non-zero rejected count and does not go RED on it. | +| Daily task reported stale on every run | daily-health-check | fixed 2026-08-16 | 2026-08-16 | 2 reports | The nightly email named `detect-star-surges` stale on 2026-08-15 and 2026-08-16 while `sync_jobs` recorded `last_status = 'ok'` at 2026-08-15T04:30Z, exactly on its `30 4 * * *` schedule. `staleAfterSeconds` was a hand-kept list of cron prefixes with a 3-hour default, and a daily cron that was not one of the two literals it named fell to that default, so the check called it stale three hours after running. Any task added later with an unlisted shape inherits the same false positive, which is the actual defect. Fixed 2026-08-16, undeployed: the window is derived from the cron's own period, one full period plus half a period of slack, floored at 20 minutes. The two shapes the list did cover land on the same 20 minutes and 36 hours. Covered by `cronPeriodSeconds` cases and a health-check case that returns RED without the fix. Exit condition: one nightly report that names no stale task while every task is on schedule. Deployed 2026-08-16 on `23aecea`. Exit condition still open: it needs the next nightly run at 22:00 Melbourne. | +| Skipped deploy guard read as a broken gate | daily-checkin observability | fixed 2026-08-16 | 2026-08-16 | 1 archive | The 2026-08-16 archive recorded `Deploy to Cloudflare` in state `failure` with `consecutiveFailures: 1`. The run it read had conclusion `skipped`: a `workflow_run` event from every branch triggers the deploy workflow, and its guard skips itself off `main`, which is the guard working. `completedState` treated any non-`success` conclusion as a failure. Fixed 2026-08-16: a `skipped` run carries no verdict, so it is read through to the last run that reached one, and a workflow that only ever skipped reports `missing` rather than `failure`. Covered by two cases in `test/unit/daily-checkin-observability.test.ts`. Exit condition: one archive where a skipped guard run does not move the workflow state. Deployed 2026-08-16 on `23aecea`. Following this merge surfaced four `skipped` deploy runs on `e9379f3`, the exact shape the classifier now reads through; the exit condition needs the next check-in archive to record it. | +| Rejected submission counted as a job failure | repo-review-sync | watch | 2026-08-16 | 73 | 61 `no_supported_skill_paths` and 12 `root_skill_unsupported` rows in `failed_jobs` over the 2026-08-16 window. These are terminal decisions, not faults: `handleRegistryRepoJob` calls `ctx.fail(reason)`{lang="ts"} when a submitted repository has no supported SKILL.md. Because they land in `failed_jobs` they count toward the "N jobs failed in 24 hours" reason that drives the nightly email RED, so a healthy day of correctly rejecting unsuitable submissions reads as an incident. Fixed 2026-08-16, undeployed: the health check splits `failed_jobs` into faults and decisions, applying the principle its own discovery-candidate alarm states three lines below in the same query. The reason list is the existing `TERMINAL_DISCOVERY_REJECTION_REASONS`, bound as JSON so the values stay parameters, plus the `skill_parse_rejected:` prefix that carries a path. `failedJobs24h` counts faults only, `rejectedJobs24h` carries the decisions and appears in both rendered reports so the number stays visible. `ctx.fail` is unchanged, because the submission UI reads its reason back out of `failed_jobs`. Exit condition: one nightly report whose job line shows a non-zero rejected count and does not go RED on it. Deployed 2026-08-16 on `23aecea`. Exit condition still open: it needs the next nightly run. | ## 2026-08-10 local remediation