diff --git a/docs/ops/triage-ledger.md b/docs/ops/triage-ledger.md index ac0e9a3f..20e988a5 100644 --- a/docs/ops/triage-ledger.md +++ b/docs/ops/triage-ledger.md @@ -109,5 +109,5 @@ Run `pnpm registry:convergence` for a read-only production snapshot. Incomplete The 2026-07-25 report first attributed the 4 remaining sync failures to repository renames, on the strength of an exact correlation: the 5 stuck repos were precisely the renamed ones. That was wrong. Worker logs showed `blob_batch_failed:502`, and direct measurement confirmed the cause was unchunked GraphQL batches. The renamed repos simply happen to be the largest. A supporting test (`getTree` returning 301 under curl) was also misleading, because it omitted `-L` while the Workers runtime follows redirects. The 2026-08-03 report first blamed the embedding drift partly on head-of-line blocking, reasoning from the code that `embedding_attempts` is write-only, so failing skills would re-consume a slot every hour. The code reading is accurate and the conclusion was still wrong. Querying the table showed 480 completions in the last 24 hours against a 480 per day cap, with no attempt in any failed state for a week: no slot is being wasted. The 2,455 `rejected` rows that made the theory look plausible are all `malformed_acknowledgement` from 2026-07-23 and 2026-07-24, the incident already closed on 2026-07-25, and reading the row count without its date range turned closed history into an apparent live fault. Cumulative state tables need a recency filter before they can support any claim about today. -| `Insufficient AI Gateway credits` degrades the digest summary | `send-digests` scheduled task | watch | 2026-09-02 | 2 users | The 2026-09-02 archive records `send-digests` with `last_status: partial`, last run 2026-09-01T09:01:12Z, error `user 39 AI fallback: provider_failure: 2021: Insufficient AI Gateway credits; user 42 AI fallback: provider_failure: 2021: Insufficient AI Gateway credits`. Both digests were delivered (`digests_sent` 2, `digests_failed` 0); the AI summary is what failed, and `send-digests.ts:77` records it as `aiFallbackReason` while still sending, which is the designed degradation. The nightly operator email for 2026-09-02 went AMBER on the single reason `Scheduled tasks partially failed: send-digests.`, sent 2026-09-01T22:00:31Z on worker version `805ccae2`. Hypothesis, unconfirmed: error `2021` is an account-level credit exhaustion code, so every AI Gateway consumer is affected and the digest is only the lane that happened to run. The archive cannot separate a blocked AI generation lane from an idle one, because `ai-generate-submit` and `ai-generate-poll` both report `ok` and `ai_submitted` is 0, which is what a healthy quiet window also looks like. This is Loop 2 retention impact: the monthly digest is the retention artefact and it shipped summary-less to 100% of recipients. Exit condition: credits restored, one digest send records no `aiFallbackReason`, and a check of `ai_batches` over the same period shows submissions were idle rather than blocked. If they were blocked, `ai-generate-submit` reporting `ok` on a provider failure is a second defect and needs its own row. | +| `Insufficient AI Gateway credits` degrades the digest summary | `send-digests` scheduled task | paused 2026-09-02 | 2026-09-02 | 2 users | The 2026-09-02 archive records `send-digests` with `last_status: partial`, last run 2026-09-01T09:01:12Z, error `user 39 AI fallback: provider_failure: 2021: Insufficient AI Gateway credits; user 42 AI fallback: provider_failure: 2021: Insufficient AI Gateway credits`. Both digests were delivered (`digests_sent` 2, `digests_failed` 0); the AI summary is what failed, and `send-digests.ts:77` records it as `aiFallbackReason` while still sending, which is the designed degradation. The nightly operator email for 2026-09-02 went AMBER on the single reason `Scheduled tasks partially failed: send-digests.`, sent 2026-09-01T22:00:31Z on worker version `805ccae2`. Hypothesis, unconfirmed: error `2021` is an account-level credit exhaustion code, so every AI Gateway consumer is affected and the digest is only the lane that happened to run. The archive cannot separate a blocked AI generation lane from an idle one, because `ai-generate-submit` and `ai-generate-poll` both report `ok` and `ai_submitted` is 0, which is what a healthy quiet window also looks like. This is Loop 2 retention impact: the monthly digest is the retention artefact and it shipped summary-less to 100% of recipients. Decided 2026-09-02: pause the summary rather than restore the credits. The sentence is one ~20-word line per changed repo, Loop 2 has 2 recipients, and the model is a partner model, so there is no audience worth paying to tune it for yet. Paused 2026-09-02 in [#121](https://github.com/skilld-dev/skilld.dev/pull/121): `DIGEST_AI_SUMMARY_PAUSED` in `layers/identity/server/tasks/send-digests.ts` gates it, mirroring `HAIKU_GENERATION_PAUSED` in `layers/registry/server/tasks/ai-generate-submit.ts`. While it is true the task never calls `summariseChanges`: `resolveDigestSummariser` returns `{ _tag: 'fallback', reason: 'paused' }`, and `summariseDigestRun` keeps a paused fallback out of `errors`, so the run reports `ok` and the digest still sends on the same no-summary template. The AMBER was therefore the false alarm. Every recipient already got the summary-less digest, and the defect was reporting a chosen state as a partial failure. The genuine `aiFallbackReason` path is unchanged and still reports `partial` when the flag goes back to false. Pinned by `test/unit/digest-ai-pause.test.ts`, verified failing first. The `ai-generate-submit` question is answered separately: the scaled-content deindexing already paused Haiku generation there, so `ok` with `ai_submitted` 0 is the paused state and not a swallowed provider failure. Exit condition: one `send-digests` run reports `ok` with the digest delivered. | | `SKILLD-1E` skills-raw 503 on a live source | `GET /api/skills-raw/:owner/:repo/:name` | fixed 2026-09-02 | 2026-09-02 | 2 events | [`SKILLD-1E`](https://harlan-zw.sentry.io/issues/7705072099/), `Error: Skill source is unavailable upstream`, culprit `GET /api/skills-raw/dimillian/skills/app-store-changelog`, 2 events, 1 user, 2026-09-01T17:12:50Z to 2026-09-02T00:13:55Z. Verified read-only 2026-09-02: the endpoint answers 200, `api.github.com/repos/dimillian/skills` answers 200, and `ungh.cc/repos/dimillian/skills` answers 200, so the source is live and serving. Hypothesis: a transient ungh outage, the same shape as the recovered `neondatabase/agent-skills` events in the row above, landing in the 503 lane that [#109](https://github.com/skilld-dev/skilld.dev/pull/109) deliberately kept for genuine outages. The classifier is behaving as designed. What is new is the id: this family previously grouped under `SKILLD-11`, and the post-#109 handler raises a differently shaped error, so the split is most likely Sentry grouping rather than a second mechanism. Exit condition: one window with no growth on the id. If it grows on further culprits whose sources are live, the outage classification is over-firing and needs the treatment the gone-source path already received. Confirmed 2026-09-02: the tree read had no retry. `fetchUpstreamText` has retried a transient upstream since 2026-08-17, three attempts with jittered backoff, but `fetchUpstreamTree` called ungh once and turned any rejection that was not a 404 into `unavailable`. ofetch retries a GET once with no delay, which is too fast to clear a blip, so one bad answer from ungh became a 503 with `retry-after` on the run-command surface while the source was live. Fixed 2026-09-02 in [#119](https://github.com/skilld-dev/skilld.dev/pull/119): the retry policy moved to `layers/registry/server/utils/upstream-retry.ts` and both readers share it, so the tree read now retries a transient status or a connection that never answers, and reports `unavailable` only after the last attempt. The lanes the earlier rows built are unchanged: a 404 is still `gone` and answers 410 on the first attempt, and a sustained outage still answers 503 with `retry-after`. The tree body is also parsed at the boundary now, so a body of the wrong shape reports as an outage rather than as an empty repository. Seven cases in `test/unit/upstream-tree.test.ts` and one end-to-end case on the culprit in `test/unit/skills-raw-tree-outage.test.ts`, verified failing on `main` first. Exit condition: deploy, then one window with no growth on the id. | diff --git a/layers/identity/server/tasks/send-digests.ts b/layers/identity/server/tasks/send-digests.ts index 2c33c1e2..8676f99d 100644 --- a/layers/identity/server/tasks/send-digests.ts +++ b/layers/identity/server/tasks/send-digests.ts @@ -1,3 +1,4 @@ +import type { DigestUserOutcome } from '../utils/digest-run-summary' import type { AiBinding } from '../utils/digest-summary' import { resolveCloudflareBindings } from '@harlan-zw/nuxt-cloudflare/bindings' import { runObservedScheduledTask } from '~~/server/utils/scheduled-run' @@ -6,15 +7,26 @@ import { observedSchedulePolicy } from '#shared/schedule-policy' import { runDigestDeliveryForUser, } from '../utils/digest-delivery' +import { summariseDigestRun } from '../utils/digest-run-summary' import { loadDigestEligibleUsers, selectDigestForUser, } from '../utils/digest-select' -import { summariseChanges } from '../utils/digest-summary' +import { resolveDigestSummariser } from '../utils/digest-summary' import { renderDigest } from '../utils/digest-template' import { sendEmailWithEnv, signUnsubToken } from '../utils/email' const CRON = '0 9 1 * *' +// Kill switch: the per-repo digest sentence runs `anthropic/claude-haiku-4.5`, +// a partner model brokered through the Workers AI binding, so every call bills +// against AI Gateway credits. Those credits are exhausted, so each monthly run +// returned `provider_failure: 2021: Insufficient AI Gateway credits`, delivered +// the digest on the no-summary template, and still reported `partial`. That +// raised a monthly AMBER on the operator health email for a feature nobody had +// decided to keep. Paused by decision until Loop 2 has an audience worth tuning +// the sentence for. Re-enabling costs AI Gateway credits and one send to +// verify. Flip back to false to resume. +const DIGEST_AI_SUMMARY_PAUSED = true export default defineScheduledTask({ name: 'send-digests', @@ -42,18 +54,11 @@ export default defineScheduledTask({ const startedAt = Date.now() const scheduledAt = Math.floor(startedAt / 1_000) const users = await loadDigestEligibleUsers(db) - const summary = { - eligible: users.length, - fired: users.length, - sent: 0, - skipped: 0, - failed: 0, - claimed: 0, - uncertain: 0, - alreadyProcessed: 0, - aiFallbacks: 0, - errors: [] as string[], - } + const summarise = resolveDigestSummariser({ + paused: DIGEST_AI_SUMMARY_PAUSED, + ai, + }) + const outcomes: DigestUserOutcome[] = [] for (const user of users) { const result = await runDigestDeliveryForUser({ @@ -61,9 +66,7 @@ export default defineScheduledTask({ now: () => Math.floor(Date.now() / 1_000), newClaimToken: () => crypto.randomUUID(), select: selectDigestForUser, - summarise: ai - ? input => summariseChanges({ ai, ...input }) - : async () => ({ _tag: 'fallback', reason: 'binding_missing' }), + summarise, render: renderDigest, signUnsubscribe: userId => signUnsubToken(userId, tokenKey), send: input => sendEmailWithEnv(env, { ...input, from: emailFrom }), @@ -71,38 +74,12 @@ export default defineScheduledTask({ scheduledAt, siteUrl, }) - - if (result._tag === 'sent') { - summary.sent += 1 - if (result.aiFallbackReason) { - summary.aiFallbacks += 1 - summary.errors.push(`user ${user.id} AI fallback: ${result.aiFallbackReason}`) - } - } - else if (result._tag === 'skipped') { - summary.skipped += 1 - } - else if (result._tag === 'failed') { - summary.failed += 1 - summary.errors.push(`user ${user.id} ${result.stage}: ${result.error}`) - } - else if (result._tag === 'claimed') { - summary.claimed += 1 - } - else if (result._tag === 'delivery_uncertain') { - summary.uncertain += 1 - summary.errors.push(`user ${user.id} delivery uncertain: ${result.reason}: ${result.error}`) - } - else { - summary.alreadyProcessed += 1 - } + outcomes.push({ userId: user.id, result }) } - const status = summary.failed > 0 || summary.uncertain > 0 - ? (summary.sent > 0 || summary.skipped > 0 ? 'partial' : 'error') - : summary.aiFallbacks > 0 - ? 'partial' - : 'ok' + const { summary, status } = summariseDigestRun(outcomes, { + aiSummaryPaused: DIGEST_AI_SUMMARY_PAUSED, + }) await reportJobRun(db, 'send-digests', { cron: CRON, status, diff --git a/layers/identity/server/utils/digest-run-summary.ts b/layers/identity/server/utils/digest-run-summary.ts new file mode 100644 index 00000000..c68960e8 --- /dev/null +++ b/layers/identity/server/utils/digest-run-summary.ts @@ -0,0 +1,84 @@ +import type { DigestDeliveryResult } from './digest-delivery' + +// Pure accounting for one `send-digests` run. The task collects an outcome per +// user, this turns the set into the counters and the reported status. + +export interface DigestUserOutcome { + userId: number + result: DigestDeliveryResult +} + +export interface DigestRunOptions { + // True when the task never called the summariser. A fallback is then the + // chosen state, so it must not count as an error and must not raise the + // status above `ok`. + aiSummaryPaused: boolean +} + +export interface DigestRunSummary { + eligible: number + fired: number + sent: number + skipped: number + failed: number + claimed: number + uncertain: number + alreadyProcessed: number + aiFallbacks: number + errors: string[] +} + +export type DigestRunStatus = 'ok' | 'partial' | 'error' + +export function summariseDigestRun( + outcomes: DigestUserOutcome[], + options: DigestRunOptions, +): { summary: DigestRunSummary, status: DigestRunStatus } { + const summary: DigestRunSummary = { + eligible: outcomes.length, + fired: outcomes.length, + sent: 0, + skipped: 0, + failed: 0, + claimed: 0, + uncertain: 0, + alreadyProcessed: 0, + aiFallbacks: 0, + errors: [], + } + + for (const { userId, result } of outcomes) { + if (result._tag === 'sent') { + summary.sent += 1 + if (result.aiFallbackReason && !options.aiSummaryPaused) { + summary.aiFallbacks += 1 + summary.errors.push(`user ${userId} AI fallback: ${result.aiFallbackReason}`) + } + } + else if (result._tag === 'skipped') { + summary.skipped += 1 + } + else if (result._tag === 'failed') { + summary.failed += 1 + summary.errors.push(`user ${userId} ${result.stage}: ${result.error}`) + } + else if (result._tag === 'claimed') { + summary.claimed += 1 + } + else if (result._tag === 'delivery_uncertain') { + summary.uncertain += 1 + summary.errors.push(`user ${userId} delivery uncertain: ${result.reason}: ${result.error}`) + } + else { + summary.alreadyProcessed += 1 + } + } + + const status: DigestRunStatus = summary.failed > 0 || summary.uncertain > 0 + ? (summary.sent > 0 || summary.skipped > 0 ? 'partial' : 'error') + : summary.aiFallbacks > 0 + ? 'partial' + : 'ok' + + return { summary, status } +} diff --git a/layers/identity/server/utils/digest-summary.ts b/layers/identity/server/utils/digest-summary.ts index 18867914..49c0efec 100644 --- a/layers/identity/server/utils/digest-summary.ts +++ b/layers/identity/server/utils/digest-summary.ts @@ -68,10 +68,29 @@ export type SummariseResult } | { _tag: 'fallback' - reason: 'no_changes' | 'binding_missing' | 'provider_failure' | 'empty_response' | 'invalid_response' + reason: 'no_changes' | 'binding_missing' | 'paused' | 'provider_failure' | 'empty_response' | 'invalid_response' error?: string } +// The digest delivery only needs the changes, so the binding and the pause +// switch are bound once by the caller. +export type DigestSummariser = (input: Omit) => Promise + +// Single place that decides whether a digest run talks to the model. When the +// summary is paused, or the binding is missing, the caller gets a fallback and +// the provider is never called. +export function resolveDigestSummariser(input: { + paused: boolean + ai: AiBinding | undefined +}): DigestSummariser { + if (input.paused) + return async () => ({ _tag: 'fallback', reason: 'paused' }) + const ai = input.ai + if (!ai) + return async () => ({ _tag: 'fallback', reason: 'binding_missing' }) + return async changes => summariseChanges({ ai, ...changes }) +} + export async function summariseChanges(input: SummariseInput): Promise { if (!input.changes.length) return { _tag: 'fallback', reason: 'no_changes' } diff --git a/test/unit/digest-ai-pause.test.ts b/test/unit/digest-ai-pause.test.ts new file mode 100644 index 00000000..b15d2b40 --- /dev/null +++ b/test/unit/digest-ai-pause.test.ts @@ -0,0 +1,103 @@ +import type { AiBinding } from '../../layers/identity/server/utils/digest-summary' +import { describe, expect, it, vi } from 'vitest' +import { summariseDigestRun } from '../../layers/identity/server/utils/digest-run-summary' +import { resolveDigestSummariser } from '../../layers/identity/server/utils/digest-summary' + +describe('paused digest AI summary', () => { + it('never calls the provider and reports the pause as the fallback reason', async () => { + const run = vi.fn(async () => ({ content: [{ type: 'text', text: '{"summaries":[]}' }] })) + const ai: AiBinding = { run } + + const summarise = resolveDigestSummariser({ paused: true, ai }) + const result = await summarise({ + subscriptions: [{ + owner: 'acme', + repo: 'skills', + skills: [{ name: 'alpha', description: 'Alpha', changeCount: 2 }], + }], + changes: [{ + owner: 'acme', + repo: 'skills', + totalChangeCount: 2, + skills: [{ name: 'alpha', changeCount: 2, commitMessages: ['one'] }], + diffExcerpt: '', + }], + }) + + expect(result).toEqual({ _tag: 'fallback', reason: 'paused' }) + expect(run).not.toHaveBeenCalled() + }) + + it('still calls the provider when the pause is off', async () => { + const run = vi.fn(async () => ({ + content: [{ type: 'text', text: `{"summaries":[{"owner":"acme","repo":"skills","sentence":"Alpha changed."}]}` }], + })) + + const summarise = resolveDigestSummariser({ paused: false, ai: { run } }) + const result = await summarise({ + subscriptions: [{ + owner: 'acme', + repo: 'skills', + skills: [{ name: 'alpha', description: 'Alpha', changeCount: 2 }], + }], + changes: [{ + owner: 'acme', + repo: 'skills', + totalChangeCount: 2, + skills: [{ name: 'alpha', changeCount: 2, commitMessages: ['one'] }], + diffExcerpt: '', + }], + }) + + expect(result).toMatchObject({ _tag: 'summarized' }) + expect(run).toHaveBeenCalledTimes(1) + }) + + it('reports ok for a delivered digest when the summary is paused', () => { + const run = summariseDigestRun([{ + userId: 39, + result: { + _tag: 'sent', + deliveryKey: 'skilld-digest:39:7200', + providerMessageId: 'message-39', + aiFallbackReason: 'paused', + }, + }], { aiSummaryPaused: true }) + + expect(run.status).toBe('ok') + expect(run.summary).toMatchObject({ sent: 1, aiFallbacks: 0, errors: [] }) + }) + + it('reports partial for a genuine AI fallback when the summary is live', () => { + const run = summariseDigestRun([{ + userId: 39, + result: { + _tag: 'sent', + deliveryKey: 'skilld-digest:39:7200', + providerMessageId: 'message-39', + aiFallbackReason: 'provider_failure: 2021: Insufficient AI Gateway credits', + }, + }], { aiSummaryPaused: false }) + + expect(run.status).toBe('partial') + expect(run.summary.aiFallbacks).toBe(1) + expect(run.summary.errors).toEqual([ + 'user 39 AI fallback: provider_failure: 2021: Insufficient AI Gateway credits', + ]) + }) + + it('reports error when every user failed', () => { + const run = summariseDigestRun([{ + userId: 39, + result: { + _tag: 'failed', + deliveryKey: 'skilld-digest:39:7200', + stage: 'provider', + error: 'provider_failure', + }, + }], { aiSummaryPaused: true }) + + expect(run.status).toBe('error') + expect(run.summary).toMatchObject({ failed: 1, sent: 0 }) + }) +})