chore(digest): pause the AI summary behind an explicit flag

docs/ops/triage-ledger.md

@@ -109,5 +109,5 @@ Run `pnpm registry:convergence` for a read-only production snapshot. Incomplete
 The 2026-07-25 report first attributed the 4 remaining sync failures to repository renames, on the strength of an exact correlation: the 5 stuck repos were precisely the renamed ones. That was wrong. Worker logs showed `blob_batch_failed:502`, and direct measurement confirmed the cause was unchunked GraphQL batches. The renamed repos simply happen to be the largest. A supporting test (`getTree` returning 301 under curl) was also misleading, because it omitted `-L` while the Workers runtime follows redirects.
 
 The 2026-08-03 report first blamed the embedding drift partly on head-of-line blocking, reasoning from the code that `embedding_attempts` is write-only, so failing skills would re-consume a slot every hour. The code reading is accurate and the conclusion was still wrong. Querying the table showed 480 completions in the last 24 hours against a 480 per day cap, with no attempt in any failed state for a week: no slot is being wasted. The 2,455 `rejected` rows that made the theory look plausible are all `malformed_acknowledgement` from 2026-07-23 and 2026-07-24, the incident already closed on 2026-07-25, and reading the row count without its date range turned closed history into an apparent live fault. Cumulative state tables need a recency filter before they can support any claim about today.
-| `Insufficient AI Gateway credits` degrades the digest summary | `send-digests` scheduled task | watch | 2026-09-02 | 2 users | The 2026-09-02 archive records `send-digests` with `last_status: partial`, last run 2026-09-01T09:01:12Z, error `user 39 AI fallback: provider_failure: 2021: Insufficient AI Gateway credits; user 42 AI fallback: provider_failure: 2021: Insufficient AI Gateway credits`. Both digests were delivered (`digests_sent` 2, `digests_failed` 0); the AI summary is what failed, and `send-digests.ts:77` records it as `aiFallbackReason` while still sending, which is the designed degradation. The nightly operator email for 2026-09-02 went AMBER on the single reason `Scheduled tasks partially failed: send-digests.`, sent 2026-09-01T22:00:31Z on worker version `805ccae2`. Hypothesis, unconfirmed: error `2021` is an account-level credit exhaustion code, so every AI Gateway consumer is affected and the digest is only the lane that happened to run. The archive cannot separate a blocked AI generation lane from an idle one, because `ai-generate-submit` and `ai-generate-poll` both report `ok` and `ai_submitted` is 0, which is what a healthy quiet window also looks like. This is Loop 2 retention impact: the monthly digest is the retention artefact and it shipped summary-less to 100% of recipients. Exit condition: credits restored, one digest send records no `aiFallbackReason`, and a check of `ai_batches` over the same period shows submissions were idle rather than blocked. If they were blocked, `ai-generate-submit` reporting `ok` on a provider failure is a second defect and needs its own row. |
+| `Insufficient AI Gateway credits` degrades the digest summary | `send-digests` scheduled task | paused 2026-09-02 | 2026-09-02 | 2 users | The 2026-09-02 archive records `send-digests` with `last_status: partial`, last run 2026-09-01T09:01:12Z, error `user 39 AI fallback: provider_failure: 2021: Insufficient AI Gateway credits; user 42 AI fallback: provider_failure: 2021: Insufficient AI Gateway credits`. Both digests were delivered (`digests_sent` 2, `digests_failed` 0); the AI summary is what failed, and `send-digests.ts:77` records it as `aiFallbackReason` while still sending, which is the designed degradation. The nightly operator email for 2026-09-02 went AMBER on the single reason `Scheduled tasks partially failed: send-digests.`, sent 2026-09-01T22:00:31Z on worker version `805ccae2`. Hypothesis, unconfirmed: error `2021` is an account-level credit exhaustion code, so every AI Gateway consumer is affected and the digest is only the lane that happened to run. The archive cannot separate a blocked AI generation lane from an idle one, because `ai-generate-submit` and `ai-generate-poll` both report `ok` and `ai_submitted` is 0, which is what a healthy quiet window also looks like. This is Loop 2 retention impact: the monthly digest is the retention artefact and it shipped summary-less to 100% of recipients. Decided 2026-09-02: pause the summary rather than restore the credits. The sentence is one ~20-word line per changed repo, Loop 2 has 2 recipients, and the model is a partner model, so there is no audience worth paying to tune it for yet. Paused 2026-09-02 in [#121](https://github.com/skilld-dev/skilld.dev/pull/121): `DIGEST_AI_SUMMARY_PAUSED` in `layers/identity/server/tasks/send-digests.ts` gates it, mirroring `HAIKU_GENERATION_PAUSED` in `layers/registry/server/tasks/ai-generate-submit.ts`. While it is true the task never calls `summariseChanges`: `resolveDigestSummariser` returns `{ _tag: 'fallback', reason: 'paused' }`, and `summariseDigestRun` keeps a paused fallback out of `errors`, so the run reports `ok` and the digest still sends on the same no-summary template. The AMBER was therefore the false alarm. Every recipient already got the summary-less digest, and the defect was reporting a chosen state as a partial failure. The genuine `aiFallbackReason` path is unchanged and still reports `partial` when the flag goes back to false. Pinned by `test/unit/digest-ai-pause.test.ts`, verified failing first. The `ai-generate-submit` question is answered separately: the scaled-content deindexing already paused Haiku generation there, so `ok` with `ai_submitted` 0 is the paused state and not a swallowed provider failure. Exit condition: one `send-digests` run reports `ok` with the digest delivered. |
 | `SKILLD-1E` skills-raw 503 on a live source | `GET /api/skills-raw/:owner/:repo/:name` | fixed 2026-09-02 | 2026-09-02 | 2 events | [`SKILLD-1E`](https://harlan-zw.sentry.io/issues/7705072099/), `Error: Skill source is unavailable upstream`, culprit `GET /api/skills-raw/dimillian/skills/app-store-changelog`, 2 events, 1 user, 2026-09-01T17:12:50Z to 2026-09-02T00:13:55Z. Verified read-only 2026-09-02: the endpoint answers 200, `api.github.com/repos/dimillian/skills` answers 200, and `ungh.cc/repos/dimillian/skills` answers 200, so the source is live and serving. Hypothesis: a transient ungh outage, the same shape as the recovered `neondatabase/agent-skills` events in the row above, landing in the 503 lane that [#109](https://github.com/skilld-dev/skilld.dev/pull/109) deliberately kept for genuine outages. The classifier is behaving as designed. What is new is the id: this family previously grouped under `SKILLD-11`, and the post-#109 handler raises a differently shaped error, so the split is most likely Sentry grouping rather than a second mechanism. Exit condition: one window with no growth on the id. If it grows on further culprits whose sources are live, the outage classification is over-firing and needs the treatment the gone-source path already received. Confirmed 2026-09-02: the tree read had no retry. `fetchUpstreamText` has retried a transient upstream since 2026-08-17, three attempts with jittered backoff, but `fetchUpstreamTree` called ungh once and turned any rejection that was not a 404 into `unavailable`. ofetch retries a GET once with no delay, which is too fast to clear a blip, so one bad answer from ungh became a 503 with `retry-after` on the run-command surface while the source was live. Fixed 2026-09-02 in [#119](https://github.com/skilld-dev/skilld.dev/pull/119): the retry policy moved to `layers/registry/server/utils/upstream-retry.ts` and both readers share it, so the tree read now retries a transient status or a connection that never answers, and reports `unavailable` only after the last attempt. The lanes the earlier rows built are unchanged: a 404 is still `gone` and answers 410 on the first attempt, and a sustained outage still answers 503 with `retry-after`. The tree body is also parsed at the boundary now, so a body of the wrong shape reports as an outage rather than as an empty repository. Seven cases in `test/unit/upstream-tree.test.ts` and one end-to-end case on the culprit in `test/unit/skills-raw-tree-outage.test.ts`, verified failing on `main` first. Exit condition: deploy, then one window with no growth on the id. |

layers/identity/server/tasks/send-digests.ts

@@ -1,3 +1,4 @@
+import type { DigestUserOutcome } from '../utils/digest-run-summary'
 import type { AiBinding } from '../utils/digest-summary'
 import { resolveCloudflareBindings } from '@harlan-zw/nuxt-cloudflare/bindings'
 import { runObservedScheduledTask } from '~~/server/utils/scheduled-run'
@@ -6,15 +7,26 @@ import { observedSchedulePolicy } from '#shared/schedule-policy'
 import {
   runDigestDeliveryForUser,
 } from '../utils/digest-delivery'
+import { summariseDigestRun } from '../utils/digest-run-summary'
 import {
   loadDigestEligibleUsers,
   selectDigestForUser,
 } from '../utils/digest-select'
-import { summariseChanges } from '../utils/digest-summary'
+import { resolveDigestSummariser } from '../utils/digest-summary'
 import { renderDigest } from '../utils/digest-template'
 import { sendEmailWithEnv, signUnsubToken } from '../utils/email'
 
 const CRON = '0 9 1 * *'
+// Kill switch: the per-repo digest sentence runs `anthropic/claude-haiku-4.5`,
+// a partner model brokered through the Workers AI binding, so every call bills
+// against AI Gateway credits. Those credits are exhausted, so each monthly run
+// returned `provider_failure: 2021: Insufficient AI Gateway credits`, delivered
+// the digest on the no-summary template, and still reported `partial`. That
+// raised a monthly AMBER on the operator health email for a feature nobody had
+// decided to keep. Paused by decision until Loop 2 has an audience worth tuning
+// the sentence for. Re-enabling costs AI Gateway credits and one send to
+// verify. Flip back to false to resume.
+const DIGEST_AI_SUMMARY_PAUSED = true
 
 export default defineScheduledTask({
   name: 'send-digests',
@@ -42,67 +54,32 @@ export default defineScheduledTask({
       const startedAt = Date.now()
       const scheduledAt = Math.floor(startedAt / 1_000)
       const users = await loadDigestEligibleUsers(db)
-      const summary = {
-        eligible: users.length,
-        fired: users.length,
-        sent: 0,
-        skipped: 0,
-        failed: 0,
-        claimed: 0,
-        uncertain: 0,
-        alreadyProcessed: 0,
-        aiFallbacks: 0,
-        errors: [] as string[],
-      }
+      const summarise = resolveDigestSummariser({
+        paused: DIGEST_AI_SUMMARY_PAUSED,
+        ai,
+      })
+      const outcomes: DigestUserOutcome[] = []
 
       for (const user of users) {
         const result = await runDigestDeliveryForUser({
           db,
           now: () => Math.floor(Date.now() / 1_000),
           newClaimToken: () => crypto.randomUUID(),
           select: selectDigestForUser,
-          summarise: ai
-            ? input => summariseChanges({ ai, ...input })
-            : async () => ({ _tag: 'fallback', reason: 'binding_missing' }),
+          summarise,
           render: renderDigest,
           signUnsubscribe: userId => signUnsubToken(userId, tokenKey),
           send: input => sendEmailWithEnv(env, { ...input, from: emailFrom }),
         }, user, {
           scheduledAt,
           siteUrl,
         })
-
-        if (result._tag === 'sent') {
-          summary.sent += 1
-          if (result.aiFallbackReason) {
-            summary.aiFallbacks += 1
-            summary.errors.push(`user ${user.id} AI fallback: ${result.aiFallbackReason}`)
-          }
-        }
-        else if (result._tag === 'skipped') {
-          summary.skipped += 1
-        }
-        else if (result._tag === 'failed') {
-          summary.failed += 1
-          summary.errors.push(`user ${user.id} ${result.stage}: ${result.error}`)
-        }
-        else if (result._tag === 'claimed') {
-          summary.claimed += 1
-        }
-        else if (result._tag === 'delivery_uncertain') {
-          summary.uncertain += 1
-          summary.errors.push(`user ${user.id} delivery uncertain: ${result.reason}: ${result.error}`)
-        }
-        else {
-          summary.alreadyProcessed += 1
-        }
+        outcomes.push({ userId: user.id, result })
       }
 
-      const status = summary.failed > 0 || summary.uncertain > 0
-        ? (summary.sent > 0 || summary.skipped > 0 ? 'partial' : 'error')
-        : summary.aiFallbacks > 0
-          ? 'partial'
-          : 'ok'
+      const { summary, status } = summariseDigestRun(outcomes, {
+        aiSummaryPaused: DIGEST_AI_SUMMARY_PAUSED,
+      })
       await reportJobRun(db, 'send-digests', {
         cron: CRON,
         status,

layers/identity/server/utils/digest-run-summary.ts

@@ -0,0 +1,84 @@
+import type { DigestDeliveryResult } from './digest-delivery'
+
+// Pure accounting for one `send-digests` run. The task collects an outcome per
+// user, this turns the set into the counters and the reported status.
+
+export interface DigestUserOutcome {
+  userId: number
+  result: DigestDeliveryResult
+}
+
+export interface DigestRunOptions {
+  // True when the task never called the summariser. A fallback is then the
+  // chosen state, so it must not count as an error and must not raise the
+  // status above `ok`.
+  aiSummaryPaused: boolean
+}
+
+export interface DigestRunSummary {
+  eligible: number
+  fired: number
+  sent: number
+  skipped: number
+  failed: number
+  claimed: number
+  uncertain: number
+  alreadyProcessed: number
+  aiFallbacks: number
+  errors: string[]
+}
+
+export type DigestRunStatus = 'ok' | 'partial' | 'error'
+
+export function summariseDigestRun(
+  outcomes: DigestUserOutcome[],
+  options: DigestRunOptions,
+): { summary: DigestRunSummary, status: DigestRunStatus } {
+  const summary: DigestRunSummary = {
+    eligible: outcomes.length,
+    fired: outcomes.length,
+    sent: 0,
+    skipped: 0,
+    failed: 0,
+    claimed: 0,
+    uncertain: 0,
+    alreadyProcessed: 0,
+    aiFallbacks: 0,
+    errors: [],
+  }
+
+  for (const { userId, result } of outcomes) {
+    if (result._tag === 'sent') {
+      summary.sent += 1
+      if (result.aiFallbackReason && !options.aiSummaryPaused) {
+        summary.aiFallbacks += 1
+        summary.errors.push(`user ${userId} AI fallback: ${result.aiFallbackReason}`)
+      }
+    }
+    else if (result._tag === 'skipped') {
+      summary.skipped += 1
+    }
+    else if (result._tag === 'failed') {
+      summary.failed += 1
+      summary.errors.push(`user ${userId} ${result.stage}: ${result.error}`)
+    }
+    else if (result._tag === 'claimed') {
+      summary.claimed += 1
+    }
+    else if (result._tag === 'delivery_uncertain') {
+      summary.uncertain += 1
+      summary.errors.push(`user ${userId} delivery uncertain: ${result.reason}: ${result.error}`)
+    }
+    else {
+      summary.alreadyProcessed += 1
+    }
+  }
+
+  const status: DigestRunStatus = summary.failed > 0 || summary.uncertain > 0
+    ? (summary.sent > 0 || summary.skipped > 0 ? 'partial' : 'error')
+    : summary.aiFallbacks > 0
+      ? 'partial'
+      : 'ok'
+
+  return { summary, status }
+}

layers/identity/server/utils/digest-summary.ts

@@ -68,10 +68,29 @@ export type SummariseResult
   }
   | {
     _tag: 'fallback'
-    reason: 'no_changes' | 'binding_missing' | 'provider_failure' | 'empty_response' | 'invalid_response'
+    reason: 'no_changes' | 'binding_missing' | 'paused' | 'provider_failure' | 'empty_response' | 'invalid_response'
     error?: string
   }
 
+// The digest delivery only needs the changes, so the binding and the pause
+// switch are bound once by the caller.
+export type DigestSummariser = (input: Omit<SummariseInput, 'ai'>) => Promise<SummariseResult>
+
+// Single place that decides whether a digest run talks to the model. When the
+// summary is paused, or the binding is missing, the caller gets a fallback and
+// the provider is never called.
+export function resolveDigestSummariser(input: {
+  paused: boolean
+  ai: AiBinding | undefined
+}): DigestSummariser {
+  if (input.paused)
+    return async () => ({ _tag: 'fallback', reason: 'paused' })
+  const ai = input.ai
+  if (!ai)
+    return async () => ({ _tag: 'fallback', reason: 'binding_missing' })
+  return async changes => summariseChanges({ ai, ...changes })
+}
+
 export async function summariseChanges(input: SummariseInput): Promise<SummariseResult> {
   if (!input.changes.length)
     return { _tag: 'fallback', reason: 'no_changes' }

test/unit/digest-ai-pause.test.ts

@@ -0,0 +1,103 @@
+import type { AiBinding } from '../../layers/identity/server/utils/digest-summary'
+import { describe, expect, it, vi } from 'vitest'
+import { summariseDigestRun } from '../../layers/identity/server/utils/digest-run-summary'
+import { resolveDigestSummariser } from '../../layers/identity/server/utils/digest-summary'
+
+describe('paused digest AI summary', () => {
+  it('never calls the provider and reports the pause as the fallback reason', async () => {
+    const run = vi.fn(async () => ({ content: [{ type: 'text', text: '{"summaries":[]}' }] }))
+    const ai: AiBinding = { run }
+
+    const summarise = resolveDigestSummariser({ paused: true, ai })
+    const result = await summarise({
+      subscriptions: [{
+        owner: 'acme',
+        repo: 'skills',
+        skills: [{ name: 'alpha', description: 'Alpha', changeCount: 2 }],
+      }],
+      changes: [{
+        owner: 'acme',
+        repo: 'skills',
+        totalChangeCount: 2,
+        skills: [{ name: 'alpha', changeCount: 2, commitMessages: ['one'] }],
+        diffExcerpt: '',
+      }],
+    })
+
+    expect(result).toEqual({ _tag: 'fallback', reason: 'paused' })
+    expect(run).not.toHaveBeenCalled()
+  })
+
+  it('still calls the provider when the pause is off', async () => {
+    const run = vi.fn(async () => ({
+      content: [{ type: 'text', text: `{"summaries":[{"owner":"acme","repo":"skills","sentence":"Alpha changed."}]}` }],
+    }))
+
+    const summarise = resolveDigestSummariser({ paused: false, ai: { run } })
+    const result = await summarise({
+      subscriptions: [{
+        owner: 'acme',
+        repo: 'skills',
+        skills: [{ name: 'alpha', description: 'Alpha', changeCount: 2 }],
+      }],
+      changes: [{
+        owner: 'acme',
+        repo: 'skills',
+        totalChangeCount: 2,
+        skills: [{ name: 'alpha', changeCount: 2, commitMessages: ['one'] }],
+        diffExcerpt: '',
+      }],
+    })
+
+    expect(result).toMatchObject({ _tag: 'summarized' })
+    expect(run).toHaveBeenCalledTimes(1)
+  })
+
+  it('reports ok for a delivered digest when the summary is paused', () => {
+    const run = summariseDigestRun([{
+      userId: 39,
+      result: {
+        _tag: 'sent',
+        deliveryKey: 'skilld-digest:39:7200',
+        providerMessageId: 'message-39',
+        aiFallbackReason: 'paused',
+      },
+    }], { aiSummaryPaused: true })
+
+    expect(run.status).toBe('ok')
+    expect(run.summary).toMatchObject({ sent: 1, aiFallbacks: 0, errors: [] })
+  })
+
+  it('reports partial for a genuine AI fallback when the summary is live', () => {
+    const run = summariseDigestRun([{
+      userId: 39,
+      result: {
+        _tag: 'sent',
+        deliveryKey: 'skilld-digest:39:7200',
+        providerMessageId: 'message-39',
+        aiFallbackReason: 'provider_failure: 2021: Insufficient AI Gateway credits',
+      },
+    }], { aiSummaryPaused: false })
+
+    expect(run.status).toBe('partial')
+    expect(run.summary.aiFallbacks).toBe(1)
+    expect(run.summary.errors).toEqual([
+      'user 39 AI fallback: provider_failure: 2021: Insufficient AI Gateway credits',
+    ])
+  })
+
+  it('reports error when every user failed', () => {
+    const run = summariseDigestRun([{
+      userId: 39,
+      result: {
+        _tag: 'failed',
+        deliveryKey: 'skilld-digest:39:7200',
+        stage: 'provider',
+        error: 'provider_failure',
+      },
+    }], { aiSummaryPaused: true })
+
+    expect(run.status).toBe('error')
+    expect(run.summary).toMatchObject({ failed: 1, sent: 0 })
+  })
+})