Skip to content

IndexNow: notify changed pages after each production deploy - #76

Open
mattwebhub wants to merge 1 commit into
mainfrom
technical/indexnow-on-deploy
Open

mattwebhub wants to merge 1 commit into
mainfrom
technical/indexnow-on-deploy

Conversation

@mattwebhub

Copy link
Copy Markdown
Contributor

What this does

Every time a production deploy of trytoone.com goes live, the site tells IndexNow (Bing, Yandex and the other engines that use it) which pages changed, so they recrawl them sooner. Owner decision, 2026-11-02: "the whole site, every deploy".

When it runs. Vercel already reports each deploy to GitHub. When a Production deploy succeeds, a GitHub Actions job starts. Preview deploys and toone-landing*.vercel.app never trigger it. Runs go one at a time, so two merges close together can't send the same page twice.

What it sends. The job reads the live sitemap.xml, sitemap-explore.xml and sitemap-journal.xml on trytoone.com. A URL is sent only if it has never been sent before, or if its <lastmod> is later than the lastmod it was last sent with. An unchanged page is never sent again (AUTH-013). Everything goes out in one POST to https://api.indexnow.org/indexnow with the same payload as the Journal sender: host trytoone.com, key kpdkez871rpzxfa3q11m2d2fvpqq4cyg, keyLocation https://trytoone.com/kpdkez871rpzxfa3q11m2d2fvpqq4cyg.txt.

What it never sends

  • /en/journal and Journal posts. The Journal webhook already sends those on every publish (JOURNAL_INDEXNOW, unchanged). The hub pages /en/journal/releases and /en/journal/launches are included because the webhook doesn't send them.
  • Any URL with no <lastmod>, for example after a CLI upload with no git history (PR Sitemap: omit lastmod when the git date is unknown, never the build date (TECH-013) #74). With no date there's no proof the page changed, so nothing is sent for it.
  • Any URL that fails a live check right before sending. The check requires HTTP 200 with no redirect, no noindex in X-Robots-Tag or a robots/bingbot meta tag, and exactly one canonical that equals the URL (TECH-010, TECH-013). Dropped URLs are logged with the reason and checked again on the next run. They aren't marked as sent.
  • Pages that aren't on https://trytoone.com.

First run. It starts from the 49 URLs Authority sent by hand on 2026-11-02 at 14:31:44 UTC (HTTP 200, list SHA-256 9d3b0db2…3965). Each of them is sent again only once its lastmod is later than 14:31:44 UTC. A dry run against the live site at 14:39 UTC gave: 0 URLs to send (49 unchanged, 4 left to the Journal webhook, 0 with no lastmod).

Explore approvals (EXPLORE_INDEXNOW_ON_APPROVAL): left OFF on purpose

This PR does not turn the on-approval sender on. The variable isn't set on toone-landing today, so it stays off. Explore record pages are still covered: they're listed in sitemap-explore.xml with the record's update time as lastmod, so this job sends a newly approved or changed record once, at the next production deploy, after the same live check. Turning both on would send an approved record twice. A comment in lib/explore/indexnow.ts and in .env.example now says to leave it unset.

The trade-off is that a record approved on a day with no deploy waits for the next deploy. If you'd rather not wait, a follow-up can add a daily run of this same job. The rules stay the same, so nothing is sent twice. It isn't included here because the decision was "every deploy".

Kill switch and record

  • Off switch: set the GitHub repository variable SITE_INDEXNOW to false (Settings → Secrets and variables → Actions → Variables, or gh variable set SITE_INDEXNOW --body false -R io-hexagonal/Toone). Delete it or set it to anything else to turn it back on. The Journal's own switch JOURNAL_INDEXNOW is unchanged.
  • Record of each submission: every run writes its time, the URLs sent, IndexNow's HTTP status, dropped URLs with reasons and counts of skipped URLs to the run's job summary (Actions tab → "IndexNow site notice"). It also uploads the artifact indexnow-site-record, which has log.md (every run so far, newest first) and state.json (each URL with the lastmod it was sent with and when). The next run starts from that artifact.
  • Fail-safe: if the last record can't be read or has expired (no production deploy for 90 days), or a sitemap can't be read, the job stops and sends nothing. If IndexNow answers anything other than 200/202, the URLs stay unsent, the run is marked failed and they're retried on the next deploy.
  • Manual run: Actions → "IndexNow site notice" → Run workflow, from main only. "Dry run" is ticked by default, so it only lists what would be sent.

Known limits

  • sitemap.xml dates are day-only (git commit day). If a page is changed again on the same day it was last sent, that second change waits until the page next changes on a later day. A follow-up can switch those dates to full timestamps if you want same-day precision.
  • CLI deploys (vercel --prod) aren't reported to GitHub, so they don't trigger the job. They also carry no lastmod (PR Sitemap: omit lastmod when the git date is unknown, never the build date (TECH-013) #74), so there would be nothing to send. The next Git deploy catches up.

Files

  • .github/workflows/indexnow-site.yml: the job (trigger, record load/save, kill switch).
  • scripts/indexnow/notify-site.ts: reads the sitemaps, selects, runs the live check, sends, writes the record.
  • lib/indexnow/site-notice.ts: selection rules, live check, log text. lib/indexnow/baseline.ts: the 49 URLs from 2026-11-02.
  • lib/indexnow/submit.ts: the shared payload. lib/explore/indexnow.ts now re-exports it, so the Journal and Explore senders send exactly what they did before.
  • tests/indexnow/site-notice.test.ts: 11 tests (added to npm test).

Checks

  • npm test: all suites pass, typecheck clean. npm run build: clean.
  • Live dry runs (nothing sent): from the baseline, 0 to send. From an empty record, all 49 non-Journal sitemap URLs pass the live check. Known-bad pages are dropped correctly: /en/downloads (meta noindex), /explore (308), /en/marketplace (404), /fr (X-Robots-Tag noindex).
  • Workflow file validated against the GitHub Actions schema.
  • Playbook 02/04: no page or copy change. Playbook 03: TECH-010 and TECH-013 are enforced by the live check. No change to robots, sitemaps or metadata.

Merging releases this. From the first production deploy after the merge (the merge's own deploy), changed pages are sent automatically with no further approval. At today's state that first run sends nothing.

…UTH-013)

After Vercel reports a successful Production deployment, a GitHub Actions job
reads the live sitemaps and sends IndexNow only the URLs whose lastmod is new
or later than their last notice, after a live check of each (200, no redirect,
indexable, self-canonical). Journal posts and /en/journal stay with the
Journal webhook. URLs with no lastmod are never sent. The first run starts
from the 49 URLs sent by hand on 2026-11-02 at 14:31:44Z. Kill switch: repo
variable SITE_INDEXNOW=false. Each run keeps a record artifact (state + log)
and a job summary.

The payload moves to lib/indexnow/submit.ts and is shared by all senders;
Explore's on-approval path is unchanged and stays off.
@vercel

vercel Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
toone-landing Building Building Preview Nov 2, 2026 2:42pm UTC

This branch was successfully deployed

1 active deployment
Preview — b1fcd44f Deployed Nov 2, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant