Skip to content
CannObservPublic

About

Internet Information Monitoring Infrastructure

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

2,205 Commits

Folders and files

Repository files navigation

watcher

Monitors web resources for change. Each Watched Item is a URL that Archiver registers; Watcher has it fetched on a schedule (Replicator performs the fetch), fingerprints the content and reports each change on the bus; Archiver notifies from those reports.

Architecture

The WatchedItem is the single monitored entity (#191): one row = one URL = one fingerprint = one change signal. It owns its effective_url, source_specs, schedule (the cadence Archiver announces, else a 1d system default — #375), domain_name / domain_suspended, and health/timestamps. A periodic schedule_tick enqueues check_watched_item for each due item; the processor extracts and fingerprints, and Watcher writes a ChangeRevision and stages one change_detected fact per change.

Canonical content provenance (InfoItem / InfoSource / SourceRevision) lives in the sibling Archiver service (separate repo, its own VM), consumed over the bus via info.registry announcements — Watcher holds no Archiver SDK and makes no HTTP calls to it (#254). archiver_info_item_id and archiver_info_source_id on a WatchedItem are required cross-service references (#251) — every WatchedItem is an InfoItem being watched, and Archiver provisions it. Every detected change is reported to Archiver as a source_revision_observed fact on the content.revisions bus stream (#253), and the latest revision is re-announced when a full fetch renews the blob reference behind it (#293), with a local pending_archiver_sync outbox + drain worker guaranteeing delivery across broker outages. Its change, error and recovery edges go to Archiver as facts on info.watch-events through a second outbox (#369; docs/WATCH-EVENTS.md), and Archiver notifies from those; Watcher sends no notification (#376, #379).

Operate WatchedItems from Archiver's control plane; Watcher serves only /health and /ready (#378). Full conventions: AGENTS.md. Operator install: docs/DEPLOYMENT.md; env files and every variable: docs/ENVIRONMENT.md. Fetch/revision path and the Archiver linkage: docs/CONTENT-PIPELINE.md. The WatchedItem entity in full: docs/WATCHED-ITEMS.md. Module layout and bus topology: docs/ARCHITECTURE.md. Collapse design: docs/plans/2026-06-16-collapse-watcheditem-watch-design.md.

Setup

git submodule update --init --recursive
uv sync

Development

# Dev server on port 8001 against a non-production database (#233).
# Port 8000 belongs to the systemd service — never bind it by hand.
bash scripts/dev_server.sh

Testing

uv run pytest

See docs/COMMANDS.md for full command reference.

About

Internet Information Monitoring Infrastructure

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages