Repository navigation
ci: flag links to archived repositories - #169
Conversation
Add scripts/check_archived.py, which asks the GitHub API for the archived repositories of the organization and reports links to them in the docs unless marked <!-- historical --> on the line or the line above. Run it weekly as the job archived (Archived repos) in links.yml, reporting through the shared action, and mark the existing links to protes-middleware-crypt4gh and demo-workflows. Closes #160 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reviewer's GuideImplementationAdd the archived-repository checker and extend the shared file-discovery and JSON-fetching helpers it relies on.
ConfigurationSchedule the archived-repository check in the links workflow and configure its permissions, issue reporting, and failure behavior.
Update ignored files and coverage configuration for the new checker.
DocumentationExplain the new archived-repository check, its historical marker rules, and how to run it locally.
Mark the intentional archived-repository links in the demo list and Crypt4GH tutorial as historical.
TestsTest archived-repository discovery, link matching and marking, reporting, command behavior, and the shared request-header changes.
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
Documentation build overview
|
Linked issuesChecked at commit d01bc56 #160 ci: flag mentions of archived repositories · 3 of 3 met
|
There was a problem hiding this comment.
Hey - I've found 8 issues
Prompt for AI Agents
Please address the comments from this code review:
## Individual Comments
### Comment 1
<location path="scripts/check_archived.py" line_range="178" />
<code_context>
+ description="Report links to archived repositories in the docs."
+ )
+ parser.add_argument("paths", nargs="*", help="files and directories")
+ parser.add_argument("--api-url", default=API_URL, help="GitHub REST API")
+ parser.add_argument("--output", type=Path, help="also write the report here")
+ return parser.parse_args(args)
</code_context>
<issue_to_address>
**Custom API receives GitHub token**
When the caller sets `--api-url` to a non-GitHub endpoint while `GITHUB_TOKEN` is set, `main` passes the caller-supplied API URL and `GITHUB_TOKEN` to `archived_repos`, which adds the token as an `Authorization` header; `fetch_json` then sends it to that host. A caller using an untrusted endpoint exposes the token to its operator, and an HTTP endpoint also exposes it in transit.
Only send the token to the trusted HTTPS GitHub API host, or omit it when a custom API URL is used.
Also at `scripts/check_archived.py:96`.
</issue_to_address>
### Comment 2
<location path="scripts/check_archived.py" line_range="161" />
<code_context>
+ ),
+ "",
+ ]
+ lines += [f"- `{m.location}`: `{ORG}/{m.repo}`" for m in mentions]
+ return "\n".join(lines) + "\n"
+
</code_context>
<issue_to_address>
**Filenames inject links into reports**
When a merged documentation change includes an unmarked archived-repository link in a file with a Markdown-delimiter-containing name, a contributor can add a file whose name contains Markdown delimiters and a deceptive link, then include an archived-repository URL in it. `find_mentions` carries that filename into `report` without escaping it, and the report-issue action publishes the resulting attacker-controlled link in an issue or comment.
Escape Markdown metacharacters in filenames before rendering them, or encode paths in a format that cannot create active Markdown.
</issue_to_address>
### Comment 3
<location path="scripts/check_archived.py" line_range="98-99" />
<code_context>
+ # Not sent again on redirects, which may lead to other hosts
+ secret = {"Authorization": f"Bearer {token}"} if token else {}
+ names = set()
+ for page in range(1, MAX_PAGES + 1):
+ url = f"{api_url}/orgs/{ORG}/repos?type=public&per_page={PER_PAGE}&page={page}"
+ try:
+ repos = fetch_json(url, headers, secret)
</code_context>
<issue_to_address>
**Archived links go unreported**
When a repository is deleted from an earlier page while the checker is fetching later pages, `archived_repos` fetches numbered pages from a live repository list. If a repository is deleted from an earlier page between requests, a still-existing archived repository can shift across a page boundary and be skipped, so `find_mentions` reports its links as clear for that run.
Use a listing strategy that detects page-boundary changes, or verify the complete listing before treating the result as clean.
</issue_to_address>
### Comment 4
<location path="scripts/check_archived.py" line_range="107-109" />
<code_context>
+ if not isinstance(repos, list):
+ raise CheckError("GitHub API: expected a list of repositories")
+ for repo in repos:
+ if not isinstance(repo, dict) or not isinstance(repo.get("name"), str):
+ raise CheckError("GitHub API: malformed repository")
+ if repo.get("archived") is True:
+ names.add(repo["name"].lower())
+ if len(repos) < PER_PAGE:
</code_context>
<issue_to_address>
**Malformed API data passes silently**
When the GitHub API returns a repository entry with a missing or non-Boolean `archived` field, `archived_repos` accepts entries with a valid `name` but no Boolean `archived` field, then ignores them; an unmarked link to an archived repository can therefore be reported as clean and the check exits 0.
Validate that every repository entry has a Boolean `archived` value and raise `CheckError` otherwise.
</issue_to_address>
### Comment 5
<location path="scripts/check_archived.py" line_range="132" />
<code_context>
+ for number, line in enumerate(
+ path.read_text(encoding="utf-8").splitlines(), start=1
+ ):
+ if not MARKER.search(line) and not ONLY_MARKER.match(previous):
+ for match in REPO_LINK.finditer(line):
+ if match["repo"].lower() in archived:
</code_context>
<issue_to_address>
**Archived links are hidden by literal markers**
When a scanned line contains an archived-repository link and marker spelling inside JSON, code or string content, `MARKER.search` treats the spelling as a real comment even inside content, so `find_mentions` skips the line and the check reports the archived link as clean.
Recognize historical markers only when they are actual comments, not literal content.
</issue_to_address>
### Comment 6
<location path="scripts/check_archived.py" line_range="53" />
<code_context>
+MAX_PAGES = 50
+
+REPO_LINK = re.compile(
+ rf"(?:github\.com[/:]|raw\.githubusercontent\.com/){re.escape(ORG)}/"
+ # A dot ends the name only if no name character follows (end of sentence)
+ r"(?P<repo>[A-Za-z0-9_.-]+?)(?:\.git)?(?![A-Za-z0-9_-]|\.[A-Za-z0-9_-])",
</code_context>
<issue_to_address>
**Unrelated hosts are reported**
When a scanned URL uses a different hostname containing `github.com` before the organization path, `REPO_LINK` matches the `github.com` substring without checking a hostname boundary, so it extracts the repository name and reports an unrelated-host URL as an archived-repository link.
Require `github.com` to begin at a hostname boundary before accepting the organization path.
</issue_to_address>
### Comment 7
<location path="tests/unit/test_check_services.py" line_range="382-390" />
<code_context>
+ """Sends default headers, which given headers may replace."""
+ Server.routes = {"/x": (200, [])}
+ check_services.fetch_json(f"{server}/x")
+ check_services.fetch_json(
+ f"{server}/x", {"Accept": "text/plain"}, {"Authorization": "Bearer t"}
+ )
+ first, second = Server.headers_seen
+ assert first["Accept"] == "application/json"
+ assert first["User-Agent"] == "elixir-on-cloud-docs-checks"
+ assert "Authorization" not in first
+ assert second["Accept"] == "text/plain"
+ assert second["Authorization"] == "Bearer t"
+
+
</code_context>
<issue_to_address>
**Custom headers drop default agent**
When custom headers are supplied, `test_fetch_json_headers` checks the default User-Agent only on the first request, then checks only Accept and Authorization on the customized request. Replacing the default headers wholesale when custom headers are supplied would pass this test while dropping the User-Agent.
Assert that the customized request also retains the default User-Agent.
</issue_to_address>
### Comment 8
<location path="tests/unit/test_check_services.py" line_range="398-402" />
<code_context>
+) -> None:
+ """Doesn't send secret headers to the target of a redirect."""
+ monkeypatch.setattr(check_services, "placeholder", lambda host: False)
+ Server.redirects = {"/x": f"{server}/y"}
+ Server.routes = {"/y": (200, [])}
+ check_services.fetch_json(f"{server}/x", None, {"Authorization": "Bearer t"})
+ assert Server.headers_seen[0]["Authorization"] == "Bearer t"
+ assert "Authorization" not in Server.headers_seen[1]
+
+
</code_context>
<issue_to_address>
**Cross-host token leak goes undetected**
When a redirect points to a different host, `test_fetch_json_secret_headers_not_redirected` redirects to `f"{server}/y"`, the same host as the original request. A redirect handler that strips Authorization only for same-host redirects would pass this test while leaking the token on a cross-host redirect.
Redirect to a different host in the test and assert that the target receives no Authorization header.
</issue_to_address>Sourcery assessment
Needs a human reviewer. 8 findings to address first, and if the checker wrongly flags links, the workflow can create or comment on a public repository issue, sending notifications that closing the issue cannot retract. Reverting stops future reports, but does not undo notifications already sent.
Blocking findings: scripts/check_archived.py:178, scripts/check_archived.py:161, scripts/check_archived.py:99, scripts/check_archived.py:109, scripts/check_archived.py:132, and 3 more
Send GITHUB_TOKEN to the default API URL only, require a boolean archived field, match github.com at a host boundary, keep file names from breaking out of inline code in reports, and test the default User-Agent with custom headers and a cross-host redirect. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Summary
Weekly check that the docs don't recommend archived repositories of the organization.
scripts/check_archived.py(standard library only): collects links togithub.com/elixir-cloud-aai/<repo>(also SSH clone URLs andraw.githubusercontent.com) from the Markdown, YAML, JSON and shell files ofREADME.md,mkdocs.yml,docs/andincludes/, asks the GitHub REST API for the organization's repositories (GET /orgs/elixir-cloud-aai/repos, paginated,archivedfield), and reports each link to an archived repository that is not marked as historical. Nothing is hard-coded: today the API lists 29 archived repositories.<!-- historical -->on the same line, or alone on the line above (needed above reference definitions; in YAML and shell files as# <!-- historical -->). A marker covers every link on its line.protes-middleware-crypt4gh(Crypt4GH tutorial and demo list, which already say the repository is archived; the "More information" link now says "(archived)") anddemo-workflows(retired demos). The build renders all of them as links, with the comments invisible.links.yml: new jobarchived/Archived repos,contents: readandissues: write, skipped in forks.GITHUB_TOKENonly raises the rate limit; it is sent as an unredirected header, so a redirect never carries it to another host.scripts/check_services.py:fetch_jsontakes extra and secret headers (User-Agent is nowelixir-on-cloud-docs-checks),find_filestakes the suffixes to list.Closes #160
Verification
python scripts/check_archived.py(live API): before marking, 6 unmarked links in 2 pages (exit 2); after: "The organization has 29 archived repositories. No page links to them without ahistoricalmarker.", exit 0.pytest: 302 passed, 100 % line and branch coverage (check_archived,check_names,check_services,registry,last_verified); the API is a local HTTP server.python scripts/check_names.py,mkdocs build --strict: pass; the built pages link the marked repositories.mypy --stricton the changed scripts and tests: clean.Review
Two rounds of adversarial and quality review. Fixed: names followed by a full stop,
*.github.iorepositories, SSH clone URLs,#-comment markers for YAML, JSON and shell files scanned, token not sent on redirects, README on schedule and marker scope, tests of the header handling, split error tests, report wording.Fixed from the Sourcery review (d01bc56):
GITHUB_TOKENsent to the default API URL only, a booleanarchivedrequired,github.commatched at a host boundary, file names can't break out of inline code in the reports of both checks, tests of the default User-Agent with custom headers and of a cross-host redirect.Declined review findings
Open questions
None.
🤖 Generated with Claude Code