A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine
-
Updated
Oct 16, 2026 - Python
#
cdx
Here are 12 public repositories matching this topic...
Summarize web archive capture index (CDX) files.
-
Updated
Mar 28, 2026 - Python
Python tools to retrieve text from CommonCrawl WARC files based on cdx index.
-
Updated
Feb 18, 2022 - Python
Fetch and normalize parameterized URLs from the Wayback CDX API (OSINT, inspired by ParamSpider).
python url cli security osint pypi pip recon bugbounty wayback-machine kali-linux wayback parrot cdx pypi-package
-
Updated
Nov 10, 2026 - Python
Enables Mac roundtrip editing for ChemDraw scheme-contaning PowerPoints made in Windows
-
Updated
Jun 10, 2021 - Python
Zero-install, server-preserving web archiver and ISO 28500 forensic engine for offline replication and digital preservation.
python offline-first warc web-archiving digital-preservation cdx anti-ddos mcp-server solo-maintainer server-preservation iso-28500
-
Updated
Nov 11, 2026 - Python
View cdx and warc files, caching them locally as needed
-
Updated
May 30, 2025 - Python
A web crawler built from scratch in Python, one concept per stage: frontier, robots.txt, politeness, async workers, dedup, rendering, freshness, a distributed frontier, WARC archives, a CDX index and replay. Thirteen stages, 222 tests, verified against a corpus that declares its own ground truth.
python crawler web-crawler simhash teaching web-scraping robots-txt asyncio learning-by-doing warc web-archiving cdx
-
Updated
Sep 22, 2026 - Python
Unofficial agent skills for the Internet Archive: CORS-aware HTTP/API access, browser widgets, Wayback/CDX and degraded-mode handling, per-mediatype metadata patterns. Complements internetarchive/internet-archive-skills.
-
Updated
Sep 15, 2026 - Python
The solution to extend the deadline for the virtual machines on CDX.
-
Updated
Nov 16, 2020 - Python
Wayback Machine scraper for Python and the command line: archived snapshots of a URL or a whole domain with capture time, HTTP status, MIME type, size, content digest and playback links, filtered by date range, as JSON or CSV. Free and open source; sign in with your QuanticData account, no proxies needed.
python cli scraper osint seo internet-archive web-scraping wayback-machine historical-data snapshots cdx web-archive archive-org wayback-machine-api quanticdata wayback-machine-scraper
-
Updated
Oct 30, 2026 - Python
Add this topic to your repo
To associate your repository with the cdx topic, visit your repo's landing page and select "manage topics."