Collect web data from your terminal for scripts, agents and data pipelines. Extract pages, crawl sites, discover URLs, search the web and run batches with the Geonode CLI. JSON output and exit codes make the results easy to use in your own automation.
Integrations page · Geonode API docs · CLI setup guide · View package on PyPI · View source on GitHub · Complete guide
Quick start · Choose a command · Example: save a page for a scheduled pipeline · Setup questions
- Schedule a content refresh. Run extraction commands from your scheduler, inspect exit codes, and save JSON results.
- Inspect a site from the terminal. Map URLs, select pages and extract Markdown without writing an API client.
Requires Python 3.10 or newer and a Geonode API key from the dashboard.
pipx install geonode-scraper-cli
export GEONODE_SCRAPER_API_KEY="your-api-key"
gscraper extract https://example.com --format markdownUse pip install geonode-scraper-cli if you already manage your own virtual environment. On Windows PowerShell, set $env:GEONODE_SCRAPER_API_KEY = "your-api-key".
For machine-readable output:
gscraper extract https://example.com --format markdown --jsonA successful response has ok: true and the API result under result; a failed request has ok: false and an error object. API requests require an active account and use your plan's allowance.
| Task | Command |
|---|---|
| Extract a page | gscraper extract URL --format markdown |
| Extract several URLs | gscraper batch create URL [URL ...] |
| Crawl a website | gscraper crawl create URL --depth 2 --limit 50 |
| Discover URLs | gscraper map run URL |
| Search the web | gscraper search run "your query" |
| Wait for a job | gscraper jobs wait JOB_ID |
| Check concurrency | gscraper usage concurrency |
Use gscraper --help or gscraper COMMAND --help for options. See the complete guide for background jobs, profiles, output files, shell completion and exit codes.
Set credentials through the environment or a local config file; command-line keys can remain in shell history. gscraper config show masks keys. Flags override environment variables, which override the selected profile.
A missing/invalid key or blocked target exits with code 4; invalid request/extraction failure uses 6, a network failure uses 7, and a wait timeout uses 8. See all exit codes when handling failures in scripts.
After configuring GEONODE_SCRAPER_API_KEY, run one extraction and save its structured response. This Bash example writes to a local file; it does not create a scheduler or a storage service.
if gscraper extract https://example.com --format markdown --json > page-result.json; then
echo "Inspect ok and result.data.markdown in page-result.json"
else
echo "Extraction failed; inspect page-result.json" >&2
fiExpected result: Successful synchronous JSON output has ok: true and content under result.data.markdown. Inspect failure output before importing it into a dataset.
Command not found? Check that the environment where you installed the package is active and that its Scripts or bin directory is on PATH. pipx manages a separate command environment.
Wrong account or endpoint? Inspect gscraper config show, which masks keys. Flags override environment variables, which override the selected configuration profile.
Need reliable scheduling? Let your scheduler inspect the command's exit code, preserve failed output and apply a bounded retry policy. A submitted background job still needs a completion check.
Does the CLI run an MCP server? No. It calls the Geonode API; the hosted MCP endpoint is a separate assistant connection.
These recipes describe source-supported workflows. They have not been run as authenticated end-to-end examples in this review.
See CONTRIBUTING.md for local checks and releases.
For API or account questions, contact Geonode. For security issues, follow SECURITY.md. Licensed under MIT.