Repository navigation
DIE-363: Serve pygeoapi from pipeline GeoParquet baked into the image - #138
Merged
Conversation
collection_to_geoparquet() writes a product FeatureCollection as GeoParquet in the shape pygeoapi's Parquet provider needs: - GeoParquet 1.1 metadata with a bbox covering column (bbox queries) - feature_id: the top-level GeoJSON id, made unique with :2, :3, ... suffixes (timeseries ids are source:site:date and repeat when a well is read several times a day: 499,711 distinct ids in 882,280 rows) - datetime as tz-aware UTC timestamp[ms], rows sorted by it; parsed in Python so values past pandas' 2262 limit (e.g. 2316) survive - plain string columns (the provider can't describe large_string), dict/list values JSON-encoded, and columns mixing bool/number/string stored as strings (e.g. approval_status booleans with provider text) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
GCSResource.upload_product now writes a GeoParquet copy of every
product to products/{id}/latest.parquet, stamped with the GeoJSON's
content hash.
- Unchanged GeoJSON still gets its Parquet written when it is missing
or its hash differs, so existing products backfill on their next run.
- A conversion failure never fails the product: the GeoJSON is already
published, any stale latest.parquet is deleted so pygeoapi falls back
to the GeoJSON, and the asset records parquet_status/parquet_error.
- The GeoJSON is parsed once and reused for the hash, the feature
count, and the conversion (it was parsed twice before).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
generate_config.py --check-parquet reads each product's latest.parquet footer at image build time and uses the Parquet provider when the file is usable, falling back to OGR + latest.geojson otherwise. Usable means GeoParquet metadata with a bbox covering column, point geometry only, an id column (feature_id, else id), and a tz-aware datetime for timeseries products. Polygon layers stay on GeoJSON: the Parquet provider answers bbox by containment, so county and basin polygons crossing the query box were dropped (0-2 results vs 5). A missing or unreadable file falls back to GeoJSON; permission or network errors fail the build. Cloud Build runs the check with --network=cloudbuild and CHECK_PARQUET=1, so the build account needs storage.objectViewer on the products bucket. Local builds bake a GeoJSON-only config. --source-root points at a local folder for tests. Locally, 28 of 31 layers moved to Parquet and every layer answered items, bbox, and item-by-id in 12-240 ms (vs 1-6 s from GeoJSON on the live service), peaking at 1,733 MiB under the 2 GiB limit. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Update §4.4, §5.1 and §6 for latest.parquet: how the pipeline writes and backfills it, the file shape, the build-time usability check and GeoJSON fallback, the polygon exclusion, the Cloud Build GCS access it needs, and local config rendering with --source-root. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Cloud Run doesn't compress responses. With server.gzip set, pygeoapi compresses whenever the client sends Accept-Encoding: gzip. One 5,000-row timeseries page for a single well drops from 3.85 MB to 83 KB. Clients that don't ask for gzip still get plain JSON. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Pin geopython/pygeoapi by digest (0.25.dev0). Patch the Parquet provider at build time with patch_pygeoapi.py. Each fix must match its target source exactly once, or the build fails. That way a base-image bump can't silently drop a fix. - Paging: when a filtered page spanned several batches, the provider sliced the last batch to limit + 1 rows and ignored rows already collected. Pages held up to ~9,600 rows for limit=5000 and overlapped. Now every page but the last is exactly `limit` rows. - Date-only datetime bounds (2020-01-01/2020-12-31) parsed as naive datetimes and returned 500 against the tz-aware column. They are now read as UTC. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The hydrograph client asks for one well at a time (?id=). With rows sorted by datetime alone, that filter had to scan every row group. Rows are now sorted by id, then source, then datetime, so one well sits in one or two row groups. id comes first because clients filter on id alone. With source first, row-group id ranges overlapped and well 9033 still hit 4 of 9 groups. Locally, the whole 48,809-row series for well 9033 went from 1.6-2.5 s to 1.3-1.4 s. Datetime queries across all wells can no longer skip row groups. They still return the right results. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
_sync_parquet skipped the write whenever latest.parquet carried the GeoJSON's content hash. A converter change, such as the new sort order, never reached files whose GeoJSON hadn't changed. latest.parquet is now also stamped with PARQUET_FORMAT_VERSION, and a mismatch rewrites the file. The version starts at "2", so every existing file is rewritten on its next run, including the manually uploaded timeseries file. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On Cloud Run, reading products from GCS cost ~1.2 s per request. pygeoapi builds a new provider on every request, and opening a gs:// dataset takes several round trips. generate_config.py --bake-to DIR now copies each product's chosen file into the image and points the config at the copy. It copies latest.parquet when usable, otherwise latest.geojson. A product with neither fails the build, so the previous revision keeps serving. Cloud Build sets BAKE_PRODUCTS=1. Local builds without the arg still read GCS as before. Locally, all 31 collections answer from the baked files (88 MB), and limit=10 takes 13-35 ms. New data is now served only after a rebuild. Automating that rebuild is deferred; for now it's manual. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Under a sustained load of 10,000-row pages, the base image's 4 workers pushed the container to its 2 GiB memory cap locally, with each worker at 470-650 MB. Cloud Run has no swap, so that would restart the instance on out-of-memory. With WSGI_WORKERS=2, the same load peaked at ~1.8 GiB with the same timings. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- docs/pygeoapi-performance-plan.md: the plan (tasks T1-T7), progress, local measurements, and the memory finding. It also records that T5, rebuilding automatically after a pipeline run, is deferred. - SPEC.md §1/§6/§V: products are baked into the image at build time. Updates cover the Parquet provider patch, the site-then-time sort, PARQUET_FORMAT_VERSION, and WSGI_WORKERS=2. New §6.6 says a manual rebuild is needed after pipeline runs that change data. The embedded template and Dockerfile are refreshed. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The Parquet provider built responses with geopandas __geo_interface__. That adds a collection bbox, which is [NaN, NaN, NaN, NaN] when nothing matches, plus a bbox on every feature. NaN isn't valid JSON, so any empty query failed to parse in the browser, such as ?id=xyz or a bbox with no wells. That breaks Planning regions, the inspector, and hydrographs for wells without readings. Build responses with to_geo_dict(show_bbox=False) for items and single items. This drops both bboxes and matches the OGR (GeoJSON) collections. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Item pages reported numberMatched as offset + rows returned. While more rows existed, that was one more than offset + limit, so it was an estimate on every page but the last. Count the filtered rows with the page's scanner, as resulttype=hits already does. Clients can then fetch pages in parallel without a separate count request. The next link, which compares numberMatched with offset + limit, is unaffected. The count costs ~20-110 ms of server time per request (measured live with resulttype=hits). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- SPEC.md §6.1: list the four provider patches (paging, date-only datetime, no bbox, true numberMatched). OGR layers have no numberMatched. - Plan: live measurements from the baked deploy, the client feedback and its fixes, and the WQP source-label mismatch. The source labels are left for a separate ticket. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Shorten comments and docstrings added on this branch. Each one now says what the code does. Rationale and measurements stay in SPEC.md and the performance plan. The Dockerfile patch comment is corrected (the patch now applies four fixes), and the copies of the Dockerfile and template embedded in SPEC.md are resynced. No behavior changes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
Your pull request is automatically being deployed to Dagster Cloud.
|
chasetmartin
added this pull request to stack #139
September 25, 2026 16:03
jirhiker
approved these changes
Sep 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Live service: https://die-pygeoapi-95715287188.us-central1.run.app
Stacked on #137.
What changed
latest.parquetnext tolatest.geojson. It uses GeoParquet 1.1 with a bbox covering column, has a uniquefeature_id, and stores UTC timestamps. The timeseries is sorted by well, then time. A failed conversion doesn't fail the product. The file carries a format version, so converter changes rewrite existing files.bboxfilter is containment-only.datetimequeries, which returned 500"bbox": [NaN, ...]) on empty resultsnumberMatched, which is now a true totalResults (live endpoint, this branch deployed)
Well 9033, all 48,809 readings:
A
limit=10request went from at least ~1.2 s to about 0.2 s. All 31 collections return 200 for items,bbox, and item-by-id.Before merging
docs/pygeoapi-performance-plan.mdcovers the remaining work.🤖 Generated with Claude Code