DIE-359: Deploy pygeoapi on Cloud Run

[chasetmartin]

Live service: https://die-pygeoapi-95715287188.us-central1.run.app

Summary

Deploys pygeoapi as a Cloud Run service (die-pygeoapi) serving all 31 DIE products as OGC API - Features collections from GCS. It is live, and all 31 collections return features for plain and bbox item queries.

• 30 products are served as GeoJSON through pygeoapi's OGR provider (/vsigs/).
• nm_waterlevels_timeseries is served as GeoParquet through pygeoapi's Parquet provider (gs://). As GeoJSON (733 MB) every request took ~9–10 s, and the OGR provider ignores datetime filters.

A latest.parquet for the timeseries product was uploaded to the bucket manually. The pipeline doesn't produce it yet.

Changes

orchestration/pygeoapi/Dockerfile

• Build context is now orchestration/ so config/products.yaml can be copied (the old COPY ../config/... reached outside the context).
• Config is generated with /venv/bin/python3 (the Noble base image has no python and blocks system pip).
• Installs pyarrow==25.0.1 and geopandas==1.1.4 for the Parquet provider.

orchestration/pygeoapi/config.yml.j2 — fixes for fields current pygeoapi requires, each of which failed at boot or first request:

• layer: latest (GDAL names a GeoJSON layer after the file basename, not OGRGeoJSON)
• metadata.contact.name, server.map, and extents (plural)
• server.limits (default_items: 500, max_items: 10000) replacing the deprecated server.limit, which broke the HTML items page and capped pages at 10
• temporal.begin/end instead of the interval array (datetime= requests crashed with KeyError: 'begin')
• ogc_timeseries products use the Parquet provider with time_field: datetime
• Landing-page metadata: new description, identification.url from PYGEOAPI_SERVER_URL, contact email

orchestration/pygeoapi/generate_config.py — the GCS path check expects gs://.../latest.parquet for timeseries products.

orchestration/pygeoapi/cloudbuild.yaml

• Build context orchestration
• PYGEOAPI_SERVER_URL derived from $PROJECT_NUMBER/$_REGION instead of a placeholder
• CPL_MACHINE_IS_GCE=YES — without it GDAL doesn't detect Cloud Run as GCE, skips the metadata-server credentials, and every GeoJSON collection returned 500 (No valid GCS credentials found) on the first deploy
• Runs as a dedicated die-pygeoapi service account (storage.objectViewer on the products bucket)
• Memory 512Mi → 2Gi for the Parquet provider

SPEC.md — §3.3 and §6 updated to describe both providers, the GeoParquet requirements, GCS auth, and the Cloud Run settings.

Local format benchmark (nm_waterlevels_timeseries)

The 733 MB timeseries GeoJSON was converted to FlatGeobuf, GeoPackage, and GeoParquet, and the same requests were timed against each in the local container (files on local disk, no CPU limit). GeoParquet is shown in the deployed configuration (UTC timestamps, 2 GB memory).

Format (provider)	Size	Startup	limit=5	bbox	datetime	offset=100000	limit=10000	Peak mem
GeoJSON (OGR)	733 MB	10s	8.66s	10.26s	8.80s, not filtered	9.08s	10.24s	343 MiB
FlatGeobuf (OGR)	296 MB	5s	0.019s	0.013s	0.005s, not filtered	0.004s	1.36s	321 MiB
GeoPackage (OGR)	263 MB	4s	0.031s	0.017s	0.010s, not filtered	0.010s	1.36s	330 MiB
GeoParquet (Parquet) — deployed	24 MB	4s	0.22s	0.22s	0.21s, filtered ✅	0.030s	0.25s	1,838 MiB

• The OGR provider accepts datetime but never applies it, so those requests returned unfiltered results. GeoParquet is the only format where the filter works, and it returns ISO 8601 datetimes (OGR returns YYYY/MM/DD HH:MM:SS+00).
• The OGR formats ran with a 512 MB limit. GeoParquet exceeded 512 MB with pygeoapi's 4 workers (workers were killed on offset and large pages), which is why the service now has 2 GB.
• GeoParquet stores datetime as UTC so standard RFC 3339 queries work (2020-01-01T00:00:00Z/..); date-only queries return 500 (see Known issues).

On the live service, timeseries queries take ~0.8–1.5 s (reading from GCS on 1 vCPU) and a 10,000-feature page ~3.5–4 s (7.8 MB response).

Known issues

• Parquet provider: date-only datetime queries (2020-01-01/2020-12-31) return 500 against the tz-aware column; numberMatched is wrong (reported 6 for 2020; the file has 10,518 matching rows).
• Duplicate ids on the timeseries layer: id_field: id is the well id, which repeats across observations, so /items/{id} is unreliable. Needs a unique feature id column.
• Large GeoJSON layers are slow: the OGR provider re-reads the whole file from GCS on every request, so layers over ~15 MB take 4–6 s even for limit=5 (e.g. nm_well_correlation, 44 MB, ~5.8 s).
• Data: one timeseries observation is dated 2316-03-02.
• The base image is still geopython/pygeoapi:latest.

Next step

Update the processing pipeline to build GeoParquet for each layer and write latest.parquet to GCS alongside latest.geojson, then move the remaining collections to the Parquet provider. That replaces the manual upload, fixes the slow large layers, and is the place to add a unique feature id. The files should have a bbox covering column, tz-aware UTC timestamps, plain string columns, and schema_version="1.1.0".

🤖 Generated with Claude Code

[github-actions]

Your pull request is automatically being deployed to Dagster Cloud.

Location	Status	Link	Updated
die-orchestration		View in Cloud	Sep 24, 2026 at 10:00 PM (UTC)

[chasetmartin]

Note: the live die-pygeoapi endpoint is now deployed from #138, which is stacked on this PR. It includes pipeline-built GeoParquet, the Parquet provider fixes, and baked product files. Testing against the live endpoint reflects #138, not this branch alone.