diff --git a/SPEC.md b/SPEC.md
index c4f5a3ea..b808d66f 100644
--- a/SPEC.md
+++ b/SPEC.md
@@ -94,10 +94,11 @@ Cloud Scheduler (cron)
GCS bucket
(public read)
│
- GDAL /vsigs/ virtual FS
+ GDAL /vsigs/ (GeoJSON) · PyArrow gs:// (GeoParquet)
│
Cloud Run Service ← pygeoapi (always-on)
- (OGR provider reads GeoJSON from GCS)
+ (OGR provider: GeoJSON products;
+ Parquet provider: timeseries products)
│
HTTP clients
(OGC API - Features)
@@ -105,10 +106,11 @@ Cloud Scheduler (cron)
**Two Cloud Run deployments:**
- **Cloud Run Job** (Dagster, stateless): triggered by Cloud Scheduler, runs pipeline per product
-- **Cloud Run Service** (pygeoapi, always-on): serves OGC API - Features via GDAL OGR + GCS
+- **Cloud Run Service** (pygeoapi, always-on): serves OGC API - Features from GCS
-No PostgreSQL required. GCS is authoritative store. pygeoapi reads GeoJSON directly from
-GCS via GDAL's `/vsigs/` virtual filesystem — no proxy, no DB, no sync step.
+No PostgreSQL required. GCS is authoritative store. pygeoapi reads product files directly
+from GCS — GeoJSON via GDAL's `/vsigs/` virtual filesystem, timeseries GeoParquet via
+PyArrow's `gs://` filesystem — no proxy, no DB, no sync step (§6).
> GeoServer/PostGIS persister in `backend/persisters/geoserver.py` is unchanged and
> still usable via CLI `--output-format geoserver`. Not part of this pipeline.
@@ -261,16 +263,40 @@ Structure supports later migration to persistent Dagster daemon (change Cloud Ru
### §6.1 Role
-pygeoapi serves the GCS-stored GeoJSON files as OGC API - Features collections.
-No DB. pygeoapi uses the **OGR provider** backed by GDAL's `/vsigs/` virtual filesystem,
-which reads GeoJSON directly from GCS using Application Default Credentials on Cloud Run.
+pygeoapi serves the GCS-stored product files as OGC API - Features collections. No DB.
+The provider depends on the product's `output_type`:
+
+| Products | Provider | Source |
+|---|---|---|
+| `ogc_timeseries` | **Parquet** (`time_field: datetime`) | `gs://{bucket}/products/{id}/latest.parquet` |
+| all others | **OGR** (`source_type: GeoJSON`) | `/vsigs/{bucket}/products/{id}/latest.geojson` |
+
+Timeseries products use Parquet because the OGR provider ignores `datetime` filters and
+parses the whole GeoJSON on every request (~9–10 s per request for the 733 MB
+`nm_waterlevels_timeseries` file in local testing, vs. under 1 s from GeoParquet).
+
+The timeseries GeoParquet file must have:
+- GeoParquet `geo` metadata with WKB geometry and a **bbox covering column** (required for
+ `bbox` queries; geopandas `to_parquet(..., write_covering_bbox=True)`)
+- `datetime` as a **tz-aware UTC** timestamp
+- `string` rather than `large_string` columns (the provider can't describe `large_string`
+ fields)
+
+The pipeline does not write `latest.parquet` yet; it is uploaded manually. It must exist
+in the bucket before pygeoapi starts — the entrypoint fails if any collection can't open.
+
+Known provider limitations:
+- Parquet: date-only `datetime` queries (`2020-01-01/2020-12-31`) return 500 against a
+ tz-aware column — full RFC 3339 values (`2020-01-01T00:00:00Z/..`) work.
+ `numberMatched` is wrong.
+- OGR: no `numberReturned`; DateTime properties are returned as `YYYY/MM/DD HH:MM:SS+00`.
```
GET /collections
GET /collections/{id}/items
GET /collections/{id}/items/{feature_id}
GET /collections/{id}/items?bbox=-107,32,-103,37
-GET /collections/{id}/items?datetime=2020-01-01/2024-12-31 ← timeseries only
+GET /collections/{id}/items?datetime=2020-01-01T00:00:00Z/2024-12-31T23:59:59Z ← timeseries only
```
### §6.2 pygeoapi Config Template (`orchestration/pygeoapi/config.yml.j2`)
@@ -286,7 +312,12 @@ server:
language: en-US
cors: true
pretty_print: false
- limit: 500
+ limits:
+ default_items: 500
+ max_items: 10000 # NewWeaver requests limit=10000
+ map:
+ url: https://tile.openstreetmap.org/{z}/{x}/{y}.png
+ attribution: '© OpenStreetMap contributors'
logging:
level: ERROR
@@ -295,7 +326,11 @@ metadata:
identification:
title: NM Unified Water Data
description: OGC API - Features for New Mexico water data
- keywords: [water, groundwater, "New Mexico", NMBGMR]
+ keywords:
+ - water
+ - groundwater
+ - New Mexico
+ - NMBGMR
keywords_type: theme
terms_of_service: https://creativecommons.org/licenses/by/4.0/
url: https://waterdata.nmt.edu
@@ -305,92 +340,167 @@ metadata:
provider:
name: NM Bureau of Geology & Mineral Resources
url: https://geoinfo.nmt.edu
+ contact:
+ name: NM Bureau of Geology & Mineral Resources
+ address: 801 Leroy Place
+ city: Socorro
+ stateorprovince: New Mexico
+ postalcode: "87801"
+ country: USA
resources:
{% for product in products %}
{{ product.id }}:
type: collection
- title: {{ product.title }}
- description: {{ product.description }}
- keywords: [water, groundwater, "New Mexico"]
- extent:
+ title: {{ product.title | tojson }}
+ description: {{ product.description | tojson }}
+ keywords:
+ - water
+ - groundwater
+ - New Mexico
+ extents:
spatial:
- bbox: [-109.05, 31.33, -103.00, 37.00]
+ bbox:
+ - -109.05
+ - 31.33
+ - -103.00
+ - 37.00
crs: http://www.opengis.net/def/crs/OGC/1.3/CRS84
{% if product.output_type == 'ogc_timeseries' %}
temporal:
- interval: [["1900-01-01T00:00:00Z", null]]
+ begin: 1900-01-01T00:00:00Z # unquoted: pygeoapi needs a tz-aware datetime, not a string
+ end: null
{% endif %}
providers:
+{% if product.output_type == 'ogc_timeseries' %}
+ # GeoParquet via the Parquet provider: the OGR provider ignores datetime
+ # filters and parses the whole GeoJSON on every request. The file must be
+ # GeoParquet with a bbox covering column and a tz-aware UTC datetime.
+ - type: feature
+ name: Parquet
+ data:
+ source: gs://{{ gcs_bucket }}/products/{{ product.id }}/latest.parquet
+ id_field: id
+ time_field: datetime
+{% else %}
- type: feature
name: OGR
data:
source_type: GeoJSON
- source: /vsigs/{{ gcs_bucket }}/{{ product.id }}/latest.geojson
+ source: /vsigs/{{ gcs_bucket }}/products/{{ product.id }}/latest.geojson
source_options:
- GDAL_HTTP_UNSAFESSL: NO
+ GDAL_HTTP_UNSAFESSL: "NO"
gdal_ogr_options:
- EMPTY_AS_NULL: NO
- GDAL_CACHEMAX: 64
+ EMPTY_AS_NULL: "NO"
+ GDAL_CACHEMAX: "64"
id_field: id
- layer: OGRGeoJSON
-{% if product.output_type == 'ogc_timeseries' %}
- time_field: datetime
+ layer: latest # GDAL names a GeoJSON layer after the file basename
{% endif %}
{% endfor %}
```
+Notes on fields current pygeoapi requires (each fails at boot or first request if wrong):
+`metadata.contact.name`, `server.map`, `extents` (plural), `server.limits` (the old
+`server.limit` is deprecated and leaves the HTML items page broken), `temporal.begin`/`end`
+(not the `interval` array used in API responses; `begin` unquoted so YAML parses a
+tz-aware datetime), and `layer: latest` for GeoJSON (GDAL names the layer after the file
+basename).
+
### §6.3 Config Generation (`orchestration/pygeoapi/generate_config.py`)
```python
-import yaml
-from jinja2 import Environment, FileSystemLoader
-from pathlib import Path
-
-def generate(products_path: Path, template_path: Path, output_path: Path):
- products = yaml.safe_load(products_path.read_text())
- env = Environment(loader=FileSystemLoader(str(template_path.parent)))
+def generate(products_path: Path, template_path: Path, output_path: Path) -> None:
+ products_config = yaml.safe_load(products_path.read_text())
+ env = Environment(
+ loader=FileSystemLoader(str(template_path.parent)),
+ keep_trailing_newline=True,
+ )
tmpl = env.get_template(template_path.name)
- output_path.write_text(tmpl.render(
- products=products["products"],
- gcs_bucket=products["gcs_bucket"],
- ))
+ rendered = tmpl.render(
+ products=products_config["products"],
+ gcs_bucket=products_config["gcs_bucket"],
+ )
+
+ # Sanity check: every product must produce a GCS source entry
+ for product in products_config["products"]:
+ pid = product["id"]
+ bucket = products_config["gcs_bucket"]
+ if product.get("output_type") == "ogc_timeseries":
+ expected = f"gs://{bucket}/products/{pid}/latest.parquet"
+ else:
+ expected = f"/vsigs/{bucket}/products/{pid}/latest.geojson"
+ assert expected in rendered, (
+ f"§V violated: provider for '{pid}' missing {expected} in generated config"
+ )
+
+ output_path.parent.mkdir(parents=True, exist_ok=True)
+ output_path.write_text(rendered)
+ print(f"Generated {output_path} ({len(products_config['products'])} collections)")
```
-Run at Docker build time in `cloudbuild.yaml` — baked into image, not runtime.
+Run at Docker build time — baked into the image, not runtime.
-### §6.4 GDAL + GCS Auth
+### §6.4 GCS Auth
-On Cloud Run, GDAL `/vsigs/` uses the service account's ADC automatically.
-No credentials file needed. Require the pygeoapi Cloud Run Service account to have
-`roles/storage.objectViewer` on the `die-products` bucket.
+On Cloud Run, both GDAL `/vsigs/` and PyArrow's `gs://` filesystem use the service
+account's ADC. No credentials file needed. GDAL only uses the metadata server when it
+detects GCE, which it can't on Cloud Run — `CPL_MACHINE_IS_GCE=YES` must be set or every
+OGR collection fails with `No valid GCS credentials found`. PyArrow needs no setting. Require the pygeoapi Cloud Run
+Service account to have `roles/storage.objectViewer` on the products bucket
+(`gcs_bucket` in `products.yaml`).
-For local dev:
+For local dev, render the config and rewrite GCS paths to a mounted folder instead:
```bash
-export GOOGLE_APPLICATION_CREDENTIALS=/path/to/sa-key.json
+sed -i '' -E 's#(/vsigs/|gs://)dataservices-die-products#/data#' local.config.yml
```
### §6.5 Dockerfile (`orchestration/pygeoapi/Dockerfile`)
+Build context is `orchestration/` so `config/products.yaml` can be copied.
+
```dockerfile
FROM geopython/pygeoapi:latest
-# Generate config from products.yaml at build time
-COPY ../config/products.yaml /tmp/products.yaml
-COPY config.yml.j2 /tmp/config.yml.j2
-COPY generate_config.py /tmp/generate_config.py
-RUN python /tmp/generate_config.py \
+# pygeoapi base image includes GDAL with /vsigs/ GCS support.
+# Auth uses Application Default Credentials — no key file needed on Cloud Run.
+
+WORKDIR /pygeoapi
+
+# pygeoapi's Parquet provider needs pyarrow and geopandas, which the base
+# image doesn't ship (s3fs, shapely, pandas are already in /venv).
+# Pinned to the versions tested locally against pygeoapi 0.25.
+RUN /venv/bin/python3 -m pip install --no-cache-dir --quiet \
+ pyarrow==25.0.1 geopandas==1.1.4
+
+# Copy generation inputs
+# Build context is orchestration/ (see cloudbuild.yaml)
+COPY pygeoapi/config.yml.j2 /tmp/config.yml.j2
+COPY pygeoapi/generate_config.py /tmp/generate_config.py
+COPY config/products.yaml /tmp/products.yaml
+
+# Bake config into image at build time.
+# §V: config generated from products.yaml, not hand-edited.
+# jinja2 and PyYAML are pygeoapi dependencies, already in /venv.
+RUN /venv/bin/python3 /tmp/generate_config.py \
--products /tmp/products.yaml \
--template /tmp/config.yml.j2 \
--output /pygeoapi/local.config.yml
EXPOSE 80
+
+# pygeoapi reads PYGEOAPI_CONFIG env var; default is local.config.yml
+ENV PYGEOAPI_CONFIG=/pygeoapi/local.config.yml
```
-Cloud Run Service env vars:
-- `PYGEOAPI_SERVER_URL` — public Cloud Run URL
+Cloud Run Service (`orchestration/pygeoapi/cloudbuild.yaml`):
+- `PYGEOAPI_SERVER_URL` — `https://die-pygeoapi-$PROJECT_NUMBER.$_REGION.run.app`
+- `CPL_MACHINE_IS_GCE=YES` — lets GDAL `/vsigs/` use Cloud Run credentials (§6.4)
- Port: 80
+- Service account: `die-pygeoapi@$PROJECT_ID.iam.gserviceaccount.com` — needs
+ `roles/storage.objectViewer` on the products bucket; Cloud Build's account needs
+ `roles/iam.serviceAccountUser` on it
+- Memory: 2Gi — the Parquet provider peaked at ~1.8 GB across 4 gunicorn workers locally
---
@@ -940,6 +1050,7 @@ feature/composition-refactor ← branch off main after §T.9 merged
- All existing tests pass under `uv run pytest`
- pygeoapi config MUST be generated from `products.yaml` — never hand-edited
- pygeoapi OGR provider MUST use `/vsigs/` path (GCS), never local filesystem path
+- pygeoapi Parquet provider (`ogc_timeseries` products) MUST use `gs://` path (GCS), never local filesystem path
- No database introduced in orchestration pipeline — GCS is sole store
- `latest.geojson` MUST be overwritten atomically (upload to tmp key, then copy/rename)
diff --git a/orchestration/pygeoapi/Dockerfile b/orchestration/pygeoapi/Dockerfile
index b87d49b2..5c6a0886 100644
--- a/orchestration/pygeoapi/Dockerfile
+++ b/orchestration/pygeoapi/Dockerfile
@@ -5,15 +5,22 @@ FROM geopython/pygeoapi:latest
WORKDIR /pygeoapi
+# pygeoapi's Parquet provider needs pyarrow and geopandas, which the base
+# image doesn't ship (s3fs, shapely, pandas are already in /venv).
+# Pinned to the versions tested locally against pygeoapi 0.25.
+RUN /venv/bin/python3 -m pip install --no-cache-dir --quiet \
+ pyarrow==25.0.1 geopandas==1.1.4
+
# Copy generation inputs
-COPY config.yml.j2 /tmp/config.yml.j2
-COPY generate_config.py /tmp/generate_config.py
-COPY ../config/products.yaml /tmp/products.yaml
+# Build context is orchestration/ (see cloudbuild.yaml)
+COPY pygeoapi/config.yml.j2 /tmp/config.yml.j2
+COPY pygeoapi/generate_config.py /tmp/generate_config.py
+COPY config/products.yaml /tmp/products.yaml
# Bake config into image at build time.
# §V: config generated from products.yaml, not hand-edited.
-RUN pip install jinja2 pyyaml --quiet && \
- python /tmp/generate_config.py \
+# jinja2 and PyYAML are pygeoapi dependencies, already in /venv.
+RUN /venv/bin/python3 /tmp/generate_config.py \
--products /tmp/products.yaml \
--template /tmp/config.yml.j2 \
--output /pygeoapi/local.config.yml
diff --git a/orchestration/pygeoapi/cloudbuild.yaml b/orchestration/pygeoapi/cloudbuild.yaml
index dabf0800..b7bcb32f 100644
--- a/orchestration/pygeoapi/cloudbuild.yaml
+++ b/orchestration/pygeoapi/cloudbuild.yaml
@@ -8,7 +8,7 @@ steps:
- '$_REGION-docker.pkg.dev/$PROJECT_ID/die/die-pygeoapi:latest'
- '-f'
- 'orchestration/pygeoapi/Dockerfile'
- - 'orchestration/pygeoapi' # build context — allows COPY ../config/products.yaml
+ - 'orchestration' # build context holds pygeoapi/ and config/products.yaml
- name: 'gcr.io/cloud-builders/docker'
args:
@@ -27,12 +27,14 @@ steps:
- '--platform=managed'
- '--allow-unauthenticated'
- '--port=80'
- - '--memory=512Mi'
- - '--set-env-vars=PYGEOAPI_SERVER_URL=https://die-pygeoapi-$_SUFFIX.run.app'
+ - '--service-account=die-pygeoapi@$PROJECT_ID.iam.gserviceaccount.com' # needs storage.objectViewer on the products bucket
+ - '--memory=2Gi' # Parquet provider peaked at 1.2-2.0 GB across 4 workers locally
+ # CPL_MACHINE_IS_GCE: GDAL can't detect Cloud Run as GCE, so /vsigs/ skips the
+ # metadata-server credentials and every GeoJSON collection fails without it.
+ - '--set-env-vars=PYGEOAPI_SERVER_URL=https://die-pygeoapi-$PROJECT_NUMBER.$_REGION.run.app,CPL_MACHINE_IS_GCE=YES'
substitutions:
_REGION: us-central1
- _SUFFIX: placeholder # replace with Cloud Run URL suffix after first deploy
images:
- '$_REGION-docker.pkg.dev/$PROJECT_ID/die/die-pygeoapi:$COMMIT_SHA'
diff --git a/orchestration/pygeoapi/config.yml.j2 b/orchestration/pygeoapi/config.yml.j2
index 6eb32baf..ea00c1a9 100644
--- a/orchestration/pygeoapi/config.yml.j2
+++ b/orchestration/pygeoapi/config.yml.j2
@@ -8,15 +8,24 @@ server:
language: en-US
cors: true
pretty_print: false
- limit: 500
+ limits:
+ default_items: 500
+ max_items: 10000 # NewWeaver requests limit=10000
+ map:
+ url: https://tile.openstreetmap.org/{z}/{x}/{y}.png
+ attribution: '© OpenStreetMap contributors'
logging:
level: ERROR
metadata:
identification:
- title: NM Unified Water Data
- description: OGC API - Features for New Mexico water data
+ title: New Mexico Unified Water Data
+ description: >-
+ This tool integrates groundwater-level and water-quality data for New Mexico
+ from state and federal sources, including NMBGMR, USGS, and the Water Quality Portal.
+ Its collections provide per-well summaries, trends, time series, and derived
+ water-quality indicators through the OGC API - Features standard.
keywords:
- water
- groundwater
@@ -24,7 +33,7 @@ metadata:
- NMBGMR
keywords_type: theme
terms_of_service: https://creativecommons.org/licenses/by/4.0/
- url: https://waterdata.nmt.edu
+ url: ${PYGEOAPI_SERVER_URL} # the live Cloud Run URL, set in cloudbuild.yaml
license:
name: CC-BY 4.0
url: https://creativecommons.org/licenses/by/4.0/
@@ -32,11 +41,13 @@ metadata:
name: NM Bureau of Geology & Mineral Resources
url: https://geoinfo.nmt.edu
contact:
+ name: NM Bureau of Geology & Mineral Resources
address: 801 Leroy Place
city: Socorro
stateorprovince: New Mexico
postalcode: "87801"
country: USA
+ email: data-services-nmbg@nmt.edu
resources:
{% for product in products %}
@@ -48,7 +59,7 @@ resources:
- water
- groundwater
- New Mexico
- extent:
+ extents:
spatial:
bbox:
- -109.05
@@ -58,11 +69,21 @@ resources:
crs: http://www.opengis.net/def/crs/OGC/1.3/CRS84
{% if product.output_type == 'ogc_timeseries' %}
temporal:
- interval:
- - - "1900-01-01T00:00:00Z"
- - null
+ begin: 1900-01-01T00:00:00Z # unquoted: pygeoapi needs a tz-aware datetime, not a string
+ end: null
{% endif %}
providers:
+{% if product.output_type == 'ogc_timeseries' %}
+ # GeoParquet via the Parquet provider: the OGR provider ignores datetime
+ # filters and parses the whole GeoJSON on every request. The file must be
+ # GeoParquet with a bbox covering column and a tz-aware UTC datetime.
+ - type: feature
+ name: Parquet
+ data:
+ source: gs://{{ gcs_bucket }}/products/{{ product.id }}/latest.parquet
+ id_field: id
+ time_field: datetime
+{% else %}
- type: feature
name: OGR
data:
@@ -74,9 +95,7 @@ resources:
EMPTY_AS_NULL: "NO"
GDAL_CACHEMAX: "64"
id_field: id
- layer: OGRGeoJSON
-{% if product.output_type == 'ogc_timeseries' %}
- time_field: datetime
+ layer: latest # GDAL names a GeoJSON layer after the file basename
{% endif %}
{% endfor %}
diff --git a/orchestration/pygeoapi/generate_config.py b/orchestration/pygeoapi/generate_config.py
index a9e4a906..46102002 100644
--- a/orchestration/pygeoapi/generate_config.py
+++ b/orchestration/pygeoapi/generate_config.py
@@ -2,7 +2,8 @@
Generate pygeoapi config.yml from products.yaml + Jinja2 template.
§V: pygeoapi config MUST be generated from products.yaml — never hand-edited.
-§V: pygeoapi OGR provider MUST use /vsigs/ path (GCS).
+§V: pygeoapi OGR provider MUST use /vsigs/ path (GCS). Timeseries products use
+ the Parquet provider and read gs://.../latest.parquet instead.
Usage:
python generate_config.py \
@@ -29,13 +30,16 @@ def generate(products_path: Path, template_path: Path, output_path: Path) -> Non
gcs_bucket=products_config["gcs_bucket"],
)
- # Sanity check: every product must produce an OGR /vsigs/ entry
+ # Sanity check: every product must produce a GCS source entry
for product in products_config["products"]:
pid = product["id"]
bucket = products_config["gcs_bucket"]
- expected = f"/vsigs/{bucket}/products/{pid}/latest.geojson"
+ if product.get("output_type") == "ogc_timeseries":
+ expected = f"gs://{bucket}/products/{pid}/latest.parquet"
+ else:
+ expected = f"/vsigs/{bucket}/products/{pid}/latest.geojson"
assert expected in rendered, (
- f"§V violated: OGR provider for '{pid}' missing /vsigs/ path in generated config"
+ f"§V violated: provider for '{pid}' missing {expected} in generated config"
)
output_path.parent.mkdir(parents=True, exist_ok=True)