Skip to content

feat(flows): uploads go into the catalog as datasets ready to run - #454

Merged
wmobley merged 1 commit into
developfrom
feat/flow-uploads
Nov 20, 2026
Merged

wmobley merged 1 commit into
developfrom
feat/flow-uploads

Conversation

@wmobley

@wmobley wmobley commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

C4.1 of the plan in #441, reworked on the project owner's decision that a user's uploads go through CKAN ("User uploads should route through ckan. Temporary data we can store somewhere else").

Web app

  • The data step uploads to CKAN with the user's Tapis token, sent as Authorization: Bearer (CKAN's CORS allows that header, not X-Tapis-Token). The user picks an organisation from those CKAN says they can create datasets in; the dataset is private unless they tick public. For a model input it must be public, since a model's job fetches its own files.
  • A collection input takes several files; any other input takes one.
  • Each upload is checked against its input before anything is created: the input's variable on every file, the file type matching the input's format, the thread's dates and region as coverage. The dataset is read back before it is bound.
  • The data step's search includes the user's private datasets when signed in.

Adapter

  • The gather task sends the runner's token to the configured catalog host only (flow_collection_catalog_host, empty = off), compared exactly and decided again on every redirect, so it can read private files.
  • Any flow input bound to a web link is now gathered: a single file reaches the steps as a file in the run's scratch folder, where today the pieces could not read it at all.

Checked live from the dev web app: CKAN accepts the Tapis token as a bearer token; a private dataset's file downloads with it and is refused without it. Not yet run live: the multipart upload from the browser, package_create with spatial, and a real gather of a private file.

Run-time working copies and outputs stay in the run's scratch folder, as before.

Tests: adapter 1114 passing with the four failures that predate this work; web app 1404 passing, type check clean.

A user's uploads are their own data, so they become catalog datasets rather
than files in scratch, which is cleared after a time.

- The data step uploads to CKAN with the user's Tapis token: the user picks an
  organisation they can create datasets in, and the dataset is private unless
  they tick public. For a model it must be public, since a model's job fetches
  its own files.
- A collection input takes several files, any other input one. Each upload is
  checked against its input first: the input's variable on every file, the
  file type matching the input's format, the thread's dates and region as
  coverage; the dataset is read back before it is bound.
- The data step's search includes the user's private datasets when signed in.
- The gather step sends the runner's token to the configured catalog host
  only, decided again on every redirect, so it can read private files. Any
  flow input bound to a web link is now gathered, so a single file reaches the
  steps as a file in the run's scratch folder.
@vercel

vercel Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
monorepo Ready Ready Preview Nov 20, 2026 4:15am UTC

@wmobley
wmobley merged commit ab652da into develop Nov 20, 2026
41 checks passed

This branch was successfully deployed

1 active deployment
Preview — cb2cbd50 Deployed Nov 20, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant