Skip to content

feat(flows): a flow input that takes a collection of files - #446

Merged
wmobley merged 1 commit into
developfrom
feat/flow-collection-input
Nov 20, 2026
Merged

wmobley merged 1 commit into
developfrom
feat/flow-collection-input

Conversation

@wmobley

@wmobley wmobley commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

C1(b) of the plan in #441, the backend. The web app half is a separate pull request.

A flow can mark an input as taking a collection (thread_bindings.collections). The files bound to it in a thread are then one input and one run, where several files on an input otherwise mean one run each.

  • Catalog API, adapter check: accept and validate the marker. No table change.
  • Ensemble Manager: sends a marked input's files to the adapter as one collection. An unmarked input with several files is still refused.
  • Adapter: a gather task, written by the adapter, copies the files into a folder under the run's staging folder in the runner's scratch (dynamo-runs/<run>/inputs/<gather id>/) and hands that folder to the steps that read the input. The run's id is now chosen before the pipeline is made, so the folder can be named for it. A required output_uri nobody gave defaults to the run's outputs/ folder. A run's stages list the gather step first.

The gather task fetches https links with no credential; resolves each host and connects only to public addresses it has checked, with the certificate still checked against the name; follows at most five redirects, re-checking each; counts bytes as they arrive against a limit; gives each file at most 30 minutes; and prints no token, query string or response body.

From an independent security review of this change:

  • The Tapis address was an ordinary run argument, so a submission could name the host that receives the runner's token. For a flow it is now refused from the request and set from the adapter's setting, and the gather task refuses any other host. This applies to every flow run.
  • A tapis:// file in a collection is read only from systems listed in flow_collection_tapis_systems; none by default. The file list comes from a thread others may edit and is read with the runner's token.

Not yet run against real Tapis: making the staging folder, the upload, a real fetch, and inserting a run under a chosen id in the real database. To be checked on dev.

Tests: adapter 958 passing with the four failures that predate this work; Ensemble Manager 459; catalog API 186; type checks clean.

A flow can mark an input as taking a collection (thread_bindings.collections).
The files bound to it in a thread are then one input and there is one run,
where several files on an input otherwise mean one run each.

- catalog API and the adapter's check accept and validate the marker
- Ensemble Manager sends a marked input's files to the adapter as one
  collection; an unmarked input with several files is still refused
- adapter: a gather task, written by the adapter, copies the files into a
  folder under the run's staging folder in the runner's scratch and hands that
  folder to the steps that read the input. The run's id is chosen before the
  pipeline is made, so the folder can be named for it. A required output_uri
  nobody gave defaults to the run's outputs folder
- a run's stages list the gather step first

The gather task fetches https links with no credential, connects only to
public addresses it has checked, follows at most five redirects re-checking
each, and counts bytes as they arrive against a limit. The Tapis token goes
only to the adapter's own Tapis address, which a submission can no longer
choose. A tapis:// file is read only from systems the adapter lists; none by
default.
@vercel

vercel Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
monorepo Ready Ready Preview Nov 20, 2026 2:42am UTC

@wmobley
wmobley merged commit 94fb7c5 into develop Nov 20, 2026
42 checks passed

This branch was successfully deployed

1 active deployment
Preview — 9890c235 Deployed Nov 20, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant