Repository navigation
feat(flows): a flow input that takes a collection of files - #446
Merged
Merged
Conversation
A flow can mark an input as taking a collection (thread_bindings.collections). The files bound to it in a thread are then one input and there is one run, where several files on an input otherwise mean one run each. - catalog API and the adapter's check accept and validate the marker - Ensemble Manager sends a marked input's files to the adapter as one collection; an unmarked input with several files is still refused - adapter: a gather task, written by the adapter, copies the files into a folder under the run's staging folder in the runner's scratch and hands that folder to the steps that read the input. The run's id is chosen before the pipeline is made, so the folder can be named for it. A required output_uri nobody gave defaults to the run's outputs folder - a run's stages list the gather step first The gather task fetches https links with no credential, connects only to public addresses it has checked, follows at most five redirects re-checking each, and counts bytes as they arrive against a limit. The Tapis token goes only to the adapter's own Tapis address, which a submission can no longer choose. A tapis:// file is read only from systems the adapter lists; none by default.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
C1(b) of the plan in #441, the backend. The web app half is a separate pull request.
A flow can mark an input as taking a collection (
thread_bindings.collections). The files bound to it in a thread are then one input and one run, where several files on an input otherwise mean one run each.dynamo-runs/<run>/inputs/<gather id>/) and hands that folder to the steps that read the input. The run's id is now chosen before the pipeline is made, so the folder can be named for it. A requiredoutput_urinobody gave defaults to the run'soutputs/folder. A run's stages list the gather step first.The gather task fetches
httpslinks with no credential; resolves each host and connects only to public addresses it has checked, with the certificate still checked against the name; follows at most five redirects, re-checking each; counts bytes as they arrive against a limit; gives each file at most 30 minutes; and prints no token, query string or response body.From an independent security review of this change:
tapis://file in a collection is read only from systems listed inflow_collection_tapis_systems; none by default. The file list comes from a thread others may edit and is read with the runner's token.Not yet run against real Tapis: making the staging folder, the upload, a real fetch, and inserting a run under a chosen id in the real database. To be checked on dev.
Tests: adapter 958 passing with the four failures that predate this work; Ensemble Manager 459; catalog API 186; type checks clean.