Repository navigation
feat(flows): uploads go into the catalog as datasets ready to run - #454
Merged
Merged
Conversation
A user's uploads are their own data, so they become catalog datasets rather than files in scratch, which is cleared after a time. - The data step uploads to CKAN with the user's Tapis token: the user picks an organisation they can create datasets in, and the dataset is private unless they tick public. For a model it must be public, since a model's job fetches its own files. - A collection input takes several files, any other input one. Each upload is checked against its input first: the input's variable on every file, the file type matching the input's format, the thread's dates and region as coverage; the dataset is read back before it is bound. - The data step's search includes the user's private datasets when signed in. - The gather step sends the runner's token to the configured catalog host only, decided again on every redirect, so it can read private files. Any flow input bound to a web link is now gathered, so a single file reaches the steps as a file in the run's scratch folder.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
C4.1 of the plan in #441, reworked on the project owner's decision that a user's uploads go through CKAN ("User uploads should route through ckan. Temporary data we can store somewhere else").
Web app
Authorization: Bearer(CKAN's CORS allows that header, notX-Tapis-Token). The user picks an organisation from those CKAN says they can create datasets in; the dataset is private unless they tick public. For a model input it must be public, since a model's job fetches its own files.Adapter
flow_collection_catalog_host, empty = off), compared exactly and decided again on every redirect, so it can read private files.Checked live from the dev web app: CKAN accepts the Tapis token as a bearer token; a private dataset's file downloads with it and is refused without it. Not yet run live: the multipart upload from the browser,
package_createwithspatial, and a real gather of a private file.Run-time working copies and outputs stay in the run's scratch folder, as before.
Tests: adapter 1114 passing with the four failures that predate this work; web app 1404 passing, type check clean.