Skip to content

feat: upload a dataset page, flow runs record their inputs and settings, sub-tasks under every task - #461

Merged
wmobley merged 4 commits into
developfrom
feat/ui-uploads-provenance-subtasks
Nov 20, 2026
Merged

wmobley merged 4 commits into
developfrom
feat/ui-uploads-provenance-subtasks

Conversation

@wmobley

@wmobley wmobley commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

Three changes asked for during today's live flow tests, in one PR so dev runs a single build.

1. Upload a dataset (Datasets → "Upload a dataset", /datasets/upload)

A signed-in user loads one file, or a collection of files, into CKAN, ready for DYNAMO to run, without starting from a thread.

  • Dataset: an organisation they can publish to, private by default with a "Make public" box, plus a title and description. The page explains that a model's own job reads only public files, and a flow can read private ones.
  • Ready to run:
    • a standard variable from the existing picker, written on every file of a collection;
    • the format from the file extensions, with a warning on mixed types;
    • required start and end dates;
    • spatial coverage that is optional: none, a MINT region, or a bounding box.
  • Before upload: a preview of the registered flows and models with an input of that variable, and whether the format fits. It reuses the app's model tree query; nothing goes to the adapter.
  • After upload: the dataset is read back, then shown with links to its page and to "Use in a thread".
  • Shared code: it reuses uploadFilesToCatalog from feat(flows): uploads go into the catalog as datasets ready to run #454, which gains optional title and spatial params and exports boundingBoxToSpatial. Thread uploads behave as before.

2. Flow runs record what they read and the settings they used (Ensemble Manager)

  • The gap: a flow run's execution row had results but no data_bindings and no parameter_bindings. The Results page showed "—" for every input, and the run's provenance was missing what it read.
  • Inputs: submitFlow builds the record from the adapter's stored plan. A collection input gets one execution_data_binding per resource.
  • Settings: user-set and flow-fixed settings are recorded. A linked setting is stored as a reference such as linked: step 2 output 0.
  • Secrets: a value is withheld when the plan's secrets list it, when its name looks like a credential, or when the value looks like a token.
  • Resumed runs: a resumed run records the edited file the user uploaded.
  • Failure handling: recording never blocks a run. If the binding write is refused, the execution is recorded without bindings.
  • UI: the Results table shows a collection input as "N files" with the file names.
  • Backfill: mint-ensemble-manager/scripts/backfill-flow-provenance.ts.
    • It is a dry run unless --apply is given, it only inserts, with ON CONFLICT DO NOTHING, and it takes the admin secret from the environment.
    • It has not been run against dev.
    • It rebuilds runs whose adapter plan still exists, and reports the rest as partial or skipped.

3. Sub-tasks under every task (problem statement page)

  • Every task lists its own sub-tasks again.
  • Each list collapses and expands with a chevron, using the existing Radix accordion with a height animation. The animation is off under prefers-reduced-motion.
  • Selecting a sub-task under any task also selects that task, and a ?threadId= deep link opens its task.
  • Creating a sub-task works under any task.
  • tailwind.config.ts gains the accordion-down and accordion-up keyframes, which ui/accordion.tsx already used, so the existing Accordion now animates too.

Tests

  • Ensemble Manager: 623 passed; tsc clean.
    • CI does not lint the Ensemble Manager, which carries existing no-explicit-any errors. The new files follow the service's existing style, and the changed files add two anys.
  • UI: 1538 passed; tsc, eslint and prettier clean.
  • Not run against live CKAN: the upload page's network paths, and the accordion animation, which jsdom can't render. Both are checked in the browser after deploy.

🤖 Generated with Claude Code

Loads one file or a collection into CKAN with a standard variable, dates
and spatial coverage written as the thread upload writes them, previews
the flows and models that take the variable, and reads the dataset back
before reporting success.
…t used

A flow run's execution row had results but empty data and parameter bindings,
so the Results page showed a dash for every input. Record one data binding per
resource (a collection input is several rows under one input), the settings the
run used as parameter bindings (a linked setting as a reference, never a
credential), and the file a person edited at a pause as an input of the resumed
run. Results shows a collection input as an N files list. Add a dry-run-by-default
backfill script for runs already recorded.
Each task on the problem statement page now lists its own sub-tasks, with a
chevron to collapse or expand them (Radix Accordion, animated height, honours
prefers-reduced-motion). Selecting a sub-task under any task also selects that
task, and sub-tasks can be created under any task. Adds the accordion
keyframes the shared accordion component already referenced.
@vercel

vercel Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
monorepo Ready Ready Preview Nov 20, 2026 1:43pm UTC

This branch was successfully deployed

1 active deployment
Preview — ffc34605 Deployed Nov 20, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant