Skip to content

feat(flows): a step fetches the runner's own secret when it starts - #450

Merged
wmobley merged 2 commits into
developfrom
feat/flow-step-secrets
Nov 20, 2026
Merged

wmobley merged 2 commits into
developfrom
feat/flow-step-secrets

Conversation

@wmobley

@wmobley wmobley commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

The rest of the design in #440 (a runner's own secrets for flow pieces); step 1, the routes where a user stores their own secrets, is #449.

  • A piece declares the secrets it needs (runtime_json.secrets) and what it sends outside DYNAMO (sends_data_to). Approval covers both. The catalog sync keeps them; a piece whose registry copy has lost a declaration is refused at plan.
  • A task input a secret fills cannot also be a run argument, a flow setting, a link or a managed input, compared without regard to case, on each step's final inputs.
  • A flow plan lists the secrets it needs and whether each is stored (secrets_ready), and what it sends outside DYNAMO. The Ensemble Manager passes these through as required_secrets, secrets_ready and sends_data_to; no value passes through it.
  • Submit refuses a missing or changed secret, a piece kind that cannot fetch one, a run whose submitter did not make the plan, and a run that binds a secret but holds a piece that is not built in, approved or the runner's own.
  • Each step bound to a secret gets a one-time credential made by the adapter for that run and step. GET /runs/{id}/steps/{step}/secrets needs the runner's token and that credential, answers once, and re-checks the piece's trust and the secret's owner, scope and version.
  • The wrapper adds the values to the piece's inputs, prints nothing secret, and masks a value echoed in the result.

From an independent security review, fixed here: cross-user submit; the fetch rate limit now counts per caller and run after the ownership check; a run binding a secret is refused while the same user has another active run holding an untrusted piece, and the other way round (this narrows, but does not close, a race through readable run arguments; documented); Tapis app steps are held to the same trust rule; the fixed inputs cannot be bound in any letter case; the signing key must be at least 32 bytes.

Needs a new setting on the adapter pod (step_secret_key, at least 32 bytes) for secrets to bind; without it a secret-bearing run is refused with 503 and nothing else changes.

Not yet run against a live service: the one-time claim on real Hasura, Tapis handing each step its own credential, a task reaching the adapter's public address.

Tests: adapter 1085 passing with the four failures that predate this work; Ensemble Manager 501, type check clean.

@vercel

vercel Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
monorepo Ready Ready Preview Nov 20, 2026 3:53am UTC

The rest of the design for a runner's own secrets for flow pieces.

- A piece declares the secrets it needs (runtime_json.secrets) and what it
  sends outside DYNAMO (sends_data_to). Approval covers both.
- A task input a secret fills cannot also be a run argument, a flow setting, a
  link or a managed input, compared without regard to case.
- A flow plan lists the secrets it needs and whether each is stored
  (secrets_ready), and what it sends outside DYNAMO. Submit refuses a missing
  or changed secret, a piece kind that cannot fetch one, and a run that binds a
  secret but holds a piece that is not built in, approved or the runner's own.
- Submit binds each step's secrets to the run by id and version. Each such
  step gets a one-time credential made by the adapter for that run and step.
- GET /runs/{id}/steps/{step}/secrets needs the runner's token and that
  credential, and answers once. The wrapper adds the values to the piece's
  inputs, prints nothing secret, and masks a value echoed in the result.
- The Ensemble Manager passes required_secrets, secrets_ready and
  sends_data_to through. No value passes through it.
- A plan records who made it; a run binding a secret is refused when its
  submitter is someone else, and when a bound secret has no record of the
  version the plan saw.
- The fetch rate limit counts per caller and run, after the ownership check,
  in a bounded table.
- A run binding a secret is refused while the same user has another active run
  holding a piece that is not built in, approved or their own, and the other
  way round; the fetch checks again. The race this narrows is documented.
- A Tapis app step, which holds the runner's token, is held to the same trust
  rule as stored code.
- The issued event is written before the one-time claim; submitting again is
  the recovery for a lost answer.
- The fixed inputs cannot be bound in any letter case, and the signing key
  must be at least 32 bytes.
@wmobley
wmobley force-pushed the feat/flow-step-secrets branch from 48effc2 to e6a781a Compare November 20, 2026 03:45
@wmobley
wmobley merged commit 3504361 into develop Nov 20, 2026
37 checks passed

This branch was successfully deployed

1 active deployment
Preview — e6a781a0 Deployed Nov 20, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant