Repository navigation
bug(policy): auto-approval leaves unmergeable proposals pending without an actionable reason #2821
Description
User Story
As a sandbox developer using automatic policy proposals, I want safe proposals to either merge automatically or explain why they could not, so that I can iterate on a workload without repeatedly diagnosing opaque 403 failures.
Problem Statement
When a host and port already exist as an inspected endpoint for one binary, a denial from another binary can produce a mechanistic proposal for a separate generic L4 endpoint. The proposed endpoint conflicts with the existing endpoint metadata for the same host and port.
The prover can report no new findings while auto-approval subsequently fails during merge or policy validation. The proposal remains pending, but openshell rule get only shows the successful prover result. It does not expose the auto-approval failure. Repeated attempts continue failing and incrementing the proposal hit count.
Impact / Why This Matters
Auto mode appears enabled and the proposal appears safe, but the workload does not progress. Users must inspect gateway logs, infer an endpoint-ambiguity conflict, and manually author a compatible rule. This defeats the quicker deny-propose-approve-retry workflow and is especially costly for dependency resolvers that repeatedly retry the same request.
The current presentation is also misleading: prover: no new findings is shown beside a pending proposal without the separate merge-validation failure that explains why it was not approved.
Acceptance Criteria
- A mechanistic proposal for a new binary on an existing host and port preserves compatible endpoint metadata or otherwise proposes a mergeable binary expansion without weakening L7 enforcement.
- With
proposal_approval_mode=auto, a mergeable proposal with an empty prover delta and no security notes is approved, persisted as a new base-policy revision, and hot-reloaded. - If auto-approval cannot merge or validate a proposal, the failure reason is persisted with the chunk and displayed by
openshell rule get. - The displayed status distinguishes prover success from merge or policy-validation failure.
- Repeated observations deduplicate against the existing chunk without hiding the failure reason or creating an indefinite unexplained retry loop.
- If a later policy revision already covers the pending proposal, the proposal is reconciled into an appropriate terminal or superseded state.
- Existing L7 endpoint behavior is not silently downgraded to L4 passthrough.
Reproduction Steps
-
Create
repro-policy.yamlwith a REST endpoint for one binary:version: 1 network_policies: cargo_registry: name: cargo-registry endpoints: - host: index.crates.io port: 443 protocol: rest enforcement: enforce access: read-only binaries: - path: /usr/bin/cargo
-
Create a sandbox with the policy and auto approval:
openshell sandbox create \ --policy repro-policy.yaml \ --approval-mode auto \ auto-approval-repro \ -- sleep infinity openshell settings set auto-approval-repro \ --key agent_policy_proposals_enabled \ --value true
-
From a different binary, request the same host and port:
openshell sandbox exec auto-approval-repro -- \ /usr/bin/curl https://index.crates.io/config.json -
Wait for policy analysis to flush, then inspect pending rules:
sleep 15 openshell rule get auto-approval-repro --status pending
-
Observe a pending L4 proposal for
/usr/bin/curlonindex.crates.io:443withprover: no new findings. The request remains denied. Gateway logs contain an auto-approval failure, but the chunk output does not contain its reason.
Environment
- OpenShell CLI: 0.0.83
- Host OS: macOS Darwin 25.5.0, ARM64
- Compute runtime: Podman
- Policy advisor: enabled at sandbox scope
- Proposal approval mode: auto at sandbox scope
- Providers v2: enabled globally
Logs
Status: pending
Rule: allow_index_crates_io_443
Binary: /usr/bin/curl
Validation: prover: no new findings
Endpoints: index.crates.io:443 [L4]
[gateway] WARN auto-approval failed; chunk remains pending for human review
Activity
zredlined commentedon Aug 20, 2026
Related follow-up: #2841. Our long-horizon Policy Advisor experiments reproduced this failure class repeatedly and exposed the broader cause: proposal review and application do not evaluate the same complete effective-policy candidate. #2841 proposes validating the exact candidate before review and binding approval to that evaluated candidate. That should prevent the post-review endpoint-ambiguity failure described here. I am leaving this issue open as the concrete regression case and for its separate failure-persistence and presentation requirements; implementation and regression testing can determine whether one change fully resolves both.
johntmyers commentedon Aug 20, 2026
Visual explanation of the three-part failure
In plain language: OpenShell drafts the wrong kind of change, performs an incomplete safety review that looks successful, then discovers during application that the change is impossible—but only records that discovery in logs.
flowchart TD A["Existing policy<br/>cargo may use index.crates.io:443<br/>through inspected REST, read-only rules"] --> B["curl tries the same address"] B --> C["Denied<br/>The rule permits cargo, not curl"] subgraph P1["Part 1 — The proposal describes the wrong shape"] C --> D["Mechanistic mapper sees:<br/>host + port + curl"] D --> E["It does not consult the existing policy"] E --> F["Proposes a new rule:<br/>curl → generic L4 endpoint<br/>advisor_proposed = true"] G["What it should propose:<br/>Add curl to the existing REST rule<br/>and preserve its inspection settings"] end subgraph P2["Part 2 — The initial review gives a misleading green light"] F --> H["Gateway tries to combine the proposal<br/>with the existing REST rule"] H --> I{"Can it combine them<br/>without unintended changes?"} I -->|"No"| J["Safety guard keeps them separate<br/>to avoid silently widening or changing<br/>the existing cargo authorization"] J --> K["Candidate now contains two entries<br/>for index.crates.io:443:<br/>① existing REST endpoint<br/>② proposed L4 endpoint"] K --> L["Initial checks are incomplete<br/>They do not run the full endpoint-ambiguity validation"] L --> M["Prover asks:<br/>Does this introduce a modeled security finding?"] M --> N["Answer: no new findings ✅"] end subgraph P3["Part 3 — Applying uses stricter checks, then hides the failure"] N --> O["Auto-approval tries to save and hot-reload the policy"] O --> P["Full candidate validation finally runs"] P --> Q["Collision detected ❌<br/>Same host and port, incompatible provenance:<br/>advisor_proposed = false vs true"] Q --> R["Policy is not saved<br/>The original REST policy remains active"] R --> S["Failure is written only to gateway logs"] S --> T["The proposal stays pending<br/>but the CLI still shows:<br/>prover: no new findings"] T --> U["curl retries → another 403<br/>Same proposal is deduplicated<br/>Hit count increases indefinitely"] end V["User's experience:<br/>Auto mode looks enabled and safe,<br/>but nothing progresses and no reason is shown"]:::user U --> V classDef user fill:#7f1d1d,color:#fff,stroke:#ef4444,stroke-width:2px; classDef bad fill:#3f1d1d,color:#fff,stroke:#f87171; classDef misleading fill:#422006,color:#fff,stroke:#fbbf24; classDef normal fill:#172554,color:#fff,stroke:#60a5fa; class F,J,K bad; class L,M,N,T misleading; class A,B,C,D,E,H,I,O,P,R,S,U normal;The intended flow is:
flowchart LR A["curl is denied"] --> B["Consult the current effective policy"] B --> C["Convert the denial into:<br/>add curl to the existing REST rule"] C --> D["Run the complete candidate validation<br/>before showing or auto-approving it"] D --> E{"Valid?"} E -->|"Yes"| F["Save revision and hot-reload"] E -->|"No"| G["Keep pending or reject<br/>and persist the exact application error"] F --> H["curl retries through<br/>the existing REST/read-only enforcement"] G --> I["CLI shows separate results:<br/>Prover: …<br/>Application: failed because …"]johntmyers commentedon Aug 20, 2026
Proposed tracking plan
Use this issue as the implementation and regression tracking issue. #2841 remains related design context for the broader invariant: proposal review and application must evaluate the same complete effective-policy candidate.
flowchart TD A["Denied request<br/>curl → index.crates.io:443"] --> B["1. Build the canonical candidate"] B --> C{"Compatible endpoint<br/>already exists?"} C -->|"No"| D["Create a separate narrow rule"] C -->|"Yes"| E{"Who owns the<br/>existing rule?"} E -->|"Sandbox policy"| F["Create a binary expansion<br/>Add curl to the existing REST rule<br/>Preserve L7, access, TLS, and provenance"] E -->|"Provider policy"| G["Treat provider rule as immutable<br/>Read its contract, but create a compatible<br/>sandbox-owned overlay for curl"] D --> H["2. Evaluate the complete effective candidate"] F --> H G --> H H --> I["Merge and structural validation"] I --> J["Endpoint, L7, middleware,<br/>provider, and credential validation"] J --> K["Run the prover on that exact valid candidate"] K --> L{"Candidate valid and<br/>prover gate satisfied?"} L -->|"No"| M["Return and persist the exact reason<br/>Keep prover and application results separate"] L -->|"Yes"| N["3. Store the evaluation hash/token<br/>and its validation + prover result<br/>Review or auto-approve this exact candidate"] N --> O["Before applying, rebuild from live inputs<br/>and recompute the evaluation hash"] O --> P{"Evaluation hash<br/>still matches?"} P -->|"No"| Q["Inputs changed<br/>Run complete validation + prover again"] Q --> L P -->|"Yes"| R["Reuse the stored validation + prover result<br/>Do not rerun the prover"] R --> S["Atomically persist revision and hot-reload"] S --> T["4. Reconcile pending proposals<br/>Supersede anything now covered"] T --> U["Retry succeeds through the retained<br/>REST/read-only enforcement"] M --> V["CLI, TUI, and policy.local show:<br/>Prover: …<br/>Application: failed because …"]Implementation slices
Canonical proposal construction
/usr/bin/curlto the existing REST rule,” not a second L4 endpoint._provider_*rule. When a provider endpoint supplies the compatible contract, create a sandbox-owned overlay while leaving the provider layer unchanged.One complete candidate evaluator
Actionable proposal state
validation_resultandrejection_reason.policy.local, CLI, TUI, and generated SDKs.Reconciliation
Regression coverage and docs
e2e/policy-advisor/mechanistic-smoke.shand update the policy-advisor architecture and user documentation.Completion criteria
zredlined commentedon Aug 20, 2026
This looks solid and covers both the concrete failure and the broader reviewer/application mismatch. Keeping advisor provenance scoped to the new binary while retaining the existing endpoint contract is the right shape. Two implementation details to preserve: treat provider policy rules as immutable inputs and materialize any compatible expansion in the sandbox base policy; and avoid rerunning the prover when the recomputed complete-candidate hash is unchanged and the stored result is still applicable. I'm good using this issue as the implementation and regression tracker and closing #2841 as superseded.
johntmyers commentedon Aug 20, 2026
Implementation Plan
Issue type:
fixComplexity: High
Confidence: Medium — the main flow is clear; the remaining uncertainty is the boundary between gateway preflight and runtime-only supervisor validation
Summary
Build one gateway-owned proposal pipeline that canonicalizes a denial against the live effective policy, validates the complete candidate, stores that evaluation with an opaque review token, and applies only that same candidate. This fixes the cargo/curl regression and absorbs #2841's review/apply mismatch while preserving provider rules as immutable inputs and reusing prover results whenever the recomputed complete-candidate hash is unchanged.
flowchart TD A["Denial or agent-authored AddRule"] --> B["Read live base policy, immutable provider layers,<br/>credential capabilities, and relevant settings"] B --> C["Canonicalize the requested authorization"] C --> D["Build base-policy candidate<br/>and complete effective candidate"] D --> E["Run merge, structural, endpoint, L7,<br/>middleware, provider, credential, and prover checks"] E --> F{"Complete candidate valid?"} F -->|"No"| G["Return/store exact application failure<br/>Do not present it as prover-clean"] F -->|"Yes"| H["Store candidate context, review token,<br/>validator result, and prover result"] H --> I["Human/SDK approval supplies token<br/>or auto mode selects stored evaluation"] I --> J["Rebuild from live inputs<br/>and recompute token"] J --> K{"Token unchanged?"} K -->|"No"| L["FAILED_PRECONDITION<br/>Refresh evaluation and require fresh review"] K -->|"Yes"| M["Reuse stored prover result<br/>Do not rerun prover"] M --> N["CAS-write the reviewed base-policy revision"] N --> O["Hot-reload and reconcile<br/>now-covered pending chunks"]Scope
crates/openshell-server/src/grpc/policy.rs: introduce the canonical candidate evaluator and use it for submission, auto approval, single approval, bulk approval, editing, persistence, and post-write reconciliation.crates/openshell-policy/src/merge.rs: add a policy-aware canonicalization helper for compatible binary expansion and sandbox-owned overlays; never mutate or merge into_provider_*rules.crates/openshell-policy/src/lib.rs,crates/openshell-policy/src/l7_validate.rs, and the corresponding supervisor L7 validation call sites: share all deterministic endpoint/L7/middleware semantics needed to reject gateway-known invalid candidates before review; leave runtime/OPA construction as defense in depth.proto/openshell.proto: add candidate/evaluation metadata, a distinct application failure, and review tokens to chunk and approval messages.crates/openshell-server/src/policy_store.rs,crates/openshell-server/src/persistence/sqlite.rs, andcrates/openshell-server/src/persistence/postgres.rs: round-trip evaluation state and provide compare-and-set updates without losing status, failure text, or dedup hit counts. The generic protobuf payload means no relational schema migration should be necessary.crates/openshell-cli/src/run.rsandcrates/openshell-tui/src/ui/sandbox_draft.rs: show candidate status, prover result, and application failure separately; refetch and submit the current review token on approval.crates/openshell-supervisor-network/src/policy_local.rs: return candidate context/token and distinct application feedback through proposal and/waitpayloads.sdk/go/openshell/v1/types/policy.go,sdk/go/openshell/v1/internal/converter/policy.go,sdk/go/openshell/v1/policy_client.go, generated Go protobufs, and generated TypeScript raw bindings: let SDK reviewers read candidate context/token and supply the token for approval. Add a curated TypeScript policy API only if one is introduced independently; today the repository exposes this surface through raw generated bindings.e2e/policy-advisor/,architecture/security-policy.md,docs/sandboxes/policy-advisor.mdx, and relevant CLI skill/reference text: cover the regression and document the new review/apply contract and status output.Implementation Steps
Lock the canonical merge behavior with policy tests.
openshell-policyhelper that takes the proposedAddRule, current sandbox base policy, and composed provider policy and returns a canonical sandbox-base operation._provider_*.Create one complete
evaluate_proposal_candidatepath in the gateway.Define the hash and atomicity contract.
FAILED_PRECONDITION, refresh the stored evaluation, and leave the chunk pending for fresh review.apply_merge_operations_with_retrysilently rebase a reviewed proposal onto newer inputs; a conflict is stale review, not a retry opportunity.Add wire and stored proposal state.
PolicyChunk,DraftChunkPayload, andStoredDraftChunk: opaquereview_token, current/candidate policy hashes plus a canonical candidate representation/diff, and evaluator version.application_errorseparately fromvalidation_result(prover) andrejection_reason(reviewer decision).review_tokentoApproveDraftChunkRequest. ForApproveAllDraftChunksRequest, add repeated(chunk_id, review_token)selections so each human-reviewed chunk is bound explicitly; auto approval uses the stored token internally.FAILED_PRECONDITIONfor reviewable chunks, with a refetch instruction. Regenerate committed Go protobufs and ephemeral TypeScript bindings using repository tasks.Route every proposal transition through the evaluator.
Persist actionable outcomes and reconcile covered work.
policy_covers_rule(including binary and endpoint contract, not host/port alone).rejectedstate with a reason such ascovered by active policy revision N, and notify draft/policy.local watchers. A distinctsupersededenum can remain follow-up work.Update user and reviewer surfaces.
Candidate,Prover, andApplicationindependently.policy.local, Go, and TypeScript expose the exact candidate context/token used for the decision and the refreshed context after a stale-token response.Test Plan
openshell-policy: compatible cargo REST/read-only binary expansion; no L7 downgrade; provider rule immutability plus sandbox overlay; incompatible contracts rejected; malformed GraphQL/L7 and middleware shapes rejected by the shared validator; canonical coverage behavior.openshell-server: submit uses the complete evaluator; invalid agent proposal bypasses inbox; mechanistic failure persists; unchanged hash reuses an instrumented/counted prover result; changed base/provider/credential/settings/evaluator inputs produce a new token andFAILED_PRECONDITION; CAS conflict cannot apply a different candidate; edit invalidates evaluation; dedup preserves errors; later coverage reconciles terminally.policy_reloaded. Add replay fixtures for endpoint ambiguity, missing binary, and malformed GraphQL preflight failures with the active policy unchanged.e2e/policy-advisor/mechanistic-smoke.shor add a sibling test that creates the cargo policy, triggers curl, waits for auto approval/hot reload, verifies no unexplained pending chunk remains, retries successfully, and confirms the effective endpoint is still REST/read-only. Run the Docker gateway lane because this changes policy application/hot reload.mise run pre-commit,mise run test,mise run go:ci,mise run sdk:ts:ci, and the relevantmise run e2e:dockerpolicy-advisor lane before the PR.Risks & Open Questions
openshell-policyvalidation andopenshell-supervisor-network::l7::validate_l7_policies. Any deterministic rule the gateway can know must move behind a shared validator; runtime/environment-dependent checks remain defense in depth.FAILED_PRECONDITIONand coordinate CLI/TUI/Go/TypeScript releases in the same change./proc, exec labeling, Landlock, or inter-process visibility.Documentation Impact
architecture/security-policy.mdwith canonical proposal, evaluation-token, immutable-provider, and reconciliation invariants.docs/sandboxes/policy-advisor.mdxwith exact-candidate review, stale-review behavior, auto-mode prover reuse, and distinct status fields.openshell rule getor policy.local proposal payloads.docs/reference/gateway-config.mdxis not affected unless implementation unexpectedly adds configuration, which this plan does not recommend.Revision 1 — initial plan
johntmyers commentedon Aug 20, 2026
Implementation is ready in #2850.
The PR implements the tracking plan discussed above:
Verification includes the full
mise run cigate, focused Rust/Go/TypeScript tests, the standard Docker E2E, and a new #2821 Docker regression proving that adding/usr/bin/curlto an existing REST/read-only endpoint auto-approves, hot-reloads, and does not downgrade the endpoint to L4.fix(policy): bind reviews to applicable candidates
fix(policy): bind reviews to applicable candidates
fix(policy): bind reviews to applicable candidates
fix(policy): validate the applicable advisor candidate (NVIDIA#2850)