Skip to content

[BC-Bench] Add Sales & Operations Control Tower BCal challenge - #992

Open
Esben Nyhuus Kristoffersen (esbenk) wants to merge 2 commits into
esbenk-bcal-scenario-evidencefrom
esbenk-bcal-controltower-challenge
Open

Esben Nyhuus Kristoffersen (esbenk) wants to merge 2 commits into
esbenk-bcal-scenario-evidencefrom
esbenk-bcal-controltower-challenge

Conversation

@esbenk

Copy link
Copy Markdown
Collaborator

Stacked on #984.

Why

The ControlTower3 BCal run (Service 30.0.44.53019) ended at the absolute turn safety limit after 416 tool calls, with an uncompiled trailing edit, an early false 7/7 completion, repeated completion-review gaps (allocation, simulation, prioritization, profitability, AI recommendations) and manufactured progress (e.g. a …Health version codeunit). The benchmark had no case that regresses that behavior.

What

  • Adds nl2al__collected-sales-operations-control-tower-1 to the challenge panel using the planner's original prompt verbatim.
  • 13 generic assertions (no object/file names), judged against output plus [BC-Bench] Use BCal scenario evidence in evaluations #984 scenario evidence (context):
    • genuine terminal completion (no safety/iteration-limit interruption, no unsupported blocked/discussion-only handover, no completion claim contradicted by completion review)
    • compiles after the last source mutation
    • real netting of open sales demand vs inventory and open supply documents; allocation across competing demand
    • rule-table-driven prioritization; side-effect-free what-if simulation
    • service level, profitability (price/cost/margin) and working capital impact
    • planner-facing recommendations; honest AI labeling; re-runnable evaluation
    • no credit for manufactured progress (health/version/utility objects, overclaiming)
  • New sales-operations-control-tower cohort for scoped runs.
  • Manifest: challenge count/SHA updated, version 2026-10-14.candidate-v2.bcbench-1, and a bcbench_additions provenance record (BC-Bench-local entry, to be upstreamed to BC-DeveloperExperience).
  • Docs and tests (entry/prompt identity, per-requirement critical-assertion coverage, no name-matching, manifest/cohort validity, completed vs safety-limit-interrupted export discrimination).

Validation

  • Full suite: 1304 passed, 2 skipped, 1 deselected. Ruff check/format clean.
  • Note: assertion semantics are scored by the bc-eval LLM judge; local tests verify plumbing/coverage, the baseline run verifies judge behavior.

Adds the open-ended ControlTower prompt verbatim to the challenge panel with generic, evidence-aware assertions (genuine terminal completion, compile after last mutation, real demand/supply netting and allocation, rule-driven prioritization, what-if simulation, service/profit/working-capital impact, no manufactured progress), a single-entry cohort, manifest provenance, and tests.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
… BCal timeout

- Accept evidenceContractVersion 1.0 and 1.1; persist the outcome envelope beside
  evidence in result rows and the judge context; export workStatus/interruption metadata.
- On invalid or contract-violating results, keep salvaged outcome, evidence, failure,
  metrics, generated diff and bounded raw result JSON instead of dropping everything.
- Resolve BCal timeout from run override, entry bcal_timeout_minutes, then default;
  add evaluate bcal --timeout-minutes, dataset bcal-timeout command and workflow input;
  derive the step timeout from it. Timeout rows keep partial diff and salvaged evidence.
- ControlTower entry: 75 min budget and outcome-aware terminal-completion assertion;
  bump dataset to candidate-v2.bcbench-2.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant