Skip to content

[BC-Bench] Use BCal compilation evidence in NL2AL scoring - #985

Open
Esben Nyhuus Kristoffersen (esbenk) wants to merge 3 commits into
esbenk-bcal-scenario-evidencefrom
esbenk-plan-compile-evidence
Open

Esben Nyhuus Kristoffersen (esbenk) wants to merge 3 commits into
esbenk-bcal-scenario-evidencefrom
esbenk-plan-compile-evidence

Conversation

@esbenk

@esbenk Esben Nyhuus Kristoffersen (esbenk) commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Why

BC-Bench persists the generated workspace git diff as evaluator output and raw BCal scenario evidence as context. The internal checklist scorer did not reliably use that large context payload for artifact-level assertions.

In evidence run c633665c-dea7-4777-9555-b3c262d67e58, nl2al__block-purchase-order-without-vendor-vat-2 compiled successfully against the target symbols, but the supported-event-signature assertion failed because the scored artifact exposed neither the authoritative compilation outcome nor target execution identity. Scoring needs explicit, trusted BCal facts without accepting agent-authored claims, changing persisted diffs, or altering legacy/non-BCal behavior.

What

  • Normalize validated BCal scenario results and trace events into typed execution facts, including compilation status, execution identity, evidence source, and structured terminal failure information.
  • Prefer the scenario step's compileSucceeded; use the latest correlated compilation_outcome only for valid, ordered, untruncated legacy evidence.
  • Persist normalized facts alongside the raw git diff and raw BCal context.
  • Export a BCal-only, scorer-facing envelope containing authoritative execution facts and the exact stored git diff.
  • Add scalar provenance and failure metadata for JSONL/Kusto analysis while preserving incomplete generated artifacts.
  • Leave non-BCal and evidence-free legacy exports unchanged.

How

Execution facts are derived only from validated BCal scenario output and evidence and must match the result's execution identity. The implementation never infers compilation success from assistant prose, generated AL, comments, stdout/stderr, error text, deployment transitions, or claims embedded in the diff.

For BCal NL2AL rows with trusted facts, export constructs this scorer-only shape:

{
  "authoritative_execution_facts": {
    "source": "bcal_scenario_result",
    "agent_authored": false,
    "compile_succeeded": true
  },
  "artifact": {
    "kind": "git_diff",
    "content": "<exact stored diff>"
  }
}

The raw result still stores the exact git diff as output, and raw evidence remains the bc-eval context. JSON serialization keeps artifact content inside its boundary so it cannot overwrite trusted fields. Missing, malformed, inconsistent, truncated, or identity-mismatched evidence produces an unknown compilation state rather than a success claim. Published bc-eval/Kusto rows are not accepted as raw BC-Bench result JSONL because they no longer retain the typed source boundary required for safe re-export.

Verification

  • 156 targeted tests passed across BCal provider parsing, scorer selection/export, result serialization, and result writing.
  • uv run ruff check passed.
  • uv run ruff format --check passed.
  • uv run ty check passed.
  • Full test suite: 1,325 passed, 1 skipped, 1 deselected.
  • Archived nl2al__block-purchase-order-without-vendor-vat-2 evidence normalizes to compile_succeeded=true from the latest correlated legacy compilation outcome while preserving the exact git diff.
  • Coverage includes successful and failed compilation, missing/malformed/truncated evidence, failed-then-successful compilation, execution-identity mismatch, structured incomplete failure, non-BCal providers, and evidence-free legacy results.

Stack

Targets esbenk-bcal-scenario-evidence and depends on #984. Historical re-scoring must start from retained per-instance raw result artifacts under a new run ID; already-published bc-eval/Kusto rows cannot be safely backfilled as raw inputs.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: dbb1e449-481d-469d-a112-b6878673eb0e
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: dbb1e449-481d-469d-a112-b6878673eb0e
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: dbb1e449-481d-469d-a112-b6878673eb0e
@esbenk Esben Nyhuus Kristoffersen (esbenk) changed the title Use BCal compilation evidence in NL2AL scoring [BC-Bench] Use BCal compilation evidence in NL2AL scoring Oct 27, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The legacy fallback can incorrectly publish authoritative compilation success from uncorrelated attempts, invalid event ordering, or traces with omitted events.

3 open issues
What changed in this PR

This PR builds on BCal scenario evidence support to expose compilation facts directly to the NL2AL checklist scorer while preserving stored workspace diffs and raw evidence.

Changes:

  • Normalize and persist typed compilation, execution-identity, and failure facts.
  • Add a BCal-only scoring envelope and scalar analysis metadata, leaving non-BCal and evidence-free legacy exports unchanged.
  • Expand parsing, serialization, and export tests, and document safe historical re-scoring.
File Description
tests/​test_result_serialization.py Tests execution-facts round trips and published-row rejection.
tests/​test_nl2al_panel_selection.py Tests scoring envelopes, legacy evidence, and export compatibility.
tests/​test_bcal_agent_provider.py Tests compilation normalization and context propagation.
src/​bcbench/​types.py Defines typed execution facts and compilation events.
src/​bcbench/​results/​bceval_export.py Adds scorer-facing envelopes and execution metadata.
src/​bcbench/​results/​base.py Persists execution facts and rejects published rows as raw inputs.
src/​bcbench/​commands/​evaluate.py Transfers execution facts into evaluation context.
src/​bcbench/​bcal_evidence.py Normalizes scenario and trace compilation evidence.
src/​bcbench/​agent/​bcal/​agent.py Parses compilation status and derives execution facts.
src/​bcbench/​agent/​__init__.py Reformats the existing export list.
docs/​bcal-datasets.md Documents compilation evidence and re-scoring boundaries.

Review effort: Balanced

Comment on lines +37 to +40
outcomes = [event for event in compilation_events if event.type == "compilation_outcome"]
if outcomes and outcomes[-1].succeeded is not None:
compile_succeeded = outcomes[-1].succeeded
compile_evidence_source = "trace_latest_outcome"
Comment on lines +72 to +74
truncated = trace["truncated"]
if not isinstance(truncated, bool):
raise TypeError("Trace truncated must be a boolean")
Comment on lines +80 to +88
events = tuple(
sorted(
(BCalCompilationEvent.model_validate(event) for event in raw_events if isinstance(event, dict) and event.get("type") in {"compilation_attempt", "compilation_outcome"}),
key=lambda event: event.sequence,
)
)
if any(not isinstance(event, dict) for event in raw_events):
raise TypeError("Trace events must be objects")
return "present", truncated, events

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants