Repository navigation
[BC-Bench] Use BCal compilation evidence in NL2AL scoring - #985
Open
Esben Nyhuus Kristoffersen (esbenk) wants to merge 3 commits into
Open
Esben Nyhuus Kristoffersen (esbenk) wants to merge 3 commits into
Esben Nyhuus Kristoffersen (esbenk) wants to merge 3 commits into
Conversation
Esben Nyhuus Kristoffersen (esbenk)
force-pushed
the
esbenk-plan-compile-evidence
branch
from
October 25, 2026 13:45
4b1e0d5 to
bcee22c
Compare
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: dbb1e449-481d-469d-a112-b6878673eb0e
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: dbb1e449-481d-469d-a112-b6878673eb0e
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: dbb1e449-481d-469d-a112-b6878673eb0e
Esben Nyhuus Kristoffersen (esbenk)
force-pushed
the
esbenk-plan-compile-evidence
branch
from
October 25, 2026 15:45
bcee22c to
fd926a2
Compare
vesakr
approved these changes
Oct 27, 2026
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The legacy fallback can incorrectly publish authoritative compilation success from uncorrelated attempts, invalid event ordering, or traces with omitted events.
3 open issues
What changed in this PR
This PR builds on BCal scenario evidence support to expose compilation facts directly to the NL2AL checklist scorer while preserving stored workspace diffs and raw evidence.
Changes:
- Normalize and persist typed compilation, execution-identity, and failure facts.
- Add a BCal-only scoring envelope and scalar analysis metadata, leaving non-BCal and evidence-free legacy exports unchanged.
- Expand parsing, serialization, and export tests, and document safe historical re-scoring.
| File | Description |
|---|---|
tests/test_result_serialization.py |
Tests execution-facts round trips and published-row rejection. |
tests/test_nl2al_panel_selection.py |
Tests scoring envelopes, legacy evidence, and export compatibility. |
tests/test_bcal_agent_provider.py |
Tests compilation normalization and context propagation. |
src/bcbench/types.py |
Defines typed execution facts and compilation events. |
src/bcbench/results/bceval_export.py |
Adds scorer-facing envelopes and execution metadata. |
src/bcbench/results/base.py |
Persists execution facts and rejects published rows as raw inputs. |
src/bcbench/commands/evaluate.py |
Transfers execution facts into evaluation context. |
src/bcbench/bcal_evidence.py |
Normalizes scenario and trace compilation evidence. |
src/bcbench/agent/bcal/agent.py |
Parses compilation status and derives execution facts. |
src/bcbench/agent/__init__.py |
Reformats the existing export list. |
docs/bcal-datasets.md |
Documents compilation evidence and re-scoring boundaries. |
Review effort: Balanced
Comment on lines
+37
to
+40
| outcomes = [event for event in compilation_events if event.type == "compilation_outcome"] | ||
| if outcomes and outcomes[-1].succeeded is not None: | ||
| compile_succeeded = outcomes[-1].succeeded | ||
| compile_evidence_source = "trace_latest_outcome" |
Comment on lines
+72
to
+74
| truncated = trace["truncated"] | ||
| if not isinstance(truncated, bool): | ||
| raise TypeError("Trace truncated must be a boolean") |
Comment on lines
+80
to
+88
| events = tuple( | ||
| sorted( | ||
| (BCalCompilationEvent.model_validate(event) for event in raw_events if isinstance(event, dict) and event.get("type") in {"compilation_attempt", "compilation_outcome"}), | ||
| key=lambda event: event.sequence, | ||
| ) | ||
| ) | ||
| if any(not isinstance(event, dict) for event in raw_events): | ||
| raise TypeError("Trace events must be objects") | ||
| return "present", truncated, events |
This was referenced Nov 2, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Why
BC-Bench persists the generated workspace git diff as evaluator output and raw BCal scenario evidence as context. The internal checklist scorer did not reliably use that large context payload for artifact-level assertions.
In evidence run
c633665c-dea7-4777-9555-b3c262d67e58,nl2al__block-purchase-order-without-vendor-vat-2compiled successfully against the target symbols, but the supported-event-signature assertion failed because the scored artifact exposed neither the authoritative compilation outcome nor target execution identity. Scoring needs explicit, trusted BCal facts without accepting agent-authored claims, changing persisted diffs, or altering legacy/non-BCal behavior.What
compileSucceeded; use the latest correlatedcompilation_outcomeonly for valid, ordered, untruncated legacy evidence.How
Execution facts are derived only from validated BCal scenario output and evidence and must match the result's execution identity. The implementation never infers compilation success from assistant prose, generated AL, comments, stdout/stderr, error text, deployment transitions, or claims embedded in the diff.
For BCal NL2AL rows with trusted facts, export constructs this scorer-only shape:
{ "authoritative_execution_facts": { "source": "bcal_scenario_result", "agent_authored": false, "compile_succeeded": true }, "artifact": { "kind": "git_diff", "content": "<exact stored diff>" } }The raw result still stores the exact git diff as
output, and raw evidence remains the bc-evalcontext. JSON serialization keeps artifact content inside its boundary so it cannot overwrite trusted fields. Missing, malformed, inconsistent, truncated, or identity-mismatched evidence produces an unknown compilation state rather than a success claim. Published bc-eval/Kusto rows are not accepted as raw BC-Bench result JSONL because they no longer retain the typed source boundary required for safe re-export.Verification
uv run ruff checkpassed.uv run ruff format --checkpassed.uv run ty checkpassed.nl2al__block-purchase-order-without-vendor-vat-2evidence normalizes tocompile_succeeded=truefrom the latest correlated legacy compilation outcome while preserving the exact git diff.Stack
Targets
esbenk-bcal-scenario-evidenceand depends on #984. Historical re-scoring must start from retained per-instance raw result artifacts under a new run ID; already-published bc-eval/Kusto rows cannot be safely backfilled as raw inputs.