Skip to content

Refresh BCAL gold dataset criteria and remove unused manifest - #996

Merged
Sun Haoran (haoranpb) merged 2 commits into
mainfrom
dataset/bcal-gold-criteria-fixes
Nov 9, 2026
Merged

Sun Haoran (haoranpb) merged 2 commits into
mainfrom
dataset/bcal-gold-criteria-fixes

Conversation

@haoranpb

Copy link
Copy Markdown
Collaborator

Lands the BCAL gold dataset correction that was stranded on dataset/sync-bcal-2026-10-14 (no PR), so it can serve as the base of the BCal evidence stack (#984 → #985, #992).

Changes

  • Refresh BCAL gold dataset (Rui's 9d444126, cherry-picked): evaluator assertion corrections synced from BC-DeveloperExperience candidate-v2. Only expected[].text changes, on 4 of 110 entries; prompts and IDs are unchanged. The criteria now accept valid alternative AL solutions:
    • persona-item-low-stock-warning-consultant-2: drops the "message identifies the current item" requirement
    • gl-account-needs-review-flag-2: an explicit InitValue = false is also accepted
    • collected-shipment-complaint-log-1: Item Card has no General group, so "after Item" counts as "after General"; Method = Year/Month/Count query columns are valid
    • expanded-left-outer-slot-directory-1: TableRelation with default validation satisfies the aisle check; both OrderBy forms are valid
  • Remove dataset/nl2al_manifest.json: nothing has read it since Replace NL2AL dataset panels with a generic metadata tier filter #974 replaced panels and their hash/version checks with tiers.

Dropped from the original branch

Copilot AI balanced review requested due to automatic review settings November 2, 2026 15:14
@github-code-quality

github-code-quality Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Code coverage overview

Languages: Python

Python / code-coverage/pytest

main 2b281bd dataset/bcal-gold-cr... b25bc3f +/-
88% 88% 0%

ℹ️ Calculations represent line coverage. Results may change as calculations are updated.


Updated November 09, 2026 13:09 UTC

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The changed pass criteria lack the required major benchmark version bump, allowing runs using old and corrected criteria to be aggregated together.

1 open issue
What changed in this PR

Refreshes BCAL gold evaluation criteria to accept valid alternative AL implementations, providing a corrected dataset baseline for the scenario-evidence and scoring work.

Changes:

  • Updates assertion text for 4 of 110 gold entries without changing prompts or IDs.
  • Removes the obsolete NL2AL manifest following the transition from panels to tiers.
File Description
dataset/​nl2al.jsonl Corrects criteria for notifications, Boolean defaults, complaint summaries, and aisle queries.
dataset/​nl2al_manifest.json Deletes unused dataset version, hash, and panel metadata.

Review effort: Balanced

Comment thread dataset/nl2al.jsonl Resolved
martinsrui-msft and others added 2 commits November 9, 2026 14:07
Sync evaluator assertion corrections from BC-DeveloperExperience candidate-v2.

(cherry picked from commit 9d44412; dataset version and manifest hash
changes dropped because main no longer tracks them since #974)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Nothing reads dataset/nl2al_manifest.json since #974 replaced dataset panels and their hash/version checks with tiers.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@haoranpb
Sun Haoran (haoranpb) force-pushed the dataset/bcal-gold-criteria-fixes branch from 46dfa79 to b25bc3f Compare November 9, 2026 13:08
Sun Haoran (haoranpb) added a commit that referenced this pull request Nov 9, 2026
Linearize #1000 on the refreshed #996 base while retaining the current base changes.

Co-authored-by: esbenk <esbenk@microsoft.com>
Co-authored-by: Rui Martins <martinsrui@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@haoranpb
Sun Haoran (haoranpb) merged commit c27c3f2 into main Nov 9, 2026
17 checks passed
@haoranpb
Sun Haoran (haoranpb) deleted the dataset/bcal-gold-criteria-fixes branch November 9, 2026 13:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants