Skip to content

Evaluation: keep standalone BDD tag lines in prepared testing samples - #267

Merged
Jesper Schulz-Wedde (JesperSchulz) merged 1 commit into
microsoft:mainfrom
Nikolaj1407:fixtures-keep-bdd-tags
Nov 6, 2026
Merged

Jesper Schulz-Wedde (JesperSchulz) merged 1 commit into
microsoft:mainfrom
Nikolaj1407:fixtures-keep-bdd-tags

Conversation

@Nikolaj1407

Copy link
Copy Markdown
Contributor

Second PR for #201, the testing family, following the design agreed there: one narrowly anchored global rule that keeps standalone BDD tag lines in every prepared sample, plus complete, neutral tags in the selected testing samples. The scorer, the oracle, keepComments, Good/Bad neutralization and the manifest are unchanged. Based on main at 9b72501.

Preparation rule

The text preparation that was inline in Test-ReviewFixtures.ps1 moves to tools/Review-FixtureText.ps1, dot-sourced by the harness and by its regression check, with one change to the comment-stripping expression:

before: (?m)^\s*//.*(?:\r?\n|$)
after:  (?m)^\s*//(?![ \t]*\[(?:FEATURE|SCENARIO|GIVEN|WHEN|THEN)\](?:[ \t]|\r?$)).*(?:\r?\n|$)

A full-line comment survives only when // is followed by optional spaces or tabs, one of the five uppercase tags, and then a space, a tab or the end of the line. Everything else is stripped exactly as before, including [Given], [WHENEVER], [WHEN]:, [WHEN]x, [ WHEN ], /// [WHEN], // - [WHEN] and // Note: [WHEN]. Inline comments after code were never stripped and are unchanged. Good/Bad neutralization still runs on the whole text afterwards, and keepComments cases still keep every comment.

Validation now fails when a retained tag line in a selected sample contains Good, Bad or anti-pattern (case-insensitive, word-bounded). It reads the source text, before neutralization could turn a label into Eval, and reports file:line.

Samples

Ten selected testing articles have [Test] procedures. Their 19 samples now carry [FEATURE] above each test codeunit (none for the bare-procedure fragments, which have no codeunit), [SCENARIO] first in each test, [GIVEN] on setup, one [WHEN] on the action and [THEN] on the assertion. A test with no assertion gets no [THEN], so no tag claims a check the code does not make. Where the action and the assertion are one statement (asserterror ..., Item.Get), [WHEN] and [THEN] stand together above it. Bad and good samples of a pair use the same tag text wherever their code does the same thing. No AL statement and no ordinary comment changed.

Kept deliberately:

  • test-feature-scenario-tags.bad.al stays untagged; its good sample is unchanged.
  • test-one-when-per-test.bad.al keeps its two [WHEN] blocks.
  • given-blocks-must-cover-full-precondition-chain.bad.al keeps a [GIVEN] that covers only the sales order.

Rewritten because they gave the answer away or would dangle after stripping. The explanation stays for human readers as an ordinary comment directly below, which preparation removes:

  • given-blocks-must-cover-full-precondition-chain.bad.al:8 "a sales order — posting groups left to whatever exists in the test company" became "a sales order".
  • test-one-when-per-test.bad.al:22 "second action — this is a second test in disguise" and :28 "asserting two unrelated things" became neutral descriptions of the step.
  • asserterror-needs-expectederror-and-code.good.al:16 "verify the SPECIFIC failure through a reusable Library helper" became "the blank name is rejected by the mandatory field check", the same [THEN] as the bad sample.
  • The bare [WHEN]/[THEN] lines in the given-blocks and test-one-when samples, and test-one-when-per-test.bad.al's [GIVEN] ..., got neutral descriptions.

Not in this PR: the six unselected testing articles with [Test] samples (permission-tests-must-lower-the-execution-context, test-data-must-be-random-and-complete, testisolation-belongs-on-the-test-runner, use-library-codeunits-for-test-fixtures, use-testpage-editable-to-verify-field-editability, use-testpage-visible-enabled-to-verify-field-ui-state). Their samples are not prepared, so none of their lines reaches a model today. The changed-file coverage check also rejects a change to an unselected paired article unless it is selected or waived, which would change the oracle. I reviewed them anyway: use-testpage-editable-to-verify-field-editability.bad.al:17-18 and :22, your example, do give the defect away and need the same treatment before that article is selected. I have neutral tags for all six ready and can send them with the selection, one article at a time, if you want that follow-up.

One observation for the article owner, not changed here: with tags now visible, given-blocks-must-cover-full-precondition-chain.good.al shows its posting-group chain only through LibrarySales.CreateCustomer (which assigns the gen. bus., VAT bus. and customer posting groups from existing setups) and an explicit WorkDate() posting date, and its [GIVEN] lines now say so. If the article's Best Practice ("set up the full posting-group chain") means explicit setup records, that clean control needs more setup code than a tag change can give it.

Checks

On fixtures-keep-bdd-tags, both as checked out on Windows (CRLF) and as a git archive export with LF line endings, which is what CI sees:

  • tools/Test-ReviewFixtures.ps1 with the CI arguments (-PrepareDirectory, -ChangedPathsFile from git diff --name-only main, -CoverageReportPath): 299 cases, 149/368 paired articles, 21 leaf domains; review-contract checks passed.
  • tools/Test-ReviewFixtureContexts.ps1, extended with the targeted checks below: passed, 19 tagged inputs.
  • tools/Test-ReviewContract.ps1, .github/scripts/Test-KnowledgeIndex.ps1 (437 articles, 759 samples), .github/scripts/Test-SkillIndex.ps1, validate_frontmatter.py (0 errors, 0 warnings), test_knowledge_response.py and test_knowledge_response_semantics.py: passed.

New targeted checks in Test-ReviewFixtureContexts.ps1:

  • Exact and near-miss tags, each with LF and CRLF: seven exact forms are kept and recognized; thirteen near misses and ordinary comments are stripped and not recognized; an inline tag after code is left alone.
  • Unchanged non-tag behavior: text without tag lines prepares byte-for-byte as with the previous expression, and keepComments keeps every comment and still neutralizes Good/Bad.
  • Labels: five label variants are detected and three neutral lookalikes (badge, goodwill) are not. An injected // [FEATURE] Good pricing in a scratch copy of the repository makes the real harness fail at test-one-when-per-test.good.al:1.
  • Emitted inputs: outside keepComments, no prepared case keeps any full-line comment other than a tag line. Every selected testing case is mapped to its source through its manifest ID, not through the preparer. For each one, the prepared file keeps exactly its source tag lines, removing them gives exactly the previous preparation, and no kept tag contains Eval. Every such sample with [Test] procedures has tags, except the tags positive. Each case appears once in review-request.json, request-testing.json and its request-<case>.json, all pointing at the same file.
  • Two mutations of Review-FixtureText.ps1 each fail these checks: restoring the old expression, and matching tags case-insensitively without the closing boundary.

These are preparation and contract checks only. No model was run, and no score change is claimed.

Candidate coverage

Kept tags are part of the case text the deterministic ranking reads, so some per-case candidate lists change. The selected article stays within the cutoff for every case, which the harness enforces. Compared with main, per testing case:

Case Kept tag lines (before -> after) Candidates removed Candidates added Tag article offered
asserterror-needs-expectederror-and-code.bad 0 -> 5 testisolation-belongs-on-the-test-runner table-relation-test-exclude-known-invalid-relations-via-event yes
asserterror-needs-expectederror-and-code.good 0 -> 5 use-generateguid-for-unique-test-fixture-values, use-library-codeunits-for-test-fixtures given-blocks-must-cover-full-precondition-chain, test-one-when-per-test yes
bcpt-scenarios-must-be-app-specific.bad 0 -> 0 - - yes
bcpt-scenarios-must-be-app-specific.good 0 -> 0 - - yes
commit-shared-test-fixture-inside-lazy-initialize.bad 0 -> 9 ui-test-codeunit-naming given-blocks-must-cover-full-precondition-chain yes
commit-shared-test-fixture-inside-lazy-initialize.good 0 -> 9 ui-test-codeunit-naming given-blocks-must-cover-full-precondition-chain yes
given-blocks-must-cover-full-precondition-chain.bad 0 -> 4 bcpt-scenarios-must-be-app-specific, use-library-codeunits-for-test-fixtures test-feature-scenario-tags, testisolation-belongs-on-the-test-runner yes
given-blocks-must-cover-full-precondition-chain.good 0 -> 5 bcpt-scenarios-must-be-app-specific, permission-tests-must-lower-the-execution-context test-feature-scenario-tags, use-generateguid-for-unique-test-fixture-values yes
recordref-open-temp-parameter-defeats-real-table-checks.bad 0 -> 0 - - no
recordref-open-temp-parameter-defeats-real-table-checks.good 0 -> 0 - - no
reset-per-test-state-before-the-isinitialized-guard.bad 0 -> 10 use-testpage-editable-to-verify-field-editability, use-testpage-visible-enabled-to-verify-field-ui-state given-blocks-must-cover-full-precondition-chain, test-one-when-per-test yes
reset-per-test-state-before-the-isinitialized-guard.good 0 -> 10 use-testpage-editable-to-verify-field-editability, use-testpage-visible-enabled-to-verify-field-ui-state given-blocks-must-cover-full-precondition-chain, test-one-when-per-test yes
table-relation-test-exclude-known-invalid-relations-via-event.bad 0 -> 0 - - no
table-relation-test-exclude-known-invalid-relations-via-event.good 0 -> 0 - - no
test-feature-scenario-tags.bad 0 -> 0 - - yes
test-feature-scenario-tags.good 5 -> 5 - - yes
test-one-when-per-test.bad 0 -> 5 use-library-codeunits-for-test-fixtures given-blocks-must-cover-full-precondition-chain yes
test-one-when-per-test.good 0 -> 8 bcpt-scenarios-must-be-app-specific, use-library-codeunits-for-test-fixtures given-blocks-must-cover-full-precondition-chain, test-feature-scenario-tags yes
transactionmodel-attribute-governs-test-transactions.bad 0 -> 3 - - yes
transactionmodel-attribute-governs-test-transactions.good 0 -> 6 - - yes
ui-handlers-in-tests.bad 0 -> 14 reset-per-test-state-before-the-isinitialized-guard given-blocks-must-cover-full-precondition-chain yes
ui-handlers-in-tests.good 0 -> 9 recordref-open-temp-parameter-defeats-real-table-checks, reset-per-test-state-before-the-isinitialized-guard given-blocks-must-cover-full-precondition-chain, test-one-when-per-test yes
ui-test-codeunit-naming.bad 0 -> 6 use-library-codeunits-for-test-fixtures test-one-when-per-test yes
ui-test-codeunit-naming.good 0 -> 9 reset-per-test-state-before-the-isinitialized-guard, use-library-codeunits-for-test-fixtures given-blocks-must-cover-full-precondition-chain, test-one-when-per-test yes
use-assert-isfalse-not-asserterror-for-boolean-checks.bad 0 -> 4 bcpt-scenarios-must-be-app-specific use-testpage-visible-enabled-to-verify-field-ui-state yes
use-assert-isfalse-not-asserterror-for-boolean-checks.good 0 -> 4 testisolation-belongs-on-the-test-runner use-testpage-visible-enabled-to-verify-field-ui-state yes

Overlap with #181 and #248

Both are unmerged, and this PR does not depend on them. #181 changes ui-handlers-in-tests.bad.al/.good.al and #248 changes transactionmodel-attribute-governs-test-transactions.good.al, which this PR also tags, so whichever lands second needs a small textual merge. If either lands first, I will rebase onto main, re-tag the changed tests and rerun these checks on the emitted inputs.

Retained-line inventory: every kept tag line in the changed samples, with its line number
  • asserterror-needs-expectederror-and-code.bad.al
    • 1: // [FEATURE] Customer master data
    • 11: // [SCENARIO] A customer record with a blank name raises a mandatory field error
    • 12: // [GIVEN] a customer record with a blank name
    • 16: // [WHEN] the name field is checked as mandatory
    • 17: // [THEN] the blank name is rejected by the mandatory field check
  • asserterror-needs-expectederror-and-code.good.al
    • 1: // [FEATURE] Customer master data
    • 11: // [SCENARIO] A customer record with a blank name raises a mandatory field error
    • 12: // [GIVEN] a customer record with a blank name
    • 16: // [WHEN] the name field is checked as mandatory
    • 19: // [THEN] the blank name is rejected by the mandatory field check
  • commit-shared-test-fixture-inside-lazy-initialize.bad.al
    • 1: // [FEATURE] Item master data
    • 36: // [SCENARIO] A test changes the description of the shared item and reverts the change
    • 37: // [GIVEN] the shared item fixture is initialized
    • 40: // [WHEN] the description of the shared item is changed
    • 45: // [THEN] the scratch change is rolled back
    • 57: // [SCENARIO] The shared item can be read by a second test
    • 58: // [GIVEN] the shared item fixture is initialized
    • 61: // [WHEN] the shared item is read by its number
    • 62: // [THEN] the shared item is found
  • commit-shared-test-fixture-inside-lazy-initialize.good.al
    • 1: // [FEATURE] Item master data
    • 35: // [SCENARIO] A test changes the description of the shared item and reverts the change
    • 36: // [GIVEN] the shared item fixture is initialized
    • 39: // [WHEN] the description of the shared item is changed
    • 44: // [THEN] the scratch change is rolled back
    • 55: // [SCENARIO] The shared item can be read by a second test
    • 56: // [GIVEN] the shared item fixture is initialized
    • 62: // [WHEN] the shared item is read by its number
    • 63: // [THEN] the shared item is found
  • given-blocks-must-cover-full-precondition-chain.bad.al
    • 8: // [SCENARIO] Posting a sales order results in a posted sales invoice record
    • 9: // [GIVEN] a sales order
    • 12: // [WHEN] the sales order is posted
    • 14: // [THEN] the posted sales invoice record exists
  • given-blocks-must-cover-full-precondition-chain.good.al
    • 9: // [SCENARIO] Posting a sales order results in a posted sales invoice record
    • 10: // [GIVEN] a customer with the posting groups assigned by the sales library
    • 12: // [GIVEN] a sales order for that customer with the work date as posting date
    • 16: // [WHEN] the sales order is posted
    • 18: // [THEN] the posted sales invoice record exists
  • reset-per-test-state-before-the-isinitialized-guard.bad.al
    • 1: // [FEATURE] Customer and sales setup
    • 12: // [SCENARIO] The customer card opens for the customer that is passed to the page
    • 13: // [GIVEN] the test codeunit is initialized
    • 15: // [GIVEN] a customer whose number is queued for the customer card handler
    • 19: // [WHEN] the customer card is opened for that customer
    • 22: // [THEN] the queued customer number is consumed by the handler
    • 31: // [SCENARIO] The stockout warning on the sales setup can be switched off
    • 32: // [GIVEN] the test codeunit is initialized
    • 35: // [WHEN] the stockout warning is switched off
    • 38: // [THEN] the sales setup holds the stockout warning as disabled
  • reset-per-test-state-before-the-isinitialized-guard.good.al
    • 1: // [FEATURE] Customer and sales setup
    • 12: // [SCENARIO] The customer card opens for the customer that is passed to the page
    • 13: // [GIVEN] the test codeunit is initialized
    • 15: // [GIVEN] a customer whose number is queued for the customer card handler
    • 19: // [WHEN] the customer card is opened for that customer
    • 22: // [THEN] the queued customer number is consumed by the handler
    • 31: // [SCENARIO] The stockout warning on the sales setup can be switched off
    • 32: // [GIVEN] the test codeunit is initialized
    • 35: // [WHEN] the stockout warning is switched off
    • 38: // [THEN] the sales setup holds the stockout warning as disabled
  • test-one-when-per-test.bad.al
    • 11: // [SCENARIO] Customer price list lines apply to a sales line for the customer and item
    • 12: // [GIVEN] a customer and an item with a customer price list line
    • 20: // [WHEN] a sales line is created for the customer and item
    • 23: // [WHEN] the price list line gets a minimum quantity of 10 and another price list line is created
    • 30: // [THEN] the sales line unit price matches the price list line and the price list holds two lines
  • test-one-when-per-test.good.al
    • 11: // [SCENARIO] A customer price list line sets the unit price on a new sales line
    • 12: // [GIVEN] a customer with a price list line for the item
    • 23: // [WHEN] a sales line is created for the customer and item
    • 26: // [THEN] the sales line picks up the unit price of the price list line
    • 38: // [SCENARIO] A price list with two minimum-quantity lines for one item holds both lines
    • 39: // [GIVEN] a customer price list with two minimum-quantity price lines for the same item
    • 54: // [WHEN] the lines of the price list are read
    • 56: // [THEN] the filtered set contains two price list lines
  • transactionmodel-attribute-governs-test-transactions.bad.al
    • 1: // [FEATURE] Customer maintenance
    • 12: // [SCENARIO] A customer record is created by the customer routine
    • 13: // [WHEN] the routine that creates the customer is run
  • transactionmodel-attribute-governs-test-transactions.good.al
    • 1: // [FEATURE] Customer maintenance
    • 12: // [SCENARIO] A customer record is created in the test body
    • 13: // [GIVEN] a new customer record
    • 16: // [WHEN] the customer record is inserted
    • 26: // [SCENARIO] A customer record is created by the customer routine
    • 27: // [WHEN] the routine that creates the customer is run
  • ui-handlers-in-tests.bad.al
    • 1: // [FEATURE] Customer management
    • 12: // [SCENARIO] The customer card is opened for a newly created customer
    • 13: // [GIVEN] a newly created customer and a status flag
    • 17: // [WHEN] the customer card is opened for the customer
    • 20: // [THEN] the status flag is checked
    • 31: // [SCENARIO] A message follows the customer card for a newly created customer
    • 32: // [GIVEN] a newly created customer
    • 35: // [WHEN] the customer card is opened for the customer and a message is raised afterwards
    • 46: // [SCENARIO] The customer card is opened and closed for a newly created customer
    • 47: // [GIVEN] a newly created customer
    • 50: // [WHEN] the customer card is opened for the customer
    • 60: // [SCENARIO] The customer card is opened and closed for a newly created customer
    • 61: // [GIVEN] a newly created customer
    • 64: // [WHEN] the customer card is opened for the customer
  • ui-handlers-in-tests.good.al
    • 1: // [FEATURE] Customer management
    • 12: // [SCENARIO] Opening the customer card for a customer passes that customer's number to the card handler
    • 13: // [GIVEN] a newly created customer and an empty captured customer number
    • 17: // [WHEN] the customer card is opened for the customer
    • 20: // [THEN] the customer number captured by the card handler matches the customer
    • 30: // [SCENARIO] The customer card is opened for a newly created customer
    • 31: // [GIVEN] a newly created customer and an empty captured customer number
    • 35: // [WHEN] the customer card is opened for the customer
    • 38: // [THEN] the customer number captured by the card handler matches the customer
  • ui-test-codeunit-naming.bad.al
    • 1: // [FEATURE] Sales
    • 11: // [SCENARIO] A ten percent discount lowers a unit price of 100
    • 12: // [WHEN] a 10 percent discount is applied to a unit price of 100
    • 13: // [THEN] the discounted price is 90
    • 28: // [SCENARIO] A new customer can be started from the customer card
    • 29: // [WHEN] the customer card is opened for a new record
  • ui-test-codeunit-naming.good.al
    • 1: // [FEATURE] Item pricing
    • 12: // [SCENARIO] A ten percent discount lowers a unit price of 100
    • 13: // [WHEN] a 10 percent discount is applied to a unit price of 100
    • 15: // [THEN] the discounted price is 90
    • 25: // [FEATURE] Customer card
    • 38: // [SCENARIO] A customer name entered on the customer card is saved to the customer
    • 39: // [GIVEN] a customer
    • 41: // [WHEN] the name is changed on the customer card
    • 47: // [THEN] the customer holds the new name
  • use-assert-isfalse-not-asserterror-for-boolean-checks.bad.al
    • 1: // [FEATURE] Purchase document amount verification
    • 12: // [SCENARIO] A purchase header with no lines does not pass document amount verification
    • 13: // [WHEN] the document amount of a purchase header with no lines is verified
    • 14: // [THEN] the document amount is not verified
  • use-assert-isfalse-not-asserterror-for-boolean-checks.good.al
    • 1: // [FEATURE] Purchase document amount verification
    • 12: // [SCENARIO] A purchase header with no lines does not pass document amount verification
    • 13: // [WHEN] the document amount of a purchase header with no lines is verified
    • 14: // [THEN] the document amount is not verified

Fixture preparation removed every full-line comment, so the [FEATURE],
[SCENARIO], [GIVEN], [WHEN] and [THEN] tags that three testing articles read
never reached the model, and samples written for other articles tripped
test-feature-scenario-tags. Preparation now keeps a full-line comment that
starts with one of the five exact uppercase tags followed by whitespace or the
end of the line, in every sample; all other comment stripping, keepComments,
Good/Bad neutralization and scoring are unchanged. Validation fails when a
retained tag line in a selected sample names Good, Bad or anti-pattern,
checked before neutralization.

The selected testing samples get complete, neutral tag lines. Existing tag
lines that gave the answer away or continued on a stripped line are rewritten
to one neutral line, with the explanation kept as an ordinary comment. The
deliberate multiple [WHEN] blocks and incomplete [GIVEN] setup stay, and the
test-feature-scenario-tags positive stays untagged. Relates to microsoft#201.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved content at 93336a63c804a3f227a74230830b9a8ecc0a3b77 after independent Sol and Opus reviews of the actual patch. The narrowly bounded, case-sensitive five-tag retention, pre-neutralization label guard and extracted helper preserve keepComments, Good/Bad neutralization and the scoring/oracle contracts. Full retained-prose review found no new material blocker, and the deliberately untagged, multiple-WHEN and incomplete-GIVEN controls retain their source evidence.

Actual changed-path preparation, native regressions, LF/CRLF and independently failing mutation controls support the mechanism. All primary targets remain offered; non-testing prepared inputs are unchanged. This establishes preparation behavior, not measured model recall/precision or BC runtime behavior.

The existing GIVEN pair concern you noted is separate: the standard LibrarySales.CreateSalesOrder also delegates through CreateCustomerNo to the same posting-group-assigning CreateCustomer used by the good example (pinned helper source). That pre-existing contrast deserves a bounded article/example follow-up, not a new assertion that this comment-only patch breaks the clean control or a silent oracle change here.

The four substantive hosted validation workflows remain permission-blocked and must pass before merging. No workflow execution or merge is authorized by this review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants