Repository navigation
Evaluation: keep standalone BDD tag lines in prepared testing samples - #267
Conversation
Fixture preparation removed every full-line comment, so the [FEATURE], [SCENARIO], [GIVEN], [WHEN] and [THEN] tags that three testing articles read never reached the model, and samples written for other articles tripped test-feature-scenario-tags. Preparation now keeps a full-line comment that starts with one of the five exact uppercase tags followed by whitespace or the end of the line, in every sample; all other comment stripping, keepComments, Good/Bad neutralization and scoring are unchanged. Validation fails when a retained tag line in a selected sample names Good, Bad or anti-pattern, checked before neutralization. The selected testing samples get complete, neutral tag lines. Existing tag lines that gave the answer away or continued on a stripped line are rewritten to one neutral line, with the explanation kept as an ordinary comment. The deliberate multiple [WHEN] blocks and incomplete [GIVEN] setup stay, and the test-feature-scenario-tags positive stays untagged. Relates to microsoft#201.
Jesper Schulz-Wedde (JesperSchulz)
left a comment
There was a problem hiding this comment.
Approved content at 93336a63c804a3f227a74230830b9a8ecc0a3b77 after independent Sol and Opus reviews of the actual patch. The narrowly bounded, case-sensitive five-tag retention, pre-neutralization label guard and extracted helper preserve keepComments, Good/Bad neutralization and the scoring/oracle contracts. Full retained-prose review found no new material blocker, and the deliberately untagged, multiple-WHEN and incomplete-GIVEN controls retain their source evidence.
Actual changed-path preparation, native regressions, LF/CRLF and independently failing mutation controls support the mechanism. All primary targets remain offered; non-testing prepared inputs are unchanged. This establishes preparation behavior, not measured model recall/precision or BC runtime behavior.
The existing GIVEN pair concern you noted is separate: the standard LibrarySales.CreateSalesOrder also delegates through CreateCustomerNo to the same posting-group-assigning CreateCustomer used by the good example (pinned helper source). That pre-existing contrast deserves a bounded article/example follow-up, not a new assertion that this comment-only patch breaks the clean control or a silent oracle change here.
The four substantive hosted validation workflows remain permission-blocked and must pass before merging. No workflow execution or merge is authorized by this review.
2971be3
into
microsoft:main
Second PR for #201, the testing family, following the design agreed there: one narrowly anchored global rule that keeps standalone BDD tag lines in every prepared sample, plus complete, neutral tags in the selected testing samples. The scorer, the oracle,
keepComments, Good/Bad neutralization and the manifest are unchanged. Based onmainat9b72501.Preparation rule
The text preparation that was inline in
Test-ReviewFixtures.ps1moves totools/Review-FixtureText.ps1, dot-sourced by the harness and by its regression check, with one change to the comment-stripping expression:A full-line comment survives only when
//is followed by optional spaces or tabs, one of the five uppercase tags, and then a space, a tab or the end of the line. Everything else is stripped exactly as before, including[Given],[WHENEVER],[WHEN]:,[WHEN]x,[ WHEN ],/// [WHEN],// - [WHEN]and// Note: [WHEN]. Inline comments after code were never stripped and are unchanged. Good/Bad neutralization still runs on the whole text afterwards, andkeepCommentscases still keep every comment.Validation now fails when a retained tag line in a selected sample contains
Good,Badoranti-pattern(case-insensitive, word-bounded). It reads the source text, before neutralization could turn a label intoEval, and reportsfile:line.Samples
Ten selected testing articles have
[Test]procedures. Their 19 samples now carry[FEATURE]above each test codeunit (none for the bare-procedure fragments, which have no codeunit),[SCENARIO]first in each test,[GIVEN]on setup, one[WHEN]on the action and[THEN]on the assertion. A test with no assertion gets no[THEN], so no tag claims a check the code does not make. Where the action and the assertion are one statement (asserterror ...,Item.Get),[WHEN]and[THEN]stand together above it. Bad and good samples of a pair use the same tag text wherever their code does the same thing. No AL statement and no ordinary comment changed.Kept deliberately:
test-feature-scenario-tags.bad.alstays untagged; its good sample is unchanged.test-one-when-per-test.bad.alkeeps its two[WHEN]blocks.given-blocks-must-cover-full-precondition-chain.bad.alkeeps a[GIVEN]that covers only the sales order.Rewritten because they gave the answer away or would dangle after stripping. The explanation stays for human readers as an ordinary comment directly below, which preparation removes:
given-blocks-must-cover-full-precondition-chain.bad.al:8"a sales order — posting groups left to whatever exists in the test company" became "a sales order".test-one-when-per-test.bad.al:22"second action — this is a second test in disguise" and:28"asserting two unrelated things" became neutral descriptions of the step.asserterror-needs-expectederror-and-code.good.al:16"verify the SPECIFIC failure through a reusable Library helper" became "the blank name is rejected by the mandatory field check", the same[THEN]as the bad sample.[WHEN]/[THEN]lines in the given-blocks and test-one-when samples, andtest-one-when-per-test.bad.al's[GIVEN] ..., got neutral descriptions.Not in this PR: the six unselected testing articles with
[Test]samples (permission-tests-must-lower-the-execution-context,test-data-must-be-random-and-complete,testisolation-belongs-on-the-test-runner,use-library-codeunits-for-test-fixtures,use-testpage-editable-to-verify-field-editability,use-testpage-visible-enabled-to-verify-field-ui-state). Their samples are not prepared, so none of their lines reaches a model today. The changed-file coverage check also rejects a change to an unselected paired article unless it is selected or waived, which would change the oracle. I reviewed them anyway:use-testpage-editable-to-verify-field-editability.bad.al:17-18and:22, your example, do give the defect away and need the same treatment before that article is selected. I have neutral tags for all six ready and can send them with the selection, one article at a time, if you want that follow-up.One observation for the article owner, not changed here: with tags now visible,
given-blocks-must-cover-full-precondition-chain.good.alshows its posting-group chain only throughLibrarySales.CreateCustomer(which assigns the gen. bus., VAT bus. and customer posting groups from existing setups) and an explicitWorkDate()posting date, and its[GIVEN]lines now say so. If the article's Best Practice ("set up the full posting-group chain") means explicit setup records, that clean control needs more setup code than a tag change can give it.Checks
On
fixtures-keep-bdd-tags, both as checked out on Windows (CRLF) and as agit archiveexport with LF line endings, which is what CI sees:tools/Test-ReviewFixtures.ps1with the CI arguments (-PrepareDirectory,-ChangedPathsFilefromgit diff --name-only main,-CoverageReportPath): 299 cases, 149/368 paired articles, 21 leaf domains; review-contract checks passed.tools/Test-ReviewFixtureContexts.ps1, extended with the targeted checks below: passed, 19 tagged inputs.tools/Test-ReviewContract.ps1,.github/scripts/Test-KnowledgeIndex.ps1(437 articles, 759 samples),.github/scripts/Test-SkillIndex.ps1,validate_frontmatter.py(0 errors, 0 warnings),test_knowledge_response.pyandtest_knowledge_response_semantics.py: passed.New targeted checks in
Test-ReviewFixtureContexts.ps1:keepCommentskeeps every comment and still neutralizes Good/Bad.badge,goodwill) are not. An injected// [FEATURE] Good pricingin a scratch copy of the repository makes the real harness fail attest-one-when-per-test.good.al:1.keepComments, no prepared case keeps any full-line comment other than a tag line. Every selected testing case is mapped to its source through its manifest ID, not through the preparer. For each one, the prepared file keeps exactly its source tag lines, removing them gives exactly the previous preparation, and no kept tag containsEval. Every such sample with[Test]procedures has tags, except the tags positive. Each case appears once inreview-request.json,request-testing.jsonand itsrequest-<case>.json, all pointing at the same file.Review-FixtureText.ps1each fail these checks: restoring the old expression, and matching tags case-insensitively without the closing boundary.These are preparation and contract checks only. No model was run, and no score change is claimed.
Candidate coverage
Kept tags are part of the case text the deterministic ranking reads, so some per-case candidate lists change. The selected article stays within the cutoff for every case, which the harness enforces. Compared with
main, per testing case:asserterror-needs-expectederror-and-code.badtestisolation-belongs-on-the-test-runnertable-relation-test-exclude-known-invalid-relations-via-eventasserterror-needs-expectederror-and-code.gooduse-generateguid-for-unique-test-fixture-values,use-library-codeunits-for-test-fixturesgiven-blocks-must-cover-full-precondition-chain,test-one-when-per-testbcpt-scenarios-must-be-app-specific.badbcpt-scenarios-must-be-app-specific.goodcommit-shared-test-fixture-inside-lazy-initialize.badui-test-codeunit-naminggiven-blocks-must-cover-full-precondition-chaincommit-shared-test-fixture-inside-lazy-initialize.goodui-test-codeunit-naminggiven-blocks-must-cover-full-precondition-chaingiven-blocks-must-cover-full-precondition-chain.badbcpt-scenarios-must-be-app-specific,use-library-codeunits-for-test-fixturestest-feature-scenario-tags,testisolation-belongs-on-the-test-runnergiven-blocks-must-cover-full-precondition-chain.goodbcpt-scenarios-must-be-app-specific,permission-tests-must-lower-the-execution-contexttest-feature-scenario-tags,use-generateguid-for-unique-test-fixture-valuesrecordref-open-temp-parameter-defeats-real-table-checks.badrecordref-open-temp-parameter-defeats-real-table-checks.goodreset-per-test-state-before-the-isinitialized-guard.baduse-testpage-editable-to-verify-field-editability,use-testpage-visible-enabled-to-verify-field-ui-stategiven-blocks-must-cover-full-precondition-chain,test-one-when-per-testreset-per-test-state-before-the-isinitialized-guard.gooduse-testpage-editable-to-verify-field-editability,use-testpage-visible-enabled-to-verify-field-ui-stategiven-blocks-must-cover-full-precondition-chain,test-one-when-per-testtable-relation-test-exclude-known-invalid-relations-via-event.badtable-relation-test-exclude-known-invalid-relations-via-event.goodtest-feature-scenario-tags.badtest-feature-scenario-tags.goodtest-one-when-per-test.baduse-library-codeunits-for-test-fixturesgiven-blocks-must-cover-full-precondition-chaintest-one-when-per-test.goodbcpt-scenarios-must-be-app-specific,use-library-codeunits-for-test-fixturesgiven-blocks-must-cover-full-precondition-chain,test-feature-scenario-tagstransactionmodel-attribute-governs-test-transactions.badtransactionmodel-attribute-governs-test-transactions.goodui-handlers-in-tests.badreset-per-test-state-before-the-isinitialized-guardgiven-blocks-must-cover-full-precondition-chainui-handlers-in-tests.goodrecordref-open-temp-parameter-defeats-real-table-checks,reset-per-test-state-before-the-isinitialized-guardgiven-blocks-must-cover-full-precondition-chain,test-one-when-per-testui-test-codeunit-naming.baduse-library-codeunits-for-test-fixturestest-one-when-per-testui-test-codeunit-naming.goodreset-per-test-state-before-the-isinitialized-guard,use-library-codeunits-for-test-fixturesgiven-blocks-must-cover-full-precondition-chain,test-one-when-per-testuse-assert-isfalse-not-asserterror-for-boolean-checks.badbcpt-scenarios-must-be-app-specificuse-testpage-visible-enabled-to-verify-field-ui-stateuse-assert-isfalse-not-asserterror-for-boolean-checks.goodtestisolation-belongs-on-the-test-runneruse-testpage-visible-enabled-to-verify-field-ui-stateOverlap with #181 and #248
Both are unmerged, and this PR does not depend on them. #181 changes
ui-handlers-in-tests.bad.al/.good.aland #248 changestransactionmodel-attribute-governs-test-transactions.good.al, which this PR also tags, so whichever lands second needs a small textual merge. If either lands first, I will rebase ontomain, re-tag the changed tests and rerun these checks on the emitted inputs.Retained-line inventory: every kept tag line in the changed samples, with its line number
asserterror-needs-expectederror-and-code.bad.al1:// [FEATURE] Customer master data11:// [SCENARIO] A customer record with a blank name raises a mandatory field error12:// [GIVEN] a customer record with a blank name16:// [WHEN] the name field is checked as mandatory17:// [THEN] the blank name is rejected by the mandatory field checkasserterror-needs-expectederror-and-code.good.al1:// [FEATURE] Customer master data11:// [SCENARIO] A customer record with a blank name raises a mandatory field error12:// [GIVEN] a customer record with a blank name16:// [WHEN] the name field is checked as mandatory19:// [THEN] the blank name is rejected by the mandatory field checkcommit-shared-test-fixture-inside-lazy-initialize.bad.al1:// [FEATURE] Item master data36:// [SCENARIO] A test changes the description of the shared item and reverts the change37:// [GIVEN] the shared item fixture is initialized40:// [WHEN] the description of the shared item is changed45:// [THEN] the scratch change is rolled back57:// [SCENARIO] The shared item can be read by a second test58:// [GIVEN] the shared item fixture is initialized61:// [WHEN] the shared item is read by its number62:// [THEN] the shared item is foundcommit-shared-test-fixture-inside-lazy-initialize.good.al1:// [FEATURE] Item master data35:// [SCENARIO] A test changes the description of the shared item and reverts the change36:// [GIVEN] the shared item fixture is initialized39:// [WHEN] the description of the shared item is changed44:// [THEN] the scratch change is rolled back55:// [SCENARIO] The shared item can be read by a second test56:// [GIVEN] the shared item fixture is initialized62:// [WHEN] the shared item is read by its number63:// [THEN] the shared item is foundgiven-blocks-must-cover-full-precondition-chain.bad.al8:// [SCENARIO] Posting a sales order results in a posted sales invoice record9:// [GIVEN] a sales order12:// [WHEN] the sales order is posted14:// [THEN] the posted sales invoice record existsgiven-blocks-must-cover-full-precondition-chain.good.al9:// [SCENARIO] Posting a sales order results in a posted sales invoice record10:// [GIVEN] a customer with the posting groups assigned by the sales library12:// [GIVEN] a sales order for that customer with the work date as posting date16:// [WHEN] the sales order is posted18:// [THEN] the posted sales invoice record existsreset-per-test-state-before-the-isinitialized-guard.bad.al1:// [FEATURE] Customer and sales setup12:// [SCENARIO] The customer card opens for the customer that is passed to the page13:// [GIVEN] the test codeunit is initialized15:// [GIVEN] a customer whose number is queued for the customer card handler19:// [WHEN] the customer card is opened for that customer22:// [THEN] the queued customer number is consumed by the handler31:// [SCENARIO] The stockout warning on the sales setup can be switched off32:// [GIVEN] the test codeunit is initialized35:// [WHEN] the stockout warning is switched off38:// [THEN] the sales setup holds the stockout warning as disabledreset-per-test-state-before-the-isinitialized-guard.good.al1:// [FEATURE] Customer and sales setup12:// [SCENARIO] The customer card opens for the customer that is passed to the page13:// [GIVEN] the test codeunit is initialized15:// [GIVEN] a customer whose number is queued for the customer card handler19:// [WHEN] the customer card is opened for that customer22:// [THEN] the queued customer number is consumed by the handler31:// [SCENARIO] The stockout warning on the sales setup can be switched off32:// [GIVEN] the test codeunit is initialized35:// [WHEN] the stockout warning is switched off38:// [THEN] the sales setup holds the stockout warning as disabledtest-one-when-per-test.bad.al11:// [SCENARIO] Customer price list lines apply to a sales line for the customer and item12:// [GIVEN] a customer and an item with a customer price list line20:// [WHEN] a sales line is created for the customer and item23:// [WHEN] the price list line gets a minimum quantity of 10 and another price list line is created30:// [THEN] the sales line unit price matches the price list line and the price list holds two linestest-one-when-per-test.good.al11:// [SCENARIO] A customer price list line sets the unit price on a new sales line12:// [GIVEN] a customer with a price list line for the item23:// [WHEN] a sales line is created for the customer and item26:// [THEN] the sales line picks up the unit price of the price list line38:// [SCENARIO] A price list with two minimum-quantity lines for one item holds both lines39:// [GIVEN] a customer price list with two minimum-quantity price lines for the same item54:// [WHEN] the lines of the price list are read56:// [THEN] the filtered set contains two price list linestransactionmodel-attribute-governs-test-transactions.bad.al1:// [FEATURE] Customer maintenance12:// [SCENARIO] A customer record is created by the customer routine13:// [WHEN] the routine that creates the customer is runtransactionmodel-attribute-governs-test-transactions.good.al1:// [FEATURE] Customer maintenance12:// [SCENARIO] A customer record is created in the test body13:// [GIVEN] a new customer record16:// [WHEN] the customer record is inserted26:// [SCENARIO] A customer record is created by the customer routine27:// [WHEN] the routine that creates the customer is runui-handlers-in-tests.bad.al1:// [FEATURE] Customer management12:// [SCENARIO] The customer card is opened for a newly created customer13:// [GIVEN] a newly created customer and a status flag17:// [WHEN] the customer card is opened for the customer20:// [THEN] the status flag is checked31:// [SCENARIO] A message follows the customer card for a newly created customer32:// [GIVEN] a newly created customer35:// [WHEN] the customer card is opened for the customer and a message is raised afterwards46:// [SCENARIO] The customer card is opened and closed for a newly created customer47:// [GIVEN] a newly created customer50:// [WHEN] the customer card is opened for the customer60:// [SCENARIO] The customer card is opened and closed for a newly created customer61:// [GIVEN] a newly created customer64:// [WHEN] the customer card is opened for the customerui-handlers-in-tests.good.al1:// [FEATURE] Customer management12:// [SCENARIO] Opening the customer card for a customer passes that customer's number to the card handler13:// [GIVEN] a newly created customer and an empty captured customer number17:// [WHEN] the customer card is opened for the customer20:// [THEN] the customer number captured by the card handler matches the customer30:// [SCENARIO] The customer card is opened for a newly created customer31:// [GIVEN] a newly created customer and an empty captured customer number35:// [WHEN] the customer card is opened for the customer38:// [THEN] the customer number captured by the card handler matches the customerui-test-codeunit-naming.bad.al1:// [FEATURE] Sales11:// [SCENARIO] A ten percent discount lowers a unit price of 10012:// [WHEN] a 10 percent discount is applied to a unit price of 10013:// [THEN] the discounted price is 9028:// [SCENARIO] A new customer can be started from the customer card29:// [WHEN] the customer card is opened for a new recordui-test-codeunit-naming.good.al1:// [FEATURE] Item pricing12:// [SCENARIO] A ten percent discount lowers a unit price of 10013:// [WHEN] a 10 percent discount is applied to a unit price of 10015:// [THEN] the discounted price is 9025:// [FEATURE] Customer card38:// [SCENARIO] A customer name entered on the customer card is saved to the customer39:// [GIVEN] a customer41:// [WHEN] the name is changed on the customer card47:// [THEN] the customer holds the new nameuse-assert-isfalse-not-asserterror-for-boolean-checks.bad.al1:// [FEATURE] Purchase document amount verification12:// [SCENARIO] A purchase header with no lines does not pass document amount verification13:// [WHEN] the document amount of a purchase header with no lines is verified14:// [THEN] the document amount is not verifieduse-assert-isfalse-not-asserterror-for-boolean-checks.good.al1:// [FEATURE] Purchase document amount verification12:// [SCENARIO] A purchase header with no lines does not pass document amount verification13:// [WHEN] the document amount of a purchase header with no lines is verified14:// [THEN] the document amount is not verified