[cachafla]

Pull Request Description

What and why?

Adds notebooks/quickstart/quickstart_documentation_run_test.ipynb, a variant of quickstart_documentation.ipynb that doesn't rely on a document template. The user creates an empty Development document. The notebook then runs each test with vm.tests.run_test() and logs it with result.log(), so the results are attached to the document without registered content blocks.

It installs validmind[xgboost,explainability] (explainability provides shap for SHAPGlobalImportance). It trains the same kind of customer churn XGBoost model as the standard quickstart, on the first 500 rows to keep the run quick, then:

• Shows vm.tests.list_tests() as a way to explore tests, and links to the Explore tests notebook.
• Runs and logs, in one loop, the same tests as the binary classification template's test blocks: data quality, descriptive statistics and correlations, model training, evaluation (including ClassifierPerformance:in_sample / :out_of_sample), explainability (PermutationFeatureImportance, SHAPGlobalImportance) and diagnosis. The inputs match what run_documentation_tests() would assign.

This gives us an end-to-end path for testing AI document drafting from logged results, without a template.

How to test

1. Register a model, add an empty Development document (no template), and put its snippet in .env.
2. Run the notebook top to bottom.
3. In the platform, the Development document should have every logged result attached, ready to insert.

Locally: in a fresh py3.11 venv with only validmind[xgboost,explainability] from this branch, all 30 tests in the final loop ran without errors on the 500-row sample. Platform logging was stubbed out, so this run didn't upload results. With [xgboost] alone, SHAPGlobalImportance fails to load, which is why the install line names both extras.

What needs special review?

• Wording in "Run and log individual tests": the tests are named explicitly because the document has no test-driven blocks, not because a template is "missing".
• The sample is trimmed to load_data().head(500) to keep the run fast, and the notebook says so. The standard quickstart uses the full dataset.
• list_tests(filter="classification", pretty=True) depends on [SC-19101] Fix list_tests crash on installs without optional extras #573 (already merged) to work on a plain validmind[xgboost] install.

Dependencies, breaking changes, and deployment notes

Depends on #573 (merged, not yet released). The docs-site entry is in validmind/documentation#1555, a draft until the release that ships this notebook. Not added to scripts/run_e2e_notebooks.py; happy to add it if we want it in the integration run.

Release notes

Added a quickstart notebook that shows how to document a model by running and logging individual tests with run_test() and result.log(), without applying a document template.

Checklist

• What and why
• Screenshots or videos (Frontend)
• How to test
• What needs special review
• Dependencies, breaking changes, and deployment notes
• Labels applied
• PR linked to Shortcut
• Unit tests added (Backend)
• Tested locally
• Documentation updated (if required)
• Environment variable additions/changes documented (if required)

🤖 Generated with Claude Code

[juanmleng]

Merge with recommendations · 0 blocking · 3 small fixes · 1 question for you

The test list matches the Binary classification template exactly, and the inputs follow the same rules as the standard quickstart.

Small fixes: FN1 install line misses explainability, so a clean install stops at SHAP and the last three tests never log · FN2 .head(500) is unexplained and shrinks the test set to about 100 rows · FN3 stale contents entry ("Apply documentation template").

❓ @cachafla — Q1: is this notebook meant for users (then it needs a docs-site entry) or mainly as a fixture for testing drafting (then how_to/ may fit better than quickstart/)?

Ready for your approval, @juanmleng.

_{calibrated-review v2 · review · run by @juanmleng · head 2564b30}

Agent handoff · calibrated-review v2 · review · 2564b30

calibrated-review: 2
mode: review
pr: validmind/validmind-library#576
head: 2564b30d3
merge_base: 6c176a831
run_by: juanmleng
verdict: merge-with-recs # merge | merge-with-recs | request-changes
downgraded: 1 # FN1: real and reproduced, but fails visibly and retryably (safe direction)
items:
  - id: FN1
    class: fix-now
    type: correctness
    direction: safe
    title: Install line lacks the explainability extra; clean installs stop at SHAPGlobalImportance
    scenario: >-
      Fresh environment, run the notebook as written. The install cell installs
      validmind[xgboost], which does not include shap. In the final loop,
      run_test("validmind.model_validation.sklearn.SHAPGlobalImportance") raises
      LoadTestError. The 26 tests before it are logged; WeakspotsDiagnosis,
      OverfitDiagnosis and RobustnessDiagnosis never run.
    evidence:
      - pyproject.toml puts shap only in the explainability (and all) extras; the xgboost extra is xgboost + scikit-learn
      - validmind/tests/run.py:321 calls load_test directly, so the skip-on-missing-dependency handling in tests/load.py _load_tests does not apply
      - "fresh py3.11 venv, pip install <head>[xgboost]: shap not installed; load_test(SHAPGlobalImportance) -> LoadTestError"
    fix: '%pip install -q "validmind[xgboost,explainability]" in the install cell'
    done_when: the install cell names both extras
  - id: FN2
    class: fix-now
    type: hygiene
    direction: safe
    title: raw_df is truncated to 500 rows with no explanation
    scenario: >-
      raw_df = customer_churn.load_data().head(500) is the only code change from
      quickstart_documentation.ipynb besides the test loop. With the 60/20/20
      split the test set is about 100 rows, so threshold, diagnosis and PSI
      results are much noisier than the standard quickstart's, and the PR body
      says it trains "the same" model.
    evidence:
      - cell 22 of the new notebook; the sibling uses load_data() with no head()
      - customer_churn.preprocess splits 80/20 and then 75/25
    fix: drop .head(500), or add one sentence to the markdown above the cell saying why it is there
    done_when: the truncation is gone or explained in the notebook text
  - id: FN3
    class: fix-now
    type: hygiene
    direction: safe
    title: Contents cell still lists "Apply documentation template"
    scenario: >-
      The contents cell links toc3_2_2 as "Apply documentation template" while
      that section is now "Create an empty document"; toc4_2 is listed with no
      toc4_1 after the preview section was removed.
    evidence:
      - notebook source lines 36 and 40 (contents cell)
    fix: regenerate the contents cell, or rename the entry and renumber
    done_when: every contents entry matches its heading
  - id: Q1
    class: question
    owner: cachafla
    title: Audience and location of the notebook
    scenario: >-
      The PR body gives its purpose as an end-to-end path for testing AI
      drafting from logged results. quickstart/ is the user-facing entry folder,
      and the docs site lists quickstart notebooks explicitly, so as merged this
      notebook is neither listed for users nor separated from them.
    fix: decide user-facing (add a docs-site entry) or fixture (consider how_to/tests/run_tests/)
accepted_risks:
  - checked: test list and inputs vs the platform's Binary classification template and customer_churn.get_demo_test_config()
    why: identical 30 test IDs, including the ClassifierPerformance in_sample/out_of_sample pair; same input-assignment rules
  - checked: the bare loop aborts on the first failing test, unlike run_documentation_tests
    why: acceptable in a teaching notebook; FN1 is the only reachable failure found
  - checked: committed outputs and copyright cell
    why: every code cell has no outputs and a null execution_count; the copyright cell is present
  - checked: vm.init document key, init_dataset and assign_predictions
    why: unchanged from quickstart_documentation.ipynb
verification:
  - run: fresh py3.11 venv, install <head>[xgboost], load_test SHAPGlobalImportance
    result: LoadTestError; shap not installed
  - claim: "the 30 test runs in the final loop all completed without errors" (PR body)
    result: NOT CHECKED
  - claim: "the Development document should have every logged result attached, ready to insert" (PR body)
    result: NOT CHECKED

[cachafla]

@juanmleng thanks, all addressed in 9727dd8:

• FN1: the install cell is now validmind[xgboost,explainability], with a sentence saying the extra is for the SHAP test. I reproduced your finding: in a fresh py3.11 venv with only [xgboost], SHAPGlobalImportance fails to load. With both extras, all 30 tests load. All 30 also ran end to end on the notebook's 500-row sample, with platform logging stubbed out.
• FN2: I kept .head(500) on purpose, to keep the run quick. The dataset section now says so, and that removing .head(500) uses the full dataset like the standard quickstart.
• FN3: I rebuilt the contents cell from the actual headings, so there's no "Apply documentation template" entry, and renumbered toc4_2 → toc4_1 to close the gap.
• Q1: it's meant for users, so it stays in quickstart/. The docs entries (sidebar + Quickstart listing) are in List the individual-tests quickstart notebook documentation#1555. That PR is a draft until the next library release: the notebook's list_tests cell depends on [SC-19101] Fix list_tests crash on installs without optional extras #573, which isn't in v2.14.0, and the notebook file reaches the docs site through make get-source for that release.

[juanmleng]

Thanks @cachafla, all verified at 9727dd8:

• FN1: the install cell now pulls in explainability, and the sentence above it explains why.
• FN2: .head(500) is now explained in the dataset section, with how to switch to the full dataset. Fine as it is.
• FN3: all 34 contents entries now match their headings and anchors.
• Q1: answered. It's for users and stays in quickstart/, and the docs entries are in validmind/documentation#1555, which waits for the release that ships #573.

CI is green and no outputs are committed. Approving.