[juanmleng]

Pull Request Description

What and why?

Adds a quickstart for documenting and validating a linear regression model end to end, and the tests a validator needs for it.

• New notebook and template: a credit card balance model is developed, compared with a challenger model, and validated — data, model specification, assumptions, uncertainty, sensitivity, segments, and stability over time. The template ships next to the notebook.
• New tests for linear regression models: 18 tests covering the checks a validator makes that the library did not have — for example multicollinearity, influential observations, prediction intervals, stress scenarios, bias by segment, and performance over time. They work with any linear regression model.
• Synthetic dataset: a bundled credit card balance dataset, with the script that generates it.
• Fixes: the coefficient chart drew its confidence intervals incorrectly; the regression model summary test failed on recent scikit-learn; generating test descriptions could hang in Jupyter when a test produced several Plotly figures; and the existing regression quickstart pointed users to the wrong template.

How to test

• Unit tests: make test-unit, or the new files under tests/unit_tests/model_validation/ and tests/unit_tests/model_validation/statsmodels/.
• Notebook: apply notebooks/use_cases/regression/linear_regression_template.yaml to a model, then run notebooks/use_cases/regression/quickstart_linear_regression.ipynb top to bottom. Expect no cell errors and every template section populated.

What needs special review?

• The PR is large on purpose, so the use case can be seen working end to end. Suggested reading order: the fixes (RegressionCoeffs, RegressionModelSummary, validmind/ai/test_descriptions.py), then the shared helpers (_segments.py, statsutils.py), the statsmodels tests, the model-agnostic tests, and finally the template and notebook.
• When LLM descriptions are on, figures are now rendered before the description jobs start. This fixes the hang in Jupyter, but makes run_test slightly slower.
• Should this template become a built-in model documentation template? It currently ships only as a YAML file next to the notebook, so users have to upload it themselves. Making it available to every organization, next to the existing Regression template, would be a separate backend change.

Dependencies, breaking changes, and deployment notes

Release notes

New linear regression use case: a quickstart notebook and template for documenting and validating a linear regression model, 18 new tests for linear regression models, and a synthetic credit card balance dataset. Also fixes the confidence-interval bars in RegressionCoeffs and a RegressionModelSummary failure on scikit-learn 1.6 and later.

Checklist

• What and why
• Screenshots or videos (Frontend)
• How to test
• What needs special review
• Dependencies, breaking changes, and deployment notes
• Labels applied
• PR linked to Shortcut
• Unit tests added (Backend)
• Tested locally
• Documentation updated (if required)
• Environment variable additions/changes documented (if required)

[cachafla]

@juanmleng thanks for this. The fixes to existing tests are correct: the RegressionCoeffs error bars are now distances rather than bounds, RegressionModelSummary works on recent scikit-learn, and the Regression template link in the existing quickstart is fixed. All 50 tests in the template match tests the notebook runs, and the notebook outputs are cleared.

Requesting changes for the new tests below. Each bug was reproduced with a short script against 49bebf44. They mostly show up on inputs the bundled dataset never hits: missing values, models fitted on numpy arrays, and models already fitted with robust errors.

Bugs

1. validmind/tests/model_validation/statsmodels/statsutils.py:67-71 (design_data): models fitted on plain numpy arrays (sm.OLS(y.values, sm.add_constant(X.values))) get regressor names x1, x2, which aren't dataset columns. df[exog_names] then raises KeyError, breaking PredictionIntervals, FeatureSensitivity and ScenarioAnalysis. The docstring says array models are supported, but only array models fitted on a DataFrame work.
2. validmind/tests/model_validation/statsmodels/PredictionIntervals.py:83-90: for formula models, get_prediction silently drops rows with a missing regressor, but dataset.y keeps every row. The test then fails with a shape-mismatch ValueError (e.g. 500 vs 300 rows).
3. validmind/tests/model_validation/statsmodels/RobustStandardErrors.py:64-67: the "OLS" errors come from results.bse, which are already robust if the model was fitted with .fit(cov_type="HC3"). Every ratio is then 1.0 and the test always passes, which is the case it should catch. The classical errors should be computed explicitly, e.g. get_robustcov_results("nonrobust") or results.model.fit().
4. validmind/tests/model_validation/statsmodels/CoefficientStability.py:76-82: when a bootstrap resample contains no rows for a rare dummy variable, lstsq returns 0 for its coefficient, and np.sign(0) is counted as a sign flip. A dummy covering 2 of 500 rows fails the stability threshold even though its sign never flips, and the zero draws also pull the confidence interval toward 0. Suggest skipping rank-deficient draws, or counting only non-zero draws.
5. validmind/tests/model_validation/Heteroscedasticity.py:104-110: numeric features go straight into het_breuschpagan/het_white without dropping missing values. Any NaN in a feature raises MissingDataError. That matters for models that handle NaN natively, such as HistGradientBoostingRegressor.
6. validmind/tests/model_validation/FeatureSensitivity.py:104-111: std, min and max are computed without skipping NaN, so a feature with any missing value gets its whole shocked column set to NaN. The table then shows "± nan" and ranks that feature as highly influential. np.nanstd/np.nanmin/np.nanmax (and leaving NaN cells unshocked) would fix it. _is_binary also treats a 0/1 column with NaNs as non-binary.

Smaller issues

• validmind/tests/model_validation/_segments.py:42-49: segment bins are grouped by their display label, which is rounded to 4 significant digits. Large-valued numeric columns such as YYYYMMDD dates end up with every bin under the same label (2.024e+07 – 2.024e+07), so one "segment" holds the whole dataset. This affects ResidualBiasBySegment and RegressionWeakspots. Grouping by the bin itself and formatting the label only for display fixes it.
• validmind/tests/model_validation/statsmodels/VarianceInflationFactor.py:94-101 vs 145-147: the status label uses the unrounded VIF but pass/fail uses the rounded value. A VIF of 9.996 shows as "10.0 / Moderate" yet the test fails.
• validmind/tests/model_validation/TailPerformance.py:74-77: floating-point error in n * tail_share makes the two tails different sizes; n=100 with tail_share=0.29 gives 29 top rows and 28 bottom. round(n * tail_share) fixes it.
• validmind/tests/model_validation/statsmodels/ExpectedCoefficientSigns.py:108-112: an empty expected_signs raises KeyError: 'Result' instead of a clear error.

Notes

• validmind/ai/test_descriptions.py:223-229: rendering the figures before the background jobs start is reasonable. The cached image would go stale if a figure were changed afterwards, but nothing does that today.
• Unit tests: with the stats extra installed, the 61 statsmodels tests and 52 tests for the other new checks pass. make test-unit fails on a default environment with No module named 'statsmodels' unless the stats extra is installed, so CI may need that too.

[juanmleng]

@cachafla thanks for the careful review, and for the repro scripts. I reproduced all ten and fixed them in 6f4b2cc. Each fix has a unit test that fails without it.

Bugs

1. Numpy-fitted array models: design_data now maps x1, x2, … onto the dataset's feature_columns by position, so the feature_columns order must match the fit. If the counts differ, the test skips with a message explaining how to fix it. The ols_results docstring now says both DataFrame and numpy fits are supported.
2. PredictionIntervals with missing regressors: predictions are aligned to the dataset by row label, and rows with a missing prediction, bound or actual are excluded and counted in a new Excluded (Missing) column. This also fixes a related case: an array model with a NaN regressor used to count the row as inside the interval.
3. RobustStandardErrors on a model fitted with robust errors: it now refits plain OLS (results.model.fit()) and compares against that, so a model fitted with cov_type="HC3" gives the same table as a plain fit. Your first suggestion, get_robustcov_results("nonrobust"), raises cov_type not recognized on statsmodels 0.14, so I used the refit.
4. CoefficientStability with rare dummies: resamples that cannot estimate every coefficient (rank-deficient) are skipped, and their count is in RawData. If more than half are skipped, the test skips. The 2-of-500 dummy now has sign stability 1.0.
5. Heteroscedasticity with NaN features: rows with a missing feature or residual are excluded from Breusch-Pagan/White, and the table reports the observations used.
6. FeatureSensitivity with NaN features: shocks use nanstd/nanmin/nanmax, missing cells stay missing (including binary flips), and _is_binary ignores NaN. In my repro the feature ranked last, not first (up and down were equal, so its range was 0), but the "± nan" table was wrong either way.

Smaller issues

• Segment labels: digits are added until every bin has its own label, so YYYYMMDD columns keep one segment per bin.
• VIF: the value is rounded once, and the status and pass/fail both use that rounded value, so 9.996 shows as 10.0 / Severe / Fail.
• TailPerformance: the tail size is round(n * tail_share), so both tails are the same size.
• ExpectedCoefficientSigns: an empty expected_signs raises a clear ValueError.

Notes

• Figure caching: agreed, nothing mutates a figure after run_test today.
• statsmodels in CI: the unit test job already runs uv sync --all-extras (.github/workflows/python.yaml), and main already has unit tests that import statsmodels, so CI has it. A local make test-unit without the extras already failed before this PR.

[cachafla]

@juanmleng thanks. I re-ran my reproduction scripts against 6f4b2ccd, and 9 of the 10 points are fixed. The PR's unit tests pass (116 passed with the stats extra). Your CI note checks out: uv sync --all-extras installs statsmodels.

Still open

2. PredictionIntervals with missing regressors (validmind/tests/model_validation/statsmodels/PredictionIntervals.py:84-90). The crash is gone, but predictions are now paired with the wrong rows. After dropping NaN rows, get_prediction(...).summary_frame() returns a positional index 0..k-1, not df's labels. frame.reindex(df.index) therefore shifts every prediction onto an earlier row and marks the last rows as missing.

• Default index: 400 rows with x missing on the first 50. The PR reports Observations 350, Excluded 50, Coverage 0.434. With predictions kept on their own rows, coverage is 0.954.
• Non-default index: for example the shuffled index train_test_split returns, which the notebook's preprocess uses. Any NaN regressor makes every row miss the reindex, so the test skips with "found no rows without missing values".
• Duplicated index (e.g. after pd.concat): the result shows Excluded 0 and coverage 0.40.

test_rows_with_a_missing_regressor_are_excluded only checks the counts (300/200), so it doesn't catch this. One option is to build the design matrix with patsy.dmatrix(results.model.data.design_info, df, return_type="dataframe"), which keeps df's labels on the rows it keeps, then predict on it and align by that index. Either way, the test should assert coverage against a hand-computed value, and include a non-default index.

Fixed (verified on 6f4b2ccd)

1. Numpy-fitted array models: PredictionIntervals and predict_frame now work. A model with a different number of regressors than feature_columns skips with a clear message.
2. RobustStandardErrors: an HC3-fitted model now gives the same ratios as a plain fit (1.00 / 1.52 / 1.08).
3. CoefficientStability: the rare dummy now has sign stability 1.0, with no zero draws.
4. Heteroscedasticity: it runs with NaN features (1949 of 2000 observations used).
5. FeatureSensitivity: the shock is finite, the table no longer shows "± nan", and the NaN feature isn't ranked as influential.

• Segment labels: the YYYYMMDD column now gives 5 distinct segments.
• VIF: 9.996 shows as 10.0 / Severe / Fail, so the label and pass/fail agree.
• TailPerformance: with n=100, both tails are 29 rows at 0.29 and 7 rows at 0.07.
• ExpectedCoefficientSigns: an empty expected_signs raises a clear ValueError.
• Notes: acknowledged; no action needed.

[juanmleng]

@cachafla thanks for checking again, you're right. get_prediction numbers the rows it keeps 0..k-1, so reindexing by the dataset's labels shifted predictions onto other rows. PredictionIntervals now builds the formula design with patsy.dmatrix(design_info, ...) on the dataset's row positions, so shuffled and duplicated indexes line up too. test_rows_with_a_missing_regressor_are_excluded now checks coverage against a hand-computed value on the complete rows, for default, shuffled and duplicated indexes and for formula and array models; the formula cases fail on 6f4b2ccd. Fixed in 312bda2

[cachafla]

@juanmleng thanks, verified on 312bda26. All ten points from my review are now addressed, and from my side this is ready for approval.

2. PredictionIntervals with missing regressors: fixed. Building the formula design with patsy.dmatrix on row positions keeps each prediction on its own row. On my repro (400 rows, 50 with x missing), coverage is now 0.954 with 50 rows excluded. That matches the coverage computed by hand on the complete rows (0.954); 6f4b2ccd reported 0.434. Shuffled and duplicated indexes give the same result, for both formula and array models. The updated test_rows_with_a_missing_regressor_are_excluded now checks coverage, not just counts.

The other nine points were verified on 6f4b2ccd (see my previous review), and 312bda26 only touches PredictionIntervals and its test. The PR's unit tests pass: 116 passed with the stats extra.

[cachafla]

LGTM, thanks @juanmleng! Verified on 312bda2: all review points addressed, and CI is green.