Skip to content

ARMA optim scoring and dynamic weights: mirror the later R fixes - #166

Closed
gitRasheed wants to merge 3 commits into
rasheed/mirror-dependencefrom
rasheed/mirror-timeseries
Closed

gitRasheed wants to merge 3 commits into
rasheed/mirror-dependencefrom
rasheed/mirror-timeseries

Conversation

@gitRasheed

@gitRasheed gitRasheed commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

Four fixes from the R NNS.ARMA.optim and NNS.ARMA PRs that came after the first mirror. When equal weights won, obj.fn and the shrink and nns.regress comparisons still used the weights = None score. The nns.regress candidate smoothed the whole series and was scored on the validation window, so the smooth had seen the values it was scored against. With dynamic=True, user weights were silently replaced by the NNS.seas weights.

Stacked on #165; merge with a merge commit.

Changes

  • obj.fn is the equal-weights score when equal weights win. On the two-cycle seed-33 series it goes from 1.054731 to 1.021186.
  • The nns.regress decision and the validation results use a smooth of the training window only. The out-of-sample forecast still smooths the full series. On random walk seed 13 with training_set=80, nns.regress goes from True to False.
  • The smoothed candidate is compared with the best score so far, after shrink.
  • Numeric weights with dynamic=True raise, as in R. weights="equal" applies to whichever lags each step selects: the forecast goes from 181.788 184.092 186.380 to 181.495 183.784 186.057, which equals repeated h = 1 calls.

weights=None and static calls are unchanged.

Tests

The two changed NNS.ARMA.optim.custom entries are refilled from R and two are added. R's refilled values also carry the book-alignment change in OVVO-Financial/NNS#88, so for now these cases compare the search outcome (method, periods, weights, obj.fn, shrink, nns.regress) and leave the value keys to #174. A new invariant covers dynamic weights. The parity cache encodes R 13.3 plus the fixes in OVVO-Financial/NNS#73-#92.

Mirrors OVVO-Financial/NNS#77 and OVVO-Financial/NNS#78
Part of OVVO-Financial/NNS#63 and OVVO-Financial/NNS#64

Review

GPT-6.1 Sol reviewed this slice and raised one finding, which we took: the new walk-16 case had no forecast assertion, so reintroducing the validation-window leak still passed. A separate test now asserts results - bias.shift, which matches R to 1.8e-11; the leaking variant is 0.14 off and fails. The fix commit has not been re-reviewed.

…hrink (mirror of the R fixes)

Three fixes from R NNS.ARMA.optim (arma-optim-scoring):
- When equal weights win, obj.fn and the later shrink and nns.regress
  comparisons used the weights = None score. Take the equal-weights score.
- The nns.regress candidate smoothed the whole series and was scored on
  the validation window, so the smooth had seen the values it was scored
  against. Smooth only the training window for the decision and the
  validation-window results; the out-of-sample forecast (h given) still
  smooths the full series.
- The smoothed candidate was compared with the score before shrink.
  Compare it with the best score so far.

Before -> after (decisions and obj.fn = R):
  equal-weights case (two-cycle series, seed 33), obj.fn
                                           1.054731 -> 1.021186
  random walk seed 13, training_set 80, nns.regress   True -> False
  random walk seed 16, nns.regress stays True, results
                          39.095 39.301 39.504 -> 39.173 39.348 39.523

The two changed NNS.ARMA.optim.custom entries are refilled from R and two
are added. R's refilled values also carry optim-pi-from-returned-model
(bias and errors from the returned model), so for these cases the test
compares the search outcome (method, periods, weights, obj.fn, shrink,
nns.regress) and leaves the value keys to that change.
…tep (mirror of the R fix)

The dynamic loop re-resolved the lags at each step with weights=None, so
user weights were silently replaced by the NNS.seas weights. As in R
NNS.ARMA (arma-forecasts), numeric weights with dynamic=True now raise
(they belong to fixed lags, like a numeric seasonal_factor, which already
raises), and weights="equal" is applied to whichever lags each step
selects.

Before -> after, 100 + t + 15 sin(pi t / 6), t = 0..59, h = 3,
seasonal_factor=False, best_periods=2, dynamic=True:
  weights="equal"      181.788 184.092 186.380
                       -> 181.495 183.784 186.057 (= repeated h = 1 calls)
  weights=[1, 0]       weights ignored -> ValueError (R: error)
weights=None and static calls are unchanged.
walk-16 compared only the search outcome, so a smooth that saw the validation
window still passed; results - bias.shift now matches R to 1.8e-11, the mutant is 0.14 off.
@github-actions

github-actions Bot commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

Incorporated through NNS R 14.0 integration #192, merged as bd19c3303222af46da79d8c97f258248eb5674d2. Verified exact head d2767c8b5add85ba44f51dfe72c832e9199b0df0 is an ancestor of main; original commits and authorship are preserved. The integration includes the subsequent R14 compatibility corrections and passed the complete GitHub Python 3.11-3.14 matrix. This supersedes the separate landing, not the work or review history.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant