Skip to content

diff: dy_dx uses every bandwidth when small samples repeat one - #186

Closed
gitRasheed wants to merge 1 commit into
rasheed/py-mc-exp-boundsfrom
rasheed/py-dydx-duplicate-bands
Closed

gitRasheed wants to merge 1 commit into
rasheed/py-mc-exp-boundsfrom
rasheed/py-dydx-duplicate-bands

Conversation

@gitRasheed

Copy link
Copy Markdown
Collaborator

R's dy.dx uses five bandwidths h_s = round(exp(seq(log(2), log(floor(sqrt(n))), length.out = 5))), which repeat for n < 49. It looped over the values and assigned each block at which(h == h_s), a length-2 index for a repeat, so rbind dropped those blocks. Below n = 16 every value repeats and dy.dx failed. Python copied this, using only the bandwidths that occur once, or the distinct ones below 16. OVVO-Financial/NNS#147 (merged) loops over the positions, so every bandwidth counts. This slice mirrors it.

Stacked on #185; merge with a merge commit.

Changes

  • _dy_dx_numeric (diff.py) builds the derivative windows for all five entries of h_s, repeats included.
  • dy_dx(seq(0, 3, length.out = 10), x^2, 1.5): first derivative 2.91851 -> 2.91861 and second 0.15957 -> 0.16054, equal to R main (2.9186064885499, 0.1605427056170) to 1e-11. For n = 12, 20 and 30 Python differs from R main because R main's smoothed NNS.reg fit itself changed with its dependence work in Fix #95-#101: NNS.dep, NNS.caus and NNS.seas redesigns, with the follow-up fixes NNS#107 (the point estimates for n = 20 already differ in the second decimal). Python takes that with the core re-vendor.

Tests

  • New invariants mirroring R's test-dydx-duplicate-bands.R: finite derivatives near 3 for n = 10 and 12, and for n = 10, 20, 30 and 50 the result equals R's own construction of the five-bandwidth mean built from Python's nns_reg (rtol 1e-12), plus R main's literal values for n = 10.
  • 3 dy.dx.numeric cache entries predate the R change (n = 24 at one and three eval points, and n = 30). I refilled them from an R build of the cache's base plus only Thread settings, non-finite lpm targets and removed submodule names #147's R/dy_dx.R hunk. The other 4 came back identical; docs/parity.md records it. The parity test comment that described the old rule now describes the new one.
  • Gates on the top of this stack (rasheed/ss-constant-group, which includes this slice), cache only: native 2935 passed, 4 skipped, 3 xfailed, 2 xpassed. With NNS_DISABLE_NATIVE=1: 2913 passed, 29 skipped, 2 xpassed. The 3 xfails are mode: break tied discrete modes by gravity, not their mean (mirrors R #111) #178's strict ones and the 2 xpasses the existing non-strict live-R VAR xfails. tests/parity/test_practical_examples.py against the usual R library: 9 passed, 2 xfailed, as on mode: break tied discrete modes by gravity, not their mean (mirrors R #111) #178. ruff check is clean on every changed file.

Review

GPT-6.1 Sol reviewed the #146, #147 and #148 mirrors together: SHIP. Its one nit, docs/conventions.md still saying repeated bandwidths are skipped, is fixed in this slice.

Mirrors OVVO-Financial/NNS#147

…x-duplicate-bands)

Mirrors R dy.dx (OVVO-Financial/NNS dydx-duplicate-bands). The bandwidths
h_s = round(exp(seq(log 2, log floor(sqrt(n)), length.out = 5))) repeat
for n < 49. R looped over the values and assigned each block at
which(h == h_s), a length-2 index for a repeat, so rbind dropped those
blocks, and below n = 16 every value repeats and dy.dx failed. Python
copied that, using only the bandwidths that occur once (or the distinct
ones below 16). Both now average over all five positions.

Before -> after (= R main), dy_dx(seq(0, 3, length.out = 10), x^2, 1.5):
  first derivative   2.91851 -> 2.91861 (R main 2.9186064885499)
  second derivative  0.15957 -> 0.16054 (R main 0.1605427056170)
For n = 12, 20 and 30 Python now differs from R main because R main's
smoothed NNS.reg fit changed with its later dependence work, which Python
mirrors in the re-vendor, so the tests check those against R's own
construction from Python's nns_reg instead of literal values.

3 dy.dx.numeric cache entries (n = 24 and 30, which repeat bandwidths)
predate the R change and were refilled from the cache's R build plus that
change alone; the other 4 came back identical.
@github-actions

github-actions Bot commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

Incorporated through NNS R 14.0 integration #192, merged as bd19c3303222af46da79d8c97f258248eb5674d2. Verified exact head af6a47f9130b4d4328a87f2d5b879ac42cb78210 is an ancestor of main; original commits and authorship are preserved. The integration includes the subsequent R14 compatibility corrections and passed the complete GitHub Python 3.11-3.14 matrix. This supersedes the separate landing, not the work or review history.

@github-actions github-actions Bot closed this Nov 7, 2026
@github-actions
github-actions Bot deleted the rasheed/py-dydx-duplicate-bands branch November 7, 2026 18:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant