Skip to content

Partial moments and CDF: native paths, R's input rules, vector percentiles - #149

Closed
gitRasheed wants to merge 10 commits into
rasheed/test-ci-prepfrom
rasheed/partial-moments
Closed

gitRasheed wants to merge 10 commits into
rasheed/test-ci-prepfrom
rasheed/partial-moments

Conversation

@gitRasheed

@gitRasheed gitRasheed commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

nns_cdf built dense n x n matrices for fractional degrees, degree >= 4 or any NaN, and its unshifted prefix expansion lost precision on level-shifted data. lpm and friends dropped to a dense NumPy path on NaN input and miscounted at degree 0. lpm_var took only a scalar percentile. We move these onto the native kernels and follow R 13.3's input rules.

Stacked on #148; merge with a merge commit.

Changes

  • Univariate nns_cdf goes through lpm_ratio (the native prefix backend with R's median shift) at every degree. On 1e6 + N(0,1) the old expansion gave CDF values outside [0, 1], non-monotone steps and inf at degree 3. No input builds an n x n matrix any more; that took 11-13 s and 2.5 GB at n = 1e4. Multivariate nns_cdf uses the bound clpm_nd_batch kernel instead of (n, n, d) tensors: n = 4000 goes from 7.8 s and 1.1 GB to 0.08 s.
  • lpm, upm, lpm_ratio and upm_ratio raise ValueError for NaN or Inf in x, as R's LPM/UPM do. Before, NaN took a dense path (4.5 s and 2.3 GB for 1000 targets x 1e5 observations) and degree-0 moments no longer summed to 1. target=None now means mean(x), R's default, instead of returning NaN. NaN targets still propagate to NaN, because internal callers rely on it.
  • skew_pm, kurt_pm and nns_moments return R's NaN/Inf pattern for constant input and for n <= 3 with population=False, instead of raising ZeroDivisionError. Finite results are unchanged.
  • lpm_var and upm_var accept array-like percentiles, as R's LPM.VaR/UPM.VaR do. For 100 percentiles at n = 1e5, degree 1 goes from 2964 ms to 24 ms (R 53 ms). Scalar results are bit-identical on 980 combinations.
  • co_lpm_nd, co_upm_nd and dpm_nd no longer emit RuntimeWarnings at fractional degrees. Values are unchanged.
  • Docstrings for the partial-moment functions, fixes to api_reference.md, and the nns_mode default (multi=False, R's is TRUE) is now documented. The default itself is unchanged.

Tests

New parity tests and cache entries for the NNS.moments edge cases and vector LPM.VaR/UPM.VaR, and an invariant for the fractional-degree co-moments. The parity cache encodes unpatched R 13.3.

See #147

Update

This push makes a non-finite target raise "target must be finite." instead of returning NaN.

Review

GPT-6.1 Sol reviewed this slice and raised three findings. We took two and rejected one.

  • Taken: the pure-Python nns_cdf built an n x n matrix (763 MiB at n = 1e4). The fallback now works through the targets in blocks. Peak memory for the fallback nns_cdf at n = 1e4, degree 2 goes from 2289 MiB to 235 MiB, and results are bit-identical on 216 random cases.
  • Taken: DataFrame column names that collide with "CDF" or with each other dropped a coordinate. Names that collide now take R's make.unique suffixes with "CDF" reserved (CDF.1, a.2), so all coordinates survive.
  • Rejected: lpm_ratio/upm_ratio now reject NaN in x, where R 13.3 skips it. The R fixes in Fix cancellation in the >32-target LPM/UPM backend, and bound-check native kernels NNS#73-Inspect R NNS API update f834b7bec8201ef4d5585dbb01ea3039192c3753 #92 stop on non-finite input too ("variable must be finite"), and this slice follows them.

The fix commits have not been re-reviewed.

- Univariate: every degree now goes through lpm_ratio (native prefix backend
  with R's median shift). The unshifted prefix-power expansion lost precision on
  level-shifted data (1e6 + N(0,1) gave CDF values outside [0, 1], non-monotone
  steps and inf at degree 3), and fractional degree, degree >= 4 or any NaN built
  an n x n matrix (11-13 s and 2.5 GB at n = 1e4).
- R's NA quirk (NA kept in n) is reproduced by rescaling the finite-value ratio
  instead of materialising the target x observation matrix.
- Multivariate: use the already-bound clpm_nd_batch kernel (R's
  Co.LPM_nD.batch) instead of (n, n, d) tensors: n = 4000 goes from 7.8 s /
  1.1 GB to 0.08 s. The NumPy fallback is chunked.
- Vectorise the hazard proxy loop; take column names from a DataFrame.

Results are unchanged wherever they were previously correct.
- lpm, upm, lpm_ratio and upm_ratio raise ValueError for NaN/Inf in x, as R's
  LPM/UPM do ("variable must be finite"). Previously any NaN dropped to an
  (m x n) NumPy path (4.5 s and 2.3 GB for 1000 targets x 1e5 obs) and at
  degree 0 silently returned moments that no longer summed to 1.
- target=None now means mean(x) (R's ratio and co-moment wrappers) instead of
  silently returning NaN; NaN targets raise.
- Add docstrings to the four public partial-moment functions.
Internal callers (e.g. causation on degenerate normalised inputs) pass NaN
targets and relied on a NaN result; keep that behaviour for every degree.
Non-finite x still raises, as in R.
skew_pm, kurt_pm and nns_moments divided Python floats, so a constant
series (variance 0) raised ZeroDivisionError, and nns_moments(population=
False) raised for n <= 3 because of the (n - 2) and (n - 3) factors. R's
NNS.moments uses IEEE arithmetic and returns NaN (or +-Inf):

  NNS.moments(rep(1, 5))              -> mean 1, variance 0, skew NaN, kurt NaN
  NNS.moments(c(1, 2, 4), FALSE)      -> skew 1.718, kurt NaN
  NNS.moments(5, FALSE)               -> variance NaN, skew NaN, kurt NaN

The arithmetic now runs on np.float64 scalars under np.errstate, so these
inputs give R's NaN/Inf pattern without warnings. Finite results are
unchanged (same operations in the same order).
R's LPM.VaR(percentile, degree, x) and UPM.VaR are vectorised over
percentile: the integer-degree path sorts x and builds the prefix powers
once and only solves one interval per percentile. Python called
float(percentile), so lpm_var(np.array([0.05, 0.5]), 1, x) raised
TypeError, and a Python loop over percentiles rebuilt the sort and prefix
sums every time.

- lpm_var/upm_var take a scalar (returns float, as before) or array-like
  percentile (returns an ndarray of the same shape), clamped to [0, 1].
- The integer-degree inversion is split into a builder
  (_lpm_var_integer_solver) and a per-percentile interval solve; degree 0
  uses one np.quantile call; fractional degrees still optimise per
  percentile, as R does.
- Docstrings describe the semantics.

A/B against the previous scalar implementation: 980 (data, degree,
percentile) combinations, including clamping, constant/length-1 input and
NaN/Inf dropping, identical bit for bit, and the vector result equals the
scalar calls. 100 percentiles, n = 1e5 t3 returns (loaded box):

  degree 1: 2964 ms (scalar loop) -> 24 ms (R 53 ms)
  degree 2: 2333 ms -> 24 ms
  degree 0:  319 ms -> 13 ms

Scalar results are unchanged.
co_lpm_nd, co_upm_nd and dpm_nd (via its normalisation) raised every row's
deviation to the degree before masking the rows that are not entirely
below/above the target, so any fractional degree emitted "invalid value
encountered in power" for the negative deviations. The deviations
are now clamped at zero before the power; the masked rows are discarded as
before, so values are unchanged (checked identical against the previous
code on 200 random matrices x degrees 0, 0.5, 1, 1.5, 2, 3).
- co_lpm, co_upm, d_lpm and d_upm had no docstrings. They now state the
  formula, which variable uses which degree, the degree-0 tie convention
  and the target/None handling (checked against direct formulas).
- api_reference.md called D.LPM/D.UPM/DPM "distance"/"directional"
  partial moments; they are the divergent quadrants. It named R's
  NNS.cdf (it is NNS.CDF) and said ecdf_pm is "used by moment routines"
  (nothing uses it). It now also notes that the nns.pm_matrix attribute
  (the function) shadows the submodule.
- lpm_var/upm_var: describe the percentile semantics and the vector form.
- nns_mode defaults to multi=False while R's NNS.mode defaults to
  multi = TRUE. The default is unchanged; the divergence is now stated in
  the docstring, api_reference.md and a conventions.md section.
- conventions.md: nns_moments returns NaN for undefined moments, like R.

Documentation only.
Without the native kernels nns_cdf(n=1e4, degree=2) peaked at 2289 MiB
(tracemalloc); blocks of 1024 targets peak at 235 MiB, bit-identical on 216 random cases.
A DataFrame column named "CDF" (or two named "a") dropped a coordinate: 2 of 3
output columns before, 3 of 3 now, named like R's make.unique.
R's LPM and UPM stop with "`target` must be finite." for NaN and Inf
targets; lpm/upm returned NaN (or a count at degree 0 with Inf) instead. Raise
the same error, as lpm_ratio/upm_ratio and non-finite x already do.
@github-actions

github-actions Bot commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

Incorporated through NNS R 14.0 integration #192, merged as bd19c3303222af46da79d8c97f258248eb5674d2. Verified exact head 266a05494a012ec3da1b1cb021725270e28d4141 is an ancestor of main; original commits and authorship are preserved. The integration includes the subsequent R14 compatibility corrections and passed the complete GitHub Python 3.11-3.14 matrix. This supersedes the separate landing, not the work or review history.

@github-actions github-actions Bot closed this Nov 7, 2026
@github-actions
github-actions Bot deleted the rasheed/partial-moments branch November 7, 2026 18:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant