Make (Rcpp)Eigen use threads the way (Rcpp)Armadillo does

@jaganmn

While doing some work updating and overhauling (Rcpp)Armadillo last summer and fall, we also revisited / simplified some of the threading (including as always a redo of what macOS does; let's leave that aside for now).

I carried some of that over to RcppEigen as well so that by default we get all cores (if a user opts in). I still have that code here and had been meaning to reach out to 'interested parties': @jaganmn @bbolker @bgoodri @SteveBronder come to mind.

My current thinking is that we should be able to set something up that 'does no harm' (i.e. defaults to a max of two threads unless told otherwise to not cause harm at CRAN) but goes beyond if suitable local environment variables or config settings are found. So the current status quo, as I recall it, is that

package RcppEigen ignores the existence of OpenMP around R, RcppEigen:::EigenNbThreads() (which existed for years) return 1
just by adding 'use OpenMP where is exists' we can do better, i.e. on my machine I now get all cores (which would upset CRAN)
we just decide which throttling mechanism we like
- ie maybe borrow the data.table one which uses 50%
- and/or combine with a configuration variable as I once did for tiledb

Thoughts? I'll try what I have here (uncommitted, and it isn't much yet) into another branch

I.e. following data.table may be enough (though we may want to use 100% instead of 50% or...)

$ Rscript -e 'data.table:::getDTthreads()'   # by default 50% of all, here 0.5 * 32 threads
[1] 16
$ OMP_THREAD_LIMIT=2 Rscript -e 'data.table:::getDTthreads()'   # CRAN setting
[1] 2
$

And I have a simple version of this now:

$ Rscript -e 'RcppEigen:::EigenNbThreads()'   # no limit by default
[1] 32
$ OMP_THREAD_LIMIT=4 Rscript -e 'RcppEigen:::EigenNbThreads()'
[1] 4
$ Rscript -e 'options(Ncpus=6); RcppEigen:::EigenNbThreads()'
[1] 6
$

This would 'do the Right Thing (TM)' at CRAN via OMP_THREAD_LIMIT being set to two, also allows option Ncpus to be set, and if all else fails uses all machine cores.

PS In the branch feature/openmp.

PPS And once more just comparison in a 'pristine' r2u container:

$ Rscript -e 'RcppEigen:::EigenNbThreads()'        # using CRAN RcppEigen         
$ # .... installing from the feature/openmp branch ... not shown
$ Rscript -e 'RcppEigen:::EigenNbThreads()'       # new behavior using OpenMP
[1] 32
$ OMP_THREAD_LIMIT=2 Rscript -e 'RcppEigen:::EigenNbThreads()'   # 'CRAN-like' when limit is 2
[1] 2
$

The diff against master looks fine to me and indeed everything is simple enough once one sets aside macOS. I just wonder - is there a precedent for using Ncpus for purposes other than package download/install/check? I do set Ncpus in my .Rprofile and past me would not have anticipated effects on a package like RcppEigen. Any reason to not name the option EigenNbThreads (or similar)?

Edit: Oh, I looked before the Autoconf stuff was added. I'll look again when I get a moment ...

I adopted Ncpus years ago because it is in Writing R Extensions and quite helpful for several cases of standard parallelism with R. Try a quick GitHub search with organization CRAN for getOption("Ncpus") or option(Ncpus or option("Ncpus ...
It seems best not to reinvent a wheel here.

The code logic in configure.ac, needed only for macOS, evolved over several PRs in the fall and benefited from a close look at data.table. I am quite confident this is 'good' now over in RcppArmadillo and can therefore be used here. (Over there we also need to export an additional flag 'yes have OpenMP' which is not needed here: it is all internal to Eigen.

Ncpus no longer (?) appears in the manuals, but I'm happy to go ahead with it. Same for the Autoconf stuff - for your maintenance it makes sense to mirror RcppArmadillo by default and adapt later if there seems to be some issue.

Weird. Just grep'ed doc/manuals and I cannot see it.

But packages tools and utils still use (and document) it. ?options only mentions it for downloads though. Maybe I misremembered and it was never in WRE? It was always used, and I have a vague recollection it was also a suitable default for snow and/or parallel use over several processors or cores.

It really is too bad we do not have a better 'general' variable for 'use this much of my general multiple core resources'.

Navigation Menu

Repository navigation

Make (Rcpp)Eigen use threads the way (Rcpp)Armadillo does #144

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Make (Rcpp)Eigen use threads the way (Rcpp)Armadillo does #144

Description

Activity

eddelbuettel commented on Dec 27, 2025

eddelbuettel commented on Dec 27, 2025

jaganmn commented on Dec 28, 2025

eddelbuettel commented on Dec 28, 2025

jaganmn commented on Dec 28, 2025

eddelbuettel commented on Dec 28, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions