Best practice for using max 2 cores on CRAN?

[tdhock]

CRAN has a policy that limits the number of cores that can be used during package checks, and they have recently implemented a mechanism for checking compliance. Upon recent CRAN submission of one of my R packages which imports data.table, I got the following rejection message from CRAN:

Flavor: r-devel-linux-x86_64-debian-gcc
Check: examples, Result: NOTE
  Examples with CPU (user + system) or elapsed time > 5s
                         user system elapsed
  aum_line_search      12.349  0.322   1.935
  aum_line_search_grid 10.033  0.308   1.781
  Examples with CPU time > 2.5 times elapsed time
                         user system elapsed ratio
  aum_line_search      12.349  0.322   1.935 6.548
  aum_line_search_grid 10.033  0.308   1.781 5.806
  aum_diffs_penalty     4.730  0.169   1.635 2.996

These messages can be suppressed by adding data.table::setDTthreads(1) at the start of each example.
Is there another/recommended way to avoid using too many cores when checking a CRAN package which uses data.table? The default number of threads in data.table is max cores/2 which is over the CRAN limit (max 2 cores I believe).
Also is there documentation about this somewhere? Probably would be good to mention in the datatable-importing vignette.

[eddelbuettel]

they have recently implemented a mechanism for checking compliance

I don't think it is that recent. Package tiledb got dinged three or so years ago, and I then implemented a scheme that works.

Here is what the package does now. In essence, in every example file or unit test file (i.e. files that CRAN runs) I explicitly call a function that throttles cores down to

• the given function value, if there is one, in this throttler function
• or the value of option Ncpu with a fallback value of 2 (and that is what CRAN gets; as I recall they set Ncpu)
• or the value of environment variable OMP_THREAD_LIMIT common to set OpenMP and MKL thread counts.

So this gets us the max of 2 in everything that CRAN runs yet it can be overriden (if, they, your CI has more cores). I already had Ncpu set locally to e.g. install multiple packages in parallel via install.packages() / update.package() ).

Importantly, it leaves general startup (i.e. .onLoad(), .onAttach() ) alone: it still gets full core count and performance.

So normal users are not affected and get full speed, but CRAN gets its limit. I can detail that some more you if you care; it is in tiledb file R/config.R (but a little tied to how the config object progagates for us).

[jangorecki]

Also reading DT news file can point to commits that set that up for DT

[tdhock]

@eddelbuettel thanks for the quick feedback, that approach seems similar to what I did, but more flexible. You wrote that CRAN sets option Ncpu -- do you know where is that documented? On https://cran.r-project.org/web/packages/policies.html I see "If running a package uses multiple threads/cores it must never use more than two simultaneously: the check farm is a shared resource and will typically be running many checks simultaneously. " but no mention of option Ncpu.

Also I was thinking that if CRAN indeed sets option Ncpu then the data.table default number of threads should respect that (avoids having to change example/test code in 1000+ packages with hard dependency on data.table).

@jangorecki I checked https://github.com/Rdatatable/data.table/blob/master/NEWS.md but I did not see mention of commits, can you please clarify?

[tdhock]

An alternative to option Ncpu would be to just detect if on CRAN, for example using code below, and then throttle to two cores for data.table default.

> testthat:::on_cran
function () 
{
    !env_var_is_true("NOT_CRAN")
}

I do not see mention of NOT_CRAN env var on CRAN repository policy either, do you know where that is documented?

[eddelbuettel]

Relying on absence of NOT_CRAN has always been and still is a hack I do not recommended, or use. YYMV.

I do not recall where they document but it is documented somewhere that they allow two threads and AFAICR Ncpus is the one vessel for that payload.

[tdhock]

There is a related R-devel thread, https://stat.ethz.ch/pipermail/r-devel/2021-November/081289.html that explains the env var _R_CHECK_LIMIT_CORES_=TRUE is set when running R CMD check --as-cran.

[tdhock]

?tools::check_packages_in_dir says Ncpus option is used to specify how many packages are checked in parallel,

   Ncpus: the number of parallel processes to use for parallel
          installation and checking.

[eddelbuettel]

But note that what you quoted does not limit it to package checks as your statement implies but to general 'checking'. Which is where we started: how to play nice at CRAN and not exceed limits.

[tdhock]

This check is implemented via this code https://github.com/wch/r-source/blob/1c0545ba7c6c07e8c358eda552b875b1e4d6826d/src/library/tools/R/check.R#L4123 which gets the 2.5 ratio from the environment variable _R_CHECK_EXAMPLE_TIMING_CPU_TO_ELAPSED_THRESHOLD_ so we could check to see if that is set and then round down as a default for number of threads.

[eddelbuettel]

Also:

data.table/src/openmp-utils.c

Line 95 in 8803918

Rprintf(_(" OMP_THREAD_LIMIT %s\n"), mygetenv("OMP_THREAD_LIMIT", "unset")); // CRAN sets to 2

Recall that my approach was about taking the lower limit from Ncpu amd OMP_NUM_THREADS (and I no longer recall why I do not / did not also set OMP_THREAD_LIMIT).

Also

data.table/man/openmp-utils.Rd

Line 27 in 8803918

The number of logical CPUs is determined by the OpenMP function \code{omp_get_num_procs()} whose meaning may vary across platforms and OpenMP implementations. \code{setDTthreads()} will not allow more than this limit. Neither will it allow more than \code{omp_get_thread_limit()} nor the current value of \code{Sys.getenv("OMP_THREAD_LIMIT")}. Note that CRAN's daily test system (results for data.table \href{https://cran.r-project.org/web/checks/check_results_data.table.html}{here}) sets \code{OMP_THREAD_LIMIT} to 2 and should always be respected; e.g., if you have written a package that uses data.table and your package is to be released on CRAN, you should not change \code{OMP_THREAD_LIMIT} in your package to a value greater than 2.

[tdhock]

related to #5573 #5620 about other environment variables to look at to determine default number of threads.

[TimTaylor]

I also ran in to this on a submission this week and so did others recently (see r-devel thread where Dirk gives similar guidance).

I was confused as I assumed this would always be handled on the data.table side. I wonder if the CRAN test machine is no longer setting the OMP_THREAD_LIMIT environment variable which data.table is assuming it does?

[jangorecki]

I haven't found in NEWS file anything useful as well.
#3300 seems to be related