rolling functions: adaptive left, frollmax, frollapply adaptive, partial #5441

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed

jangorecki wants to merge 59 commits into master from frollmax

.Rbuildignore

-Original file line number
+Diff line change
@@ Expand Up / @@ -12,7 +12,6 @@ @@
     ^\.ci$
     ^\.dev$
-    ^\.devcontainer$
     ^\.graphics$
     ^\.github$
@@ Expand All / @@ -23,7 +22,6 @@ @@
     ^NEWS\.0\.md$
     ^_pkgdown\.yml$
     ^src/Makevars$
-    ^CODEOWNERS$
     ^\.RData$
     ^\.Rhistory$
@@ Expand All / @@ -41,4 +39,3 @@ @@
     ^pkgdown$
     ^lib$
     ^library$
-    ^devwd$

.ci/Dockerfile.in

-Original file line number
+Diff line change
@@ -0,0 +1,9 @@
+    FROM registry.gitlab.com/jangorecki/dockerfiles/SRC_IMAGE_NAME
+    MAINTAINER Jan Gorecki j.gorecki@wit.edu.pl
+    COPY bus/build/cran/ /cran/
+    RUN Rscript -e 'install.packages("data.table", repos=file.path("file:","cran"))'
+    CMD ["R"]

.ci/README.md

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -1,50 +1,72 @@
  
    # data.table continuous integration and deployment

    On each Pull Request opened in GitHub we run GitHub Actions test jobs to provide prompt feedback about the status of PR. Our main CI pipeline runs on GitLab CI nightly. GitLab repository automatically mirrors our GitHub repository and runs pipeline on `master` branch every night. It tests more environments and different configurations. It publish variety of artifacts.

    On each Pull Request opened in GitHub we run Travis CI and Appveyor to provide prompt feedback about the status of PR. Our main CI pipeline runs on GitLab CI. GitLab repository automatically mirrors our GitHub repository and runs pipeline on `master` branch. It tests more environments and different configurations. It publish variety of artifacts.

    ## Environments

    ### [GitLab CI](./../.gitlab-ci.yml)

    Test jobs:

    - `test-lin-rel` - `r-release` on Linux, most comprehensive test environment, force all suggests, `-O3 -flto=auto -fno-common -Wunused-result`, test for no compilation warnings.

    - `test-lin-rel-vanilla` - `r-release` on Linux, no suggested deps, no zlib, no OpenMP, flags `-g -O0 -fno-openmp`, skip manual and vignettes.

    - `test-lin-rel-cran` - `--as-cran` on Linux, strict test for final status of `R CMD check`.

    - `test-lin-dev-gcc-strict-cran` - `--as-cran` on Linux, `r-devel` built with `-enable-strict-barrier --disable-long-double`, test for compilation warnings, test for new NOTEs/WARNINGs from `R CMD check`.

    - `test-lin-dev-clang-cran` - same as `gcc-strict` job but R built with `clang` and  no `--enable-strict-barrier --disable-long-double` flags.

    - `test-lin-310-cran` - R 3.1.0 on Linux, stated R dependency version.

    - `test-win-rel` - `r-release` on Windows.

    - `test-win-dev` - `r-devel` on Windows.

    - `test-win-old` - `r-oldrel` on Windows.

    - `test-mac-rel` - macOS build not yet available, see [#3326](https://github.com/Rdatatable/data.table/issues/3326) for status

    Tests jobs are allowed to fail, summary and logs of test jobs are later published at _CRAN-like checks_ page, see artifacts below.

    - `test-rel-lin` - `r-release` on Linux, most comprehensive test environment, `-O3 -flto -fno-common -Wunused-result`, extra check for no compilation warnings, includes testing [_with other packages_](./../inst/tests/other.Rraw)

    - `test-rel-cran-lin` - `--as-cran` on Linux, `-g0`, extra check for final status of `R CMD check` where we allow one NOTE (_size of tarball_).

    - `test-dev-cran-lin` - `r-devel` and `--as-cran` on Linux, `--with-recommended-packages --enable-strict-barrier --disable-long-double`, tests for compilation warnings in pkg install and new NOTEs/Warnings in pkg check, and because it is R-devel it is marked as allow_failure

    - `test-rel-vanilla-lin` - `r-release` on Linux, no suggested deps, no OpenMP, `-O0`, tracks memory usage during tests

    - `test-310-cran-lin` - R 3.1.0 on Linux

    - `test-344-cran-lin` - R 3.4.4 on Linux

    - `test-350-cran-lin` - R 3.5.0 on Linux, no `r-recommended`

    - `test-rel-win` - `r-release` on Windows

    - `test-dev-win` - `r-devel` on Windows

    - `test-old-win` - `r-oldrel` on Windows

    - `test-rel-osx` - MacOSX build not yet deployed, see [#3326](https://github.com/Rdatatable/data.table/issues/3326) for status

    Artifacts:

    - [homepage](https://rdatatable.gitlab.io/data.table) - made with [pkgdown](https://github.com/r-lib/pkgdown)

    - [html manual](https://rdatatable.gitlab.io/data.table/library/data.table/html/00Index.html)

    - [pdf manual](https://rdatatable.gitlab.io/data.table/web/packages/data.table/data.table.pdf)

    - [html vignettes](https://rdatatable.gitlab.io/data.table/library/data.table/doc/index.html)

    - R packages repository for `data.table` and all _Suggests_ dependencies, url: `https://rdatatable.gitlab.io/data.table`

    - R packages repository for `data.table` and all _Suggests_ dependencies, url: `https://Rdatatable.gitlab.io/data.table`

      - sources

      - Windows binaries for `r-release`, `r-devel` and `r-oldrel`

    - [CRAN-like homepage](https://rdatatable.gitlab.io/data.table/web/packages/data.table/index.html)

    - [CRAN-like checks results](https://rdatatable.gitlab.io/data.table/web/checks/check_results_data.table.html)

    - [CRAN-like checks results](https://rdatatable.gitlab.io/data.table/web/checks/check_results_data.table.html) - note that all artifacts, including check results page, are being published only when all test jobs successfully pass, thus one will not see an _ERROR_ status there (unless error happened on a job marked as `allow_failure`).

    - [docker images](https://gitlab.com/Rdatatable/data.table/container_registry) - copy/paste-able `docker pull` commands can be found at the bottom of our [CRAN-like homepage](https://rdatatable.gitlab.io/data.table/web/packages/data.table/index.html)

    ### [GitHub Actions](./../.github/workflows)

    ### [Travis CI](./../.travis.yml)

    TODO document

    Test jobs:

    - `r-release` on Linux, includes code coverage check

    - _(might be disabled)_ `r-release` on OSX

    Artifacts:

    - R packages repository having `data.table` sources only, url: `https://Rdatatable.github.io/data.table`

    - code coverage stats pushed to [codecov.io/gh/Rdatatable/data.table](https://codecov.io/gh/Rdatatable/data.table)

    ### [Appveyor](./../.appveyor.yml)

    TODO document

    Test jobs:

    - Windows `r-release`

    - _(might be disabled)_ Windows `r-devel`

    Artifacts:

    - Windows `r-release` binaries accessed only via web UI

    ## CI tools

    ## Tools

    ### [`ci.R`](./ci.R)

    Base R implemented helper script, [originally proposed to base R](https://svn.r-project.org/R/branches/tools4pkgs/src/library/tools/R/packages.R), that ease the process of extracting dependency information from description files, and to mirror packages and their recursive dependencies from CRAN to local CRAN-like directory. It is used in [GitLab CI pipeline](./../.gitlab-ci.yml).

    Base R implemented helper script, [originally proposed to R](https://svn.r-project.org/R/branches/tools4pkgs/src/library/tools/R/packages.R), that ease the process of extracting dependency information from description files, also to mirror packages and their recursive dependencies from CRAN to local CRAN-like directory. It is widely used in our [GitLab CI pipeline](./../.gitlab-ci.yml).

    ### [`publish.R`](./publish.R)

    Base R implemented helper script to orchestrate generation of most artifacts and to arrange them nicely. It is being used only in [_integration_ stage in GitLab CI pipeline](./../.gitlab-ci.yml).

    Base R implemented helper script to orchestrate generation of most artifacts. It is being used only in [_integration_ stage in GitLab CI pipeline](./../.gitlab-ci.yml).

    ### [`Dockerfile.in`](./Dockerfile.in)

    Template file to produce `Dockerfile` for, as of now, three docker images. Docker images are being built and published in [_deploy_ stage in GitLab CI pipeline](./../.gitlab-ci.yml).

    - `r-base-dev` using `r-release`: publish docker image of `data.table` on R-release

    - `r-builder` using `r-release`: publish on R-release and OS dependencies for building Rmarkdown vignettes

    - `r-devel`: publish docker image of `data.table` on R-devel built with `--with-recommended-packages --enable-strict-barrier --disable-long-double`

    ### [`deploy.sh`](./deploy.sh)

    Script used on Travis CI to publish CRAN-like repository of `data.table` sources. It publishes to `gh-pages` branch in GitHub repository. It depends on a token, which is provided based on `secure` environment variable in [.travis.yml](./../.travis.yml). It has been generated by @jangorecki.

.ci/ci.R

-Original file line number
+Diff line change
@@ Expand Up / @@ -185,3 +185,7 @@ function(pkgs, @@
       dp
     }
+    ## set repositories for CI tests
+    if (as.logical(Sys.getenv("GITLAB_CI","false")) && identical(Sys.getenv("CI_PROJECT_NAME"), "data.table")) {
+      options("repos" = if (.Platform$OS.type == "windows") file.path("file://",getwd(),"bus/mirror-packages/cran") else file.path("file:", normalizePath("bus/mirror-packages/cran", mustWork=FALSE)))
+    }

.ci/deploy.sh

-Original file line number
+Diff line change
@@ -0,0 +1,30 @@
+    #!/bin/bash
+    set -o errexit -o nounset
+    PKG_REPO=$PWD
+    PKG_TARBALL=$(ls -1t *.tar.gz | head -n 1)
+    cd ..
+    addToDrat(){
+      mkdir drat; cd drat
+      ## Set up Repo parameters
+      git init
+      git config user.name "addToDrat"
+      git config user.email "addToDrat@travis.ci"
+      ## Get drat repo
+      git remote add upstream "https://$GH_TOKEN@github.com/Rdatatable/data.table.git" 2>err.txt
+      git fetch upstream gh-pages 2>err.txt
+      git checkout gh-pages 2>err.txt
+      git reset --hard "88000defd316538c37af4c8dc842e73e7953f4e2" 2>err.txt
+      Rscript -e "drat::insertPackage('$PKG_REPO/$PKG_TARBALL', \
+        repodir = '.', \
+        commit='Travis publish data.table: build $TRAVIS_COMMIT', \
+        addFiles=TRUE, fields='Revision')"
+      git push --force upstream gh-pages 2>err.txt
+    }
+    addToDrat

.ci/publish.R

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -91,17 +91,26 @@ package.index <- function(package, lib.loc, repodir="bus/integration/cran") {
  
      )

      vign = tools::getVignetteInfo(pkg, lib.loc=lib.loc)

      r_rel_ver = Sys.getenv("R_REL_VERSION")

      r_dev_ver = Sys.getenv("R_DEV_VERSION")

      r_old_ver = Sys.getenv("R_OLD_VERSION")

      stopifnot(nzchar(r_rel_ver), nzchar(r_dev_ver), nzchar(r_old_ver))

      r_devel_ver = Sys.getenv("R_DEVEL_VERSION")

      r_oldrel_ver = Sys.getenv("R_OLDREL_VERSION")

      stopifnot(nzchar(r_rel_ver), nzchar(r_devel_ver), nzchar(r_oldrel_ver))

      cran.home = "../../.."

      tbl.dl = c(

        sprintf("<tr><td> Reference manual: </td><td> <a href=\"%s.pdf\">%s.pdf</a>, <a href=\"%s/library/%s/html/00Index.html\">00Index.html</a> </td></tr>", pkg, pkg, cran.home, pkg),

        if (nrow(vign)) sprintf("<tr><td>Vignettes:</td><td>%s</td></tr>", paste(sprintf("<a href=\"%s/library/data.table/doc/%s\">%s</a><br/>", cran.home, vign[,"PDF"], vign[,"Title"]), collapse="\n")), # location unline cran web/pkg/vignettes to not duplicate content, documentation is in ../../../library

        sprintf("<tr><td> Package source: </td><td> <a href=\"%s/src/contrib/%s_%s.tar.gz\"> %s_%s.tar.gz </a> </td></tr>", cran.home,pkg, version, pkg, version),

        sprintf("<tr><td> Windows binaries: </td><td> %s </td></tr>", format.bins(ver=c("r-devel","r-release","r-oldrel"), bin_ver=c(r_dev_ver, r_rel_ver, r_old_ver), cran.home=cran.home, os.type="windows", pkg=pkg, version=version, repodir=repodir)),

        sprintf("<tr><td> macOS binaries: </td><td> %s </td></tr>", format.bins(ver=c("r-release","r-oldrel"), bin_ver=c(r_rel_ver, r_old_ver), cran.home=cran.home, os.type="macosx", pkg=pkg, version=version, repodir=repodir))

        sprintf("<tr><td> Windows binaries: </td><td> %s </td></tr>", format.bins(ver=c("r-devel","r-release","r-oldrel"), bin_ver=c(r_devel_ver, r_rel_ver, r_oldrel_ver), cran.home=cran.home, os.type="windows", pkg=pkg, version=version, repodir=repodir)),

        sprintf("<tr><td> macOS binaries: </td><td> %s </td></tr>", format.bins(ver=c("r-release","r-oldrel"), bin_ver=c(r_rel_ver, r_oldrel_ver), cran.home=cran.home, os.type="macosx", pkg=pkg, version=version, repodir=repodir))

      )

      if (pkg=="data.table") { ## docker images

        registry = Sys.getenv("CI_REGISTRY", "registry.gitlab.com")

        namespace = Sys.getenv("CI_PROJECT_NAMESPACE", "Rdatatable")

        project = Sys.getenv("CI_PROJECT_NAME", "data.table")

        images = c("r-release","r-devel","r-release-builder")

        images.title = c("Base R release", "Base R development", "R release package builder")

        tags = rep("latest", 3)

        docker.dl = sprintf("<tr><td> %s: </td><td> <pre><code>docker pull %s/%s/%s/%s:%s</code></pre> </td></tr>", images.title, tolower(registry), tolower(namespace), tolower(project), tolower(images), tags)

      }

      index.file = file.path(repodir, "web/packages", pkg, "index.html")

      if (!dir.exists(dirname(index.file))) dir.create(dirname(index.file), recursive=TRUE)

      writeLines(c(

    @@ -122,6 +131,11 @@ package.index <- function(package, lib.loc, repodir="bus/integration/cran") {
  
        sprintf("<table summary=\"Package %s downloads\">", pkg),

        tbl.dl,

        "</table>",

        if (pkg=="data.table")

          c("<h4>Docker images:</h4>",

            sprintf("<table summary=\"Package %s docker images\">", pkg),

            docker.dl,

            "</table>"),

        "</body>",

        "</html>"

      ), index.file)

    @@ -134,7 +148,7 @@ lib.copy <- function(lib.from, repodir="bus/integration/cran"){
  
      pkg.copy <- function(pkg.from, lib.to) {

        pkg<-basename(pkg.from);

        dir.create(file.path(lib.to, pkg), recursive=TRUE)

        lib.dirs<-intersect(c("help","html","doc"), all.lib.dirs<-list.dirs(pkg.from, full.names=FALSE))

        lib.dirs<-intersect(c("html","doc"), all.lib.dirs<-list.dirs(pkg.from, full.names=FALSE))

        ans1<-setNames(file.copy(file.path(pkg.from, lib.dirs), file.path(lib.to, pkg), recursive=TRUE), lib.dirs)

        lib.files<-setdiff(list.files(pkg.from), all.lib.dirs)

        ans2<-setNames(file.copy(file.path(pkg.from, lib.files), file.path(lib.to, pkg)), lib.files)

    @@ -155,30 +169,24 @@ plat <- function(x) if (grepl("^.*win", x)) "Windows" else if (grepl("^.*mac", x
  
    r.ver <- function(x) {

      tmp = strsplit(x, "-", fixed=TRUE)[[1L]]

      if (length(tmp) < 3L) stop("test job names must be test-[lin|win|mac]-[r.version]-...")

      v = tmp[3L]

      if (length(tmp) < 2L) stop("test job names must be test-[r.version]-...")

      v = tmp[2L]

      if (identical(v, "rel")) "r-release"

      else if (identical(v, "dev")) "r-devel"

      else if (identical(v, "old")) "r-oldrel"

      else {

        if (grepl("\\D", v)) stop("third word in test job name must be rel/dev/old or numbers of R version")

        if (grepl("\\D", v)) stop("second word in test job name must be rel/dev/old or numbers of R version")

        paste0("r-", paste(strsplit(v, "")[[1L]], collapse="."))

      }

    }

    # this for now is constant but when we move to independent pipelines (commit, daily, weekly) those values can be different

    pkg.version <- function(job, pkg) {

      Rcheck = file.path("bus", job, paste(pkg, "Rcheck", sep="."))

      if (!dir.exists(Rcheck))

        return(NA_character_)

      dcf = read.dcf(file.path(Rcheck, "00_pkg_src", pkg, "DESCRIPTION"))

      dcf = read.dcf(file.path("bus", job, paste(pkg, "Rcheck", sep="."), pkg, "DESCRIPTION"))

      dcf[,"Version"]

    }

    pkg.revision <- function(job, pkg) {

      Rcheck = file.path("bus", job, paste(pkg, "Rcheck", sep="."))

      if (!dir.exists(Rcheck))

        return(NA_character_)

      dcf = read.dcf(file.path(Rcheck, "00_pkg_src", pkg, "DESCRIPTION"))

      dcf = read.dcf(file.path("bus", job, paste(pkg, "Rcheck", sep="."), pkg, "DESCRIPTION"))

      if ("Revision" %in% colnames(dcf)) {

        proj.url = Sys.getenv("CI_PROJECT_URL", "")

        if (!nzchar(proj.url)) {

    @@ -190,10 +198,7 @@ pkg.revision <- function(job, pkg) {
  
      } else ""

    }

    pkg.flags <- function(job, pkg) {

      Rcheck = file.path("bus", job, paste(pkg, "Rcheck", sep="."))

      if (!dir.exists(Rcheck))

        return(NA_character_)

      cc = file.path(Rcheck, pkg, "cc") ## data.table style cc file

      cc = file.path("bus", job, paste(pkg, "Rcheck", sep="."), pkg, "cc") ## data.table style cc file

      if (file.exists(cc)) {

        d = readLines(cc)

        w.cflags = substr(d, 1, 7)=="CFLAGS="

    @@ -263,34 +268,6 @@ check.flavors <- function(jobs, repodir="bus/integration/cran") {
  
      setNames(file.exists(file), file)

    }

    log.copy <- function(job, repodir="bus/integration/cran") {

      dir.create(job.checks<-file.path(repodir, "web", "checks", pkg<-"data.table", job), recursive=TRUE, showWarnings=FALSE)

      to = file.path(job.checks, "log")

      if (!file.exists(job_id_file <- file.path("bus", job, "id")))

        return(setNames(file.exists(to), "log"))

      job_id = readLines(job_id_file, warn=FALSE)[1L]

      from = sprintf("https://gitlab.com/Rdatatable/data.table/-/jobs/%s/raw", job_id)

      download.file(from, to, method="wget", quiet=TRUE)

      Sys.sleep(0.1) ## to not get ban from gitlab.com

      setNames(file.exists(to), "log")

    }

    ci.status <- function(job) {

      if (!file.exists(status_file <- file.path("bus", job, "status")))

        return(NA_character_)

      readLines(status_file, warn=FALSE)[1L]

    }

    ci.log <- function(jobs, repodir="bus/integration/cran") {

      pkg = "data.table"

      ans = vector("character", length(jobs))

      logs = sapply(jobs, log.copy, repodir=repodir)

      statuses = sapply(jobs, ci.status)

      ans[!logs] = statuses[!logs]

      ans[logs] = sprintf('<a href=\"%s/%s/log\">%s</a>', pkg[any(logs)], jobs[logs], statuses[logs])

      ans

    }

    check.index <- function(pkg, jobs, repodir="bus/integration/cran") {

      status = function(x) if (grepl("^.*ERROR", x)) "ERROR" else if (grepl("^.*WARNING", x)) "WARNING" else if (grepl("^.*NOTE", x)) "NOTE" else if (grepl("^.*OK", x)) "OK" else NA_character_

      test.files = function(job, files, trim.name=FALSE, trim.exts=0L, pkg="data.table") {

    @@ -331,18 +308,17 @@ check.index <- function(pkg, jobs, repodir="bus/integration/cran") {
  
        }

        memouts

      })

      th = "<th>Flavor</th><th>Version</th><th>Revision</th><th>Install</th><th>Status</th><th>Flags</th><th>Rout.fail</th><th>Log</th><th>Memtest</th>"

      th = "<th>Flavor</th><th>Version</th><th>Revision</th><th>Install</th><th>Status</th><th>Flags</th><th>Rout.fail</th><th>Memtest</th>"

      tbl = sprintf(

        "<tr><td><a href=\"check_flavors.html\">%s</a></td><td>%s</td><td>%s</td><td><a href=\"%s/%s/00install.out\">out</a></td><td><a href=\"%s/%s/00check.log\">%s</a></td><td>%s</td><td>%s</td><td>%s</td><td>%s</td></tr>",

        sub("test-", "", jobs, fixed=TRUE), ## Flavor

        sapply(jobs, pkg.version, pkg),     ## Version

        sapply(jobs, pkg.revision, pkg),    ## Revision

        pkg, jobs,                          ## Install

        pkg, jobs, sapply(sapply(jobs, check.test, pkg="data.table"), status), ## Status

        sapply(jobs, pkg.flags, pkg),                                          ## Flags

        mapply(test.files, jobs, routs, trim.exts=2L),                         ## Rout.fail: 1st fail, 2nd Rout, keep just: tests_x64/main

        ci.log(jobs),                                                          ## CI job logs

        mapply(test.files, jobs, memouts, trim.name=TRUE)                      ## Memtest // currently not used

        "<tr><td><a href=\"check_flavors.html\">%s</a></td><td>%s</td><td>%s</td><td><a href=\"%s/%s/00install.out\">out</a></td><td><a href=\"%s/%s/00check.log\">%s</a></td><td>%s</td><td>%s</td><td>%s</td></tr>",

        sub("test-", "", jobs, fixed=TRUE),

        sapply(jobs, pkg.version, pkg),

        sapply(jobs, pkg.revision, pkg),

        pkg, jobs, ## install

        pkg, jobs, sapply(sapply(jobs, check.test, pkg="data.table"), status), ## check

        sapply(jobs, pkg.flags, pkg),

        mapply(test.files, jobs, routs, trim.exts=2L), # 1st fail, 2nd Rout, keep just: tests_x64/main

        mapply(test.files, jobs, memouts, trim.name=TRUE)

      )

      file = file.path(repodir, "web/checks", sprintf("check_results_%s.html", pkg))

      writeLines(c(

    @@ -378,8 +354,7 @@ check.test <- function(job, pkg) {
  
      check[length(check)]

    }

    move.bin <- function(job, bin.version, os.type, file="DESCRIPTION", silent=TRUE) {

      ## currently not used, if not used for macos in future then can be removed

    move.bin <- function(job, bin.version, os.type, file="DESCRIPTION", silent=FALSE) {

      if (os.type=="unix") {

        stop("publish of linux binaries not supported")

      } else if (os.type=="windows") {

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rolling functions: adaptive left, frollmax, frollapply adaptive, partial #5441

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!