CRAN oldrel-macos install fails

[mattdowle]

It's odd that release-macos doesn't appear in the matrix.

oldrel-macos appears to be failing, on first glace, due to not linking to zlib? Our output is pretty good so I'm pleased about that at least.

I've asked Simon Urbanek on email about both.

using R version 3.6.2 (2019-12-12)
using platform: x86_64-apple-darwin15.6.0 (64-bit)
using session charset: UTF-8
checking for file ‘data.table/DESCRIPTION’ ... OK
this is package ‘data.table’ version ‘1.13.0’
checking package namespace information ... OK
checking package dependencies ... OK
checking if this is a source package ... OK
checking if there is a namespace ... OK
checking for executable files ... OK
checking for hidden files and directories ... OK
checking for portable file names ... OK
checking for sufficient/correct file permissions ... OK
checking whether package ‘data.table’ can be installed ... [30s/30s] ERROR
Installation failed.
See ‘/Volumes/SSD-Data/Builds/R-dev-web/QA/Simon/packages/el-capitan-x86_64/results/3.6/data.table.Rcheck/00install.out’ for details.
DONE
Status: 1 ERROR

See https://www.r-project.org/nosvn/R.check/r-oldrel-macos-x86_64/data.table-00install.html for details.

* installing *source* package ‘data.table’ ...
** package ‘data.table’ successfully unpacked and MD5 sums checked
** using staged installation
*** pkg-config is installed but 'pkg-config --exists zlib' did not return 0.
*** Compilation will now be attempted and if it works you can ignore this message. However,
*** if compilation fails, try 'locate zlib.h zconf.h' and ensure the zlib development library
*** is installed :
***   deb: zlib1g-dev (Debian, Ubuntu, ...)
***   rpm: zlib-devel (Fedora, EPEL, ...)
***   brew: zlib (OSX)
*** Note that zlib is required to compile R itself so you may find the advice in the R-admin
*** guide helpful regarding zlib. On Debian/Ubuntu, zlib1g-dev is a dependency of r-base as
*** shown by 'apt-cache showsrc r-base | grep ^Build-Depends | grep zlib', and therefore
*** 'sudo apt-get build-dep r-base' should be sufficient too.
*** To silence this message, please ensure that :
***   1) 'pkg-config --exists zlib' succeeds (i.e. $? -eq 0)
***   2) 'pkg-config --libs zlib' contains -lz
*** Compilation will now be attempted ...
** libs
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include  -fPIC  -Wall -g -O2  -c assign.c -o assign.o
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include  -fPIC  -Wall -g -O2  -c between.c -o between.o
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include  -fPIC  -Wall -g -O2  -c bmerge.c -o bmerge.o
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include  -fPIC  -Wall -g -O2  -c chmatch.c -o chmatch.o
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include  -fPIC  -Wall -g -O2  -c cj.c -o cj.o
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include  -fPIC  -Wall -g -O2  -c coalesce.c -o coalesce.o
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include  -fPIC  -Wall -g -O2  -c dogroups.c -o dogroups.o
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include  -fPIC  -Wall -g -O2  -c fastmean.c -o fastmean.o
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include  -fPIC  -Wall -g -O2  -c fcast.c -o fcast.o
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include  -fPIC  -Wall -g -O2  -c fifelse.c -o fifelse.o
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include  -fPIC  -Wall -g -O2  -c fmelt.c -o fmelt.o
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include  -fPIC  -Wall -g -O2  -c forder.c -o forder.o
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include  -fPIC  -Wall -g -O2  -c frank.c -o frank.o
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include  -fPIC  -Wall -g -O2  -c fread.c -o fread.o
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include  -fPIC  -Wall -g -O2  -c freadR.c -o freadR.o
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include  -fPIC  -Wall -g -O2  -c froll.c -o froll.o
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include  -fPIC  -Wall -g -O2  -c frollR.c -o frollR.o
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include  -fPIC  -Wall -g -O2  -c frolladaptive.c -o frolladaptive.o
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include  -fPIC  -Wall -g -O2  -c fsort.c -o fsort.o
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include  -fPIC  -Wall -g -O2  -c fwrite.c -o fwrite.o
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include  -fPIC  -Wall -g -O2  -c fwriteR.c -o fwriteR.o
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include  -fPIC  -Wall -g -O2  -c gsumm.c -o gsumm.o
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include  -fPIC  -Wall -g -O2  -c ijoin.c -o ijoin.o
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include  -fPIC  -Wall -g -O2  -c init.c -o init.o
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include  -fPIC  -Wall -g -O2  -c inrange.c -o inrange.o
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include  -fPIC  -Wall -g -O2  -c nafill.c -o nafill.o
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include  -fPIC  -Wall -g -O2  -c nqrecreateindices.c -o nqrecreateindices.o
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include  -fPIC  -Wall -g -O2  -c openmp-utils.c -o openmp-utils.o
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include  -fPIC  -Wall -g -O2  -c quickselect.c -o quickselect.o
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include  -fPIC  -Wall -g -O2  -c rbindlist.c -o rbindlist.o
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include  -fPIC  -Wall -g -O2  -c reorder.c -o reorder.o
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include  -fPIC  -Wall -g -O2  -c shift.c -o shift.o
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include  -fPIC  -Wall -g -O2  -c snprintf.c -o snprintf.o
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include  -fPIC  -Wall -g -O2  -c subset.c -o subset.o
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include  -fPIC  -Wall -g -O2  -c transpose.c -o transpose.o
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include  -fPIC  -Wall -g -O2  -c types.c -o types.o
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include  -fPIC  -Wall -g -O2  -c uniqlist.c -o uniqlist.o
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include  -fPIC  -Wall -g -O2  -c utils.c -o utils.o
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include  -fPIC  -Wall -g -O2  -c vecseq.c -o vecseq.o
clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include  -fPIC  -Wall -g -O2  -c wrappers.c -o wrappers.o
clang -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/Library/Frameworks/R.framework/Resources/lib -L/usr/local/lib -o data.table.so assign.o between.o bmerge.o chmatch.o cj.o coalesce.o dogroups.o fastmean.o fcast.o fifelse.o fmelt.o forder.o frank.o fread.o freadR.o froll.o frollR.o frolladaptive.o fsort.o fwrite.o fwriteR.o gsumm.o ijoin.o init.o inrange.o nafill.o nqrecreateindices.o openmp-utils.o quickselect.o rbindlist.o reorder.o shift.o snprintf.o subset.o transpose.o types.o uniqlist.o utils.o vecseq.o wrappers.o -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
installing to /Volumes/SSD-Data/Builds/R-dev-web/QA/Simon/packages/el-capitan-x86_64/results/3.6/data.table.Rcheck/00LOCK-data.table/00new/data.table/libs
generating debug symbols (dSYM)
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
Error: package or namespace load failed for ‘data.table’ in library.dynam(lib, package, package.lib):
 shared object ‘datatable.so’ not found
Error: loading failed
Execution halted
ERROR: loading failed
* removing ‘/Volumes/SSD-Data/Builds/R-dev-web/QA/Simon/packages/el-capitan-x86_64/results/3.6/data.table.Rcheck/data.table’

[mattdowle]

Although, if it couldn't find zlib, I'd expect some errors from the linking step, and those aren't there.
So maybe it's something to do with the mv in src/Makevars.in from data.table.so to datatable.so not working (permission or lock). Although, again, I would have expected some error messages in the output if that were the case.

[mattdowle]

PR #4442 is related. It isn't merged yet. If this osx install problem is related to the .so name, maybe that PR will help.

[mattdowle]

Correspondence with Simon:

---

Hi Simon,

data.table 1.13.0 was just published on CRAN.

1. On CRAN checks page r-release-macos seems to be missing. My memory might be failing but I seem to recall two rows for macos in the past, one for r-release and one for r-oldrel.

2. data.table install is failing on oldrel-macos, apparently due to not finding/linking to zlib. Any ideas please? We are passing build and tests on Travis OSX: https://travis-ci.org/github/Rdatatable/data.table

What should I do from here?

Or maybe our mv from data.table.so to datatable.so isn't working as it used to on macos. Maybe permissions. There's no output to suggest that, so I'm just trying to be helpful by guessing.
https://github.com/Rdatatable/data.table/blob/master/src/Makevars.in

Best, Matt

---

Matt,

first, you cannot assume pkg-config - why would anyone use pkg-config to check for -lz is entirely beyond me - you don't seem to rely on glib. On macOS libz is part of the system so there is no need for pkg-config, it's one-liner in autoconf to check for libz.

Also please note that blowing away user-supplied PKG_* flags is a really bad idea. You should honour either PKG_* flags or at least the standard CPPFLAGS/CFLAGS/LIBS so the user can fix any issues you have in your detection (e.g. the user user should be able to supply their own omp flags, but can't in your package - see the omp macOS link we discussed). The recommended practice is to set flags to include both, then perform tests and then substitute the result to make sure you have a consistent setup.

Finally, the reason it fails is because your script doesn't generate Makevars (due to the libz issue) so it is the standard installation that occurs. It may be helpful using standard tools like autoconf which make it much easier to log issues and make sure you don't have bugs in your configuration generation.

Out of curiosity, why do you need to re-name the dylib?

Cheers,
Simon

[mattdowle]

I suggest I write down my thoughts on this here so that others who know configuration and macos better than me can give their input before we get back to Simon...

you cannot assume pkg-config

1. We don't assume pkg-config exists, the script continues if it isn't installed.
2. But note 26 of R-exts specifically says that pkg-config is available on the macos machines that CRAN uses. So it seems reasonable to use it.

why would anyone use pkg-config to check for -lz is entirely beyond me

I guess I am pretty stupid. I did it because adding Autoconf to data.table apparently involves including the whole Autoconf program (which is a script) to the package tar.gz. If I remember correctly, and my stupidity is not failing me, the size of Autoconf approached 1MB and tripped the CRAN 5MB size limits. It seems odd to me that every package should include a copy of the whole Autoconf program, so why not use pkg-conf which does exist on each machine and is pointed to by R-exts.

Also please note that blowing away user-supplied PKG_* flags is a really bad idea

I didn't know this was even an idea we had. I don't know which flags we can and can't blow away. It is beyond me.

I looked again at PR #4374 which was included in this release but I don't think anything there was related to this new issue (it doesn't touch flags like this as far as a quick glance goes).

the user user should be able to supply their own omp flags, but can't in your package - see the omp macOS link we discussed

The link Simon is referring to I guess is https://mac.r-project.org/openmp/. Search that for "Unfortuantely, some packages don't". I did see that before but I didn't know if Simon was referring to us or not. Seems like he is then.

Finally, the reason it fails is because your script doesn't generate Makevars (due to the libz issue) so it is the standard installation that occurs.

This seems to be the key then. The script is supposed to make an attempt anyway if libz fails, and it appears to be doing so, so I don't know why Makevars might not be being generated.

Note that macos builds worked before on CRAN and we have been using pkg-config for several releases now. The macos toolchain used by CRAN has changed at roughly the same time as data.table 1.13.0 being released.

It may be helpful using standard tools like autoconf which make it much easier to log issues and make sure you don't have bugs in your configuration generation.

If that really is the best solution then I suppose we can try autoconf again. Seems a shame to lose the simpler and much lighter pkg-config approach, though. We could live with CRANs status being NOTE due to the size I guess.

Out of curiosity, why do you need to re-name the dylib?

I seem to remember it's because macos doesn't accept .so with a dot in the name: data.table.so. Or maybe it was Windows didn't like data.table.dll. And because one OS had an issue with dot, we have to rename everywhere. Or something like that. Or is it even R that doesn't like . in the name of the so/dll. It goes way back, maybe over 10 years. Pending PR #4442 changes the rename from . to _ but I don't remember the root cause of why we can't leave the .so named data.table.so.

[jangorecki]

Adding 1mb to PKG sources doesn't sound like a good idea.
We could, if nothing else works, add a nozlib flag in configure. Working example in https://github.com/Rdatatable/data.table/tree/nozlib branch.
That would also help on Solaris issue related to zlib.

[mattdowle]

---

Simon,
There is a team of people working on data.table. I have copied your reply to the issue tracker, I'll put my thoughts there, and someone (perhaps not me) will get back to you.
#4640
Matt

---

Matt,
No one needs to get back to be, just fix the bugs and re-submit to CRAN.
Cheers,
Simon

[mattdowle]

I had asked Simon why r-release-macos was missing but he didn't answer on that.
Interestingly, r-release-macos has appeared now and is passing 1.13.0 ok. So it works on r-release-macos but not r-oldrel-macos. There may be other differences too between those instances, not just r-release vs r-oldrel, such as running on a different macos machine with different environment.

[mattdowle]

Simon,
I see that r-release-macos has appeared now and it is passing 1.13.0 fully OK status.
So it doesn't seem to be a macos problem per se.
Why does it work on r-release-macos but not r-oldrel-macos?
The only error message I can see from r-oldrel-macos is "shared object ‘datatable.so’ not found". Is there any change that can be made to show more diagnosis? We are happy to help by submitting a patch to R if you can help guide us. For example, perhaps some output from some part of the installation is not being directed to the log. I know that this time you have told me over email that the Makefile is not being generated. So you can see an error somewhere that we can't. That's what I'm asking if it's possible to tee to the log file that we can see so that we can debug without needing to contact you.
As usual, I'm copying these messages to the issue tracker so others can take part too. #4640
Best, Matt

[mattdowle]

From Professor Ripley:

---

Please do comply with the CRAN policy and not send HTML as you are asked.

I do not know why this 'worked' for Simon: it fails for me with both
R-devel and R 4.0.2. As Simon has already explained, the nub is

echo "*** Compilation will now be attempted ..."
exit 0
fi

which exits without creating src/Makevars . On my macOS boxes,
removing that exit line suffices.

Telling people to use HomeBrew (which is not normally installed and
indeed having 'brew' on the path breaks more than a handful of other
CRAN packages) when there is a perfectly good system zlib which previous
versions found is unhelpful.

Also, the Solaris error is one reported and corrected before, and
documented in 'Writing R Extensions'.

BDR

---

[mattdowle]

Reply from Matt:

---

Please do comply with the CRAN policy and not send HTML as you are asked.

I wasn't even aware I send in HTML. I use gmail in the browser and on
my phone. I have researched and it seems there is an option at the
bottom under three dots to send in plain ascii, and I have selected
that this time. It is going to be difficult to remember to do that
every time just for you. If I know the reason why you can't accept
HTML, I will try. Could you add to the policy why this rule exists?

It would be great if you could raise issues and correspond on GitHub
with me and others. We find that more eyes help vs private email. As
usual, I've copied this correspondence to the issue tracker so that
the knowledge is shared: #4640.

That said, we will continue to strive for the ideal state of no
communication with you due to no problems.

I do not know why this 'worked' for Simon: it fails for me with both
R-devel and R 4.0.2. As Simon has already explained, the nub is

echo "*** Compilation will now be attempted ..."
exit 0
fi

which exits without creating src/Makevars . On my macOS boxes,
removing that exit line suffices.

Thank you for elaborating, that is very helpful.

Telling people to use HomeBrew (which is not normally installed and
indeed having 'brew' on the path breaks more than a handful of other
CRAN packages) when there is a perfectly good system zlib which previous
versions found is unhelpful.

I think it's better to phrase suggestions in the positive than the
negative. It just keeps everyone happier.
We will improve the wording of the message, and thank you for bringing
this to our attention.

Also, the Solaris error is one reported and corrected before, and
documented in 'Writing R Extensions'.

We're aware, thanks. Tracking here: #4638
It's the same issue we've seen and solved before, but in a different
place. We don't know how to add a check to our release procedures to
detect a const nth for Solaris. If you can think of any way we can add
a check for this to prevent it occuring in future, please let us know.

Best, Matt

---

NB: the R-ext documentation referred to is footnote 36 in section 1.2.1.1.