[ltsmaster] AArch64: Backport Shippable CI and va_list fix

[kinke]

No description provided.

[kinke]

This also includes a partial va_arg implementation for non-Apple AArch64, using the LLVM intrinsic for primitive types, plus a tentative support for dynamic arrays (using the LLVM intrinsic to fetch length+ptr separately, as the slice is passed as IR struct and thus probably like 2 separate args). LLVM asserts/segfaults when trying to use its va_arg intrinsic with a non-primitive type, so I left an assert(0) for all other types.

The Shippable script performs the full testsuite and ignores any test failures for now.

[kinke]

These Shippable/Partner machines are quite interesting. Single-core performance is relatively low, but we can apparently use more than 16 cores (reported: 96; memory varies between ~32-128 GB) with still measurable gains. I limited it to 16 concurrent jobs where D is involved, to prevent out-of-memory kills in case we get a 32GB machine (mostly relevant for unittest compilations).

I backported the DMD_TESTSUITE_MAKE_ARGS env variable to ltsmaster, which cut down the overall wall time dramatically (~50 mins => 15 mins).

[joakim-noah]

Heh, nice work: why don't you pull in my commit from dlang/druntime#2257 too, to get all the druntime tests to pass?

[joakim-noah]

Tried this out, breaks this function's mangling when trying to build ldc master on Android/AArch64:

Generating /usr/bin/ld: /home/src/ldc/build/lib/libldc.a(logger.cpp.o): in function `Logger::attention(Loc&, char const*, ...)':
logger.cpp:(.text._ZN6Logger9attentionER3LocPKcz+0x60): undefined reference to `vwarning(Loc const&, char const*, std::__va_list)'
clang-6.0: error: linker command failed with exit code 1 (use -v to see invocation)

readelf -sW CMakeFiles/LDCShared.dir/gen/logger.cpp.o |grep vwarning
    45: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND _Z8vwarningRK3LocPKcSt9__va_list

readelf -sW bin/ldc2.o |grep vwarning
 19335: 0000000000000000    68 FUNC    GLOBAL DEFAULT 7993 _Z20vwarningSupplementalRK3LocPKcPSt9__va_list
 19446: 0000000000000000   100 FUNC    GLOBAL DEFAULT 7987 _Z8vwarningRK3LocPKcPSt9__va_list

May want to build master from Shippable on this branch too, to catch such bootstrap issues early.

[kinke]

Thx for testing. va_list seems to be even more f*cked up for AArch64 than for x86_64. There, it's __va_list_tag[1], C++-mangled as __va_list_tag *. Here, it's apparently std::__va_list, but with compiler-magic-only byref semantics (pass a pointer, but mangle as by-value struct arg; derived from clang IR only). What a dumb mess.

Not sure how to best fix this properly. We could define our own extern(C++, std) struct __va_list { __actual_va_list* ptr; } and handle that struct wrapper in TargetABI, or have a special case in the C++ mangler. Edit: Or just use an ABIRewrite for std::__va_list params, that's probably the cleanest alternative.

[joakim-noah]

Tried this latest va_list iteration- works great!- can even build a working ldc master natively on Android/AArch64 now. All druntime tests built natively with ldc master on my smartphone now work for the first time, and all the same Phobos modules' tests pass with one exception, an assert in std.algorithm.sorting that doesn't trigger when cross-compiled (I'll look into it more and get back).

I've kicked off an ldc build on Kai's linux/AArch64 VPS to see if it makes building ldc master work on there too. I think we'll soon see a lot more green on @redstar's AArch64 buildbot. 😄

[joakim-noah]

Ldc master for linux, ie Glibc/AArch64, now works well too, when built with ltsmaster with this patch. Pretty much all the expected tests pass, except I had to disable some test blocks in core.thread, core.sys.linux.stdio, and std.concurrency, makes sense since core.thread was always flaky with Glibc/AArch64 even with ltsmaster.

I tried building ldc master with ldc master natively on Android/AArch64, hits the same va_list link error I mentioned above, so this pull will need to be applied to master also. Couldn't run the dmd testsuite for master on Android or linux/AArch64 though, some issues with d_do_test segfaulting in its own tests and then when running too, after disabling its tests. Shows that this new native ldc master compiler for AArch64 still is a bit unstable, even though most stdlib unit tests pass.

Anyway, other than the link error with master, none of these is likely related to this pull, just reporting that most everything's working well. 🕺

[kinke]

I excluded the va_arg druntime changes again, as the intrinsic sadly isn't working anyway, and synced druntime/Phobos.

[kinke]

[Ready from my side.]

@joakim-noah: Do you happen to know whether dlang/druntime#2257 is required to get master incl. stdlibs to build with ltsmaster?

[joakim-noah]

Pretty sure it isn't required- I only apply it so the druntime test runner builds- but haven't tried without it. I simply suggested it because you're running the druntime tests on Shippable, and I figure that commit will be merged unchanged eventually anyway.

[joakim-noah]

I don't know varargs or compiler magic well enough to approve this pull, but I can confirm this latest iteration still gets LDC master to work on linux and Android/AArch64.

[kinke]

I simply suggested it because you're running the druntime tests on Shippable, and I figure that commit will be merged unchanged eventually anyway.

Yeah, in case it was required I'd have opted for cherry-picking it early, but if it can wait, I'd wait.

Merging eagerly in order to improve things for master.