[mono][infra] Enable runtime tests in fullAOT LLVM mode on linux-x64#92057

Merged

kotlarmilos merged 110 commits intodotnet:mainfrom

kotlarmilos:bugfix/linux-x64-fullaot-job

Apr 5, 2024

Member

kotlarmilos commented Sep 14, 2023 •

edited

Loading

Description

This PR enables jobs to run runtime tests in fullAOT LLVM mode on linux-x64. The jobs utilize the CBL-Mariner docker image, which contains the clang toolchain instead of binutils. Due to OOM and timeout issues, the tests are split into two jobs: intrinsics and other runtime tests.

Failing tests have been disabled and will be triaged in subsequent PRs. The arm64 support will be introduced in subsequent PRs. A list of disabled tests is included in #90427.


          Use llvm toolchain instead of binutils on linux

fd56ecb

kotlarmilos added NO-REVIEW area-Infrastructure-mono labels

ghost assigned kotlarmilos

kotlarmilos added this to the 9.0.0 milestone

ghost commented Sep 14, 2023

Tagging subscribers to this area: @directhex
See info in area-owners.md if you want to be subscribed.

Issue Details

Work in progress.

This PR tests the linux-x64 fullAOT CI job.

Author:	kotlarmilos
Assignees:	-
Labels:	`NO-REVIEW`, `area-Infrastructure-mono`
Milestone:	-

kotlarmilos added the NO-MERGE label

Member Author

kotlarmilos commented Sep 14, 2023

/azp run runtime-extra-platforms

azure-pipelines bot commented Sep 14, 2023

Azure Pipelines successfully started running 1 pipeline(s).

This was referenced Sep 14, 2023

NuGet failing with Response status code does not indicate success: 503 (Service Unavailable) dotnet/arcade#11723

Open

Failed USB connection via port 54050, error 61, in tvOS arm64 Release AllSubsets_Mono #82637

Closed

[mono] Tests failing with Failed to find compatible device: arm64-v8a #87267

Closed

Tracking issue for "WORKLOAD TIMED OUT" #90309

Closed

[iOS.Simulator] Failed to determine the exit code dotnet/dnceng#639

Closed

[mono] Android x64: No pending exception expected: java.lang.ClassNotFoundException #92000

Closed

Tracking issue for CI build timeouts #76454

Closed

[6.0] Android test failure: Failed to install mobile app #83422

Open

[NativeAOT] Tests failing nativeaot/SmokeTests/StackTraceMetadata/StackTraceMetadata/StackTraceMetadata.sh on tvos-arm64 #90460

Open


          Increase job timeouts

870e2d4

Member Author

kotlarmilos commented Sep 15, 2023

/azp run runtime-extra-platforms

azure-pipelines bot commented Sep 15, 2023

Azure Pipelines successfully started running 1 pipeline(s).

build-analysis bot mentioned this pull request

System.Net.Sockets.Tests failing in CI runs #90521

Closed


          Pass docker option to increase memory limit

96c3dd5

Member Author

kotlarmilos commented Sep 15, 2023

/azp run runtime-extra-platforms

azure-pipelines bot commented Sep 15, 2023

Azure Pipelines successfully started running 1 pipeline(s).


          Pass docker option to increase memory limit

fa12bb3

Member Author

kotlarmilos commented Sep 15, 2023

/azp run runtime-extra-platforms

azure-pipelines bot commented Sep 15, 2023

Azure Pipelines successfully started running 1 pipeline(s).


          Pass docker option to increase memory limit

133f3a8

Member Author

kotlarmilos commented Sep 16, 2023

/azp run runtime-extra-platforms


          Split tests in several jobs

2bdc654

Member Author

kotlarmilos commented Apr 2, 2024

/azp run runtime-extra-platforms


          Split intrinsics and other tests

08fea48

Member Author

kotlarmilos commented Apr 3, 2024

/azp run runtime-extra-platforms

kotlarmilos added 2 commits

April 3, 2024 14:24


          Disable failing tests

acbbb8b


          Split intrinsics and other runtime tests in different jobs

889d510

Member Author

kotlarmilos commented Apr 3, 2024

/azp run runtime-extra-platforms

kotlarmilos added 2 commits

April 3, 2024 19:48


          Disable failing tests

ba408d9


          Disable failing tests

3d16867

Member Author

kotlarmilos commented Apr 3, 2024

/azp run runtime-extra-platforms

kotlarmilos added 2 commits

April 3, 2024 20:04


          Disable failing tests

e744c75


          Fix formatting

5fa6730

Member Author

kotlarmilos commented Apr 3, 2024

/azp run runtime-extra-platforms

kotlarmilos changed the title ~~[mono][infra] Enable linux-x64 Mono fullAOT llvm job on extra platforms~~ [mono][infra] Enable runtime tests in fullAOT LLVM mode on linux-x64

kotlarmilos requested review from fanyang-mono, matouskozak and steveisok

April 3, 2024 18:17

kotlarmilos marked this pull request as ready for review

April 3, 2024 18:17

kotlarmilos removed the blocked label

Member Author

kotlarmilos commented Apr 4, 2024

This PR is ready for review. Failures shouldn't be related.

Runtime tests: https://dev.azure.com/dnceng-public/public/_build/results?buildId=629337&view=logs&jobId=d29f5ac9-2776-59d5-99ad-be6d4bcd416c&j=d29f5ac9-2776-59d5-99ad-be6d4bcd416c
Runtime intrinsics tests: https://dev.azure.com/dnceng-public/public/_build/results?buildId=629337&view=logs&jobId=d29f5ac9-2776-59d5-99ad-be6d4bcd416c&j=8c8e10f8-cac0-5393-e608-4ea6cc126022

matouskozak approved these changes

View reviewed changes

Member

matouskozak left a comment

LGTM! good job

eng/pipelines/extra-platforms/runtime-extra-platforms-other.yml Outdated

+                    testGroup: innerloop
+                    nameSuffix: AllSubsets_Mono_LLVMFullAot_RuntimeTests_Tests
+                    runtimeVariant: llvmfullaot
+                    buildArgs: -s mono+libs+clr.hosts+clr.iltools -c Release /p:MonoEnableLLVM=true /p:MonoBundleLLVMOptimizer=true

Member

matouskozak Apr 4, 2024

Not sure if it is needed in this PR but we should consolidate which flags are used for fullAOT/AOT jobs across our builds. For example, for perf jobs we use /p:MonoAOTEnableLLVM=true /p:MonoEnableLLVM=true /p:BuildMonoAOTCrossCompiler=true https://github.com/matouskozak/runtime/blob/12f546416d4aeb23b980a92e407fbf693823236d/eng/pipelines/coreclr/perf-non-wasm-jobs.yml#L39

Member Author

kotlarmilos Apr 4, 2024

Good point. This command only builds the runtime.

The AOT compiler is built in an optional step which contains required parameters:

runtime/eng/pipelines/common/templates/runtimes/build-runtime-tests-and-send-to-helix.yml

Lines 42 to 63 in e610649

    
           - ${{ if and(eq(parameters.runtimeFlavor, 'mono'), in(parameters.runtimeVariant, 'llvmaot', 'llvmfullaot', 'minifullaot')) }}: 
        
             - ${{ if eq(parameters.archType, 'arm64') }}: 
        
               - ${{ if eq(parameters.runtimeVariant, 'minifullaot') }}: 
        
                 - script: ./build.sh 
        
                           -subset mono 
        
                           -c ${{ parameters.buildConfig }} 
        
                           -arch ${{ parameters.archType }} 
        
                           /p:BuildMonoAotCrossCompiler=true 
        
                           /p:BuildMonoAotCrossCompilerOnly=true 
        
                           /p:CrossBuild=true 
        
                   displayName: "Build Mono Mini AOT cross compiler" 
        
               - ${{ else }}: 
        
                 - script: ./build.sh 
        
                           -subset mono 
        
                           -c ${{ parameters.buildConfig }} 
        
                           -arch ${{ parameters.archType }} 
        
                           /p:BuildMonoAotCrossCompiler=true 
        
                           /p:BuildMonoAotCrossCompilerOnly=true 
        
                           /p:MonoLibClang="/usr/local/lib/libclang.so.16" 
        
                           /p:MonoAOTEnableLLVM=true 
        
                           /p:CrossBuild=true 
        
                   displayName: "Build Mono LLVM AOT cross compiler"

Member

matouskozak Apr 4, 2024 •

edited

Loading

~~Should we align on the use of MonoBundleLLVMOptimizer vs MonoAOTEnableLLVM then?~~ Is the MonoBundleLLVMOptimizer=true necessary then? Looks like MonoEnableLLVM=true should be enough

runtime/src/mono/mono.proj

Line 46 in c7ada75

    
           <MonoBundleLLVMOptimizer Condition="'$(MonoEnableLLVM)' == 'true'">true</MonoBundleLLVMOptimizer>

eng/pipelines/extra-platforms/runtime-extra-platforms-other.yml Outdated

+                    testGroup: innerloop
+                    nameSuffix: AllSubsets_Mono_LLVMFullAot_RuntimeIntrinsicsTests
+                    runtimeVariant: llvmfullaot
+                    buildArgs: -s mono+libs+clr.hosts+clr.iltools -c Release /p:MonoEnableLLVM=true /p:MonoBundleLLVMOptimizer=true

Member

matouskozak Apr 4, 2024 •

edited

Loading

Not sure if it is possible with Helix, but could we re-use the build from AllSubsets_Mono_LLVMFullAot_RuntimeTests_Tests job? The runtime build should be the same, the difference is only which tests get build...

Member Author

kotlarmilos Apr 4, 2024 •

edited

Loading

Correct, perhaps we can create a template. @fanyang-mono Should this be moved to runtime-llvm.yml?

Member Author

kotlarmilos Apr 5, 2024

Moved to the runtime-llvm.yml and fixed llvmaot job.

kotlarmilos added 3 commits

April 4, 2024 16:50


          Move jobs to runtime-llvm.yml

2b58f7a


          Move runtimeVariant to jobParameters

c3381cc


          Update nameSuffix for runtime tests

7619c51

Member Author

kotlarmilos commented Apr 4, 2024 •

edited

Loading

These jobs have been moved to runtime-llvm.yml. Test run: https://dev.azure.com/dnceng-public/public/_build/results?buildId=630560&view=results

kotlarmilos added 3 commits

April 4, 2024 19:38


          Fix llvmot build

a00899f


          Remove redundant parameters

fb828fe


          Merge branch 'main' into bugfix/linux-x64-fullaot-job

b449744

Member Author

kotlarmilos commented Apr 5, 2024

The failures shouldn't be related.

kotlarmilos merged commit ddae9bd into dotnet:main

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

matouskozak matouskozak approved these changes

lambdageek Awaiting requested review from lambdageek

SamMonoRT Awaiting requested review from SamMonoRT

directhex Awaiting requested review from directhex

steveisok Awaiting requested review from steveisok

fanyang-mono Awaiting requested review from fanyang-mono

+1 more reviewer

vargaz vargaz approved these changes

Labels

area-Infrastructure-mono