Cannot build for x86_64-pc-windows-gnu with nightly toolchain

[alepez]

Check this github actions run: https://github.com/alepez/lavagna/actions/runs/8694841480/job/23844594711

It fails with this errors:

     Running `rustc --crate-name build_script_build --edition=2021 'C:\Users\runneradmin\.cargo\registry\src\index.crates.io-6f17d22bba15001f\proc-macro2-1.0.80\build.rs' --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C opt-level=3 -C embed-bitcode=no -C debuginfo=2 -C debug-assertions=on --cfg 'feature="default"' --cfg 'feature="proc-macro"' -C metadata=39763306c87edd3f -C extra-filename=-39763306c87edd3f --out-dir 'D:\a\lavagna\lavagna\target\debug\build\proc-macro2-39763306c87edd3f' -L 'dependency=D:\a\lavagna\lavagna\target\debug\deps' --cap-lints allow`
     Running `rustc --crate-name unicode_ident --edition=2018 'C:\Users\runneradmin\.cargo\registry\src\index.crates.io-6f17d22bba15001f\unicode-ident-1.0.12\src\lib.rs' --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no -C debuginfo=2 -C debug-assertions=on -C metadata=64545b8510a0495e -C extra-filename=-64545b8510a0495e --out-dir 'D:\a\lavagna\lavagna\target\debug\deps' -L 'dependency=D:\a\lavagna\lavagna\target\debug\deps' --cap-lints allow`
     Running `rustc --crate-name cfg_if --edition=2018 'C:\Users\runneradmin\.cargo\registry\src\index.crates.io-6f17d22bba15001f\cfg-if-1.0.0\src\lib.rs' --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no -C debuginfo=2 -C debug-assertions=on -C metadata=426b6ce52fa3b61a -C extra-filename=-426b6ce52fa3b61a --out-dir 'D:\a\lavagna\lavagna\target\x86_64-pc-windows-gnu\debug\deps' --target x86_64-pc-windows-gnu -L 'dependency=D:\a\lavagna\lavagna\target\x86_64-pc-windows-gnu\debug\deps' -L 'dependency=D:\a\lavagna\lavagna\target\debug\deps' --cap-lints allow -C debuginfo=0 --deny warnings`
     Running `rustc --crate-name autocfg --edition=2015 'C:\Users\runneradmin\.cargo\registry\src\index.crates.io-6f17d22bba15001f\autocfg-1.2.0\src\lib.rs' --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no -C debuginfo=2 -C debug-assertions=on -C metadata=29be611d5ad42c4f -C extra-filename=-29be611d5ad42c4f --out-dir 'D:\a\lavagna\lavagna\target\debug\deps' -L 'dependency=D:\a\lavagna\lavagna\target\debug\deps' --cap-lints allow`
   Compiling getrandom v0.2.14
     Running `rustc --crate-name getrandom --edition=2018 'C:\Users\runneradmin\.cargo\registry\src\index.crates.io-6f17d22bba15001f\getrandom-0.2.14\src\lib.rs' --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no -C debuginfo=2 -C debug-assertions=on --cfg 'feature="std"' -C metadata=040fcd85f621becf -C extra-filename=-040fcd85f621becf --out-dir 'D:\a\lavagna\lavagna\target\x86_64-pc-windows-gnu\debug\deps' --target x86_64-pc-windows-gnu -L 'dependency=D:\a\lavagna\lavagna\target\x86_64-pc-windows-gnu\debug\deps' -L 'dependency=D:\a\lavagna\lavagna\target\debug\deps' --extern 'cfg_if=D:\a\lavagna\lavagna\target\x86_64-pc-windows-gnu\debug\deps\libcfg_if-426b6ce52fa3b61a.rmeta' --cap-lints allow -C debuginfo=0 --deny warnings`
   Compiling serde v1.0.197
     Running `rustc --crate-name build_script_build --edition=2018 'C:\Users\runneradmin\.cargo\registry\src\index.crates.io-6f17d22bba15001f\serde-1.0.197\build.rs' --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C opt-level=3 -C embed-bitcode=no -C debuginfo=2 -C debug-assertions=on --cfg 'feature="alloc"' --cfg 'feature="default"' --cfg 'feature="derive"' --cfg 'feature="serde_derive"' --cfg 'feature="std"' -C metadata=52b1ad7a3248ce2e -C extra-filename=-52b1ad7a3248ce2e --out-dir 'D:\a\lavagna\lavagna\target\debug\build\serde-52b1ad7a3248ce2e' -L 'dependency=D:\a\lavagna\lavagna\target\debug\deps' --cap-lints allow`
   Compiling version_check v0.9.4
     Running `rustc --crate-name version_check --edition=2015 'C:\Users\runneradmin\.cargo\registry\src\index.crates.io-6f17d22bba15001f\version_check-0.9.4\src\lib.rs' --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no -C debuginfo=2 -C debug-assertions=on -C metadata=9c303afa4680e8e7 -C extra-filename=-9c303afa4680e8e7 --out-dir 'D:\a\lavagna\lavagna\target\debug\deps' -L 'dependency=D:\a\lavagna\lavagna\target\debug\deps' --cap-lints allow`
   Compiling pin-project-lite v0.2.14
     Running `rustc --crate-name pin_project_lite --edition=2018 'C:\Users\runneradmin\.cargo\registry\src\index.crates.io-6f17d22bba15001f\pin-project-lite-0.2.14\src\lib.rs' --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no -C debuginfo=2 --warn=unreachable_pub '--warn=clippy::undocumented_unsafe_blocks' '--warn=clippy::transmute_undefined_repr' '--warn=clippy::trailing_empty_array' --warn=single_use_lifetimes --warn=rust_2018_idioms '--warn=clippy::pedantic' --warn=non_ascii_idents '--warn=clippy::inline_asm_x86_att_syntax' --warn=improper_ctypes_definitions --warn=improper_ctypes '--warn=clippy::default_union_representation' '--warn=clippy::as_ptr_cast_mut' '--warn=clippy::all' '--allow=clippy::type_complexity' '--allow=clippy::too_many_lines' '--allow=clippy::too_many_arguments' '--allow=clippy::struct_field_names' '--allow=clippy::struct_excessive_bools' '--allow=clippy::single_match_else' '--allow=clippy::single_match' '--allow=clippy::similar_names' '--allow=clippy::module_name_repetitions' '--allow=clippy::missing_errors_doc' '--allow=clippy::manual_range_contains' '--allow=clippy::manual_assert' '--allow=clippy::float_cmp' '--allow=clippy::doc_markdown' '--allow=clippy::declare_interior_mutable_const' '--allow=clippy::borrow_as_ptr' '--allow=clippy::bool_assert_comparison' -C debug-assertions=on -C metadata=315eac90ce7c4ee0 -C extra-filename=-315eac90ce7c4ee0 --out-dir 'D:\a\lavagna\lavagna\target\x86_64-pc-windows-gnu\debug\deps' --target x86_64-pc-windows-gnu -L 'dependency=D:\a\lavagna\lavagna\target\x86_64-pc-windows-gnu\debug\deps' -L 'dependency=D:\a\lavagna\lavagna\target\debug\deps' --cap-lints allow -C debuginfo=0 --deny warnings`
   Compiling windows_x86_64_gnu v0.52.5
     Running `rustc --crate-name build_script_build --edition=2021 'C:\Users\runneradmin\.cargo\registry\src\index.crates.io-6f17d22bba15001f\windows_x86_64_gnu-0.52.5\build.rs' --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C opt-level=3 -C embed-bitcode=no -C debuginfo=2 -C debug-assertions=on -C metadata=47efefe8991d7656 -C extra-filename=-47efefe8991d7656 --out-dir 'D:\a\lavagna\lavagna\target\debug\build\windows_x86_64_gnu-47efefe8991d7656' -L 'dependency=D:\a\lavagna\lavagna\target\debug\deps' --cap-lints allow`
   Compiling winapi-x86_64-pc-windows-gnu v0.4.0
     Running `rustc --crate-name build_script_build --edition=2015 'C:\Users\runneradmin\.cargo\registry\src\index.crates.io-6f17d22bba15001f\winapi-x86_64-pc-windows-gnu-0.4.0\build.rs' --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C opt-level=3 -C embed-bitcode=no -C debuginfo=2 -C debug-assertions=on -C metadata=fb4429030be33de9 -C extra-filename=-fb4429030be33de9 --out-dir 'D:\a\lavagna\lavagna\target\debug\build\winapi-x86_64-pc-windows-gnu-fb4429030be33de9' -L 'dependency=D:\a\lavagna\lavagna\target\debug\deps' --cap-lints allow`
   Compiling thiserror v1.0.58
     Running `rustc --crate-name build_script_build --edition=2021 'C:\Users\runneradmin\.cargo\registry\src\index.crates.io-6f17d22bba15001f\thiserror-1.0.58\build.rs' --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C opt-level=3 -C embed-bitcode=no -C debuginfo=2 -C debug-assertions=on -C metadata=c787188e94e5de22 -C extra-filename=-c787188e94e5de22 --out-dir 'D:\a\lavagna\lavagna\target\debug\build\thiserror-c787188e94e5de22' -L 'dependency=D:\a\lavagna\lavagna\target\debug\deps' --cap-lints allow`
     Running `D:\a\lavagna\lavagna\target\debug\build\windows_x86_64_gnu-47efefe8991d7656\build-script-build`
     Running `D:\a\lavagna\lavagna\target\debug\build\serde-52b1ad7a3248ce2e\build-script-build`
error: failed to run custom build command for `windows_x86_64_gnu v0.52.5`

Caused by:
  process didn't exit successfully: `D:\a\lavagna\lavagna\target\debug\build\windows_x86_64_gnu-47efefe8991d7656\build-script-build` (exit code: 0xc0000139, STATUS_ENTRYPOINT_NOT_FOUND)
warning: build failed, waiting for other jobs to finish...
error: failed to run custom build command for `serde v1.0.197`

Caused by:
  process didn't exit successfully: `D:\a\lavagna\lavagna\target\debug\build\serde-52b1ad7a3248ce2e\build-script-build` (exit code: 0xc0000139, STATUS_ENTRYPOINT_NOT_FOUND)
Error: Process completed with exit code 101.

[alepez]

This issue needs an in-depth analysis. For now, I disable the affected platform.

[patowen]

I decided to look into this myself, and it seems to run far deeper than Lavagna. I'm starting to get very surprised that nobody else has encountered this.

I can reproduce it with a minimal working example: https://github.com/patowen/lavagna/tree/broken-build-script (See patowen#1 for the failing workflows, although that link is less reliable).

Since I may delete my fork at some point, I'll list out the minimal reproducible example here:

.github\workflows\ci.yml:

name: ci
on:
  workflow_dispatch:
  pull_request:
  push:
    branches:
    - main
  schedule:
  - cron: '00 01 * * *'
jobs:
  build:
    name: build

    env:
      # Emit backtraces on panics.
      RUST_BACKTRACE: 1
      CARGO_INCREMENTAL: 0

    runs-on: windows-2019
    strategy:
      fail-fast: false
      matrix:
        rust_version:
          - stable
          - beta
          - nightly

    steps:
    - name: Checkout repository
      uses: actions/checkout@v2

    - name: Install Rust
      uses: actions-rs/toolchain@v1
      with:
        toolchain: ${{ matrix.rust_version }}-x86_64-pc-windows-gnu
        target: x86_64-pc-windows-gnu
        profile: minimal
        override: true

    - name: Build
      shell: bash
      run: cargo build --verbose --target x86_64-pc-windows-gnu

src\main.rs:

fn main() {
    println!("Hello world");
}

build.rs:

fn main() {
    println!("I am in a build script.");
}

Cargo.lock:

# This file is automatically @generated by Cargo.
# It is not intended for manual editing.
version = 3

[[package]]
name = "proc-macro2-testing-for-lavagna"
version = "0.1.0"

Cargo.toml:

[package]
name = "proc-macro2-testing-for-lavagna"
version = "0.1.0"
edition = "2021"

Result of "Build" step on beta branch:

Run cargo build --verbose --target x86_64-pc-windows-gnu
   Compiling proc-macro2-testing-for-lavagna v0.1.0 (D:\a\lavagna\lavagna)
     Running `rustc --crate-name build_script_build --edition=2021 build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debuginfo=2 -C metadata=b191affa0bd3624f -C extra-filename=-b191affa0bd3624f --out-dir 'D:\a\lavagna\lavagna\target\debug\build\proc-macro2-testing-for-lavagna-b191affa0bd3624f' -L 'dependency=D:\a\lavagna\lavagna\target\debug\deps'`
     Running `D:\a\lavagna\lavagna\target\debug\build\proc-macro2-testing-for-lavagna-b191affa0bd3624f\build-script-build`
error: failed to run custom build command for `proc-macro2-testing-for-lavagna v0.1.0 (D:\a\lavagna\lavagna)`

Caused by:
  process didn't exit successfully: `D:\a\lavagna\lavagna\target\debug\build\proc-macro2-testing-for-lavagna-b191affa0bd3624f\build-script-build` (exit code: 0xc0000139, STATUS_ENTRYPOINT_NOT_FOUND)
Error: Process completed with exit code 101.

[patowen]

I haven't been able to reproduce the error locally, so it seems specific to GitHub actions, possibly some configuration it uses. I tried building the minimal example with a bunch of different nightly versions on GitHub, and it seems like nightly-2024-02-26 (also known as 1.78.0-nightly (0ecbd0605 2024-02-25)) is the last good version, and nightly-2024-02-27 (also known as 1.78.0-nightly (fc3800f65 2024-02-26)) is the first bad version.

Unfortunately, it isn't as simple as checking the comparison because it's huge, and nothing seems to mention GNU (https://github.com/rust-lang/rust/compare/0ecbd0605..fc3800f65).

The only thing I've found so far that could be even semi-related is the addition of #[cfg(target_vendor = "win7")] in library/std/src/sys/pal/windows/compat.rs (which I found by searching for "entry"). Those changes come from rust-lang/rust#121317, which did seem to cause a cross compilation bug, but there's good reason to believe it's unrelated. It's probably worth just filing a bug in https://github.com/rust-lang/rust/issues and seeing if anyone more familiar with Rust finds anything.

[patowen]

Interestingly, using Windows Server 2022 instead of 2019 seems to fix the issue.

I asked in Zulip in case anyone has any better ideas, but there does seem to be an effective workaround now. Perhaps I would have been able to reproduce this locally if I owned a copy of Windows Server 2019.

[patowen]

It seems like the root cause was that the Windows 2019 version on GitHub Actions has an outdated version of MinGW, and changes were made to rustc that were incompatible with that outdated version. This compatibility has been restored in the latest nightly release (2024-04-19).

I believe any of the following actions will resolve this issue with lavagna:

• Waiting for any of the following to happen (it doesn't matter which):
  • Use raw-dylib for Windows synchronization functions rust-lang/rust#124019 is backported to the beta branch (might not happen)
  • Update MinGW in Windows Server 2019 image for (probably) future Rust support actions/runner-images#9702 is resolved (might not happen)
  • Rust 1.79.0 is released on June 13th (should happen unless the compatibility fix is reverted)
• Using os: windows-2022 instead of os: windows-2019 in ci.yml
• Seeing if there's a way to configure windows-2019 to use a different version of gcc (likely the most work for little benefit)

Now that simply waiting will likely resolve this issue, I think it's safe to ignore it for now.

I mainly took the time to investigate this because I'm thankful that this project was made and wanted to try to help out where I could on the (likely) more annoying aspects of maintaining it.

[alepez]

Nice catch! I'm trying with windows-2022 right now, which seems to be the quickest fix possible.