Fix malformed transmute handling during mir build

[Human9000-bit]

View all comments

Running dataflow const prop optimization on std::mem::transmute of mismatched sizes failed the assertion in interpreter and caused ICE.

So it's better to not let those transmutations into the interpreter at all

Fixes #149920

[rustbot]

Some changes occurred to MIR optimizations

cc @rust-lang/wg-mir-opt

[fmease]

r? mir

[oli-obk]

This error is emitted in typeck iirc. It should be tainting the body. If it isnt, it's likely that tcx.dcx() is used instead of infcx.dcx()

Please check if we can avoid changing anything in const prop by properly tainting wherever this error is emitted

[Human9000-bit]

At first glance there isn't infcx where the error is emitted, and no obvious way to get one

[oli-obk]

Where is it being emitted? If it is within the typeck query, we can probably change something further up the call stack ( you can try using -Ztreat-err-as-bug to find out where the error is emitted and what the call stack is while emitting it

[Human9000-bit]

It is emitted in check_transmutes ran by run_required_analyses:

rust/compiler/rustc_interface/src/passes.rs

Lines 1146 to 1149 in 981cf69

	if not_typeck_child {
	tcx.ensure_ok().mir_borrowck(def_id);
	tcx.ensure_ok().check_transmutes(def_id);
	}

and the very error emission happens here:

rust/compiler/rustc_hir_typeck/src/intrinsicck.rs

Lines 138 to 142 in 981cf69

	} else {
	err.note(format!("source type: `{}` ({})", from, skeleton_string(from, sk_from)));
	err.note(format!("target type: `{}` ({})", to, skeleton_string(to, sk_to)));
	err.emit();
	}

[oli-obk]

Oh wow. Didn't know that was changed (#145469)

I'll need to think on it. Unsure why it was necessary to live in a separate query

[oli-obk]

Yes that would fix this optimizer ICE, but i didn't suggest that because i assumed we'd get the same ICE in CTFE, and the only fix for that is to run the query earlier, where it would likely cycle again

[Human9000-bit]

We could check at the beginning of optimized_mir that check_transmutes does not emit an error.

I tried to do that at the start of run_pass of dataflow const prop. It ICEd for def_id being a typeck child.
So we can't just slap check_transmutes for every def_id (at least in mir opts)

[oli-obk]

You can invoke is_typeck_child to invoke check_transmutes on the parent (there's also a method for getting the typeck parent).

This will fix the ICE (if you also make the check_transmute query return a Result<(), ErrorReported> and don't do anything if it's an error).

Tho at that point it may be better to do what cjgillot suggested and check it at the start of thr optimized_mir query and just return a dummy body or the unoptimized body.

I'm just sceptical it fixed the ICE in general as it should be possibe to produce the ICE in const eval by evaluating code that has the same transmute problem

[oli-obk]

Hmm apparently not: https://play.rust-lang.org/?version=nightly&mode=release&edition=2024&gist=31291deac2b05c040502d266b33e2f69

It seems very weird that we report an error here, but that's probably an oversight. The ctfe error should be an ICE

[Human9000-bit]

Apparently your example is getting caught in well-formedness checking during HIR analysis, which uses different methods of interpreter compared to dataflow const prop opt

[rustbot]

This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

[Human9000-bit]

Follow-up idea: I am thinking it would be nice to add arena_cache modifier for check_trasnmutes query: Result<(), ErrorGuaranteed> is 1 byte size, and the query is now being reused.
Though it requires Result<(), ErrorGuaranteed> impl ArenaCache, so that's why it should better be in another PR.

[JonathanBrouwer]

@bors try @rust-timer queue

[Human9000-bit]

@oli-obk I tried the approach with checking transmutes another time before optimizing MIR, the perf results should be ready soon

[rust-bors]

☀️ Try build successful (CI)
Build commit: 8483d7e (8483d7e3a0b09a815a3d075b20976746c64cdc1d, parent: 14196dbfa3eb7c30195251eac092b1b86c8a2d84)

[rust-timer]

Finished benchmarking commit (8483d7e): comparison URL.

Overall result: no relevant changes - no action needed

Benchmarking means the PR may be perf-sensitive. It's automatically marked not fit for rolling up. Overriding is possible but disadvised: it risks changing compiler perf.

@bors rollup=never
@rustbot label: -S-waiting-on-perf -perf-regression

Instruction count

This perf run didn't have relevant results for this metric.

Max RSS (memory usage)

Results (primary 4.3%, secondary -6.7%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	4.3%	[3.0%, 5.1%]	3
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-6.7%	[-6.7%, -6.7%]	1
All ❌✅ (primary)	4.3%	[3.0%, 5.1%]	3

Cycles

Results (primary -2.3%, secondary 3.8%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	3.8%	[3.8%, 3.8%]	1
Improvements ✅ (primary)	-2.3%	[-2.9%, -1.5%]	3
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-2.3%	[-2.9%, -1.5%]	3

Binary size

This perf run didn't have relevant results for this metric.

Bootstrap: 491.114s -> 494.562s (0.70%)
Artifact size: 394.23 MiB -> 394.29 MiB (0.02%)

[oli-obk]

I think you should be able to do your check even earlier (in mir_drops_elaborated_and_const_checked), as long as you do it after borrowck is ensured

View changes since this review

[rustbot]

Reminder, once the PR becomes ready for a review, use @rustbot ready.

[oli-obk]

thinking it would be nice to add arena_cache modifier for check_trasnmutes query: Result<(), ErrorGuaranteed> is 1 byte size, and the query is now being reused.
Though it requires Result<(), ErrorGuaranteed> impl ArenaCache, so that's why it should better be in another PR.

That query modifier is for large query results: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/query/modifiers/struct.arena_cache.html

I think using it may even cause perf regressions

[rust-log-analyzer]

The job aarch64-gnu-llvm-21-1 failed! Check out the build log: (web) (plain enhanced) (plain)

Click to see the possible cause of the failure (guessed by this bot)

Executing "/scripts/stage_2_test_set1.sh"
+ /scripts/stage_2_test_set1.sh
PR_CI_JOB set; skipping tidy
+ '[' 1 == 1 ']'
+ echo 'PR_CI_JOB set; skipping tidy'
+ SKIP_TIDY='--skip tidy'
+ ../x.py --stage 2 test --skip tidy --skip compiler --skip src
##[group]Building bootstrap
    Finished `dev` profile [unoptimized] target(s) in 0.04s
##[endgroup]
downloading https://static.rust-lang.org/dist/2026-03-05/rustfmt-nightly-aarch64-unknown-linux-gnu.tar.xz
---
---- [ui] tests/ui/consts/transmute-size-mismatch-before-typeck.rs stdout ----
Saved the actual stderr to `/checkout/obj/build/aarch64-unknown-linux-gnu/test/ui/consts/transmute-size-mismatch-before-typeck/transmute-size-mismatch-before-typeck.stderr`
diff of stderr:

- error[E0080]: transmuting from word size type to 2 * word size type: `usize` -> `&[u8]`
-   --> $DIR/transmute-size-mismatch-before-typeck.rs:16:29
-    |
- LL | const ZST: &[u8] = unsafe { std::mem::transmute(1usize) };
-    |                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^ evaluation of `ZST` failed here
- 
7 error[E0512]: cannot transmute between types of different sizes, or dependently-sized types
8   --> $DIR/transmute-size-mismatch-before-typeck.rs:16:29
9    |

13    = note: source type: `usize` (word size)
14    = note: target type: `&[u8]` (2 * word size)
15 
- error: aborting due to 2 previous errors
+ error: aborting due to 1 previous error
17 
- Some errors have detailed explanations: E0080, E0512.
---
To only update this specific test, also pass `--test-args consts/transmute-size-mismatch-before-typeck.rs`

error: 1 errors occurred comparing output.
status: exit status: 1
command: env -u RUSTC_LOG_COLOR RUSTC_ICE="0" RUST_BACKTRACE="short" "/checkout/obj/build/aarch64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/tests/ui/consts/transmute-size-mismatch-before-typeck.rs" "-Zthreads=1" "-Zsimulate-remapped-rust-src-base=/rustc/FAKE_PREFIX" "-Ztranslate-remapped-path-to-local-path=no" "-Z" "ignore-directory-in-diagnostics-source-blocks=/cargo" "-Z" "ignore-directory-in-diagnostics-source-blocks=/checkout/vendor" "--sysroot" "/checkout/obj/build/aarch64-unknown-linux-gnu/stage2" "--target=aarch64-unknown-linux-gnu" "--check-cfg" "cfg(test,FALSE)" "--error-format" "json" "--json" "future-incompat" "-Ccodegen-units=1" "-Zui-testing" "-Zdeduplicate-diagnostics=no" "-Zwrite-long-types-to-disk=no" "-Cstrip=debuginfo" "--emit" "metadata" "-C" "prefer-dynamic" "--out-dir" "/checkout/obj/build/aarch64-unknown-linux-gnu/test/ui/consts/transmute-size-mismatch-before-typeck" "-A" "unused" "-W" "unused_attributes" "-A" "internal_features" "-A" "incomplete_features" "-A" "unused_parens" "-A" "unused_braces" "-Crpath" "-Cdebuginfo=0" "-Lnative=/checkout/obj/build/aarch64-unknown-linux-gnu/native/rust-test-helpers"
stdout: none
--- stderr -------------------------------
error[E0512]: cannot transmute between types of different sizes, or dependently-sized types
##[error]  --> /checkout/tests/ui/consts/transmute-size-mismatch-before-typeck.rs:16:29
   |
LL | const ZST: &[u8] = unsafe { std::mem::transmute(1usize) };
   |                             ^^^^^^^^^^^^^^^^^^^
   |
   = note: source type: `usize` (64 bits)
   = note: target type: `&[u8]` (128 bits)

error: aborting due to 1 previous error

For more information about this error, try `rustc --explain E0512`.
------------------------------------------

---- [ui] tests/ui/consts/transmute-size-mismatch-before-typeck.rs stdout end ----
---- [ui] tests/ui/layout/base-layout-is-sized-ice-123078.rs stdout ----
Saved the actual stderr to `/checkout/obj/build/aarch64-unknown-linux-gnu/test/ui/layout/base-layout-is-sized-ice-123078/base-layout-is-sized-ice-123078.stderr`
diff of stderr:

16 LL |     a: Box<[u8]>,
17    |        ++++    +
18 
- error[E0080]: the type `S` has an unknown layout
-   --> $DIR/base-layout-is-sized-ice-123078.rs:10:1
-    |
---
To only update this specific test, also pass `--test-args layout/base-layout-is-sized-ice-123078.rs`

error: 1 errors occurred comparing output.
status: exit status: 1
command: env -u RUSTC_LOG_COLOR RUSTC_ICE="0" RUST_BACKTRACE="short" "/checkout/obj/build/aarch64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/tests/ui/layout/base-layout-is-sized-ice-123078.rs" "-Zthreads=1" "-Zsimulate-remapped-rust-src-base=/rustc/FAKE_PREFIX" "-Ztranslate-remapped-path-to-local-path=no" "-Z" "ignore-directory-in-diagnostics-source-blocks=/cargo" "-Z" "ignore-directory-in-diagnostics-source-blocks=/checkout/vendor" "--sysroot" "/checkout/obj/build/aarch64-unknown-linux-gnu/stage2" "--target=aarch64-unknown-linux-gnu" "--check-cfg" "cfg(test,FALSE)" "--error-format" "json" "--json" "future-incompat" "-Ccodegen-units=1" "-Zui-testing" "-Zdeduplicate-diagnostics=no" "-Zwrite-long-types-to-disk=no" "-Cstrip=debuginfo" "--emit" "metadata" "-C" "prefer-dynamic" "--out-dir" "/checkout/obj/build/aarch64-unknown-linux-gnu/test/ui/layout/base-layout-is-sized-ice-123078" "-A" "unused" "-W" "unused_attributes" "-A" "internal_features" "-A" "incomplete_features" "-A" "unused_parens" "-A" "unused_braces" "-Crpath" "-Cdebuginfo=0" "-Lnative=/checkout/obj/build/aarch64-unknown-linux-gnu/native/rust-test-helpers"
stdout: none
--- stderr -------------------------------
error[E0277]: the size for values of type `[u8]` cannot be known at compilation time
##[error]  --> /checkout/tests/ui/layout/base-layout-is-sized-ice-123078.rs:5:8
   |
---
LL |     a: &[u8],
   |        +
help: the `Box` type always has a statically known size and allocates its contents in the heap
   |
LL |     a: Box<[u8]>,
   |        ++++    +

error[E0512]: cannot transmute between types of different sizes, or dependently-sized types
##[error]  --> /checkout/tests/ui/layout/base-layout-is-sized-ice-123078.rs:10:23
   |

[Human9000-bit]

It looks like tainting a body inside mir_drops_elaborated_and_const_checked disables some const eval analysis later on.
The error disappears if I move transmutes check into optimized_mir_inner

[Human9000-bit]

Though missing error messages don't seem very much informative to me, so we wouldn't lose much if we just bless these tests ig
@oli-obk what do you think?

[rust-bors]

☔ The latest upstream changes (presumably #155083) made this pull request unmergeable. Please resolve the merge conflicts.