[Android CoreCLR] Log managed callstacks on native crash

[mdh1418]

Android CoreCLR currently does not ship with CreateDump. When native crashes occur, it is difficult to figure out the underlying cause of the crash, as managed stacks aren't captured anywhere, and android's crash reporter doesn't understand runtime symbols when it generates a tombstone.

Until we have a solution for creating a dump on Android CoreCLR, this PR looks to log a crash report stack trace for the crashing thread whenever a dump would have been created. That way, there is some hint as to what may have caused the crash, despite not having the native callstack.

The stack trace contains both managed frames and [Native Code] markers.

The crash report is created for Android whenever PROCCreateCrashDumpIfEnabled would have been hit via a PROCCreateCrashReportAndDumpIfEnabled helper.

Example ADB log

Unhandled Managed Exception

02-10 13:09:24.654 22261 22275 E DOTNET  : Unhandled exception. System.InvalidOperationException: Test unhandled exception from managed code
02-10 13:09:24.654 22261 22275 E DOTNET  :    at Program.Level3()
02-10 13:09:24.654 22261 22275 E DOTNET  :    at Program.Level2()
02-10 13:09:24.654 22261 22275 E DOTNET  :    at Program.Level1()
02-10 13:09:24.654 22261 22275 E DOTNET  :    at Program.Main(String[] args)
02-10 13:09:24.655 22261 22275 F DOTNET  : Aborting process.

    static void Level1() { Level2(); }
    static void Level2() { Level3(); }
    static void Level3()
    {
        throw new InvalidOperationException("Test unhandled exception from managed code");
    }

    public static int Main(string[] args)
    {
        Level1();

        Console.WriteLine("Done.");
        return 42;
    }

Managed FailFast

02-10 13:03:25.765 20699 20715 E DOTNET  : Process terminated.
02-10 13:03:25.765 20699 20715 E DOTNET  : Test FailFast from managed code
02-10 13:03:25.767 20699 20715 E DOTNET  :    at System.Environment.FailFast(System.Runtime.CompilerServices.StackCrawlMarkHandle, System.String, System.Runtime.CompilerServices.ObjectHandleOnStack, System.String)
02-10 13:03:25.768 20699 20715 E DOTNET  :    at System.Environment.FailFast(System.Threading.StackCrawlMark ByRef, System.String, System.Exception, System.String)
02-10 13:03:25.768 20699 20715 E DOTNET  :    at System.Environment.FailFast(System.String)
02-10 13:03:25.768 20699 20715 E DOTNET  :    at Program.Level3()
02-10 13:03:25.768 20699 20715 E DOTNET  :    at Program.Level2()
02-10 13:03:25.768 20699 20715 E DOTNET  :    at Program.Level1()
02-10 13:03:25.768 20699 20715 E DOTNET  :    at Program.Main(System.String[])
02-10 13:03:25.768 20699 20715 E DOTNET  :    at [Native Code]
02-10 13:03:25.768 20699 20715 F DOTNET  : Aborting process.

    static void Level1() { Level2(); }
    static void Level2() { Level3(); }
    static void Level3()
    {
        Environment.FailFast("Test FailFast from managed code");
    }

    public static int Main(string[] args)
    {
        Level1();

        Console.WriteLine("Done.");
        return 42;
    }

P/Invoke Abort

02-13 22:00:20.958 25664 25742 E DOTNET  : Fatal error.
02-13 22:00:20.959 25664 25742 E DOTNET  : Got a SIGABRT while executing native code. This usually indicates
02-13 22:00:20.959 25664 25742 E DOTNET  : a fatal error in the runtime or one of the native libraries
02-13 22:00:20.959 25664 25742 E DOTNET  : used by your application.
02-13 22:00:20.974 25664 25742 E DOTNET  :    at Program.abort()
02-13 22:00:20.985 25664 25742 E DOTNET  :    at Program.abort()
02-13 22:00:20.985 25664 25742 E DOTNET  :    at Program.Level3()
02-13 22:00:20.985 25664 25742 E DOTNET  :    at Program.Level2()
02-13 22:00:20.985 25664 25742 E DOTNET  :    at Program.Level1()
02-13 22:00:20.986 25664 25742 E DOTNET  :    at Program.Main(System.String[])
02-13 22:00:20.986 25664 25742 F DOTNET  : Aborting process.

    static void Level1() { Level2(); }
    static void Level2() { Level3(); }

    [DllImport("libc", EntryPoint = "abort")]
    static extern void abort();
    static void Level3()
    {
        abort();
    }

    public static int Main(string[] args)
    {
        Level1();

        Console.WriteLine("Done.");
        return 42;
    }

SigSegV

Crashes, no logging, as expected by #123735
Now that its merged

02-13 21:58:54.758 24775 24801 E DOTNET  : Fatal error.
02-13 21:58:54.760 24775 24801 E DOTNET  : Got a SIGSEGV while executing native code. This usually indicates
02-13 21:58:54.760 24775 24801 E DOTNET  : a fatal error in the runtime or one of the native libraries
02-13 21:58:54.760 24775 24801 E DOTNET  : used by your application.
02-13 21:58:54.761 24775 24801 E DOTNET  :    at Program.memset(IntPtr, Int32, UIntPtr)
02-13 21:58:54.761 24775 24801 E DOTNET  :    at Program.memset(IntPtr, Int32, UIntPtr)
02-13 21:58:54.761 24775 24801 E DOTNET  :    at Program.Level3()
02-13 21:58:54.761 24775 24801 E DOTNET  :    at Program.Level2()
02-13 21:58:54.761 24775 24801 E DOTNET  :    at Program.Level1()
02-13 21:58:54.762 24775 24801 E DOTNET  :    at Program.Main(System.String[])
02-13 21:58:54.762 24775 24801 F DOTNET  : Aborting process.

    static void Level1() { Level2(); }
    static void Level2() { Level3(); }

    [DllImport("libc", EntryPoint = "memset")]
    static extern IntPtr memset(IntPtr dest, int c, nuint n);
    static void Level3()
    {
        memset(IntPtr.Zero, 0, 1);
    }

    public static int Main(string[] args)
    {
        Level1();

        Console.WriteLine("Done.");
        return 42;
    }

Interleaved frames

02-13 21:55:44.486 23029 23054 E DOTNET  : Fatal error.
02-13 21:55:44.487 23029 23054 E DOTNET  : Got a SIGABRT while executing native code. This usually indicates
02-13 21:55:44.487 23029 23054 E DOTNET  : a fatal error in the runtime or one of the native libraries
02-13 21:55:44.487 23029 23054 E DOTNET  : used by your application.
02-13 21:55:44.489 23029 23054 E DOTNET  :    at Program.PthreadOnceCallback()
02-13 21:55:44.489 23029 23054 E DOTNET  :    at Program.pthread_once(PthreadOnce*, Void ())
02-13 21:55:44.489 23029 23054 E DOTNET  :    at Program.pthread_once(PthreadOnce*, Void ())
02-13 21:55:44.489 23029 23054 E DOTNET  :    at Program.ManagedLevel3()
02-13 21:55:44.490 23029 23054 E DOTNET  :    at Program.ManagedLevel2()
02-13 21:55:44.490 23029 23054 E DOTNET  :    at Program.ManagedLevel1()
02-13 21:55:44.490 23029 23054 E DOTNET  :    at Program.Main(System.String[])
02-13 21:55:44.490 23029 23054 F DOTNET  : Aborting process.

    static void ManagedLevel1() { ManagedLevel2(); }
    static void ManagedLevel2() { ManagedLevel3(); }

    [DllImport("libc", EntryPoint = "abort")]
    static extern void abort();
    static void ManagedAfterNative()
    {
        abort();
    }

    [StructLayout(LayoutKind.Sequential)]
    struct PthreadOnce { public int State; }

    [DllImport("libc", EntryPoint = "pthread_once")]
    static extern unsafe int pthread_once(PthreadOnce* once_control, delegate* unmanaged<void> init_routine);

    [UnmanagedCallersOnly]
    static void PthreadOnceCallback() => ManagedAfterNative();

    static unsafe void ManagedLevel3()
    {
        PthreadOnce once = new() { State = 0 };
        _ = pthread_once(&once, &PthreadOnceCallback);
    }

    public static int Main(string[] args)
    {
        // Call stack: Main -> ManagedLevel1 -> ManagedLevel2 -> ManagedLevel3 -> [Native pthread_once] -> PthreadOnceCallback -> ManagedAfterNative -> abort
        ManagedLevel1();

        Console.WriteLine("Done.");
        return 42;
    }

[dotnet-policy-service]

Tagging subscribers to this area: @agocke
See info in area-owners.md if you want to be subscribed.

[Copilot]

Pull request overview

This PR enables Android CoreCLR to log managed stack traces when a native crash occurs, providing diagnostic information in environments where CreateDump is not available. The logging is wired through the existing crash-dump hook so that, whenever a dump would be created, the crashing managed thread’s stack is emitted to Android’s logging.

Changes:

• Add a HOST_ANDROID helper in EEPolicy that walks and logs the current managed thread’s call stack using existing LogCallstackForLogWorker infrastructure.
• Extend PROCCreateCrashDumpIfEnabled on Android to call this helper via a weak symbol and emit a formatted “Managed Stacktrace” section to minipal_log_write_fatal, followed by the existing abort message.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
src/coreclr/vm/eepolicy.cpp	Introduces LogCallstackForAndroidNativeCrash, an extern "C" Android-only helper that obtains the current Thread and logs its managed call stack, to be invoked from the PAL crash-dump path.
src/coreclr/pal/src/thread/process.cpp	On Android, adds a weak reference to LogCallstackForAndroidNativeCrash and, when present, logs a managed stacktrace banner and the managed stack in PROCCreateCrashDumpIfEnabled before logging that the process is aborting.

[jkotas]

What is the experience for unhandled managed exceptions or fail fasts with this fix? We should make sure that the stacktrace is logged just once.

[jkotas]

Example ADB log

What was the source code or runtime change for this example?

[vitek-karas]

Just curious - would we also consider printing out full mixed callstack (native/managed)? The OS stackwalker will only be able to walk the first native section of the stack, once it hits managed code it will stop. So if there's native code above the managed code which would be interesting, we will have no way to learn about it. We could possibly print out just native addresses for the native portion, and rely on symbolication later.

[mdh1418]

What was the source code or runtime change for this example?

I'm modifying the Android.Device_Emulator.JIT.Test Functional test as that was the easiest way to apply runtime changes.
I've been using the following forced native crashes:

• FailFast via RaiseFailFastException placed in eventpipe and using the in-proc native EventListener
• Segfault via *(volatile int*)0 = 0x1234; in eventpipe and using the in-proc native EventListener
• Segfault via P/Invoke memset(nuull, 0, 1); in managed
• Unhandled managed exception

That particular log in the description is the RaiseFailFastException in ep_buffer_write_event.

This log is from unhandled managed exception (haven't yet changed the format yet from discussions in above threads)

01-31 14:45:41.891  1346  1372 E DOTNET  : Unhandled exception. System.Exception: MIHW
01-31 14:45:41.891  1346  1372 E DOTNET  :    at Program.ForceNativeSegv()
01-31 14:45:41.891  1346  1372 E DOTNET  :    at Program.Main(String[] args)
01-31 14:45:41.892  1346  1372 F DOTNET  : =================================================================
01-31 14:45:41.892  1346  1372 F DOTNET  : 	Managed Stacktrace:
01-31 14:45:41.892  1346  1372 F DOTNET  : =================================================================
01-31 14:45:41.893  1346  1372 E DOTNET  :    at Program.ForceNativeSegv()
01-31 14:45:41.893  1346  1372 E DOTNET  :    at Program.Main(System.String[])
01-31 14:45:41.893  1346  1372 F DOTNET  : =================================================================
01-31 14:45:41.893  1346  1372 F DOTNET  : Aborting process.

[jkotas]

Here are my thoughts on this:

• For unhandled exceptions and fail fasts, the default experience for what you see in the Android console should match what you see on stderr on Windows or Linux. The error message and stacktrace (printed exactly once).

• For crashes in unmanaged code, the managed stacktrace is better than nothing but it will rarely help with diagnosing the problem. For example, I think it is impossible to connect the dots from the managed stacktrace to the crash in eventpipe in the test you have shared. I expect the Android thumbstone to be a lot more useful for these, yes - it needs to be symbolicated offline. I am not sure whether it is a good idea to print the managed stacktrace to the console in this case since I will likely be misleading most of the time. If we want to do it anyway, it needs to come with a good description of what it is to avoid confusion.

[lateralusX]

Just curious - would we also consider printing out full mixed callstack (native/managed)? The OS stackwalker will only be able to walk the first native section of the stack, once it hits managed code it will stop. So if there's native code above the managed code which would be interesting, we will have no way to learn about it. We could possibly print out just native addresses for the native portion, and rely on symbolication later.

This would be my next suggestion. extend our current stack walker to handler mixed native/managed frames and output symbolicated managed frames, but native frames will probably follow the same format as we get in the tombstone, address + (module + build id) in text. I think this will be beneficial, not just for Android, and it will make it simpler than combining native crash data from tombstone with the managed stack trace in logcat. If we include native frames in our internal stack walker, we also have some more capabilities to handle scenarios that the native stack-walker would have issues handle. On Android we could probably go one additional step on higher API level and use Androids libunwind and cover both native + java frames when stackwalking IP's falling outside dotnet managed code.

Do we all agree it would make sense to extend our runtime stack-walker to have an option to produce mixed stack traces, only used for scenarios like this so shouldn't impact other stack walking usage, unless enabled.

[lateralusX]

What is the experience for unhandled managed exceptions or fail fasts with this fix? We should make sure that the stacktrace is logged just once.

Agree, we should only log the stacktrace for the faulting thread once, since it will go into logcat that is a limited resource. Question is if we should do it on the call site instead of inside PROCCreateCrashDumpIfEnabled and focus that function to produce the json crash report on Android. In the scenario of unhandled managed exception and fail fast the managed stack trace is logged on call site before triggering call ending up in PROCCreateCrashDumpIfEnabled. For native crashes, that would mean logging this information as part of the signal handler, alternative is to change the logic for existing unmanaged exception and fail fast and make sure all scenarios (including signal handler) end up calling through the same function for logging the managed stack and then potentially create the crash dump when enabled. Feels like doing this change will make things a little cleaner and we centralize the logging in one function.

[lateralusX]

Here are my thoughts on this:

• For unhandled exceptions and fail fasts, the default experience for what you see in the Android console should match what you see on stderr on Windows or Linux. The error message and stacktrace (printed exactly once).
• For crashes in unmanaged code, the managed stacktrace is better than nothing but it will rarely help with diagnosing the problem. For example, I think it is impossible to connect the dots from the managed stacktrace to the crash in eventpipe in the test you have shared. I expect the Android thumbstone to be a lot more useful for these, yes - it needs to be symbolicated offline. I am not sure whether it is a good idea to print the managed stacktrace to the console in this case since I will likely be misleading most of the time. If we want to do it anyway, it needs to come with a good description of what it is to avoid confusion.

I think we should start with the managed stacktrace when hitting unhandled HW exceptions + tombstone and it should give you enough information and available in logcat or crash reports submitted to the play console. Next step is probably to do a mixed stacktraces since we will get all information in one place and handle more scenarios than what the native stack walker can handle, will stop on first managed frame. On Mono we do try to dump the native stacktrace followed by a managed stacktrace, but the native stacktrace has been disabled on Android, so Mono currently only dumps the managed stacktrace, then the Android crash daemon will generate the native crash report.

Maybe we could put a marker frame on top top frame of the managed stacktrace in case of crash in native code? Then we could do the same for transition in/out of runtime code through the stackwalk and when we have a mixed mode stackwalker, we can just replace those parts with the real native stack frames, something like this:

[External Code]
Program.CallSomeNativeCodeThatCrash()
Program.Main(System.String[])

[jkotas]

Do we all agree it would make sense to extend our runtime stack-walker to have an option to produce mixed stack traces

This should be a diagnostic specific stackwalker like what's in createdump today. I do not think we do not want to be teaching the runtime managed stack-walker that is used by the GC, EH, etc. about walking native frames for diagnostics.

if we should do it on the call site instead of inside PROCCreateCrashDumpIfEnabled

Yes. PROCCreateCrashDumpIfEnabled should keep doing what its name says. It should not print stacktraces to console.

[External Code]

Visual Studio stack window uses the same marked to indicate managed frames in managed libraries outside your project.

Something similar that's more unique like [Native Code] would work.

[jkotas]

From this PR's "current" commit - pal callback pattern, with unguarding Android ifdefs

Fatal error.
   at Program.<<Main>$>g__pthread_once|0_0(UInt32, UInt32, UInt32, UInt32, UInt32)
   at Program.<Main>$(System.String[])
   at [Native Code]

I would expect the callstack to start with [Native Code] since it crashed there. [Native Code] frame at the end is not wrong, but it is not useful either.

Maybe we should drop the Android ifdefs to add [Native Code] and just do the same as other platforms.

[Copilot]

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 3 comments.

[mdh1418]

Took a brief attempt at getting the [Native Code] markers to align more to our expectations. Without a deeper understanding of stack walking, I couldn't spot a non-heuristic/hard-coded approach to getting a [Native Code] marker at the top of the stack trace. Trying to use NOTIFY_ON_INITIAL_NATIVE_CONTEXT kind of broke stackwalkin (as the comment in threads.h suggested) to which no frames were getting printed. It looks like dacdbiimplstackwalk.cpp has a different approach, but I've decided to punt the mixed stacks for the proper json report in-proc crash dump work.

In the interim, the comprehensive error message with the signal in the commit Add descriptive native crash message similar to Mono will make it clear the crash originated from native.

Also, seems like Copilot CLI thinks the double frames for P/Invoke calls is

 Found it. Lines 2566-2578 (stackwalk.cpp): When the stack walker encounters an InlinedCallFrame for P/Invoke IL stubs, it reports the actual interop
  MethodDesc (the P/Invoke method like pthread_once). This is a "skipped frame" — the InlinedCallFrame was skipped past during
  unwinding, and the walker backtracks to report it.

  So the sequence is:

   1. Walker finds the frameless (JIT-compiled) IL stub for the P/Invoke → reports pthread_once (via GetActualInteropMethodDesc)
   2. Walker finds the InlinedCallFrame that was skipped → reports pthread_once again (same MethodDesc)

  Both pass the FUNCTIONSONLY filter since pFunc != NULL. This is the duplicate.

  This is a pre-existing behavior in StackWalkFrames — it's not specific to your changes. On Linux/Windows the LogCallstackForLogWorker
  path would have the same duplicate for FailFast scenarios after a P/Invoke. It just hasn't been noticed because native crashes use
  createdump there.

  You could deduplicate consecutive identical frames in CallStackLogger:

[AaronRobinsonMSFT]

Without a deeper understanding of stack walking,

@max-charlamb Can you help out @mdh1418 with this?

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.

[jkotas]

Thanks

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

[Copilot]

Calling PROCLogManagedCallstackForSignal(SIGABRT) immediately before PROCAbort() can cause duplicate logging when System.Runtime.CrashReportBeforeSignalChaining is enabled: the subsequent SIGABRT handler will attempt to log again and LogInfoForFatalError will emit "Fatal error while logging another fatal error." for the same thread. Consider removing this pre-log (rely on the SIGABRT handler path), or otherwise ensure only one of these sites logs for SIGABRT.

Suggested change

PROCLogManagedCallstackForSignal(SIGABRT);