Huge memory usage regression in dotnet 7 on linux

[8]

Description

A dotnet 7 application consumes much more memory on linux than a dotnet 6 or dotnet 5 application.

After upgrading our application to dotnet 7 we noticed a huge increase in memory consumption in linux, services that used 100 MB were now using twice that amount.

The base issue can be easily reproduced

1. create new console application
2. make sure it does not exit instantly, e.g. by using a while loop like:

while (true)
{
    Console.WriteLine("Hello, World!");
    Thread.Sleep(TimeSpan.FromSeconds(1));
}

3. Verify that in your .csproj file you are targeting dotnet 7:

<TargetFramework>net7.0</TargetFramework>

4. Switch to a Release configuration
5. Start it under linux, e.g. by select WSL in the dropdown to launch it.
6. Verify the used memory, e.g. by running ps aux in your wsl shell and taking a look at the RSS column:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root      5335  2.5  0.4 273549912 103944 pts/6 Sl+ 17:17   0:00 /usr/bin/dotnet /mnt/c/projects/sources/DotNet7Test/DotNet7Test/bin/Release/net7.0/DotNet7Test.dll

7. Realize in horror that the console process consumes 100+ MB
8. Stop the debugging
9. Target dotnet 6 by updating your csproj file to:

<TargetFramework>net6.0</TargetFramework>

10. Start it under linux, e.g. by select WSL in the dropdown to launch it.
11. Verify the used memory, e.g. by running ps aux in your wsl shell and taking a look at the RSS column:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root      5383  1.0  0.0 2961956 25504 pts/6   Sl+  17:21   0:00 /usr/bin/dotnet /mnt/c/projects/sources/DotNet7Test/DotNet7Test/bin/Release/net6.0/DotNet7Test.dll

12. Wonder why the same process consumes only 25MB using dotnet 6.

Configuration

Dotnet 7 on Linux x64, see above

Regression?

Yes, this is a major performance regression in dotnet 7 compared to dotnet 5 or 6.

Other information

Notes:
The memory consumption on windows on dotnet 7 is reasonable - it's just a little higher, but not by crazy amounts.
The difference between dotnet 5 and dotnet 6 on linux is negliable.

Take care,
Martin

[buyaa-n]

CC @dotnet/linux-team

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

Tagging subscribers to this area: @dotnet/gc
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

A dotnet 7 application consumes much more memory on linux than a dotnet 6 or dotnet 5 application.

After upgrading our application to dotnet 7 we noticed a huge increase in memory consumption in linux, services that used 100 MB were now using twice that amount.

The base issue can be easily reproduced

1. create new console application
2. make sure it does not exit instantly, e.g. by using a while loop like:

while (true)
{
    Console.WriteLine("Hello, World!");
    Thread.Sleep(TimeSpan.FromSeconds(1));
}

3. Verify that in your .csproj file you are targeting dotnet 7:

<TargetFramework>net7.0</TargetFramework>

4. Switch to a Release configuration
5. Start it under linux, e.g. by select WSL in the dropdown to launch it.
6. Verify the used memory, e.g. by running ps aux in your wsl shell and taking a look at the RSS column:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root      5335  2.5  0.4 273549912 103944 pts/6 Sl+ 17:17   0:00 /usr/bin/dotnet /mnt/c/projects/sources/DotNet7Test/DotNet7Test/bin/Release/net7.0/DotNet7Test.dll

7. Realize in horror that the console process consumes 100+ MB
8. Stop the debugging
9. Target dotnet 6 by updating your csproj file to:

<TargetFramework>net6.0</TargetFramework>

10. Start it under linux, e.g. by select WSL in the dropdown to launch it.
11. Verify the used memory, e.g. by running ps aux in your wsl shell and taking a look at the RSS column:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root      5383  1.0  0.0 2961956 25504 pts/6   Sl+  17:21   0:00 /usr/bin/dotnet /mnt/c/projects/sources/DotNet7Test/DotNet7Test/bin/Release/net6.0/DotNet7Test.dll

12. Wonder why the same process consumes only 25MB using dotnet 6.

Configuration

Dotnet 7 on Linux x64, see above

Regression?

Yes, this is a major performance regression in dotnet 7 compared to dotnet 5 or 6.

Other information

Notes:
The memory consumption on windows on dotnet 7 is reasonable - it's just a little higher, but not by crazy amounts.
The difference between dotnet 5 and dotnet 6 on linux is negliable.

Take care,
Martin

Author:	8
Assignees:	-
Labels:	area-GC-coreclr
Milestone:	-

[mangod9]

Hi @8,
yeah this is something we are aware of for unconstrained memory environments. We are working on resolving it in a upcoming servicing release, a workaround is to set DOTNET_GCHeapHardLimit to something reasonable (based on expected app usage or physical memory). Its an absolute increase of 64mb hence its exaggerated for hello world style apps.

It also shouldnt be an issue for memory constrainted containers

[enclave-alistair]

Just to add a note that we deploy a single-file self-contained linux application to end-users machines, so cannot realistically set DOTNET_GCHeapHardLimit, and an unrequired increase of 64mb will no doubt raise significant complaints.

[mangod9]

Ok, will update this issue once the fix is merged. Thanks

[EricStG]

We have a similar issue.

We upgraded 2 services to .NET 7 / ASPNET Code 7, both at least doubled memory usage (one went from a max of 15% memory usage to 80%)
Both are running in container with a 512mb limit (in Cloud Run if that matters)

[mangod9]

this issue shouldn't be affecting constrained memory containers (in fact, our TechEmpower measurements show a reduction in working set). Would you be able to share more details and/or a dump to investigate the increase further?

[EricStG]

@mangod9 Absolutely, let me know what you need