Extremely high and unexpected memory commit sizes using DOTNET_GCHeapHardLimit

[ANahr]

Description

We are running lots of instances of kestrel .net core processes on a server machines.
Using DOTNET_GCHeapHardLimitPercent we tried to limit the memory of a single process instance (to 128GB), so that it would be impossible for a single process to take all machine memory (256GB). This seems to work but has completely unexpected negative side effects on total memory consumption. Running the process requires about approx. 350-400MB of private memory and has a memory commit size as follows:

• Reference Windows; 8 core; 32GB; NOT using DOTNET_GCHeapHardLimit: 0.6GB
• Reference Windows; 8 core; 32GB; using DOTNET_GCHeapHardLimit: 2.0GB
• Server Windows; 16 core; 256GB; using DOTNET_GCHeapHardLimit: 6.5GB
• Server Windows; 28 core; 256GB; using DOTNET_GCHeapHardLimit: 10.5GB

We expect to run 100+ of those processes on one of the machines, but this is impossible with this commit sizes because the processes get an OutOfMemoryException (or other OOMs) instantly.
Worst case the commit size is 30 times larger than the private memory used!

Reproduction Steps

Please note that we are NOT running in containers.
Take a .net core process.
Set DOTNET_GCHeapHardLimitPercent to a high value (e.g. 75%) on a machine with lots of RAM (e.g. 256GB) many cores (e.g. 28) and observe an extremely high memory commit.

Expected behavior

Using DOTNET_GCHeapHardLimitPercent should not create extremely high memory commits compared to not using this switch. It should LIMIT memory, not enlarge it (and certainly not by unreasonable dimensions).

Actual behavior

If you set DOTNET_GCHeapHardLimit to large memory sizes it consumes high amounts of commit memory (much larger than when not setting DOTNET_GCHeapHardLimit at all) for no clear reason.
Please note that setting other switches seems to have no or little effect (see list below)

Here are some measurements with different settings on a machine with 8 cores, 32GB RAM:

[DEFAULT]
Physical: 380
Private: 340
Commit: 650

DOTNET_GCConserveMemory=9
Physical: 380
Private: 340
Commit: 650

DOTNET_GCHeapHardLimitPercent=A (10%)
Physical: 380
Private: 340
Commit: 620

DOTNET_GCHeapHardLimitPercent=5 (5%)
Physical: 350
Private: 310
Commit: 480

DOTNET_GCHeapHardLimitPercent=5A (90%)
Physical: 380
Private: 340
Commit: 2000

DOTNET_GCHeapHardLimitPercent=5A (90%)
DOTNET_GCHeapCount=4
Physical: 400
Private: 355
Commit: 2000

DOTNET_GCHeapHardLimitPercent=5A (90%)
DOTNET_GCHeapCount=1
Physical: 400
Private: 360
Commit: 2000

DOTNET_GCHeapHardLimitPercent=5A (90%)
DOTNET_GCHeapCount=1
DOTNET_GCHighMemPercent=5
Physical: 400
Private: 360
Commit: 2000

DOTNET_GCHeapHardLimitPercent=5A (90%)
DOTNET_GCHeapCount=1
DOTNET_GCConserveMemory=9
Physical: 300
Private: 350
Commit: 1900

DOTNET_GCHeapHardLimitPercent=5A (90%)
DOTNET_GCHeapCount=100
Physical: 380
Private: 340
Commit: 2000

Regression?

No response

Known Workarounds

None/not using DOTNET_GCHeapHardLimit at all

Configuration

.Net Core 6; Windows

Other information

No response

Tagging subscribers to this area: @dotnet/gc
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

We are running lots of instances of kestrel .net core processes on a server machines.
Using DOTNET_GCHeapHardLimitPercent we tried to limit the memory of a single process instance (to 128GB), so that it would be impossible for a single process to take all machine memory (256GB). This seems to work but has completely unexpected negative side effects on total memory consumption. Running the process requires about approx. 350-400MB of private memory and has a memory commit size as follows:

• Reference Windows; 8 core; 32GB; NOT using DOTNET_GCHeapHardLimit: 0.6GB
• Reference Windows; 8 core; 32GB; using DOTNET_GCHeapHardLimit: 2.0GB
• Server Windows; 16 core; 256GB; using DOTNET_GCHeapHardLimit: 6.5GB
• Server Windows; 28 core; 256GB; using DOTNET_GCHeapHardLimit: 10.5GB

We expect to run 100+ of those processes on one of the machines, but this is impossible with this commit sizes because the processes get an OutOfMemoryException (or other OOMs) instantly.
Worst case the commit size is 30 times larger than the private memory used!

Reproduction Steps

Please note that we are NOT running in containers.
Take a .net core process.
Set DOTNET_GCHeapHardLimitPercent to a high value (e.g. 75%) on a machine with lots of RAM (e.g. 256GB) many cores (e.g. 28) and observe an extremely high memory commit.

Expected behavior

Using DOTNET_GCHeapHardLimitPercent should not create extremely high memory commits compared to not using this switch. It should LIMIT memory, not enlarge it (and certainly not by unreasonable dimensions).

Actual behavior

If you set DOTNET_GCHeapHardLimit to large memory sizes it consumes high amounts of commit memory (much larger than when not setting DOTNET_GCHeapHardLimit at all) for no clear reason

Here are some measurements with different settings on a machine with 8 cores, 32GB RAM:

[DEFAULT]
Physical: 380
Private: 340
Commit: 650

DOTNET_GCConserveMemory=9
Physical: 380
Private: 340
Commit: 650

DOTNET_GCHeapHardLimitPercent=A (10%)
Physical: 380
Private: 340
Commit: 620

DOTNET_GCHeapHardLimitPercent=5 (5%)
Physical: 350
Private: 310
Commit: 480

DOTNET_GCHeapHardLimitPercent=5A (90%)
Physical: 380
Private: 340
Commit: 2000

DOTNET_GCHeapHardLimitPercent=5A (90%)
DOTNET_GCHeapCount=4
Physical: 400
Private: 355
Commit: 2000

DOTNET_GCHeapHardLimitPercent=5A (90%)
DOTNET_GCHeapCount=1
Physical: 400
Private: 360
Commit: 2000

DOTNET_GCHeapHardLimitPercent=5A (90%)
DOTNET_GCHeapCount=1
DOTNET_GCHighMemPercent=5
Physical: 400
Private: 360
Commit: 2000

DOTNET_GCHeapHardLimitPercent=5A (90%)
DOTNET_GCHeapCount=1
DOTNET_GCConserveMemory=9
Physical: 300
Private: 350
Commit: 1900

DOTNET_GCHeapHardLimitPercent=5A (90%)
DOTNET_GCHeapCount=100
Physical: 380
Private: 340
Commit: 2000

Regression?

No response

Known Workarounds

None/not using DOTNET_GCHeapHardLimit at all

Configuration

.Net Core 6; Windows

Other information

No response

Author:	ANahr
Assignees:	-
Labels:	area-GC-coreclr, untriaged
Milestone:	-

[mangod9]

Hello @ANahr, some clarifications: if you expect to run 100+ processes on these machines then you would need a fairly low GCHardLimitPercent, possibly in the single digits. So your observation with a limit of 5 shown below, assume it keeps the commit size low (appears the numbers are in mb)? With a higher setting the GC might optimize for utilizing that much memory to avoid allocation/deallocation costs.

DOTNET_GCHeapHardLimitPercent=5 (5%)
Physical: 350
Private: 310
Commit: 480

[Maoni0]

this

[DEFAULT]
Physical: 380
Private: 340
Commit: 650

DOTNET_GCHeapHardLimitPercent=5A (90%)
Physical: 380
Private: 340
Commit: 2000

DOTNET_GCHeapHardLimitPercent=5A (90%)
DOTNET_GCHeapCount=1
Physical: 400
Private: 360
Commit: 2000

seems very suspicious to me. that's saying you are committing the same amount of memory with 1 heap and 8 heaps. and that they both commit a lot more with a limit that's basically the same as all physical memory on the machine which also uses 8 heaps.

could you possibly capture some traces to diagnose this?

How to collect top level GC metrics

GC emits lightweight Informational level events that you can collect (and leave on all the time if you want) that covers all top level GC metrics. Use this PerfView commandline to collect these events –

perfview /GCCollectOnly /AcceptEULA /nogui collect

after you are done, press s in the perfview cmd window to stop it.

This should be run long enough to capture enough GC activities, e.g., if you know problems occur at times, this should cover time that leads up to when problems happen (not only during problematic time). If you are not sure when the problem starts to occur, you can just have it on for a long time.

If you know how long to run the collection, use the following (this is used much more often actually) –

perfview /GCCollectOnly /AcceptEULA /nogui /MaxCollectSec:1800 collect

and replace 1800 (half an hour) with however many seconds you need. You can of course apply this arg to other commandline as well. This will result in a file called PerfViewGCCollectOnly.etl.zip. In PerfView terms, we call this a GCCollectOnly trace.

On Linux there's sort of an equivalent which is this dotnet trace commandline:

dotnet trace collect -p -o <outputpath with .nettrace extension> --profile gc-collect --duration

It's sort of an equivalent because it collects the same GC events but only for one process that already started, whereas the perfview commandline collects ETW for the whole machine, ie, the GC events from every process on that machine, even the ones started after the beginning of the collection.

[ANahr]

Thanks @Maoni0 for your reply. I will try to get the info. However it is easily reproducable even with a super-simple application.
Please consider following application:
Program.cs:

Console.WriteLine("I need lots of memory (maybe)!");
Console.ReadLine();

On a machine with 8 cores and 32GB RAM needs the following amouts of RAM (for reference using Workstation GC):
[Default (no DOTNET config switches set)]
Working set: 14.6
Private: 3.5
Commited: 6.5

DOTNET_GCHeapHardLimitPercent=1 (1%)
Working set: 14.3
Private: 3.5
Commited: 10.1

DOTNET_GCHeapHardLimitPercent=5A (90%)
Working set: 14.6
Private: 3.8
Commited: 341.7

Note: DOTNET_GCHeapCount is ignored with non-server gc (as documented)
When using server GC we get the following:

[Default (no DOTNET config switches set)]
Working set: 15.7
Private: 5.0
Commited: 65.3

DOTNET_GCHeapHardLimitPercent=5A (90%)
DOTNET_GCHeapCount=1
Working set: 14.8
Private: 4.1
Commited: 212.3

DOTNET_GCHeapHardLimitPercent=5A (90%)
Working set: 16.1
Private: 5.5
Commited: 240.5

Yes - thats 240 or 340MB of memory for "hello world". ;)
On https://docs.microsoft.com/en-us/dotnet/core/runtime-config/garbage-collector it is written that "Heap limit Specifies the maximum commit size..."
This does not seem to be the entire truth. It also seems to specify the minimum/default commit size AND for anything but tiny values this minimum size will be a LOT LOT LOT larger than not using Heap limit at all

So takeaways are:
Setting DOTNET_GCHeapHardLimit to anything but tiny values will

• Greatly increase memory commit
• Behave even worse on Workstation GC than on Server GC
• Always deliver worse memory commit sizes that Workstation GC
• Always deliver worse memory commit sizes that Server GC

that surely does not seem correct for a switch that is named "Limit".
If this really should be intended behavior (which I cannot even imagine) then I would propose adding a warning on the page mentioned above, because this is highly unexpected.

[ANahr]

Hello @ANahr, some clarifications: if you expect to run 100+ processes on these machines then you would need a fairly low GCHardLimitPercent, possibly in the single digits. So your observation with a limit of 5 shown below, assume it keeps the commit size low (appears the numbers are in mb)? With a higher setting the GC might optimize for utilizing that much memory to avoid allocation/deallocation costs.

Hello @mangod9 , unfortunatelly the description in the initial comment was simplified to show our problem. In the real application the memory use of the processes differs greatly. We have LOTS of small ones that usually stay that small, but in general they are allowed to grow up to 128GB. We have a scheduler that heuristically predicts memory use. However it is not guaranteed to be exact, so we can only limit to 128GB.
Our assumption was that DOTNET_GCHeapHardLimit would only set the maximum commit size (see https://docs.microsoft.com/en-us/dotnet/core/runtime-config/garbage-collector).
Also its unclear if there are any good use-cases for committing 10+GB of memory for an application that only uses some hundered MBs.

[cshung]

I am on it

[Maoni0]

@ANahr setting a limit does not mean it will make the memory usage smaller. it's merely to say you are telling the GC that GC is free to use up to this much memory. in 6.0 we are still using segments which means you'll be committing a lot for GC's own bookkeeping since we commit all that up front, and that's proportional to the reserve range of the heap which is proportional to the limit you specified. in 7.0 this should be much better because this is only proportional to what you use, not the limit you specified.

however! when I looked at this in .net 7, I noticed there is a bug in the soh_segment_size calcuation. when I set it to 90%, I'm running on a machine with 64GB mem so that's about 58GB. and I'm seeing when we get to init_semi_shared, soh_segment_size is 0x1000000000, which means we are creating a mark_list of 256mb. that's clearly wrong. @cshung, could you please take a look and get a fix ASAP? thanks.

[Maoni0]

(sorry I didn't mean to delete it so I recreated it)

I think this soh_segment_size is actually the same as what we would calculate for segments as well, but we should definitely not create a mark_list that's so big!