Memory leak in master

[andrijapanicsb]

In Master, we have memory leak in the management server

Eventually, mgmt server will consume all free ram and cause server to swap etc.

[rohityadavcloud]

@andrijapanicsb what are the steps of reproduction? What do we need to do or what did you did (what operations) when you observed this? To check memory leak did you check the free -m or are you getting confused with the top/htop output for virtual/resident/shared memory?

[andrijapanicsb]

install the env and let it do something (perhaps sping a few VRs) - I did use this env actively though.
"ps aux" and read the 6th column, or HTOP from time to time and observer the RAM consumed by the process..
...
or wait for few days, then try a simple yum install which will fail (assuming no swap mounted) with no memory error (this is what happened initially to me, so I've investigated)

[rohityadavcloud]

@andrijapanicsb I'm trying to understand how you determine the memory size/leak exists and other env details (like hypervisor etc.) The details are missing in this issue.

htop shows three memory columns, I think you may be getting confused by the threads here. I'll however test latest master against visual VM and deploy some VMs with different networks on KVM.

[andrijapanicsb]

I see this in master with VMware 6.7

In my test env (like all Trillian test env), mgmt server has 4GB (no swap), and at some point I could not use yum due to not free memory - I've also seen in this env OOM killer kicking in and killing the mgmt server - this has not happene ever (to me) in any Trillian-deployed env - so I've used "ps aux | grep ......" to log the PS output to a file - see below a very short video, with RSS ever inreasing, even though no serios work was done in the env during 15h/day (while I'm not using it, as it's test env used only by me)

RSS increase video: https://screencast-o-matic.com/watch/cYQ0rCdYj4

[andrijapanicsb]

[root@ref-trl-1479-v-M7-andrija-panic-mgmt1 ~]# systemctl status cloudstack-management
● cloudstack-management.service - CloudStack Management Server
Loaded: loaded (/usr/lib/systemd/system/cloudstack-management.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Fri 2020-09-25 16:18:43 UTC; 1min 4s ago
Process: 47504 ExecStart=/usr/bin/java $JAVA_DEBUG $JAVA_OPTS -cp $CLASSPATH $BOOTSTRAP_CLASS (code=exited, status=1/FAILURE)
Main PID: 47504 (code=exited, status=1/FAILURE)

Sep 25 16:18:42 ref-trl-1479-v-M7-andrija-panic-mgmt1 java[47504]: INFO [c.c.s.StorageManagerImpl] (API-Job-Executor-17:ctx-a5ad92c7 job-2153 ctx-7252aa30) (logid:a0dad257) Storage pool null (1) ...ugh capacity
Sep 25 16:18:43 ref-trl-1479-v-M7-andrija-panic-mgmt1 java[47504]: OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000eb500000, 344981504, 0) failed; error='Not enough space' (errno=12)
Sep 25 16:18:43 ref-trl-1479-v-M7-andrija-panic-mgmt1 java[47504]: #
Sep 25 16:18:43 ref-trl-1479-v-M7-andrija-panic-mgmt1 java[47504]: # There is insufficient memory for the Java Runtime Environment to continue.
Sep 25 16:18:43 ref-trl-1479-v-M7-andrija-panic-mgmt1 java[47504]: # Native memory allocation (mmap) failed to map 344981504 bytes for committing reserved memory.
Sep 25 16:18:43 ref-trl-1479-v-M7-andrija-panic-mgmt1 java[47504]: # An error report file with more information is saved as:
Sep 25 16:18:43 ref-trl-1479-v-M7-andrija-panic-mgmt1 java[47504]: # /var/log/cloudstack/management/cloudstack-management.err
Sep 25 16:18:43 ref-trl-1479-v-M7-andrija-panic-mgmt1 systemd[1]: cloudstack-management.service: main process exited, code=exited, status=1/FAILURE
Sep 25 16:18:43 ref-trl-1479-v-M7-andrija-panic-mgmt1 systemd[1]: Unit cloudstack-management.service entered failed state.
Sep 25 16:18:43 ref-trl-1479-v-M7-andrija-panic-mgmt1 systemd[1]: cloudstack-management.service failed.

[rohityadavcloud]

@nvazquez can you advise if it's related to PR #4307, or an env issue or if you can propose a fix? Thanks

[GabrielBrascher]

Good catch, @andrijapanicsb.
Just for the record, I haven't seen a memory leak on the master. However, I tested master only with KVM nodes + mgmt on Ubuntu 18.04.

Do you know if this happens on master + XenServer as well?

[andrijapanicsb]

Haven't seen it elsewhere, only on VMware - but this needs reproduction by someone else - I will be reprorting back as I'm actively testing a few VMware stuff.

[rohityadavcloud]

We couldn't reproduce after 94+hrs:

The issue was seen from pkgs of a specific PR and not master. With #4407 fix ^^, based on this I'll close this issue. Let's reopen and re-investigate if it happens again.