Erlang garbage collection

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Erlang garbage collection

Viacheslav V. Kovalev
Hi all!
I need your help in investigation of issue with erlang memory
consumption. How typical, isn't it?

We have two different deployment schemes.
* In first scheme we running many identical nodes on small virtual
machines (in Amazon AWS),
one node per machine. Each machine has 4Gb of RAM.
* In another deployment scheme we running this nodes on big baremetal
machines (with 64 Gb of ram),
with many nodes per machine. In this deployment nodes are isolated in
docker containers
(with memory limit set to 4 Gb).

I've noticed, that heap of processes in dockerized nodes are hogging up
to 3 times much more RAM, than
heaps in non-dockerized nodes with identical load. I suspect that
garbage collection in non-dockerized nodes is more aggressive.
Unfortunately, I don't have any garbage collection statistics, but I
wanna obtain it ASAP.

To give more information, I should say that we are using HiPE R17.1 on
ubuntu 14.04 with stock kernel. In both schemes we are
running 8 schedulers per node, and using default `fullsweep_after` flag.

My blind suggestion is that erlang default garbage collection relies
(somehow) on /proc/meminfo (which is not actual in dockerized environment).
I am not C-guy and not familar with emulator internals, so could someone
point me to places in erlang sources that are responsible for garbage
collection
and some emulator options which I can use to tweak this behavior?

Viacheslav V. Kovalev
Software engineer, Tradingview.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Erlang garbage collection

dmkolesnikov
Hello,

One of the docker problem is that resources are constrained via cgroups. The ordinary system calls and  utilities reports host memory. Thus, your dockerized Erlang nodes assumes 64Gb RAM availability and they will adjust gc accordingly.

I do not see any better solution then set hard memory limit either to docker container or Erlang node.
Unfortunately, none of these options are available for you. I am afraid that you need to tune Erlang memory allocators to achieve desired effect. Here is some references:

https://erlangcentral.org/euc-2014-lukas-larsson-memory-allocators-in-the-vm-memory-management-battle-stories/#.VE39WYc3bQc

http://erlang.org/doc/man/erts_alloc.html

BTW, I am going to hit same issue during next week(s) due to my Erlang to Docker roll-out project. I’d be glad to collaborate on it to find best solution.

Best Regards,
Dmitry  


> On 25 Oct 2014, at 21:37, Viacheslav V. Kovalev <[hidden email]> wrote:
>
> Hi all!
> I need your help in investigation of issue with erlang memory consumption. How typical, isn't it?
>
> We have two different deployment schemes.
> * In first scheme we running many identical nodes on small virtual machines (in Amazon AWS),
> one node per machine. Each machine has 4Gb of RAM.
> * In another deployment scheme we running this nodes on big baremetal machines (with 64 Gb of ram),
> with many nodes per machine. In this deployment nodes are isolated in docker containers
> (with memory limit set to 4 Gb).
>
> I've noticed, that heap of processes in dockerized nodes are hogging up to 3 times much more RAM, than
> heaps in non-dockerized nodes with identical load. I suspect that garbage collection in non-dockerized nodes is more aggressive.
> Unfortunately, I don't have any garbage collection statistics, but I wanna obtain it ASAP.
>
> To give more information, I should say that we are using HiPE R17.1 on ubuntu 14.04 with stock kernel. In both schemes we are
> running 8 schedulers per node, and using default `fullsweep_after` flag.
>
> My blind suggestion is that erlang default garbage collection relies (somehow) on /proc/meminfo (which is not actual in dockerized environment).
> I am not C-guy and not familar with emulator internals, so could someone point me to places in erlang sources that are responsible for garbage collection
> and some emulator options which I can use to tweak this behavior?
>
> Viacheslav V. Kovalev
> Software engineer, Tradingview.
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Erlang garbage collection

Lukas Larsson-8
In reply to this post by Viacheslav V. Kovalev
Hello,

On Sat, Oct 25, 2014 at 8:37 PM, Viacheslav V. Kovalev <[hidden email]> wrote:
I've noticed, that heap of processes in dockerized nodes are hogging up to 3 times much more RAM, than
heaps in non-dockerized nodes with identical load. I suspect that garbage collection in non-dockerized nodes is more aggressive.

How did you measure that the heap of the processes was three times larger? erlang:memory(processes)?
 
Unfortunately, I don't have any garbage collection statistics, but I wanna obtain it ASAP.

To give more information, I should say that we are using HiPE R17.1 on ubuntu 14.04 with stock kernel. In both schemes we are
running 8 schedulers per node, and using default `fullsweep_after` flag.

My blind suggestion is that erlang default garbage collection relies (somehow) on /proc/meminfo (which is not actual in dockerized environment).

The garbage collector does very little to react to the environment it is in, so this setting should not influence how it works. 
 
 
I am not C-guy and not familar with emulator internals, so could someone point me to places in erlang sources that are responsible for garbage collection
and some emulator options which I can use to tweak this behavior?

The garbage collector is not very tweakable at all in Erlang. The only settings you can play with is min_heap_size and fullsweep_after. However, as Dmitry mentions, the erlang memory allocators do have a load of options that can be tweaked, and depending on what the root cause of the memory issue is, changing them could be effective. 

For more info about how the memory allocators work and what you can do to change their setting see the links Dmitry posted, and also I would recommend reading what Fred Hebert has written about them here https://blog.heroku.com/archives/2013/11/7/logplex-down-the-rabbit-hole and here http://ferd.github.io/recon/.

Lukas


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions