Quantcast

How can I release beam process memory?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

How can I release beam process memory?

Jack Tang
Hello list,

I run one Erlang application on Debian server and today I find the beam process consumes around 35G memory by `top` command.

```
KiB Mem:  99194912 total, 61682656 used, 37512252 free,   397380 buffers
KiB Swap:        0 total,        0 used,        0 free. 18684864 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
11858 usr1   20   0 36.850g 0.032t   6220 S  73.5 34.4   8038:49 beam.smp
```

I connect to the Erlang application using remote shell and find the mem-leaked supervisor tree and run gc on the whole tree. Code looks like blow:

```
lists:foreach(fun(E) -> PId = element(2, E), erlang:garbage_collect(PId) end, supervisor:which_children(some_thing_sup)).
```

and erlang:memory() decreases from 32G to 23G.
```
[{total,22982011544},
 {processes,12884182336},
 {processes_used,12884170336},
 {system,10097829208},
 {atom,13828705},
 {atom_used,13796692},
 {binary,170530288},
 {code,16450626},
 {ets,9637717576}]
```

However, when I input `top` command, the beam process still takes 35G memory. What can I do to release the 9G memory? Thanks

BR
-Jack

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How can I release beam process memory?

Dániel Szoboszlay
Hi Jack,

I guess the 9 GB is lost due to memory fragmentation. Erlang allocates memory in large chunks called carriers from the OS, then places the blocks your program actually needs on these carriers. A carrier can only be returned to the OS once all the blocks on it have been freed (and even then, the memory allocator may decide to keep it around for a while in case more memory is needed).

You can check with recon_alloc how much unused memory is lost due to fragmentation in the various allocators.

The bad news is that you cannot defragment the carriers, and if the selected memory allocator strategy doesn't work well for your application, you cannot change it either without restarting the emulator.

However, if the memory is wasted in the eheap_alloc, you may try to force a GC on all processes a couple of times. As the GC copies the memory, it will allocate new blocks and free up the old heap blocks. So there's a chance the allocators can compact the blocks together on fewer segments. But that's just a guess, it may or may not work at all.

Cheers,
Daniel

On Sat, 14 Jan 2017 at 08:04 Jack Tang <[hidden email]> wrote:
Hello list,

I run one Erlang application on Debian server and today I find the beam process consumes around 35G memory by `top` command.

```
KiB Mem:  99194912 total, 61682656 used, 37512252 free,   397380 buffers
KiB Swap:        0 total,        0 used,        0 free. 18684864 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
11858 usr1   20   0 36.850g 0.032t   6220 S  73.5 34.4   8038:49 beam.smp
```

I connect to the Erlang application using remote shell and find the mem-leaked supervisor tree and run gc on the whole tree. Code looks like blow:

```
lists:foreach(fun(E) -> PId = element(2, E), erlang:garbage_collect(PId) end, supervisor:which_children(some_thing_sup)).
```

and erlang:memory() decreases from 32G to 23G.
```
[{total,22982011544},
 {processes,12884182336},
 {processes_used,12884170336},
 {system,10097829208},
 {atom,13828705},
 {atom_used,13796692},
 {binary,170530288},
 {code,16450626},
 {ets,9637717576}]
```

However, when I input `top` command, the beam process still takes 35G memory. What can I do to release the 9G memory? Thanks

BR
-Jack
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How can I release beam process memory?

Frank Muller
I'm very interested in that too. Last time I asked for help only @Max Lapshin was kind enough to share some feedbacks (today's Daniel).

I would appreciate if the OTP team can teach us how to debug VM's mem issues, and  how pick the right allocator.

I'm aware of the excellent recon module. But to be honest, I was never be able to successfully use it to track my mem issue. Maybe not using it correctly. 

Thanks in advance.

Best,
/Frank

Le sam. 14 janv. 2017 à 21:47, Dániel Szoboszlay <[hidden email]> a écrit :
Hi Jack,

I guess the 9 GB is lost due to memory fragmentation. Erlang allocates memory in large chunks called carriers from the OS, then places the blocks your program actually needs on these carriers. A carrier can only be returned to the OS once all the blocks on it have been freed (and even then, the memory allocator may decide to keep it around for a while in case more memory is needed).

You can check with recon_alloc how much unused memory is lost due to fragmentation in the various allocators.

The bad news is that you cannot defragment the carriers, and if the selected memory allocator strategy doesn't work well for your application, you cannot change it either without restarting the emulator.

However, if the memory is wasted in the eheap_alloc, you may try to force a GC on all processes a couple of times. As the GC copies the memory, it will allocate new blocks and free up the old heap blocks. So there's a chance the allocators can compact the blocks together on fewer segments. But that's just a guess, it may or may not work at all.

Cheers,
Daniel

On Sat, 14 Jan 2017 at 08:04 Jack Tang <[hidden email]> wrote:
Hello list,

I run one Erlang application on Debian server and today I find the beam process consumes around 35G memory by `top` command.

```
KiB Mem:  99194912 total, 61682656 used, 37512252 free,   397380 buffers
KiB Swap:        0 total,        0 used,        0 free. 18684864 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
11858 usr1   20   0 36.850g 0.032t   6220 S  73.5 34.4   8038:49 beam.smp
```

I connect to the Erlang application using remote shell and find the mem-leaked supervisor tree and run gc on the whole tree. Code looks like blow:

```
lists:foreach(fun(E) -> PId = element(2, E), erlang:garbage_collect(PId) end, supervisor:which_children(some_thing_sup)).
```

and erlang:memory() decreases from 32G to 23G.
```
[{total,22982011544},
 {processes,12884182336},
 {processes_used,12884170336},
 {system,10097829208},
 {atom,13828705},
 {atom_used,13796692},
 {binary,170530288},
 {code,16450626},
 {ets,9637717576}]
```

However, when I input `top` command, the beam process still takes 35G memory. What can I do to release the 9G memory? Thanks

BR
-Jack


_______________________________________________


erlang-questions mailing list


[hidden email]


http://erlang.org/mailman/listinfo/erlang-questions




_______________________________________________

erlang-questions mailing list

[hidden email]

http://erlang.org/mailman/listinfo/erlang-questions


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How can I release beam process memory?

Jack Tang
In reply to this post by Dániel Szoboszlay
Hi Daniel

Thank you for the clue and excellent recon toolkits! I would like dig it deeper  :)

BR
-Jack

On Sun, Jan 15, 2017 at 4:47 AM, Dániel Szoboszlay <[hidden email]> wrote:
Hi Jack,

I guess the 9 GB is lost due to memory fragmentation. Erlang allocates memory in large chunks called carriers from the OS, then places the blocks your program actually needs on these carriers. A carrier can only be returned to the OS once all the blocks on it have been freed (and even then, the memory allocator may decide to keep it around for a while in case more memory is needed).

You can check with recon_alloc how much unused memory is lost due to fragmentation in the various allocators.

The bad news is that you cannot defragment the carriers, and if the selected memory allocator strategy doesn't work well for your application, you cannot change it either without restarting the emulator.

However, if the memory is wasted in the eheap_alloc, you may try to force a GC on all processes a couple of times. As the GC copies the memory, it will allocate new blocks and free up the old heap blocks. So there's a chance the allocators can compact the blocks together on fewer segments. But that's just a guess, it may or may not work at all.

Cheers,
Daniel

On Sat, 14 Jan 2017 at 08:04 Jack Tang <[hidden email]> wrote:
Hello list,

I run one Erlang application on Debian server and today I find the beam process consumes around 35G memory by `top` command.

```
KiB Mem:  99194912 total, 61682656 used, 37512252 free,   397380 buffers
KiB Swap:        0 total,        0 used,        0 free. 18684864 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
11858 usr1   20   0 36.850g 0.032t   6220 S  73.5 34.4   8038:49 beam.smp
```

I connect to the Erlang application using remote shell and find the mem-leaked supervisor tree and run gc on the whole tree. Code looks like blow:

```
lists:foreach(fun(E) -> PId = element(2, E), erlang:garbage_collect(PId) end, supervisor:which_children(some_thing_sup)).
```

and erlang:memory() decreases from 32G to 23G.
```
[{total,22982011544},
 {processes,12884182336},
 {processes_used,12884170336},
 {system,10097829208},
 {atom,13828705},
 {atom_used,13796692},
 {binary,170530288},
 {code,16450626},
 {ets,9637717576}]
```

However, when I input `top` command, the beam process still takes 35G memory. What can I do to release the 9G memory? Thanks

BR
-Jack
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions



--

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How can I release beam process memory?

Lukas Larsson-8
In reply to this post by Frank Muller
Hello Frank,

On Sun, Jan 15, 2017 at 12:03 AM, Frank Muller <[hidden email]> wrote:
I would appreciate if the OTP team can teach us how to debug VM's mem issues, and  how pick the right allocator.

The right allocator is the default allocator. It is only in very rare circumstances that you would want to change it. Sure you can optimize them to be more perfect, but in most cases the defaults are great.

If you suspect that you have issues that tuning erts_alloc will solve the best starting points is to read and understand Fred Hebert's excellent blogs/manuals about using erts_alloc to combat memory fragmentation. I cannot stress enough how great a resource his blog post (https://blog.heroku.com/logplex-down-the-rabbit-hole) and manual (https://www.erlang-in-anger.com/) are. 

Then you may also want to watch mine and Rickard Green's/Patrik Nyblom's talks on the subject, they basically cover the same things that Fred covers, but some parts in greater detail. There is also some internal documentation in https://github.com/erlang/otp/tree/master/erts/emulator/internal_doc that explain some of the features that you can tune in erts_alloc. There are also quite a few mail conversations on this mailing list about these types of problems.

When it comes to allocator tuning there is no magic formula. We attempted to build something that automatically tuned the settings in the instrument module (http://erlang.org/doc/man/instrument.html), but it didn't turn out very useful. The problems that to pop up with erts_alloc seem to be very different in both symptoms and solutions. 

If you explain what it is that you have issues with (backing the claims with recon_alloc statistics) and what solutions you have tried then we'll very likely answer.

Lukas

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How can I release beam process memory?

Jack Tang
In reply to this post by Jack Tang
Hello,

In the memory issue investigation, I find something interesting. As I mentioned, 
we usually adopted remote Erlang shell to interactive with nodes in production. 
Sometime we use mnesia:select(...) to return some query result, and it may contain 
billions of rows. And we leave the remote shell alive. I find the query results do 
consume the memory of remote node, since the return value is referenced. In the 
shell, we still can access it using `v(N)` command. I tried `f()` command, and it only 
unbinds the variables. So what's the mechanism of Erlang shell gc? Thanks.

BR

On Sun, Jan 15, 2017 at 2:09 PM, Jack Tang <[hidden email]> wrote:
Hi Daniel

Thank you for the clue and excellent recon toolkits! I would like dig it deeper  :)

BR
-Jack

On Sun, Jan 15, 2017 at 4:47 AM, Dániel Szoboszlay <[hidden email]> wrote:
Hi Jack,

I guess the 9 GB is lost due to memory fragmentation. Erlang allocates memory in large chunks called carriers from the OS, then places the blocks your program actually needs on these carriers. A carrier can only be returned to the OS once all the blocks on it have been freed (and even then, the memory allocator may decide to keep it around for a while in case more memory is needed).

You can check with recon_alloc how much unused memory is lost due to fragmentation in the various allocators.

The bad news is that you cannot defragment the carriers, and if the selected memory allocator strategy doesn't work well for your application, you cannot change it either without restarting the emulator.

However, if the memory is wasted in the eheap_alloc, you may try to force a GC on all processes a couple of times. As the GC copies the memory, it will allocate new blocks and free up the old heap blocks. So there's a chance the allocators can compact the blocks together on fewer segments. But that's just a guess, it may or may not work at all.

Cheers,
Daniel

On Sat, 14 Jan 2017 at 08:04 Jack Tang <[hidden email]> wrote:
Hello list,

I run one Erlang application on Debian server and today I find the beam process consumes around 35G memory by `top` command.

```
KiB Mem:  99194912 total, 61682656 used, 37512252 free,   397380 buffers
KiB Swap:        0 total,        0 used,        0 free. 18684864 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
11858 usr1   20   0 36.850g 0.032t   6220 S  73.5 34.4   8038:49 beam.smp
```

I connect to the Erlang application using remote shell and find the mem-leaked supervisor tree and run gc on the whole tree. Code looks like blow:

```
lists:foreach(fun(E) -> PId = element(2, E), erlang:garbage_collect(PId) end, supervisor:which_children(some_thing_sup)).
```

and erlang:memory() decreases from 32G to 23G.
```
[{total,22982011544},
 {processes,12884182336},
 {processes_used,12884170336},
 {system,10097829208},
 {atom,13828705},
 {atom_used,13796692},
 {binary,170530288},
 {code,16450626},
 {ets,9637717576}]
```

However, when I input `top` command, the beam process still takes 35G memory. What can I do to release the 9G memory? Thanks

BR
-Jack
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions



--



--

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Loading...