Much larger old_heap than heap

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Much larger old_heap than heap

Loïc Hoguin-4
Hello,

I am trying to understand why the total_heap_size of a few processes
is so much higher than heap_size. As can be seen in the following
snippet, the old_heap is responsible for the discrepancy:

> erlang:process_info(QPid, garbage_collection_info).  
{garbage_collection_info,[{old_heap_block_size,1439468},
                          {heap_block_size,196650},
                          {mbuf_size,289},
                          {recent_size,11674},
                          {stack_size,35},
                          {old_heap_size,940791},
                          {heap_size,86028},
                          {bin_vheap_size,36483},
                          {bin_vheap_block_size,46422},
                          {bin_old_vheap_size,34148},
                          {bin_old_vheap_block_size,46422}]}

Why?

How can I reduce this? Garbage collecting does nothing.

Cheers,

--
Loïc Hoguin

Reply | Threaded
Open this post in threaded view
|

Re: Much larger old_heap than heap

Björn-Egil Dahlberg-2
Ehum?

total_heap_size = heap_size + old_heap_size. Meaning 1st gen heap + 2nd gen heap. So total_heap_size should be equal, or more probable, higher than the heap_size.

The reason you don't see that it shrinks during a garbage collect is ofc that there's still living data on the heap. 

Den ons 17 mars 2021 kl 21:53 skrev Loïc Hoguin <[hidden email]>:
Hello,

I am trying to understand why the total_heap_size of a few processes
is so much higher than heap_size. As can be seen in the following
snippet, the old_heap is responsible for the discrepancy:

> erlang:process_info(QPid, garbage_collection_info). 
{garbage_collection_info,[{old_heap_block_size,1439468},
                          {heap_block_size,196650},
                          {mbuf_size,289},
                          {recent_size,11674},
                          {stack_size,35},
                          {old_heap_size,940791},
                          {heap_size,86028},
                          {bin_vheap_size,36483},
                          {bin_vheap_block_size,46422},
                          {bin_old_vheap_size,34148},
                          {bin_old_vheap_block_size,46422}]}

Why?

How can I reduce this? Garbage collecting does nothing.

Cheers,

--
Loïc Hoguin

Reply | Threaded
Open this post in threaded view
|

Re: Much larger old_heap than heap

Loïc Hoguin-3
Hello,

I was able to dig deeper today. It's not that there was living data
(first thing I checked of course). It's that the process was processing
so much data that the old heap quickly gets full of no longer useful
data. It sits long enough in memory to make it to old heap, but not very
long in the grand scheme of things.

Setting fullsweep_after to 0 reduces the heap size by 2-10 times
depending on the current state size.

Cheers,

On 18/03/2021 18:56, Björn-Egil Dahlberg wrote:

> Ehum?
>
> total_heap_size = heap_size + old_heap_size. Meaning 1st gen heap + 2nd
> gen heap. So total_heap_size /should/ be equal, or more probable, higher
> than the heap_size.
>
> The reason you don't see that it shrinks during a garbage collect is ofc
> that there's still living data on the heap.
>
> Den ons 17 mars 2021 kl 21:53 skrev Loïc Hoguin <[hidden email]
> <mailto:[hidden email]>>:
>
>     Hello,
>
>     I am trying to understand why the total_heap_size of a few processes
>     is so much higher than heap_size. As can be seen in the following
>     snippet, the old_heap is responsible for the discrepancy:
>
>      > erlang:process_info(QPid, garbage_collection_info).
>     {garbage_collection_info,[{old_heap_block_size,1439468},
>                                {heap_block_size,196650},
>                                {mbuf_size,289},
>                                {recent_size,11674},
>                                {stack_size,35},
>                                {old_heap_size,940791},
>                                {heap_size,86028},
>                                {bin_vheap_size,36483},
>                                {bin_vheap_block_size,46422},
>                                {bin_old_vheap_size,34148},
>                                {bin_old_vheap_block_size,46422}]}
>
>     Why?
>
>     How can I reduce this? Garbage collecting does nothing.
>
>     Cheers,
>
>     --
>     Loïc Hoguin
>

--
Loïc Hoguin
https://ninenines.eu
Reply | Threaded
Open this post in threaded view
|

Re: Much larger old_heap than heap

Dan Gudmundsson-2
You said in the first mail, that you had done a garbage collect,
at least I assumed that you erlang:garbage_collect() which does a full sweep,
and after that you should only have live data left?


On Thu, Mar 18, 2021 at 7:13 PM Loïc Hoguin <[hidden email]> wrote:
Hello,

I was able to dig deeper today. It's not that there was living data
(first thing I checked of course). It's that the process was processing
so much data that the old heap quickly gets full of no longer useful
data. It sits long enough in memory to make it to old heap, but not very
long in the grand scheme of things.

Setting fullsweep_after to 0 reduces the heap size by 2-10 times
depending on the current state size.

Cheers,

On 18/03/2021 18:56, Björn-Egil Dahlberg wrote:
> Ehum?
>
> total_heap_size = heap_size + old_heap_size. Meaning 1st gen heap + 2nd
> gen heap. So total_heap_size /should/ be equal, or more probable, higher
> than the heap_size.
>
> The reason you don't see that it shrinks during a garbage collect is ofc
> that there's still living data on the heap.
>
> Den ons 17 mars 2021 kl 21:53 skrev Loïc Hoguin <[hidden email]
> <mailto:[hidden email]>>:
>
>     Hello,
>
>     I am trying to understand why the total_heap_size of a few processes
>     is so much higher than heap_size. As can be seen in the following
>     snippet, the old_heap is responsible for the discrepancy:
>
>      > erlang:process_info(QPid, garbage_collection_info).
>     {garbage_collection_info,[{old_heap_block_size,1439468},
>                                {heap_block_size,196650},
>                                {mbuf_size,289},
>                                {recent_size,11674},
>                                {stack_size,35},
>                                {old_heap_size,940791},
>                                {heap_size,86028},
>                                {bin_vheap_size,36483},
>                                {bin_vheap_block_size,46422},
>                                {bin_old_vheap_size,34148},
>                                {bin_old_vheap_block_size,46422}]}
>
>     Why?
>
>     How can I reduce this? Garbage collecting does nothing.
>
>     Cheers,
>
>     --
>     Loïc Hoguin
>

--
Loïc Hoguin
https://ninenines.eu
Reply | Threaded
Open this post in threaded view
|

Re: Much larger old_heap than heap

Loïc Hoguin-3
Right let me be more accurate about this part. Sorry about the lack of
details of the previous messages.

When I do a garbage collect it does free up memory. I was not able to
see it because the memory is filled again very quickly. I managed to see
the effect of erlang:garbage_collect/1 by running the calls directly one
after the other. But in the grand scheme of things the memory is only
reclaimed very temporarily.

The process also does its own forced erlang:garbage_collect/0 calls from
time to time. But same situation here, the memory usage goes back up
almost immediately after that.

This is a queue process in RabbitMQ of a specific type that I am
optimizing for low memory usage in order to allow having more of them on
the same node (also looking into the memory usage being more predictable).

I am testing further to confirm but it seems like in those cases
fullsweep_after=0 would work just fine and could be enabled only for
users that have concerns with memory.

The problem I am faced with however is that it does not seem possible to
configure fullsweep_after AFTER the process has already started. In
RabbitMQ, the queue processes only know what type of queue they are
after they've started, and so I cannot easily set fullsweep_after=0 only
for those queue types.

Is there a technical reason as to why fullsweep_after cannot be set
after the process has started? Is it possible to add? For some reason
the heap flags can be set via process_flag but not this one.

Eager to try a patch or help test one if it can be done.

Cheers,

On 18/03/2021 19:53, Dan Gudmundsson wrote:

> You said in the first mail, that you had done a garbage collect,
> at least I assumed that you erlang:garbage_collect() which does a full
> sweep,
> and after that you should only have live data left?
>
>
> On Thu, Mar 18, 2021 at 7:13 PM Loïc Hoguin <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Hello,
>
>     I was able to dig deeper today. It's not that there was living data
>     (first thing I checked of course). It's that the process was processing
>     so much data that the old heap quickly gets full of no longer useful
>     data. It sits long enough in memory to make it to old heap, but not
>     very
>     long in the grand scheme of things.
>
>     Setting fullsweep_after to 0 reduces the heap size by 2-10 times
>     depending on the current state size.
>
>     Cheers,
>
>     On 18/03/2021 18:56, Björn-Egil Dahlberg wrote:
>      > Ehum?
>      >
>      > total_heap_size = heap_size + old_heap_size. Meaning 1st gen
>     heap + 2nd
>      > gen heap. So total_heap_size /should/ be equal, or more probable,
>     higher
>      > than the heap_size.
>      >
>      > The reason you don't see that it shrinks during a garbage collect
>     is ofc
>      > that there's still living data on the heap.
>      >
>      > Den ons 17 mars 2021 kl 21:53 skrev Loïc Hoguin
>     <[hidden email] <mailto:[hidden email]>
>      > <mailto:[hidden email] <mailto:[hidden email]>>>:
>      >
>      >     Hello,
>      >
>      >     I am trying to understand why the total_heap_size of a few
>     processes
>      >     is so much higher than heap_size. As can be seen in the following
>      >     snippet, the old_heap is responsible for the discrepancy:
>      >
>      >      > erlang:process_info(QPid, garbage_collection_info).
>      >     {garbage_collection_info,[{old_heap_block_size,1439468},
>      >                                {heap_block_size,196650},
>      >                                {mbuf_size,289},
>      >                                {recent_size,11674},
>      >                                {stack_size,35},
>      >                                {old_heap_size,940791},
>      >                                {heap_size,86028},
>      >                                {bin_vheap_size,36483},
>      >                                {bin_vheap_block_size,46422},
>      >                                {bin_old_vheap_size,34148},
>      >                                {bin_old_vheap_block_size,46422}]}
>      >
>      >     Why?
>      >
>      >     How can I reduce this? Garbage collecting does nothing.
>      >
>      >     Cheers,
>      >
>      >     --
>      >     Loïc Hoguin
>      >
>
>     --
>     Loïc Hoguin
>     https://ninenines.eu
>

--
Loïc Hoguin
https://ninenines.eu
Reply | Threaded
Open this post in threaded view
|

Re: Much larger old_heap than heap

Lukas Larsson-8
On Thu, Mar 18, 2021 at 8:16 PM Loïc Hoguin <[hidden email]> wrote:
Is there a technical reason as to why fullsweep_after cannot be set
after the process has started? Is it possible to add? For some reason
the heap flags can be set via process_flag but not this one.

It should be relatively easy to add. It is just a matter of changing a value within the process's struct.

I think the reason why it does not exist right now is that no one has needed it enough to implement the feature :)
Reply | Threaded
Open this post in threaded view
|

Re: Much larger old_heap than heap

Loïc Hoguin-4
> > Is there a technical reason as to why fullsweep_after cannot be set
> > after the process has started? Is it possible to add? For some reason
> > the heap flags can be set via process_flag but not this one.
>
> It should be relatively easy to add. It is just a matter of changing a value within the process's struct.
>
> I think the reason why it does not exist right now is that no one has needed it enough to implement the feature :)

Thanks! I went ahead with this:

  https://github.com/erlang/otp/pull/4651

Now to use it.

--
Loïc Hoguin