very long init:stop() and +P emulator flag

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

very long init:stop() and +P emulator flag

Vyacheslav Levytskyy-2
Hi,

I've experienced very long duration of Erlang node shutdown after
init:stop() : about 8 minutes. At last I've found a solution -- when I
remove emulator flag for the maximum number of simultaneously existing
processes (+P 134217727) the shutdown of the node return to normal
couple of seconds.

I'm using Erlang/OTP 20.0 on Ubuntu LTS 16.04, node runs in embedded
mode. The problem with long shutdown happens after all applications have
already gone, and just 30-40 processes and about 10 ports of kernel
remains running. Actually it happens even after a clean boot with just
kernel and stdlib applications started. Option -shutdown_time does not
help. The only solution that works for me at the moment is to remove +P
134217727 -- maybe decreasing the number of simultaneously existing
processes helps as well (it should), but I did not check that and did
not search after which value of +P the problem would disappear.

I'm quite puzzled with relations between +P and really long init:stop(),
and would like to ask if somebody can explain that?

Vyacheslav


.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: very long init:stop() and +P emulator flag

Tristan Sloughter-4
I checked and running erl with that +P 134217727 results in a beam
process using 1.517g of memory. So cleaning that up is probably what is
taking so much time on shutdown :)

134217727 certainly seems excessive... What size system would even be
able to handle dealing with that many processes?

--
  Tristan Sloughter
  "I am not a crackpot" - Abe Simpson
  [hidden email]

On Mon, Sep 25, 2017, at 12:02 PM, Vyacheslav Levytskyy wrote:

> Hi,
>
> I've experienced very long duration of Erlang node shutdown after
> init:stop() : about 8 minutes. At last I've found a solution -- when I
> remove emulator flag for the maximum number of simultaneously existing
> processes (+P 134217727) the shutdown of the node return to normal
> couple of seconds.
>
> I'm using Erlang/OTP 20.0 on Ubuntu LTS 16.04, node runs in embedded
> mode. The problem with long shutdown happens after all applications have
> already gone, and just 30-40 processes and about 10 ports of kernel
> remains running. Actually it happens even after a clean boot with just
> kernel and stdlib applications started. Option -shutdown_time does not
> help. The only solution that works for me at the moment is to remove +P
> 134217727 -- maybe decreasing the number of simultaneously existing
> processes helps as well (it should), but I did not check that and did
> not search after which value of +P the problem would disappear.
>
> I'm quite puzzled with relations between +P and really long init:stop(),
> and would like to ask if somebody can explain that?
>
> Vyacheslav
>
>
> .
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: very long init:stop() and +P emulator flag

Vyacheslav Levytskyy-2
In reply to this post by Vyacheslav Levytskyy-2

That's true, I don't need 134217727 processes to be running, but it is not substantial. It is just counter intuitive for me to have a beam process with only kernel+stdlib that must take 5-10 minutes to shutdown.

If you are right and this indeed happens because of 1,5Gb allocated (btw does this mean that each of 134217727 potential processes is pre-created?), I would say that setting an upper bound of something I don't expect immediate actions. Out of curiosity, I wonder what should be cleaned up in the system immediately after the clean boot, and did this behaviour exist earlier or was introduced recently (maybe as an optimization) with Erlang/OTP 19 or 20?

Vyacheslav

On 25.09.2017 23:46, Vyacheslav Levytskyy wrote:
I checked and running erl with that +P 134217727 results in a beam
process using 1.517g of memory. So cleaning that up is probably what is
taking so much time on shutdown :)

134217727 certainly seems excessive... What size system would even be
able to handle dealing with that many processes?

-- 
  Tristan Sloughter
  "I am not a crackpot" - Abe Simpson
  


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

memory used by Erlang VM

Vyacheslav Levytskyy-2
In reply to this post by Vyacheslav Levytskyy-2
Hello,

I wonder why memory used by Erlang VM as reported by the kernel via the
/proc/pid differs from erlang:memory(total). In my current configuration
I observe realistic response from erlang:memory(total) and much lower
values from the /proc/pid.

I'm not surprised by the difference itself, but rather by the fact that
the /proc/pid gives unrealistically lower values -- I'm not 100% sure,
but it looks like RabbitMQ is using the /proc/pid approach by default,
proposing also recon_alloc:memory(allocated) and erlang:memory(total) as
available options of Erlang VM memory consumption calculation strategy.

Does anybody have insights of what and why is going on with those
calculations of memory used by Erlang VM? Is it possible to select one
strategy beforehand for my Erlang app, or I must measure on each new
configuration what looks more precise? Should I compare and change the
strategy during run-time, or after I selected a strategy once for my
configuration I can be sure that selected approach always better than
other two?

Thank you,
Vyacheslav
Reply | Threaded
Open this post in threaded view
|

Re: memory used by Erlang VM

Jesper Louis Andersen-2
When you say /proc/pid, what are you looking at specifically in there? It is a bit different depending on which Unix you run on, so a simple example will help a lot.

In particular, my early guess is going to be virtual memory vs physical RSS mapping. The former can be much higher than the latter. Especially in system such as Linux, which allow overcommitting memory.

On Fri, Jan 31, 2020 at 7:22 PM Vyacheslav Levytskyy <[hidden email]> wrote:
Hello,

I wonder why memory used by Erlang VM as reported by the kernel via the
/proc/pid differs from erlang:memory(total). In my current configuration
I observe realistic response from erlang:memory(total) and much lower
values from the /proc/pid.

I'm not surprised by the difference itself, but rather by the fact that
the /proc/pid gives unrealistically lower values -- I'm not 100% sure,
but it looks like RabbitMQ is using the /proc/pid approach by default,
proposing also recon_alloc:memory(allocated) and erlang:memory(total) as
available options of Erlang VM memory consumption calculation strategy.

Does anybody have insights of what and why is going on with those
calculations of memory used by Erlang VM? Is it possible to select one
strategy beforehand for my Erlang app, or I must measure on each new
configuration what looks more precise? Should I compare and change the
strategy during run-time, or after I selected a strategy once for my
configuration I can be sure that selected approach always better than
other two?

Thank you,
Vyacheslav


--
J.
Reply | Threaded
Open this post in threaded view
|

Re: memory used by Erlang VM

Vyacheslav Levytskyy-2

It's linux (ubuntu). I read /proc/[pid]/statm, take RSS number of pages (the 2nd value in the line of that file) and calculate memory used by Erlang VM as RSS x page size (read as "getconf PAGESIZE").

On 31.01.2020 19:59, Jesper Louis Andersen wrote:
When you say /proc/pid, what are you looking at specifically in there? It is a bit different depending on which Unix you run on, so a simple example will help a lot.

In particular, my early guess is going to be virtual memory vs physical RSS mapping. The former can be much higher than the latter. Especially in system such as Linux, which allow overcommitting memory.

On Fri, Jan 31, 2020 at 7:22 PM Vyacheslav Levytskyy <[hidden email]> wrote:
Hello,

I wonder why memory used by Erlang VM as reported by the kernel via the
/proc/pid differs from erlang:memory(total). In my current configuration
I observe realistic response from erlang:memory(total) and much lower
values from the /proc/pid.

I'm not surprised by the difference itself, but rather by the fact that
the /proc/pid gives unrealistically lower values -- I'm not 100% sure,
but it looks like RabbitMQ is using the /proc/pid approach by default,
proposing also recon_alloc:memory(allocated) and erlang:memory(total) as
available options of Erlang VM memory consumption calculation strategy.

Does anybody have insights of what and why is going on with those
calculations of memory used by Erlang VM? Is it possible to select one
strategy beforehand for my Erlang app, or I must measure on each new
configuration what looks more precise? Should I compare and change the
strategy during run-time, or after I selected a strategy once for my
configuration I can be sure that selected approach always better than
other two?

Thank you,
Vyacheslav


--
J.
Reply | Threaded
Open this post in threaded view
|

Re: memory used by Erlang VM

Vyacheslav Levytskyy-2

I have also zero swappiness:

cat /proc/sys/vm/swappiness
0

so I expect that RSS shows total memory used.

On 31.01.2020 20:23, Vyacheslav Levytskyy wrote:

It's linux (ubuntu). I read /proc/[pid]/statm, take RSS number of pages (the 2nd value in the line of that file) and calculate memory used by Erlang VM as RSS x page size (read as "getconf PAGESIZE").

On 31.01.2020 19:59, Jesper Louis Andersen wrote:
When you say /proc/pid, what are you looking at specifically in there? It is a bit different depending on which Unix you run on, so a simple example will help a lot.

In particular, my early guess is going to be virtual memory vs physical RSS mapping. The former can be much higher than the latter. Especially in system such as Linux, which allow overcommitting memory.

On Fri, Jan 31, 2020 at 7:22 PM Vyacheslav Levytskyy <[hidden email]> wrote:
Hello,

I wonder why memory used by Erlang VM as reported by the kernel via the
/proc/pid differs from erlang:memory(total). In my current configuration
I observe realistic response from erlang:memory(total) and much lower
values from the /proc/pid.

I'm not surprised by the difference itself, but rather by the fact that
the /proc/pid gives unrealistically lower values -- I'm not 100% sure,
but it looks like RabbitMQ is using the /proc/pid approach by default,
proposing also recon_alloc:memory(allocated) and erlang:memory(total) as
available options of Erlang VM memory consumption calculation strategy.

Does anybody have insights of what and why is going on with those
calculations of memory used by Erlang VM? Is it possible to select one
strategy beforehand for my Erlang app, or I must measure on each new
configuration what looks more precise? Should I compare and change the
strategy during run-time, or after I selected a strategy once for my
configuration I can be sure that selected approach always better than
other two?

Thank you,
Vyacheslav


--
J.
Reply | Threaded
Open this post in threaded view
|

Re: memory used by Erlang VM

Gerhard Lazu
In reply to this post by Jesper Louis Andersen-2
I remember that battle with Erlang vs OS memory reporting well, it was such a show. This captures my thinking from 2017: https://github.com/rabbitmq/rabbitmq-server/issues/1223 . This has more context: https://github.com/rabbitmq/rabbitmq-server/pull/1259#issuecomment-308428057. There are a few other comments in that thread that you may find useful.

Erlang will either over-report or under-report the memory used. erlang:memory(total) will very rarely match the physical RSS mapping, as explained earlier by Jesper + https://stackoverflow.com/questions/7880784/what-is-rss-and-vsz-in-linux-memory-management. This detailed snapshot of Erlang memory allocators state shows this in the clearest way that I am aware of: https://grafana.gcp.rabbitmq.com/dashboard/snapshot/wM5JcgR9oQToU4CR54IbOq1NtAsa6jvu 

As it stands, the Linux OOM takes action based on the RSS value. While from an Erlang perspective it may seem OK to ask the OS for more memory, we've had years of poor RabbitMQ experience when the Erlang VM process would get killed by the Linux OOM because it was asking for memory that the system didn't have available. This has more information: https://www.rabbitmq.com/memory-use.html

Hope this helps, Gerhard.

On Fri, Jan 31, 2020 at 7:00 PM Jesper Louis Andersen <[hidden email]> wrote:
When you say /proc/pid, what are you looking at specifically in there? It is a bit different depending on which Unix you run on, so a simple example will help a lot.

In particular, my early guess is going to be virtual memory vs physical RSS mapping. The former can be much higher than the latter. Especially in system such as Linux, which allow overcommitting memory.

On Fri, Jan 31, 2020 at 7:22 PM Vyacheslav Levytskyy <[hidden email]> wrote:
Hello,

I wonder why memory used by Erlang VM as reported by the kernel via the
/proc/pid differs from erlang:memory(total). In my current configuration
I observe realistic response from erlang:memory(total) and much lower
values from the /proc/pid.

I'm not surprised by the difference itself, but rather by the fact that
the /proc/pid gives unrealistically lower values -- I'm not 100% sure,
but it looks like RabbitMQ is using the /proc/pid approach by default,
proposing also recon_alloc:memory(allocated) and erlang:memory(total) as
available options of Erlang VM memory consumption calculation strategy.

Does anybody have insights of what and why is going on with those
calculations of memory used by Erlang VM? Is it possible to select one
strategy beforehand for my Erlang app, or I must measure on each new
configuration what looks more precise? Should I compare and change the
strategy during run-time, or after I selected a strategy once for my
configuration I can be sure that selected approach always better than
other two?

Thank you,
Vyacheslav


--
J.
Reply | Threaded
Open this post in threaded view
|

Re: memory used by Erlang VM

Jesper Louis Andersen-2
In reply to this post by Vyacheslav Levytskyy-2
It is expected your RSS value is lower than what Erlang reports.

Erlang requests virtual memory from the operating system (Linux). But it is mapped into the process on-demand. RSS is the resident set size, which is the currently physical mapped pages. As Erlang hits pages it hasn't used before, a kernel trap is generated and it maps in that page to to the process.

For example, lets say we allocate a large array to hold processes, but we are only using a small smidgen of that array. Then there are many virtually mapped pages, but we aren't really "using" the memory yet. It is taken on-demand. This on-demand method is smart because it lowers memory pressure. Rather than having to do all the work up front, we can amortize the work over the course of the program running, which is much more efficient. If the application requests 1 gigabyte of memory say, and we want to give it right away, we must zero one gigabyte of memory. This takes time. But on-demand, we can keep a background process for zeroing memory and allocate it as it is needed, among other things.

Where this can create a problem is if you have several processes all wanting lots of memory, and you have less memory in the machine than what they've got promised by the kernel. Then, if they all starts wanting memory, you should expect to see various trouble.

As Erlang runs more, I would expect the RSS to go up as it populates more of the memory space.

On Fri, Jan 31, 2020 at 8:23 PM Vyacheslav Levytskyy <[hidden email]> wrote:

It's linux (ubuntu). I read /proc/[pid]/statm, take RSS number of pages (the 2nd value in the line of that file) and calculate memory used by Erlang VM as RSS x page size (read as "getconf PAGESIZE").

On 31.01.2020 19:59, Jesper Louis Andersen wrote:
When you say /proc/pid, what are you looking at specifically in there? It is a bit different depending on which Unix you run on, so a simple example will help a lot.

In particular, my early guess is going to be virtual memory vs physical RSS mapping. The former can be much higher than the latter. Especially in system such as Linux, which allow overcommitting memory.

On Fri, Jan 31, 2020 at 7:22 PM Vyacheslav Levytskyy <[hidden email]> wrote:
Hello,

I wonder why memory used by Erlang VM as reported by the kernel via the
/proc/pid differs from erlang:memory(total). In my current configuration
I observe realistic response from erlang:memory(total) and much lower
values from the /proc/pid.

I'm not surprised by the difference itself, but rather by the fact that
the /proc/pid gives unrealistically lower values -- I'm not 100% sure,
but it looks like RabbitMQ is using the /proc/pid approach by default,
proposing also recon_alloc:memory(allocated) and erlang:memory(total) as
available options of Erlang VM memory consumption calculation strategy.

Does anybody have insights of what and why is going on with those
calculations of memory used by Erlang VM? Is it possible to select one
strategy beforehand for my Erlang app, or I must measure on each new
configuration what looks more precise? Should I compare and change the
strategy during run-time, or after I selected a strategy once for my
configuration I can be sure that selected approach always better than
other two?

Thank you,
Vyacheslav


--
J.


--
J.
Reply | Threaded
Open this post in threaded view
|

Re: memory used by Erlang VM

Loïc Hoguin-3
On 31/01/2020 21:27, Jesper Louis Andersen wrote:
> It is expected your RSS value is lower than what Erlang reports.

And it would be fantastic if that was always true. But RSS is not that
simple. See Gerhard's link
https://grafana.gcp.rabbitmq.com/dashboard/snapshot/wM5JcgR9oQToU4CR54IbOq1NtAsa6jvu 
that has RSS lower than allocators report. Depending on the allocator
strategies we've seen both RSS and Virt go all over the place. I'm
afraid after spending months on this I still don't understand RSS.

Cheers,

--
Loïc Hoguin
https://ninenines.eu
Reply | Threaded
Open this post in threaded view
|

Re: memory used by Erlang VM

Max Lapshin-2
So what is RSS?

Is it a badly designed aggregate or some real value that is very hard to predict (just like it is hard to tell temperature when you know speed of several moleculas)?
Reply | Threaded
Open this post in threaded view
|

Re: memory used by Erlang VM

Jesper Louis Andersen-2
RSS is the latter.

Your processes allocate virtual memory. It is mapped to physical memory (and secondarily to the swap drive). RSS is the amount of memory currently resident in the physical memory of the computer. It is genuinely used.

What makes it hard to predict is that you don't get it all straight away. If I allocate 8 megabyte of contiguous memory via, e.g., mmap(), I'm granted permission to access it (so the VM memory goes up). But since I didn't touch the pages yet, RSS is not changing. As I start touching memory, it is mapped in and physical pages are allocated. Typically in either 4kb blocks or 2mb blocks. Contrast this with me memory mapping in a file. Then when I access memory, I'm accessing the underlying file through the disk cache. Also consider a fork. They use copy-on-write, so memory will be allocated as the processes start diverging in the memory space by doing different updates to different pages[0].

We also map libraries and the program code itself into the memory space. But this is marked as read-only, so it can be shared among several processes needing use of the same library.



[0] By default. You can memory map so things are shared.

On Sun, Feb 2, 2020 at 8:31 PM Max Lapshin <[hidden email]> wrote:
So what is RSS?

Is it a badly designed aggregate or some real value that is very hard to predict (just like it is hard to tell temperature when you know speed of several moleculas)?


--
J.