beam.smp high memory usage

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

beam.smp high memory usage

Frank Muller
Hi guys,

I’ve a fairly complex I/O bound Erlang application.

Since yesterday, the beam.smp memory usage dramatically increased from ~160MB to something between [2.5GB,  3.1GB].
I’ve no idea how, nor why :-/

Attached is snapshot taken with recon_alloc:snapshot().
Please let me know if you want anything else.

Hope someone can explain me what’s going on.

Thank you.
/Frank

INFO:
. Physical machine (not a VM)
. Erlang/OTP 20 [erts-9.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:10] [hipe] [kernel-poll:false]
. CentOS-7-x86_64
. Linux ns342284 4.9.58-xxxx-std-ipv6-64 #1 SMP Mon Oct 23 11:35:59 CEST 2017 x86_64 x86_64 x86_64 GNU/Linux
. 8x Intel(R) Xeon(R) CPU  E5504  @ 2.00GHz
. 16GB of RAM



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

recon_snapshot.terms (173K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: beam.smp high memory usage

Jesper Louis Andersen-2
First quick stab:

Most of the allocated space is binary() data.

Typical scenario: something is keeping the binary() data alive, one way or the other. Either because you need a binary:copy/1 in key locations due to sub-binaries, or because your system can't get rid of the data since you still have a pointer kept to it from some process.


On Sun, Nov 12, 2017 at 12:51 PM Frank Muller <[hidden email]> wrote:
Hi guys,

I’ve a fairly complex I/O bound Erlang application.

Since yesterday, the beam.smp memory usage dramatically increased from ~160MB to something between [2.5GB,  3.1GB].
I’ve no idea how, nor why :-/

Attached is snapshot taken with recon_alloc:snapshot().
Please let me know if you want anything else.

Hope someone can explain me what’s going on.

Thank you.
/Frank

INFO:
. Physical machine (not a VM)
. Erlang/OTP 20 [erts-9.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:10] [hipe] [kernel-poll:false]
. CentOS-7-x86_64
. Linux ns342284 4.9.58-xxxx-std-ipv6-64 #1 SMP Mon Oct 23 11:35:59 CEST 2017 x86_64 x86_64 x86_64 GNU/Linux
. 8x Intel(R) Xeon(R) CPU  E5504  @ 2.00GHz
. 16GB of RAM


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: beam.smp high memory usage

Frank Muller
Hi Jesper,

Thanks again for the feedback.

But from where should I start (Erlang newbie here)? 
I’ve no idea ... is there a guideline to follow?

I feel guessing without making any real progress for the last two days :-/

/Frank

First quick stab:

Most of the allocated space is binary() data.

Typical scenario: something is keeping the binary() data alive, one way or the other. Either because you need a binary:copy/1 in key locations due to sub-binaries, or because your system can't get rid of the data since you still have a pointer kept to it from some process.


On Sun, Nov 12, 2017 at 12:51 PM Frank Muller <[hidden email]> wrote:
Hi guys,

I’ve a fairly complex I/O bound Erlang application.

Since yesterday, the beam.smp memory usage dramatically increased from ~160MB to something between [2.5GB,  3.1GB].
I’ve no idea how, nor why :-/

Attached is snapshot taken with recon_alloc:snapshot().
Please let me know if you want anything else.

Hope someone can explain me what’s going on.

Thank you.
/Frank

INFO:
. Physical machine (not a VM)
. Erlang/OTP 20 [erts-9.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:10] [hipe] [kernel-poll:false]
. CentOS-7-x86_64
. Linux ns342284 4.9.58-xxxx-std-ipv6-64 #1 SMP Mon Oct 23 11:35:59 CEST 2017 x86_64 x86_64 x86_64 GNU/Linux
. 8x Intel(R) Xeon(R) CPU  E5504  @ 2.00GHz
. 16GB of RAM


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: beam.smp high memory usage

Jesper Louis Andersen-2
Fred's excellent "Erlang In Anger" book (https://www.erlang-in-anger.com/) has a section on hunting down binary leaks (if that is what you are working with here). At least this can give you some starting information about what to look for and how to go about handling it afterwards. Identifying which processes are holding the binary data by extension identifies the code modules which are culprits.

Since your normal operation uses 160 megabytes of memory and it suddenly shoots to gigabytes, one of the parts is to identify what is being done to the system when that happens. Perhaps you have an outside user which uses your system in a way that wasn't thought about originally. This can lead to you handling different data and in turn this can expose some of these problems.

On Mon, Nov 13, 2017 at 9:23 AM Frank Muller <[hidden email]> wrote:
Hi Jesper,

Thanks again for the feedback.

But from where should I start (Erlang newbie here)? 
I’ve no idea ... is there a guideline to follow?

I feel guessing without making any real progress for the last two days :-/

/Frank

First quick stab:

Most of the allocated space is binary() data.

Typical scenario: something is keeping the binary() data alive, one way or the other. Either because you need a binary:copy/1 in key locations due to sub-binaries, or because your system can't get rid of the data since you still have a pointer kept to it from some process.


On Sun, Nov 12, 2017 at 12:51 PM Frank Muller <[hidden email]> wrote:
Hi guys,

I’ve a fairly complex I/O bound Erlang application.

Since yesterday, the beam.smp memory usage dramatically increased from ~160MB to something between [2.5GB,  3.1GB].
I’ve no idea how, nor why :-/

Attached is snapshot taken with recon_alloc:snapshot().
Please let me know if you want anything else.

Hope someone can explain me what’s going on.

Thank you.
/Frank

INFO:
. Physical machine (not a VM)
. Erlang/OTP 20 [erts-9.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:10] [hipe] [kernel-poll:false]
. CentOS-7-x86_64
. Linux ns342284 4.9.58-xxxx-std-ipv6-64 #1 SMP Mon Oct 23 11:35:59 CEST 2017 x86_64 x86_64 x86_64 GNU/Linux
. 8x Intel(R) Xeon(R) CPU  E5504  @ 2.00GHz
. 16GB of RAM


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: beam.smp high memory usage

Frank Muller
Hi guys,

I would like to thank you for your feedbacks and pointers.

After days of unsuccessful investigations, Jesper’s hint led me directly the culprit.

All clients were sending small binaries to my app, but one. This one guy sent very large binaries for very long time (keep-alive).

A temporary workaround was to simple limit the size of each binary sent and force closing TCP connection after N seconds.

The memory usage went immediately back to ~160MB.

Thanks again 
/Frank

Fred's excellent "Erlang In Anger" book (https://www.erlang-in-anger.com/) has a section on hunting down binary leaks (if that is what you are working with here). At least this can give you some starting information about what to look for and how to go about handling it afterwards. Identifying which processes are holding the binary data by extension identifies the code modules which are culprits.

Since your normal operation uses 160 megabytes of memory and it suddenly shoots to gigabytes, one of the parts is to identify what is being done to the system when that happens. Perhaps you have an outside user which uses your system in a way that wasn't thought about originally. This can lead to you handling different data and in turn this can expose some of these problems.

On Mon, Nov 13, 2017 at 9:23 AM Frank Muller <[hidden email]> wrote:
Hi Jesper,

Thanks again for the feedback.

But from where should I start (Erlang newbie here)? 
I’ve no idea ... is there a guideline to follow?

I feel guessing without making any real progress for the last two days :-/

/Frank

First quick stab:

Most of the allocated space is binary() data.

Typical scenario: something is keeping the binary() data alive, one way or the other. Either because you need a binary:copy/1 in key locations due to sub-binaries, or because your system can't get rid of the data since you still have a pointer kept to it from some process.


On Sun, Nov 12, 2017 at 12:51 PM Frank Muller <[hidden email]> wrote:
Hi guys,

I’ve a fairly complex I/O bound Erlang application.

Since yesterday, the beam.smp memory usage dramatically increased from ~160MB to something between [2.5GB,  3.1GB].
I’ve no idea how, nor why :-/

Attached is snapshot taken with recon_alloc:snapshot().
Please let me know if you want anything else.

Hope someone can explain me what’s going on.

Thank you.
/Frank

INFO:
. Physical machine (not a VM)
. Erlang/OTP 20 [erts-9.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:10] [hipe] [kernel-poll:false]
. CentOS-7-x86_64
. Linux ns342284 4.9.58-xxxx-std-ipv6-64 #1 SMP Mon Oct 23 11:35:59 CEST 2017 x86_64 x86_64 x86_64 GNU/Linux
. 8x Intel(R) Xeon(R) CPU  E5504  @ 2.00GHz
. 16GB of RAM


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions