Hidden binaries

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Hidden binaries

Kirill Zaborsky
Hello,
I have recently found some memory "leak" in our application. Leak is in
quotes because memory load seems to be constant under constant load. But
the problem is that I see some extra binaries for which there are no signs
in process_info.
In production we use our fork of OTP16BR3-1 (including some number of httpc
fixes) but it looks like that behaviour could be reproduced locally on my
machine with OTP 17.
I have tried to get any hints from recon but with not success.
The problem could be seen from the following Erlang shell lines:
-------------
(node)87> recon:bin_leak(20),erlang:memory().
[{total,30574120},
 {processes,6765096},
 {processes_used,6738520},
 {system,23809024},
 {atom,662409},
 {atom_used,652067},
 {binary,145760},
 {code,16742599},
 {ets,1037976}]
(node)88>
lists:sum([N||{P1,N,B}<-[{P,lists:sum([S||{_,S,_}<-B]),B} || {P,{binary,B}}
<- [{Pid,(catch process_info(Pid,binary))} || Pid <- processes()]]]).
17671
(node)89>
-------------
As you see erlang:memory/0 shows 146k of memory used by binaries but
erlang:process_info/2 shows information only about 18k
Is there any way I could find out where the missing 128k of memory is?
Is it possible to get all binaries contents without gowing to some
low-level tools like gdb (I'm not quite familiar with it yet)?
Possibly there are some system-level binaries?
In production I see not kilobytes but hunderds of megabytes and sometimes
it leads even to OOM crashes.
I run Gentoo on my desktop and in production we use Ubuntu Precise.
Thanks for any advice.

Best regards,
Kirill Zaborsky
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20140528/af0144e0/attachment.html>

Reply | Threaded
Open this post in threaded view
|

Hidden binaries

Lukas Larsson-7
Hello,

Binaries can also be present in ets objects and messages in processes'
message queues. I'm sure there are some other places as well, but those
should be the major ones.

I know of no easy way (with gdb or otherwise) to get information if this is
the case.

Lukas
On Wed, May 28, 2014 at 9:07 AM, Kirill Zaborsky <qrilka> wrote:

> Hello,
> I have recently found some memory "leak" in our application. Leak is in
> quotes because memory load seems to be constant under constant load. But
> the problem is that I see some extra binaries for which there are no signs
> in process_info.
> In production we use our fork of OTP16BR3-1 (including some number of
> httpc fixes) but it looks like that behaviour could be reproduced locally
> on my machine with OTP 17.
> I have tried to get any hints from recon but with not success.
> The problem could be seen from the following Erlang shell lines:
> -------------
> (node)87> recon:bin_leak(20),erlang:memory().
> [{total,30574120},
>  {processes,6765096},
>  {processes_used,6738520},
>  {system,23809024},
>  {atom,662409},
>  {atom_used,652067},
>  {binary,145760},
>  {code,16742599},
>  {ets,1037976}]
> (node)88>
> lists:sum([N||{P1,N,B}<-[{P,lists:sum([S||{_,S,_}<-B]),B} || {P,{binary,B}}
> <- [{Pid,(catch process_info(Pid,binary))} || Pid <- processes()]]]).
> 17671
> (node)89>
> -------------
> As you see erlang:memory/0 shows 146k of memory used by binaries but
> erlang:process_info/2 shows information only about 18k
> Is there any way I could find out where the missing 128k of memory is?
> Is it possible to get all binaries contents without gowing to some
> low-level tools like gdb (I'm not quite familiar with it yet)?
> Possibly there are some system-level binaries?
> In production I see not kilobytes but hunderds of megabytes and sometimes
> it leads even to OOM crashes.
> I run Gentoo on my desktop and in production we use Ubuntu Precise.
> Thanks for any advice.
>
> Best regards,
> Kirill Zaborsky
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions
> http://erlang.org/mailman/listinfo/erlang-questions
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20140528/cba4f9e7/attachment.html>

Reply | Threaded
Open this post in threaded view
|

Hidden binaries

Robert Raschke-7
In reply to this post by Kirill Zaborsky
Hi Kirill,

is there any chance you are using binary:split to smash a larger binary
into pieces for further processing, thinking that the big thing will go
away asap?

If yes, then I can recommend reading
https://blog.heroku.com/archives/2013/11/7/logplex-down-the-rabbit-hole ,
especially from section "I Just Keep On Bleeding and I Won't Die" onwards.

Naive summary (reality is a bit more discerning :-) : when you have refs
into a larger binary, the large binary won't get gc'ed, and you need to
jump through some hoops to get rid of it.

HTH,
Robby
On May 28, 2014 8:07 AM, "Kirill Zaborsky" <qrilka> wrote:

> Hello,
> I have recently found some memory "leak" in our application. Leak is in
> quotes because memory load seems to be constant under constant load. But
> the problem is that I see some extra binaries for which there are no signs
> in process_info.
> In production we use our fork of OTP16BR3-1 (including some number of
> httpc fixes) but it looks like that behaviour could be reproduced locally
> on my machine with OTP 17.
> I have tried to get any hints from recon but with not success.
> The problem could be seen from the following Erlang shell lines:
> -------------
> (node)87> recon:bin_leak(20),erlang:memory().
> [{total,30574120},
>  {processes,6765096},
>  {processes_used,6738520},
>  {system,23809024},
>  {atom,662409},
>  {atom_used,652067},
>  {binary,145760},
>  {code,16742599},
>  {ets,1037976}]
> (node)88>
> lists:sum([N||{P1,N,B}<-[{P,lists:sum([S||{_,S,_}<-B]),B} || {P,{binary,B}}
> <- [{Pid,(catch process_info(Pid,binary))} || Pid <- processes()]]]).
> 17671
> (node)89>
> -------------
> As you see erlang:memory/0 shows 146k of memory used by binaries but
> erlang:process_info/2 shows information only about 18k
> Is there any way I could find out where the missing 128k of memory is?
> Is it possible to get all binaries contents without gowing to some
> low-level tools like gdb (I'm not quite familiar with it yet)?
> Possibly there are some system-level binaries?
> In production I see not kilobytes but hunderds of megabytes and sometimes
> it leads even to OOM crashes.
> I run Gentoo on my desktop and in production we use Ubuntu Precise.
> Thanks for any advice.
>
> Best regards,
> Kirill Zaborsky
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions
> http://erlang.org/mailman/listinfo/erlang-questions
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20140528/d0b1bc40/attachment.html>

Reply | Threaded
Open this post in threaded view
|

Hidden binaries

Jesper Louis Andersen
In reply to this post by Lukas Larsson-7
On Wed, May 28, 2014 at 10:26 AM, Lukas Larsson
<lukas>wrote:

> Binaries can also be present in ets objects and messages in processes'
> message queues. I'm sure there are some other places as well, but those
> should be the major ones.


This is a common case. Say you have a binary of size 1024 bytes and you
pick out an identifier in it which is 8 bytes. The identifier is now a
sub-binary which keeps the large binary around. If you stick the identifier
somewhere, you will be "wasting" 1024-8 bytes.

The solution is to explicitly put a binary:copy/1 in place to "lift out"
the small identifier.


--
J.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20140528/c09c7c64/attachment.html>

Reply | Threaded
Open this post in threaded view
|

Hidden binaries

Loïc Hoguin-2
On 05/28/2014 11:17 AM, Jesper Louis Andersen wrote:

>
> On Wed, May 28, 2014 at 10:26 AM, Lukas Larsson
> <lukas <mailto:lukas>> wrote:
>
>     Binaries can also be present in ets objects and messages in
>     processes' message queues. I'm sure there are some other places as
>     well, but those should be the major ones.
>
>
> This is a common case. Say you have a binary of size 1024 bytes and you
> pick out an identifier in it which is 8 bytes. The identifier is now a
> sub-binary which keeps the large binary around. If you stick the
> identifier somewhere, you will be "wasting" 1024-8 bytes.

Actually, and correct me if I'm wrong, but the sub binary optimization
breaks when you stick the identifier somewhere, so it ends up being
copied automatically.

--
Lo?c Hoguin
http://ninenines.eu

Reply | Threaded
Open this post in threaded view
|

Hidden binaries

Jesper Louis Andersen
On Wed, May 28, 2014 at 11:40 AM, Lo?c Hoguin <essen> wrote:

> Actually, and correct me if I'm wrong, but the sub binary optimization
> breaks when you stick the identifier somewhere, so it ends up being copied
> automatically.


This happens if you send the binary in a message to another process. But
AFAIK it won't happen if storing the binary in ETS.


--
J.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20140528/816042b4/attachment.html>

Reply | Threaded
Open this post in threaded view
|

Hidden binaries

Kirill Zaborsky
In reply to this post by Lukas Larsson-7
Thanks a lot, Lukas.
I was suspecting something like that but forgot to check it somehow.
It looks you are right here and problem seems to be in
ssl_otp_session_cache table - we started do a lot of ssl requests.

Regards,
Kirill.


2014-05-28 12:26 GMT+04:00 Lukas Larsson <lukas>:

> Hello,
>
> Binaries can also be present in ets objects and messages in processes'
> message queues. I'm sure there are some other places as well, but those
> should be the major ones.
>
> I know of no easy way (with gdb or otherwise) to get information if this
> is the case.
>
> Lukas
> On Wed, May 28, 2014 at 9:07 AM, Kirill Zaborsky <qrilka> wrote:
>
>> Hello,
>> I have recently found some memory "leak" in our application. Leak is in
>> quotes because memory load seems to be constant under constant load. But
>> the problem is that I see some extra binaries for which there are no signs
>> in process_info.
>> In production we use our fork of OTP16BR3-1 (including some number of
>> httpc fixes) but it looks like that behaviour could be reproduced locally
>> on my machine with OTP 17.
>> I have tried to get any hints from recon but with not success.
>> The problem could be seen from the following Erlang shell lines:
>> -------------
>> (node)87> recon:bin_leak(20),erlang:memory().
>> [{total,30574120},
>>  {processes,6765096},
>>  {processes_used,6738520},
>>  {system,23809024},
>>  {atom,662409},
>>  {atom_used,652067},
>>  {binary,145760},
>>  {code,16742599},
>>  {ets,1037976}]
>> (node)88>
>> lists:sum([N||{P1,N,B}<-[{P,lists:sum([S||{_,S,_}<-B]),B} || {P,{binary,B}}
>> <- [{Pid,(catch process_info(Pid,binary))} || Pid <- processes()]]]).
>> 17671
>> (node)89>
>> -------------
>> As you see erlang:memory/0 shows 146k of memory used by binaries but
>> erlang:process_info/2 shows information only about 18k
>> Is there any way I could find out where the missing 128k of memory is?
>> Is it possible to get all binaries contents without gowing to some
>> low-level tools like gdb (I'm not quite familiar with it yet)?
>>  Possibly there are some system-level binaries?
>> In production I see not kilobytes but hunderds of megabytes and sometimes
>> it leads even to OOM crashes.
>> I run Gentoo on my desktop and in production we use Ubuntu Precise.
>> Thanks for any advice.
>>
>> Best regards,
>> Kirill Zaborsky
>>
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions
>> http://erlang.org/mailman/listinfo/erlang-questions
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20140528/0dad7a98/attachment.html>

Reply | Threaded
Open this post in threaded view
|

Hidden binaries

Björn Gustavsson-3
In reply to this post by Jesper Louis Andersen
On Wed, May 28, 2014 at 11:42 AM, Jesper Louis Andersen
<jesper.louis.andersen> wrote:

>
> On Wed, May 28, 2014 at 11:40 AM, Lo?c Hoguin <essen> wrote:
>>
>> Actually, and correct me if I'm wrong, but the sub binary optimization
>> breaks when you stick the identifier somewhere, so it ends up being copied
>> automatically.
>
>
> This happens if you send the binary in a message to another process. But
> AFAIK it won't happen if storing the binary in ETS.
>

I think there is some confusion about here about the different
binary optimizations.

Sub-binaries are created when a binary is matched using bit
syntax or split_binary/2. There will never be any automatic
conversion from a sub binary to a copy of binary data referenced
by the sub-binary.

There is an compiler-based optimization that will delay
creation of a sub-binary to optimize loops that do
binary matching. Instead of creating a new sub-binary
for every iteration of the loop, the internal match state
structure used for binary matching is kept in the loop.
As soon as the loop is exited, the match state will be
converted to a real sub-binary.

There is another optimization when appending to
binaries. When a binary is appended to, i.e. if
you write

  <<Bin/binary, ...>>

it is assumed that the program will append to
the resulting binary again and will therefore allocate
empty space at the end of the binary. If the binary
is sent to another process or stored in an ETS table,
the extra space is deallocated and the binary is
no longer marked as appendable. That optimization
is done entirely in the run-time system.

/Bjorn

--
Bj?rn Gustavsson, Erlang/OTP, Ericsson AB