Troubleshooting binary memory usage

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
ash
Reply | Threaded
Open this post in threaded view
|

Troubleshooting binary memory usage

ash
Hi,
I've been getting high values for `erlang:memory(binary)` so I've been
looking at the following:
(1) `[erlang:process_info(Pid, binary) || Pid <- processes()]`
(2) `instrument:memory_data/0`
(3) `erlang:system_info({allocator, binary_alloc})`
Unfortunately no two of these sources agree. For example, I can try to
compare the number of live binaries according to the different data
sources:
```
(fun () ->
          ProcInfoCount = dict:size(lists:foldl(
                                      fun(Pid, D) ->
                                              case
erlang:process_info(Pid, binary) of
                                                  {binary, Bins} ->
                                                      lists:foldl(
                                                        fun({Addr,_,_},
D1) ->
                                                               
dict:update_counter(Addr,1,D1)
                                                        end,
                                                        D,
                                                        Bins);
                                                  Other -> D
                                              end
                                      end,
                                      dict:new(),
                                      processes())),
          {_,Data}=instrument:descr(instrument:memory_data()),
          InstrumentCount = lists:foldl(fun (T={binary,_,_,_},Acc) ->
Acc + 1;
                                            (_,Acc) -> Acc
                                        end,
                                        0,
                                        Data),
          GetCallCount = fun(K, Calls) ->
                                 case lists:keyfind(K, 1, Calls) of
                                     {_, X, Y} -> X*1000000000+Y;
                                     _ -> 0
                                 end
                         end,
          AllocCount = lists:foldl(fun({instance,_,PropList}, A) ->
                                           Calls =
proplists:get_value(calls, PropList, []),
                                           Allocs =
GetCallCount(binary_alloc, Calls),
                                           Frees =
GetCallCount(binary_free, Calls),
                                           A + (Allocs - Frees)
                                   end,
                                   0,
                                   erlang:system_info({allocator,
binary_alloc})),
          {ProcInfoCount, InstrumentCount, AllocCount}
  end)().
```
This yields `{2874,6371,7715}` at time of writing.
Can anyone suggest reasons for the different counts? I've found that
some processes show up in (2) but not (1), but I don't know why this is.
Thanks,
Ash
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Troubleshooting binary memory usage

Lukas Larsson-8
Hello,

On Fri, Feb 10, 2017 at 12:28 PM, ash <[hidden email]> wrote:
Hi,
I've been getting high values for `erlang:memory(binary)` so I've been looking at the following:
(1) `[erlang:process_info(Pid, binary) || Pid <- processes()]`

This describes the number of off-heap binaries created by the application belonging to the specified process.
 
(2) `instrument:memory_data/0`

This describes the amount of memory used by the allocator type binary.
 
(3) `erlang:system_info({allocator, binary_alloc})`

This describes the amount of memory allocated into the binary allocator aka binary_alloc

None of these will have the same value.

The binary allocator contains all off-heap binaries and other binary like things (for drivers, nifs and erts things), so it will always be the largest one. 

The binary allocator type is what is used to allocate a off-heap binary. However all off-heap binaries do not have to below to processes, they could also belong to ports, nifs or some other VM internal entity. This value should be smaller then binary_alloc. The pid that you see here is which process CREATED the binary, the ownership of it may later have been shared with a port, nif, or other process, or something else.

Lastly process_info describes the binaries currently referenced from the heap of the given process. This should be smaller then binary_alloc and the binary allocator type.

Lukas

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions