Experience with much memory

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Experience with much memory

Gergely Buday
Hi there,

are there experience reports on using Erlang/Elixir on systems with 200 GB of memory and more?

If you have a first-hand account on that, please share it with me.

My friends are doing that and said that only C++ was capable of handling that.

- Gergely

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Experience with much memory

Jack Tang
It is complex question. You should locate who hold the memory first :)
Below codes might help you to locate the process / ets

```
applications_memory() ->
    try application:which_applications() of
        Apps ->
            lists:map(
              fun({AppName, _, _}) ->
                      Mem = application_memory(AppName),
                      {AppName, Mem}
              end, Apps)
    catch
        _:Exception ->
            error_logger:error_msg("[~p] get application memory failed ~p", [?MODULE, Exception]),
            undefined
    end.

application_memory(App) ->
    case  application_controller:get_master(App) of
        Master when is_pid(Master) ->
            {Root, Name} = application_master:get_child(Master),
            {Memory, Children} = memory_info(Root),
            {Name, Memory div 1000000, Children};
        _ ->
            undefined
    end.

memory_info(Sup) ->
    Infos = supervisor:which_children(Sup),
    {M, E, ES} =
    lists:foldl(
      fun({Name, PId, Type, _}, {Total, Memories, MemorySets}) ->
              case Type of
                  worker ->
                      {memory, Memory} = process_info(PId, memory),
                      NTotal = Total + Memory,
                      case Name of
                          undefined ->
                              NMemorySets =
                                  case gb_sets:size(MemorySets) of
                                      Size when Size > 10 ->
                                          {_, NSets} =
                                              gb_sets:take_smallest(MemorySets),
                                          NSets;
                                      _ ->
                                          MemorySets
                                  end,
                              NNMemorySets = gb_sets:insert(
                                              {undefined, Memory, PId}, NMemorySets),
                              {NTotal, Memories, NNMemorySets};
                          Name ->
                                  {NTotal, [{Name, Memory, PId}|Memories], MemorySets}
                      end;
                  supervisor ->
                      {Memory, Each} = memory_info(PId),
                      NTotal = Total + Memory,

                      {NTotal, [{Name, {Memory, Each}, PId}|Memories], MemorySets}
              end
      end, {0, [], gb_sets:new()}, Infos),
    NE =
        case E of
            [] ->
                gb_sets:to_list(ES);
            _ ->
                E
        end,
    NNE = lists:map(
              fun({N, Mem, PId}) ->
                      case Mem of
                          Mem when is_integer(Mem) ->
                              {N, Mem div 1000000, undefined, PId};
                          {T, RestEach} ->
                              {N, T div 1000000, PId, RestEach}
                      end
              end, NE),
    NNNE = lists:reverse(lists:keysort(3, NNE)),
    {M, NNNE}.

ets_memory() ->
    All = ets:all(),
    AllMems = lists:map(
                fun(Ets) ->
                        Mem = ets:info(Ets, memory),
                        {Ets, Mem}
                end, All),
    Mems =
        lists:sort(
          fun({_, M1}, {_, M2}) ->
                  M1 > M2
          end, AllMems),
    Sum = lists:sum(lists:map(fun({_, Mem}) -> Mem end, AllMems)),
    {Sum, Mems}.
```

On Sat, Sep 28, 2019 at 10:03 PM <[hidden email]> wrote:
Hi there,

are there experience reports on using Erlang/Elixir on systems with 200 GB of memory and more?

If you have a first-hand account on that, please share it with me.

My friends are doing that and said that only C++ was capable of handling that.

- Gergely
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


--

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Experience with much memory

Richard Carlsson-3
In reply to this post by Gergely Buday
We have been running Erlang/OTP on machines with up to 3 TB of RAM, for many years. The Beam VM has no problems with this; we have seen single Erlang processes within a Beam use a few hundred GB of temporary data and then deallocate it again, while the VMs can keep running for months on end. The problems at that scale typically show up in the OS kernel, and you may need to tune it to avoid long pauses in the OS memory management, and occasionally flush the page cache to defragment the RAM.

        /Richard


Den lör 28 sep. 2019 kl 16:03 skrev <[hidden email]>:
Hi there,

are there experience reports on using Erlang/Elixir on systems with 200 GB of memory and more?

If you have a first-hand account on that, please share it with me.

My friends are doing that and said that only C++ was capable of handling that.

- Gergely
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Experience with much memory

Dániel Szoboszlay
I would like to add to Richard's points that you may have to tune the memory allocator settings of the Beam too above a few hundred GB-s. The default values may lead to some fragmentation that is becoming significant at this point, especially for long-term allocated memory (typically: ETS tables). I would advise not to use address order first fit strategies for example, but rather stick to the old default, best fit strategy. Using a super carrier may be beneficial too if you expect to use a lot of memory.

Cheers,
Daniel

On Sat, 28 Sep 2019 at 17:07, Richard Carlsson <[hidden email]> wrote:
We have been running Erlang/OTP on machines with up to 3 TB of RAM, for many years. The Beam VM has no problems with this; we have seen single Erlang processes within a Beam use a few hundred GB of temporary data and then deallocate it again, while the VMs can keep running for months on end. The problems at that scale typically show up in the OS kernel, and you may need to tune it to avoid long pauses in the OS memory management, and occasionally flush the page cache to defragment the RAM.

        /Richard


Den lör 28 sep. 2019 kl 16:03 skrev <[hidden email]>:
Hi there,

are there experience reports on using Erlang/Elixir on systems with 200 GB of memory and more?

If you have a first-hand account on that, please share it with me.

My friends are doing that and said that only C++ was capable of handling that.

- Gergely
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Experience with much memory

Max Lapshin-2
Your friends are oversimplifying things of course.

When you speak about 200GB or 3TB of RAM, you need to speak about things a bit deeper than "fast or not fast".  Here is the beginning of long list of issues to discuss:


1) is your software NUMA aware?  Many people experience strange performance degradations when they upgrade from 1CPU to 2CPU and double system memory. Numa unaware software starts migrating data between processors with serious penalty.

Erlang is NUMA aware out of the box, but you can spoil things if you allow too much data migration.



2) how big is per-process memory?  If you allocate 200GB of small integers into tuples and all of them in a single erlang process, then you will not be able even to do it in a reasonable amount of time, I suppose =)
If you do things correctly and all your 200GB are in ets, binaries and spread across different processes, then it will be ok



3) how fast is memory allocation/deallocation done?  If you just read 200GB file to memory and read data from it, then it is really easy.
If you do complicated allocations/deallocations with complex memory connections, then your memory allocator should be fast enought
to handle these allocations on 200GB of RAM. Erlang is fast enough for this.

Program written on C++ will have lot of hidden bugs that are hard to find because valgrinding 200GB is a funny task.


There are other questions about "is 200GB handling possible", these 3 are just what I could remember.

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Experience with much memory

Mikael Pettersson-5
In reply to this post by Gergely Buday
On Sat, Sep 28, 2019 at 4:03 PM <[hidden email]> wrote:
>
> Hi there,
>
> are there experience reports on using Erlang/Elixir on systems with 200 GB of memory and more?
>
> If you have a first-hand account on that, please share it with me.

The Erlang/OTP VM handles that and more without much trouble.  We have
been running Erlang on servers with multi-TB RAM for many years now.
There are a few things to look out for:

- Use a recent (say OTP-19 or above) version of the VM, older ones had
integer truncation bugs that could lead to wrong results or
segmentation faults when dealing with very large terms.
- Disable transparent hugepages in the Linux kernel, that's been known
to cause huge unpredictable pauses or even kernel crashes.  (May or
may now have been fixed now, but was certainly true in RHEL/CentOS 6.)
- Page fragmentation is an issue, also causing occasional pauses when
the Linux kernel goes busy.  Watch buddyinfo, drop caches sometimes,
and _maybe_ even compact memory.  However, dropping caches hurts file
system performance so don't do that unnecessarily.
- Increase max_map_count.  At TB sizes the VM can reach the default
limit and the failures are _not_ easy to debug.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions