How to get HugePages to be used on Linux?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

How to get HugePages to be used on Linux?

Vans S
Has anyone gotten Erlang to use hugepages on linux?  I allocated the default 2MB pages via kernel param, but I see they are not used when I run erlang, I also made it allocated in 2047 sized blocks as recommended by WhatsApp team (tho why 2047 and not 2048).  

Also what about disabling THP, anyone tested with and without?
Reply | Threaded
Open this post in threaded view
|

Re: How to get HugePages to be used on Linux?

Luke Bakken-2
Hello,

This is pretty much a guess, but you may want to read about the super
carrier - https://erlang.org/doc/apps/erts/SuperCarrier.html

On Wed, May 6, 2020 at 7:51 AM Vans S <[hidden email]> wrote:
>
> Has anyone gotten Erlang to use hugepages on linux?  I allocated the default 2MB pages via kernel param, but I see they are not used when I run erlang, I also made it allocated in 2047 sized blocks as recommended by WhatsApp team (tho why 2047 and not 2048).
>
> Also what about disabling THP, anyone tested with and without?
Reply | Threaded
Open this post in threaded view
|

Re: How to get HugePages to be used on Linux?

Mikael Pettersson-5
In reply to this post by Vans S
On Wed, May 6, 2020 at 4:51 PM Vans S <[hidden email]> wrote:
>
> Has anyone gotten Erlang to use hugepages on linux?  I allocated the default 2MB pages via kernel param, but I see they are not used when I run erlang, I also made it allocated in 2047 sized blocks as recommended by WhatsApp team (tho why 2047 and not 2048).
>
> Also what about disabling THP, anyone tested with and without?

I did some experiments a few years ago with hugepages and 2TB RAM servers.

First THP (transparent hugepages) did not work well, as they caused
major responsiveness issues whenever the kernel decided to "do stuff"
behind our back.  We always disable THP, and folks at EUC (now Code
BEAM Sto) mentioned doing the same.  Maybe the 4.x/5.x kernels have
improved, but CentOS 7 is definitely affected.  YMMV.

Second, without THP you need to tell the kernel explicitly to give you
hugepages via a flag in mmap().  Back then (R15-ish) OTP wouldn't do
that so I had to hack it in, but it didn't help much as we were
dominated by other costs.

Later on they added Super Carriers (pre-allocation), and that's a
must-have for large RAM nodes.  Mapping those with explicit hugepages
should be beneficial, but OTP would still need a patch to do so.
Reply | Threaded
Open this post in threaded view
|

Re: How to get HugePages to be used on Linux?

Vans S
This is pretty useful thanks, we noticed often we would have 30% CPU usage via microstate accounting for the allocator.  Bumped the min_heap_size but looking for other possible optimizations.  I think then we can disable hugepages set as kernel cmdline for now and just allocate a large chunk of the system ram as a supercarrier.

On Wednesday, May 6, 2020, 12:31:45 p.m. EDT, Mikael Pettersson <[hidden email]> wrote:


On Wed, May 6, 2020 at 4:51 PM Vans S <[hidden email]> wrote:

>
> Has anyone gotten Erlang to use hugepages on linux?  I allocated the default 2MB pages via kernel param, but I see they are not used when I run erlang, I also made it allocated in 2047 sized blocks as recommended by WhatsApp team (tho why 2047 and not 2048).
>
> Also what about disabling THP, anyone tested with and without?


I did some experiments a few years ago with hugepages and 2TB RAM servers.

First THP (transparent hugepages) did not work well, as they caused
major responsiveness issues whenever the kernel decided to "do stuff"
behind our back.  We always disable THP, and folks at EUC (now Code
BEAM Sto) mentioned doing the same.  Maybe the 4.x/5.x kernels have
improved, but CentOS 7 is definitely affected.  YMMV.

Second, without THP you need to tell the kernel explicitly to give you
hugepages via a flag in mmap().  Back then (R15-ish) OTP wouldn't do
that so I had to hack it in, but it didn't help much as we were
dominated by other costs.

Later on they added Super Carriers (pre-allocation), and that's a
must-have for large RAM nodes.  Mapping those with explicit hugepages
should be beneficial, but OTP would still need a patch to do so.
Reply | Threaded
Open this post in threaded view
|

Re: How to get HugePages to be used on Linux?

Vans S
In reply to this post by Luke Bakken-2
It seems the +MMscrpm true flag does not work on linux.  It seems to only reverse the virtual memory not physical.  

On Wednesday, May 6, 2020, 10:58:23 a.m. EDT, Luke Bakken <[hidden email]> wrote:


Hello,

This is pretty much a guess, but you may want to read about the super
carrier - https://erlang.org/doc/apps/erts/SuperCarrier.html

On Wed, May 6, 2020 at 7:51 AM Vans S <[hidden email]> wrote:
>
> Has anyone gotten Erlang to use hugepages on linux?  I allocated the default 2MB pages via kernel param, but I see they are not used when I run erlang, I also made it allocated in 2047 sized blocks as recommended by WhatsApp team (tho why 2047 and not 2048).
>
> Also what about disabling THP, anyone tested with and without?