stop process migration

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

stop process migration

Satish Patel
Folks,

Can i tell erlang to not load-balance process or migrate process to
different CPU?
Reply | Threaded
Open this post in threaded view
|

Re: stop process migration

Karl Velicka
Hi,

(I'm reading your question as "how to pin an Erlang process to a CPU _core_)

There is an option to erlang:spawn_opt - {scheduler, SchedNum} to pin a process to a specific scheduler in the VM. However, this option is undocumented so it's probably exposed on "caveat emptor" basis. We get the possible scheduler numbers by doing lists:seq(1, erlang:system_info(schedulers_online)) and we've been using the flag in OTP versions 20-22.

The scheduler itself is just a thread from the perspective of the OS so I assume it shouldn't be difficult to pin to a core.

All the best,
Karl

On Thu, 20 Feb 2020 at 17:05, Satish Patel <[hidden email]> wrote:
Folks,

Can i tell erlang to not load-balance process or migrate process to
different CPU?
Reply | Threaded
Open this post in threaded view
|

Using undocumented features (Was: stop process migration)

Rickard Green-2
On Fri, Feb 21, 2020 at 8:59 PM Karl Velicka <[hidden email]> wrote:
Hi,

(I'm reading your question as "how to pin an Erlang process to a CPU _core_)

There is an option to erlang:spawn_opt - {scheduler, SchedNum} to pin a process to a specific scheduler in the VM. However, this option is undocumented so it's probably exposed on "caveat emptor" basis. We get the possible scheduler numbers by doing lists:seq(1, erlang:system_info(schedulers_online)) and we've been using the flag in OTP versions 20-22.


This feature was introduced to simplify our testing. It does *not* have product quality and may cause major problems for the system as a whole. Especially if lots of processes are pinned to specific schedulers.

In general, using undocumented features or API:s may cause you major problems. They may change behavior or be removed at any time without any notice from us what so ever. You may also not fully understand the effects of using them. Some internal API:s can cause very strange side effects if not used properly. That is, I strongly discourage you from using any undocumented stuff that exist in OTP.

A much better approach is to ask us for a new feature or to document an existing feature at <https://bugs.erlang.org> or make a pull request at <https://github.com/erlang/otp>. If/when it gets implemented and taken into OTP of course depends on what it is, how we prioritize it, etc.

Before anyone asks us to document the 'scheduler' option of spawn_opt(), I can answer you *no* straight away. At least not without having solved the issues that it can cause, and currently we do not have the time to do that.

The scheduler itself is just a thread from the perspective of the OS so I assume it shouldn't be difficult to pin to a core.


See <http://erlang.org/doc/man/erl.html#+sbt> on how to bind schedulers to logical processors.
 
All the best,
Karl

On Thu, 20 Feb 2020 at 17:05, Satish Patel <[hidden email]> wrote:
Folks,

Can i tell erlang to not load-balance process or migrate process to
different CPU?

Regards,
Rickard
--
Rickard Green, Erlang/OTP, Ericsson AB
Reply | Threaded
Open this post in threaded view
|

Re: stop process migration

Satish Patel
In reply to this post by Karl Velicka
Karl,

This is what currently we are doing to pin down CPU to specific NUMA
to grain performance, i have 32 core machine and i pin down erlang to
only stay on numa1 (because when i pin down on both numa then process
migration happen and it hurt performance between cross NUMA)

/usr/mongooseim/erts-7.3/bin/beam.smp -K true -A 5 -P 10000000 -Q
1000000 -e 100000 -sct
L16t0c0p0n1:L18t0c1p0n1:L20t0c2p0n1:L22t0c3p0n1:L24t0c4p0n1:L26t0c5p0n1:L28t0c6p0n1:L30t0c7p0n1:L17t1c0p0n1:L19t1c1p0n1:L21t1c2p0n1:L23t1c3p0n1:L25t1c4p0n1:L27t1c5p0n1:L29t1c6p0n1:L31t1c7p0n1
-sbt nnts -S 16 -sbt nnts -- -root /usr/mongooseim -progname
mongooseim -- -home /usr/mongooseim -- -boot
/usr/mongooseim/releases/2.1.0beta1/mongooseim -embedded -config
/usr/mongooseim/etc/app.config

On Fri, Feb 21, 2020 at 2:58 PM Karl Velicka <[hidden email]> wrote:

>
> Hi,
>
> (I'm reading your question as "how to pin an Erlang process to a CPU _core_)
>
> There is an option to erlang:spawn_opt - {scheduler, SchedNum} to pin a process to a specific scheduler in the VM. However, this option is undocumented so it's probably exposed on "caveat emptor" basis. We get the possible scheduler numbers by doing lists:seq(1, erlang:system_info(schedulers_online)) and we've been using the flag in OTP versions 20-22.
>
> The scheduler itself is just a thread from the perspective of the OS so I assume it shouldn't be difficult to pin to a core.
>
> All the best,
> Karl
>
> On Thu, 20 Feb 2020 at 17:05, Satish Patel <[hidden email]> wrote:
>>
>> Folks,
>>
>> Can i tell erlang to not load-balance process or migrate process to
>> different CPU?
Reply | Threaded
Open this post in threaded view
|

Re: stop process migration

Jesper Louis Andersen-2
This looks far better from my end.

I was going to ask what you would like to use your pinning for, but I now see you needed it one level up from where I thought you needed it.

On Sun, Feb 23, 2020 at 5:59 PM Satish Patel <[hidden email]> wrote:
Karl,

This is what currently we are doing to pin down CPU to specific NUMA
to grain performance, i have 32 core machine and i pin down erlang to
only stay on numa1 (because when i pin down on both numa then process
migration happen and it hurt performance between cross NUMA)

/usr/mongooseim/erts-7.3/bin/beam.smp -K true -A 5 -P 10000000 -Q
1000000 -e 100000 -sct
L16t0c0p0n1:L18t0c1p0n1:L20t0c2p0n1:L22t0c3p0n1:L24t0c4p0n1:L26t0c5p0n1:L28t0c6p0n1:L30t0c7p0n1:L17t1c0p0n1:L19t1c1p0n1:L21t1c2p0n1:L23t1c3p0n1:L25t1c4p0n1:L27t1c5p0n1:L29t1c6p0n1:L31t1c7p0n1
-sbt nnts -S 16 -sbt nnts -- -root /usr/mongooseim -progname
mongooseim -- -home /usr/mongooseim -- -boot
/usr/mongooseim/releases/2.1.0beta1/mongooseim -embedded -config
/usr/mongooseim/etc/app.config

On Fri, Feb 21, 2020 at 2:58 PM Karl Velicka <[hidden email]> wrote:
>
> Hi,
>
> (I'm reading your question as "how to pin an Erlang process to a CPU _core_)
>
> There is an option to erlang:spawn_opt - {scheduler, SchedNum} to pin a process to a specific scheduler in the VM. However, this option is undocumented so it's probably exposed on "caveat emptor" basis. We get the possible scheduler numbers by doing lists:seq(1, erlang:system_info(schedulers_online)) and we've been using the flag in OTP versions 20-22.
>
> The scheduler itself is just a thread from the perspective of the OS so I assume it shouldn't be difficult to pin to a core.
>
> All the best,
> Karl
>
> On Thu, 20 Feb 2020 at 17:05, Satish Patel <[hidden email]> wrote:
>>
>> Folks,
>>
>> Can i tell erlang to not load-balance process or migrate process to
>> different CPU?


--
J.
Reply | Threaded
Open this post in threaded view
|

Re: stop process migration

Satish Patel
On hardware i am getting really good performance result but not on
openstack vm which has same compute hardware, we want vm to give us
same level of performance which we are getting from hardware (atl east
90% not 100%)

So i build single large VM (32vCPU) on kvm host machine which has (40
CPU core in HT), i did all best practices to get good performance from
kvm like cpu pinning/hugepage/sriov network etc, but when i run
application i am getting (50% benchmark result in short performance is
worst) so i have tried to use erlang CPU pinning to bind schedule with
NUMA0 (so erlang only going to use 16vCPU core located on NUMA0) in
that case i got very good benchmark result (70% result) but in this
case i am loosing NUMA1 cpu cores :(

so trying to understand how do i tell erlang when you see two NUMA
zone then please keep your process in that zone and don't migrate from
NUMA0 --> NUMA1 which causing high latency to access memory and
performance getting worst.

I am surprised no one noticed this kind of behavior before.

On Sun, Feb 23, 2020 at 12:20 PM Jesper Louis Andersen
<[hidden email]> wrote:

>
> This looks far better from my end.
>
> I was going to ask what you would like to use your pinning for, but I now see you needed it one level up from where I thought you needed it.
>
> On Sun, Feb 23, 2020 at 5:59 PM Satish Patel <[hidden email]> wrote:
>>
>> Karl,
>>
>> This is what currently we are doing to pin down CPU to specific NUMA
>> to grain performance, i have 32 core machine and i pin down erlang to
>> only stay on numa1 (because when i pin down on both numa then process
>> migration happen and it hurt performance between cross NUMA)
>>
>> /usr/mongooseim/erts-7.3/bin/beam.smp -K true -A 5 -P 10000000 -Q
>> 1000000 -e 100000 -sct
>> L16t0c0p0n1:L18t0c1p0n1:L20t0c2p0n1:L22t0c3p0n1:L24t0c4p0n1:L26t0c5p0n1:L28t0c6p0n1:L30t0c7p0n1:L17t1c0p0n1:L19t1c1p0n1:L21t1c2p0n1:L23t1c3p0n1:L25t1c4p0n1:L27t1c5p0n1:L29t1c6p0n1:L31t1c7p0n1
>> -sbt nnts -S 16 -sbt nnts -- -root /usr/mongooseim -progname
>> mongooseim -- -home /usr/mongooseim -- -boot
>> /usr/mongooseim/releases/2.1.0beta1/mongooseim -embedded -config
>> /usr/mongooseim/etc/app.config
>>
>> On Fri, Feb 21, 2020 at 2:58 PM Karl Velicka <[hidden email]> wrote:
>> >
>> > Hi,
>> >
>> > (I'm reading your question as "how to pin an Erlang process to a CPU _core_)
>> >
>> > There is an option to erlang:spawn_opt - {scheduler, SchedNum} to pin a process to a specific scheduler in the VM. However, this option is undocumented so it's probably exposed on "caveat emptor" basis. We get the possible scheduler numbers by doing lists:seq(1, erlang:system_info(schedulers_online)) and we've been using the flag in OTP versions 20-22.
>> >
>> > The scheduler itself is just a thread from the perspective of the OS so I assume it shouldn't be difficult to pin to a core.
>> >
>> > All the best,
>> > Karl
>> >
>> > On Thu, 20 Feb 2020 at 17:05, Satish Patel <[hidden email]> wrote:
>> >>
>> >> Folks,
>> >>
>> >> Can i tell erlang to not load-balance process or migrate process to
>> >> different CPU?
>
>
>
> --
> J.
Reply | Threaded
Open this post in threaded view
|

Re: Using undocumented features (Was: stop process migration)

Karl Velicka
In reply to this post by Rickard Green-2
Rickard,

Thank you for additional info regarding this flag. We think we understand the implications on using features that are not exposed via the documented APIs but perhaps my earlier message did not stress that enough.

We will look into making a feature request for this - thank you for the suggestion!

Karl

On Sat, 22 Feb 2020 at 21:49, Rickard Green <[hidden email]> wrote:
On Fri, Feb 21, 2020 at 8:59 PM Karl Velicka <[hidden email]> wrote:
Hi,

(I'm reading your question as "how to pin an Erlang process to a CPU _core_)

There is an option to erlang:spawn_opt - {scheduler, SchedNum} to pin a process to a specific scheduler in the VM. However, this option is undocumented so it's probably exposed on "caveat emptor" basis. We get the possible scheduler numbers by doing lists:seq(1, erlang:system_info(schedulers_online)) and we've been using the flag in OTP versions 20-22.


This feature was introduced to simplify our testing. It does *not* have product quality and may cause major problems for the system as a whole. Especially if lots of processes are pinned to specific schedulers.

In general, using undocumented features or API:s may cause you major problems. They may change behavior or be removed at any time without any notice from us what so ever. You may also not fully understand the effects of using them. Some internal API:s can cause very strange side effects if not used properly. That is, I strongly discourage you from using any undocumented stuff that exist in OTP.

A much better approach is to ask us for a new feature or to document an existing feature at <https://bugs.erlang.org> or make a pull request at <https://github.com/erlang/otp>. If/when it gets implemented and taken into OTP of course depends on what it is, how we prioritize it, etc.

Before anyone asks us to document the 'scheduler' option of spawn_opt(), I can answer you *no* straight away. At least not without having solved the issues that it can cause, and currently we do not have the time to do that.

The scheduler itself is just a thread from the perspective of the OS so I assume it shouldn't be difficult to pin to a core.


See <http://erlang.org/doc/man/erl.html#+sbt> on how to bind schedulers to logical processors.
 
All the best,
Karl

On Thu, 20 Feb 2020 at 17:05, Satish Patel <[hidden email]> wrote:
Folks,

Can i tell erlang to not load-balance process or migrate process to
different CPU?

Regards,
Rickard

--
Rickard Green, Erlang/OTP, Ericsson AB
Reply | Threaded
Open this post in threaded view
|

Re: stop process migration

Satish Patel
In reply to this post by Satish Patel
This is what i did to make it work, its heck but seems working, I have
two numa node and each has 16vCPU so i am running two erlang process
on two different ports and each binding to each NUMA like following

erlang-mongooseIM-1 CPU bind with NUMA0 (16 vCPU)
erlang-mongooseIM-2 CPU bind with NUMA1 (16 vCPU)

Its very ugly but seems working and i can get more performance out of it.

On Mon, Feb 24, 2020 at 10:14 AM Satish Patel <[hidden email]> wrote:

>
> On hardware i am getting really good performance result but not on
> openstack vm which has same compute hardware, we want vm to give us
> same level of performance which we are getting from hardware (atl east
> 90% not 100%)
>
> So i build single large VM (32vCPU) on kvm host machine which has (40
> CPU core in HT), i did all best practices to get good performance from
> kvm like cpu pinning/hugepage/sriov network etc, but when i run
> application i am getting (50% benchmark result in short performance is
> worst) so i have tried to use erlang CPU pinning to bind schedule with
> NUMA0 (so erlang only going to use 16vCPU core located on NUMA0) in
> that case i got very good benchmark result (70% result) but in this
> case i am loosing NUMA1 cpu cores :(
>
> so trying to understand how do i tell erlang when you see two NUMA
> zone then please keep your process in that zone and don't migrate from
> NUMA0 --> NUMA1 which causing high latency to access memory and
> performance getting worst.
>
> I am surprised no one noticed this kind of behavior before.
>
> On Sun, Feb 23, 2020 at 12:20 PM Jesper Louis Andersen
> <[hidden email]> wrote:
> >
> > This looks far better from my end.
> >
> > I was going to ask what you would like to use your pinning for, but I now see you needed it one level up from where I thought you needed it.
> >
> > On Sun, Feb 23, 2020 at 5:59 PM Satish Patel <[hidden email]> wrote:
> >>
> >> Karl,
> >>
> >> This is what currently we are doing to pin down CPU to specific NUMA
> >> to grain performance, i have 32 core machine and i pin down erlang to
> >> only stay on numa1 (because when i pin down on both numa then process
> >> migration happen and it hurt performance between cross NUMA)
> >>
> >> /usr/mongooseim/erts-7.3/bin/beam.smp -K true -A 5 -P 10000000 -Q
> >> 1000000 -e 100000 -sct
> >> L16t0c0p0n1:L18t0c1p0n1:L20t0c2p0n1:L22t0c3p0n1:L24t0c4p0n1:L26t0c5p0n1:L28t0c6p0n1:L30t0c7p0n1:L17t1c0p0n1:L19t1c1p0n1:L21t1c2p0n1:L23t1c3p0n1:L25t1c4p0n1:L27t1c5p0n1:L29t1c6p0n1:L31t1c7p0n1
> >> -sbt nnts -S 16 -sbt nnts -- -root /usr/mongooseim -progname
> >> mongooseim -- -home /usr/mongooseim -- -boot
> >> /usr/mongooseim/releases/2.1.0beta1/mongooseim -embedded -config
> >> /usr/mongooseim/etc/app.config
> >>
> >> On Fri, Feb 21, 2020 at 2:58 PM Karl Velicka <[hidden email]> wrote:
> >> >
> >> > Hi,
> >> >
> >> > (I'm reading your question as "how to pin an Erlang process to a CPU _core_)
> >> >
> >> > There is an option to erlang:spawn_opt - {scheduler, SchedNum} to pin a process to a specific scheduler in the VM. However, this option is undocumented so it's probably exposed on "caveat emptor" basis. We get the possible scheduler numbers by doing lists:seq(1, erlang:system_info(schedulers_online)) and we've been using the flag in OTP versions 20-22.
> >> >
> >> > The scheduler itself is just a thread from the perspective of the OS so I assume it shouldn't be difficult to pin to a core.
> >> >
> >> > All the best,
> >> > Karl
> >> >
> >> > On Thu, 20 Feb 2020 at 17:05, Satish Patel <[hidden email]> wrote:
> >> >>
> >> >> Folks,
> >> >>
> >> >> Can i tell erlang to not load-balance process or migrate process to
> >> >> different CPU?
> >
> >
> >
> > --
> > J.