What is "Abandon carrier utilization limit" for?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

What is "Abandon carrier utilization limit" for?

Roger Lipscombe-2
What does it mean for a carrier to be abandoned?

The documentation says that if an allocator needs another carrier,
it'll re-use an abandoned one, which doesn't sound very abandoned to
me.

Can the VM move blocks from an abandoned carrier to a new carrier? I
assume not, which means that abandoned carriers can't (easily) be
returned to the OS.

Or is it that, once a carrier falls below the threshold, the VM tries
not to use it, so that it's more likely (assuming that the remaining
content eventually gets freed) that it'll get returned to the OS?

Also: what is the default setting? The documentation says "de", but
system_info is saying "0" for all of my allocators. Is "de" choosing
zero for me?

Cheers,
Roger.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: What is "Abandon carrier utilization limit" for?

Sverker Eriksson-4
There is actually an internal doc on this subject:

https://github.com/erlang/otp/blob/master/erts/emulator/internal_doc/CarrierMigration.md


On 10/13/2017 04:41 PM, Roger Lipscombe wrote:
> What does it mean for a carrier to be abandoned?
Blocks are not allocated from abandoned carrier.
An abandoned carrier is available to be employed by any thread in which
case it's not abandoned anymore.

>
> The documentation says that if an allocator needs another carrier,
> it'll re-use an abandoned one, which doesn't sound very abandoned to
> me.
> Can the VM move blocks from an abandoned carrier to a new carrier? I
> assume not, which means that abandoned carriers can't (easily) be
> returned to the OS.
A carrier is a large contiguous piece of memory.
A block a smaller contiguous piece of memory within a carrier.
So no, moving block between carrier does not even make sense.


>
> Or is it that, once a carrier falls below the threshold, the VM tries
> not to use it, so that it's more likely (assuming that the remaining
> content eventually gets freed) that it'll get returned to the OS?
Yes, that is the idea.

> Also: what is the default setting? The documentation says "de", but
> system_info is saying "0" for all of my allocators. Is "de" choosing
> zero for me?
>
>
Hmm.... I'll be back...
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: What is "Abandon carrier utilization limit" for?

Roger Lipscombe-2
On 13 October 2017 at 18:05, Sverker Eriksson
<[hidden email]> wrote:
> There is actually an internal doc on this subject:
>
> https://github.com/erlang/otp/blob/master/erts/emulator/internal_doc/CarrierMigration.md

Thanks. I'll check that out.

> A carrier is a large contiguous piece of memory.
> A block a smaller contiguous piece of memory within a carrier.

I watched Lukas's presentation from 2014, so I got that part :)

> So no, moving block between carrier does not even make sense.

I guess I meant: can the VM copy the content to another block in a
"live" carrier, thus freeing up this carrier, but I guess that'd
involve updating all kinds of pointers, and that stuff happens at a
higher level in the VM, right?

>> Or is it that, once a carrier falls below the threshold, the VM tries
>> not to use it, so that it's more likely (assuming that the remaining
>> content eventually gets freed) that it'll get returned to the OS?
>
> Yes, that is the idea.

Cool.

>> Also: what is the default setting? The documentation says "de", but
>> system_info is saying "0" for all of my allocators. Is "de" choosing
>> zero for me?
>>
>>
> Hmm.... I'll be back...

OTP 20-rc2, btw; I'll try a newer build... I'll be back... :)
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: What is "Abandon carrier utilization limit" for?

Roger Lipscombe-2
On 13 October 2017 at 18:33, Roger Lipscombe <[hidden email]> wrote:
>>> Also: what is the default setting? The documentation says "de", but
>>> system_info is saying "0" for all of my allocators. Is "de" choosing
>>> zero for me?
>>>
>>>
>> Hmm.... I'll be back...
>
> OTP 20-rc2, btw; I'll try a newer build... I'll be back... :)

Still shows zero for OTP-20.1:

1> {_, _, _, A} = erlang:system_info(allocator).
2> lists:filtermap(fun({Id, Props}) when is_list(Props) -> {true, {Id,
proplists:get_value(acul, Props, undefined)}}; (_) -> false end, A).
[{sys_alloc,undefined},
 {temp_alloc,0},
 {sl_alloc,0},
 {std_alloc,0},
 {ll_alloc,0},
 {eheap_alloc,0},
 {ets_alloc,0},
 {fix_alloc,0},
 {literal_alloc,0},
 {exec_alloc,0},
 {binary_alloc,0},
 {driver_alloc,0},
 {test_alloc,undefined},
 {mseg_alloc,undefined},
 {alloc_util,undefined},
 {erts_mmap,undefined},
 {instr,undefined}]
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: What is "Abandon carrier utilization limit" for?

Sverker Eriksson-4
In reply to this post by Roger Lipscombe-2

On 10/13/2017 07:33 PM, Roger Lipscombe wrote:
>> So no, moving block between carrier does not even make sense.
> I guess I meant: can the VM copy the content to another block in a
> "live" carrier, thus freeing up this carrier, but I guess that'd
> involve updating all kinds of pointers, and that stuff happens at a
> higher level in the VM, right?

No, we don't do that. It can get very complicated in the general case as
the higher level code has to update all pointers to the block in a
thread-safe manner.

We did some experimental hacks for binary_alloc where it could maybe be
feasible as the data is read-only.

>>> Or is it that, once a carrier falls below the threshold, the VM tries
>>> not to use it, so that it's more likely (assuming that the remaining
>>> content eventually gets freed) that it'll get returned to the OS?
>> Yes, that is the idea.
> Cool.
>
>>> Also: what is the default setting? The documentation says "de", but
>>> system_info is saying "0" for all of my allocators. Is "de" choosing
>>> zero for me?
>>>
>>>
>> Hmm.... I'll be back...
> OTP 20-rc2, btw; I'll try a newer build... I'll be back... :)
>
system_info(allocator) only gives an overview which can be a bit misleading.
If you look at the individual allocator instances with for example

erlang:system_info({allocator, ets_alloc}).

you will see all instances except #0 have non zero values for 'acul'.

Instance #0 is a lock protected allocator used only by non-scheduler
threads.
It disables carrier migration as only scheduler threads can use the
lock-free carrier pool implementation.


/Sverker

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions