Frequency of module reloading and the constant pool

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Frequency of module reloading and the constant pool

Edmond Begumisa
Hello all,

(Not new here--just haven't posted in a while!)

I'm using merl to dynamically generate and regenerate modules at runtime  
that make use of the ERTS constant pool in order to efficiently avail data  
to all processes. I'm aware that this technique is known amongst more  
experienced Erlangers for situations where the data is updated  
infrequently (e.g. sharing global configuration parameters).

The question I'm investigating is how frequent is "infrequent" w.r.t  
reloading of modules[1]? I'm not entirely sure about the costs involved  
when (re)loading a module so I've got no idea where to start in order to  
measure the impact on the emulator. Could anybody with sufficient  
knowledge of emulator internals point me in the right direction?

Thanks in advance.

- Edmond -

[1] The more exotic use-cases for use/abuse of the constant pool and  
module reloading I'm looking into are to do with caching and incremental  
computation.

--
Using Opera's mail client: http://www.opera.com/mail/
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Frequency of module reloading and the constant pool

Edmond Begumisa
On Sun, 17 Sep 2017 03:39:50 +1000, Edmond Begumisa  
<[hidden email]> wrote:

> Hello all,
>
> (Not new here--just haven't posted in a while!)
>
> I'm using merl to dynamically generate and regenerate modules at runtime  
> that make use of the ERTS constant pool in order to efficiently avail  
> data to all processes. I'm aware that this technique is known amongst  
> more experienced Erlangers for situations where the data is updated  
> infrequently (e.g. sharing global configuration parameters).
>
> The question I'm investigating is how frequent is "infrequent" w.r.t  
> reloading of modules[1]? I'm not entirely sure about the costs involved  
> when (re)loading a module so I've got no idea where to start in order to  
> measure the impact on the emulator. Could anybody with sufficient  
> knowledge of emulator internals point me in the right direction?
>

To clarify: When measuring whether or not I'm hot-code swapping too  
frequently, what are the negative signs to look for? What does the  
emulator actually do when swapping out code w.r.t potentially  
disruptive/costly operations like locking, GC, etc which could negatively  
impact scheduling, memory and responsiveness of the system?

I'm trying to figure out when the best time to reload a dynamically  
generated module/group of modules is and how frequently I should allow  
reloading to occur.

- Edmond -

> Thanks in advance.
>
> - Edmond -
>
> [1] The more exotic use-cases for use/abuse of the constant pool and  
> module reloading I'm looking into are to do with caching and incremental  
> computation.
>


--
Using Opera's mail client: http://www.opera.com/mail/
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Frequency of module reloading and the constant pool

Adam Lindberg-7
Cost wise I'm not sure, but I would advice you to benchmark for your use case.

Operations wise the most common problem you'll see with module reloads is crashed processes. Erlang only keeps two versions of a module in memory at the same time. If you have processes lingering in the old code when reloading yet another version, those processes will get killed.

Cheers,
Adam

> On 16 Sep 2017, at 23:57, Edmond Begumisa <[hidden email]> wrote:
>
> On Sun, 17 Sep 2017 03:39:50 +1000, Edmond Begumisa <[hidden email]> wrote:
>
>> Hello all,
>>
>> (Not new here--just haven't posted in a while!)
>>
>> I'm using merl to dynamically generate and regenerate modules at runtime that make use of the ERTS constant pool in order to efficiently avail data to all processes. I'm aware that this technique is known amongst more experienced Erlangers for situations where the data is updated infrequently (e.g. sharing global configuration parameters).
>>
>> The question I'm investigating is how frequent is "infrequent" w.r.t reloading of modules[1]? I'm not entirely sure about the costs involved when (re)loading a module so I've got no idea where to start in order to measure the impact on the emulator. Could anybody with sufficient knowledge of emulator internals point me in the right direction?
>>
>
> To clarify: When measuring whether or not I'm hot-code swapping too frequently, what are the negative signs to look for? What does the emulator actually do when swapping out code w.r.t potentially disruptive/costly operations like locking, GC, etc which could negatively impact scheduling, memory and responsiveness of the system?
>
> I'm trying to figure out when the best time to reload a dynamically generated module/group of modules is and how frequently I should allow reloading to occur.
>
> - Edmond -
>
>> Thanks in advance.
>>
>> - Edmond -
>>
>> [1] The more exotic use-cases for use/abuse of the constant pool and module reloading I'm looking into are to do with caching and incremental computation.
>>
>
>
> --
> Using Opera's mail client: http://www.opera.com/mail/
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Frequency of module reloading and the constant pool

Lukas Larsson-8
On Tue, Sep 19, 2017 at 11:13 AM, Adam Lindberg <[hidden email]> wrote:
Cost wise I'm not sure, but I would advice you to benchmark for your use case.

Every time you purge a module, each process heap is scanned for any sign of the old literals (aka constants) and if a literal is found a GC is done of that process which copies the literal onto its heap.

So the cost depends on the total heap size of all running processes and the number of processes that have a reference to those literals on their heaps at the moment of the purge.

Lukas


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Frequency of module reloading and the constant pool

Edmond Begumisa
In reply to this post by Adam Lindberg-7

Thanks for the tips Lukas.

That process killing issue did actually bite me in the ar*e a few seemingly random times. Luckily, for my use-case, the processes depending on the controlled hot-swapping are known so I was able to mitigate against this by keeping track of the which versions each process knows about.

- Edmond -



On Tuesday, 19 September 2017 at 19:13 Adam Lindberg <[hidden email]> wrote:

Cost wise I'm not sure, but I would advice you to benchmark for your use case.

Operations wise the most common problem you'll see with module reloads is crashed processes. Erlang only keeps two versions of a module in memory at the same time. If you have processes lingering in the old code when reloading yet another version, those processes will get killed.

Cheers,
Adam

> On 16 Sep 2017, at 23:57, Edmond Begumisa wrote:
>
> On Sun, 17 Sep 2017 03:39:50 +1000, Edmond Begumisa wrote:
>
>> Hello all,
>>
>> (Not new here--just haven't posted in a while!)
>>
>> I'm using merl to dynamically generate and regenerate modules at runtime that make use of the ERTS constant pool in order to efficiently avail data to all processes. I'm aware that this technique is known amongst more experienced Erlangers for situations where the data is updated infrequently (e.g. sharing global configuration parameters).
>>
>> The question I'm investigating is how frequent is "infrequent" w.r.t reloading of modules[1]? I'm not entirely sure about the costs involved when (re)loading a module so I've got no idea where to start in order to measure the impact on the emulator. Could anybody with sufficient knowledge of emulator internals point me in the right direction?
>>
>
> To clarify: When measuring whether or not I'm hot-code swapping too frequently, what are the negative signs to look for? What does the emulator actually do when swapping out code w.r.t potentially disruptive/costly operations like locking, GC, etc which could negatively impact scheduling, memory and responsiveness of the system?
>
> I'm trying to figure out when the best time to reload a dynamically generated module/group of modules is and how frequently I should allow reloading to occur.
>
> - Edmond -
>
>> Thanks in advance.
>>
>> - Edmond -
>>
>> [1] The more exotic use-cases for use/abuse of the constant pool and module reloading I'm looking into are to do with caching and incremental computation.
>>
>
>
> --
> Using Opera's mail client: http://www.opera.com/mail/
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Frequency of module reloading and the constant pool

Edmond Begumisa

Sorry, I meant Adam not Lukas!

- Edmond -



On Thursday, 21 September 2017 at 08:09 Edmond Begumisa <[hidden email]> wrote:

Thanks for the tips Lukas.

That process killing issue did actually bite me in the ar*e a few seemingly random times. Luckily, for my use-case, the processes depending on the controlled hot-swapping are known so I was able to mitigate against this by keeping track of the which versions each process knows about.

- Edmond -
On Tuesday, 19 September 2017 at 19:13 Adam Lindberg wrote:
Cost wise I'm not sure, but I would advice you to benchmark for your use case.

Operations wise the most common problem you'll see with module reloads is crashed processes. Erlang only keeps two versions of a module in memory at the same time. If you have processes lingering in the old code when reloading yet another version, those processes will get killed.

Cheers,
Adam

> On 16 Sep 2017, at 23:57, Edmond Begumisa wrote:
>
> On Sun, 17 Sep 2017 03:39:50 +1000, Edmond Begumisa wrote:
>
>> Hello all,
>>
>> (Not new here--just haven't posted in a while!)
>>
>> I'm using merl to dynamically generate and regenerate modules at runtime that make use of the ERTS constant pool in order to efficiently avail data to all processes. I'm aware that this technique is known amongst more experienced Erlangers for situations where the data is updated infrequently (e.g. sharing global configuration parameters).
>>
>> The question I'm investigating is how frequent is "infrequent" w.r.t reloading of modules[1]? I'm not entirely sure about the costs involved when (re)loading a module so I've got no idea where to start in order to measure the impact on the emulator. Could anybody with sufficient knowledge of emulator internals point me in the right direction?
>>
>
> To clarify: When measuring whether or not I'm hot-code swapping too frequently, what are the negative signs to look for? What does the emulator actually do when swapping out code w.r.t potentially disruptive/costly operations like locking, GC, etc which could negatively impact scheduling, memory and responsiveness of the system?
>
> I'm trying to figure out when the best time to reload a dynamically generated module/group of modules is and how frequently I should allow reloading to occur.
>
> - Edmond -
>
>> Thanks in advance.
>>
>> - Edmond -
>>
>> [1] The more exotic use-cases for use/abuse of the constant pool and module reloading I'm looking into are to do with caching and incremental computation.
>>
>
>
> --
> Using Opera's mail client: http://www.opera.com/mail/
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Frequency of module reloading and the constant pool

Edmond Begumisa
In reply to this post by Edmond Begumisa

Thanks Lukas,

That is very very helpful information as it not only gives me knowledge of what to benchmark, it also gives me  some surprising insights on how to reorganise my (now obviously naive) code.

Apart from GC, is there anything else I should be concerned about like costly locks and memory barriers? For instance, is a fat mutex held for all schedulers when scanning all the process heaps?

- Edmond -



On Tuesday, 19 September 2017 at 19:40 Lukas Larsson <[hidden email]> wrote:

On Tue, Sep 19, 2017 at 11:13 AM, Adam Lindberg <[hidden email]> wrote:
Cost wise I'm not sure, but I would advice you to benchmark for your use case.

Every time you purge a module, each process heap is scanned for any sign of the old literals (aka constants) and if a literal is found a GC is done of that process which copies the literal onto its heap.

So the cost depends on the total heap size of all running processes and the number of processes that have a reference to those literals on their heaps at the moment of the purge.

Lukas


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Frequency of module reloading and the constant pool

Lukas Larsson-8

On Thu, Sep 21, 2017 at 12:23 AM, Edmond Begumisa <[hidden email]> wrote:

Apart from GC, is there anything else I should be concerned about like costly locks and memory barriers? For instance, is a fat mutex held for all schedulers when scanning all the process heaps?

 No, there are no locks taken. Exactly how it is done is described here: https://github.com/erlang/otp/blob/master/erts/emulator/internal_doc/CodeLoading.md

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Frequency of module reloading and the constant pool

Edmond Begumisa
On Thu, 21 Sep 2017 16:19:38 +1000, Lukas Larsson <[hidden email]> wrote:

>
> On Thu, Sep 21, 2017 at 12:23 AM, Edmond Begumisa  
> <[hidden email]> wrote:
>>
>> Apart from GC, is there anything else I should be concerned about like  
>> costly locks and memory >>barriers? For instance, is a fat mutex held  
>> for all schedulers when scanning all the process >>heaps?
>>
> No, there are no locks taken. Exactly how it is done is described here:  
> https://github.com/>erlang/otp/blob/master/erts/emulator/internal_doc/CodeLoading.md
>

Excellent resource! This was exactly what I needed.

Thanks again Lukas.

- Edmond -

--
Using Opera's mail client: http://www.opera.com/mail/
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions