sharing config via a dynamically compiled module?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

sharing config via a dynamically compiled module?

Benoit Chesneau-2
I'm temped to compile dynamically a module for some highly demanded resources
to share their config instead of using ETS for it. But I'm wondering if's not
to hackish. (I'm also tempted by just using something more pure like sharing it
via message passing).

Are other people do such thing? I can see the following pro/cons:

pro:

    * fast and constant access, it is just a call to a module function and some literals.

cons:

    * maybe too much hackish?
    * What happen if the resource is deleted when someone is using it? Will the
      module be really deleted or will it leaks?


What do you think about it? Any feedback is welcome :)

- benoît
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: sharing config via a dynamically compiled module?

Roger Lipscombe-2
On 25 October 2017 at 13:29, Benoit Chesneau <[hidden email]> wrote:
> I'm temped to compile dynamically a module for some highly demanded resources
> to share their config instead of using ETS for it. But I'm wondering if's not
> to hackish. (I'm also tempted by just using something more pure like sharing it
> via message passing).
>
> Are other people do such thing?

Yes: https://github.com/mochi/mochiweb/blob/master/src/mochiglobal.erl
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: sharing config via a dynamically compiled module?

Benoit Chesneau-2


On 25 Oct 2017, at 15:04, Roger Lipscombe <[hidden email]> wrote:

On 25 October 2017 at 13:29, Benoit Chesneau <[hidden email]> wrote:
I'm temped to compile dynamically a module for some highly demanded resources
to share their config instead of using ETS for it. But I'm wondering if's not
to hackish. (I'm also tempted by just using something more pure like sharing it
via message passing).

Are other people do such thing?

Yes: https://github.com/mochi/mochiweb/blob/master/src/mochiglobal.erl


Indeed :) I’ve also somewhat simpler code:

-include_lib("syntax_tools/include/merl.hrl").

%% @doc Utility that converts a given property list into a module that provides
%% constant time access to the various key/value pairs.
%%
%% Example:
%%
%% load_config(store_config, [{backends, [{rocksdb_ram, barrel_rocksdb},
%% {rocksdb_disk, barrel_rocksdb}]},
%% {data_dir, "/path/to_datadir"}]).
%%
%% creates the module store_config:
%% store_config:backends(). => [{rocksdb_ram,barrel_rocksdb},{rocksdb_disk,barrel_rocksdb}]
%% store_config:data_dir => "/path/to_datadir"
%%
-spec load_config(atom(), [{atom(), any()}]) -> ok.
load_config(Resource, Config) when is_atom(Resource), is_list(Config) ->
Module = ?Q("-module(" ++ atom_to_list(Resource) ++ ")."),
Functions = lists:foldl(fun({K, V}, Acc) ->
[make_function(K,V) | Acc]
end,
[], Config),
Exported = [?Q("-export([" ++ atom_to_list(K) ++ "/0]).") || {K, _V} <- Config],
Forms = lists:flatten([Module, Exported, Functions]),
merl:compile_and_load(Forms, [verbose]),
ok.

make_function(K, V) ->
Cs = [?Q("() -> _@V@")],
F = erl_syntax:function(merl:term(K), Cs),
?Q("'@_F'() -> [].").

%% @doc unload a config module loaded with the `load_config/2' function.
-spec unload_config(atom()) -> true | false.
unload_config(Resource) ->
_ = code:purge(Resource),
code:delete(Resource).

which is doing something similar but create a function for each keys in the proplists. But I’m  wondering if there are some cons to it apart the fact it is designed for case where little changes happen to the conf . It seems lot of people are using ets to share a config generally. Maybe becauseit feels a little hackish ?

- benoit

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: sharing config via a dynamically compiled module?

Dmitry Kolesnikov-2
Hello,

On 25 Oct 2017, at 16.27, Benoit Chesneau <[hidden email]> wrote:

 It seems lot of people are using ets to share a config generally. Maybe becauseit feels a little hackish ?

The usage of code-as-a-config is “valid” approach. I’ve used that for some project. However, the reloading of config is something to consider. You need to use code:purge(…) and take all pain associated with it. The code-as-a-config works as build time config but runtime config is easy with build-in application:get_env… 
 

- Dmitry


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: sharing config via a dynamically compiled module?

Erdem Aksu
Hello Benoit,

I am using following application:

It keeps object code for the module in files for later recovery. It loads the latest object code from files on application start.
One can share the object code(binary) in-between nodes and each node can load the module. (code:load_binary/3)

There is no documentation, but usage is like following if you are interested:
application:start(gb_reg),
{ok, Module} = gb_reg:new(Name), %% See new/2 that gets an Init list of Key/Values.
gb_reg:insert(Module, Key, Value),
gb_reg:insert(Module, [{K1, V1},{K2, V2}]), %% File dumped once for all entries.
Module:lookup(Key),
gb_reg:delete(Module, Key),
Module:entries(), %% Same as gb_reg:all(Module)
gb_reg:add_keys(Module, [K]), %% Two entries K -> Ref and Ref -> K are created where Ref is auto incremented integer that is encoded as unsigned integer.
gb_reg:purge(Module).

Cons:
  Slow insertion. Keeping configurations is a good use case for this if they are not updated frequently.
Pros:
  gen_server serialise addition/deletion of new entries thus the file access and code reload will be sequential.

I use this in a wide column database implementation, where I lookup Column Id <-> Column Name at writes and reads to pack/unpack the data. And I use a similar approach for keeping key (hash) ring for fast lookup on db shards.

Regards,
Erdem

On Wed, Oct 25, 2017 at 3:55 PM, Dmitry Kolesnikov <[hidden email]> wrote:
Hello,

On 25 Oct 2017, at 16.27, Benoit Chesneau <[hidden email]> wrote:

 It seems lot of people are using ets to share a config generally. Maybe becauseit feels a little hackish ?

The usage of code-as-a-config is “valid” approach. I’ve used that for some project. However, the reloading of config is something to consider. You need to use code:purge(…) and take all pain associated with it. The code-as-a-config works as build time config but runtime config is easy with build-in application:get_env… 
 

- Dmitry


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: sharing config via a dynamically compiled module?

Ilya Khaprov
In reply to this post by Benoit Chesneau-2

Hi,

 

I did this, feels good.

If this is for config,  I would say use config_change and regen.

Also code:delete/1 and code:purge/1.

 

Ilya

 

From: [hidden email]
Sent: Wednesday, October 25, 2017 03:29 PM
To: [hidden email]
Subject: [erlang-questions] sharing config via a dynamically compiled module?

 

I'm temped to compile dynamically a module for some highly demanded resources
to share their config instead of using ETS for it. But I'm wondering if's not
to hackish. (I'm also tempted by just using something more pure like sharing it
via message passing).

Are other people do such thing? I can see the following pro/cons:

pro:

    * fast and constant access, it is just a call to a module function and some literals.

cons:

    * maybe too much hackish?
    * What happen if the resource is deleted when someone is using it? Will the
      module be really deleted or will it leaks?


What do you think about it? Any feedback is welcome :)

- benoît
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

 


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: sharing config via a dynamically compiled module?

Max Lapshin-2
Take a look at thread nearby.

> All literals belong to a loaded module instance. When that (old) module instance is purged, all process heaps are scanned and heaps containing those literals will do a special garbage collection where the literals are copied.

Each config change may lead to rescanning heaps of all processes.


Frankly speaking global ets is much easier =)


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: sharing config via a dynamically compiled module?

Ilya Khaprov

Sure, then the question is how often config change happens. I doubt that often.

But having careful measurements always helps of course.

 

From: [hidden email]
Sent: Saturday, October 28, 2017 09:38 AM
To: [hidden email]
Cc: [hidden email]; [hidden email]
Subject: Re: [erlang-questions] sharing config via a dynamically compiled module?

 

Take a look at thread nearby.

 

> All literals belong to a loaded module instance. When that (old) module instance is purged, all process heaps are scanned and heaps containing those literals will do a special garbage collection where the literals are copied.

 

Each config change may lead to rescanning heaps of all processes.

 

 

Frankly speaking global ets is much easier =)

 

 


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: sharing config via a dynamically compiled module?

Benoit Chesneau-2
In reply to this post by Max Lapshin-2


> On 28 Oct 2017, at 08:38, Max Lapshin <[hidden email]> wrote:
>
> Take a look at thread nearby.
>
> > All literals belong to a loaded module instance. When that (old) module instance is purged, all process heaps are scanned and heaps containing those literals will do a special garbage collection where the literals are copied.
>
> Each config change may lead to rescanning heaps of all processes.
>
>
> Frankly speaking global ets is much easier =)
>


Right. One another advantage to ETS i that you can store any terms even refs where it's not possible . At the cost of increasing the number of ETS tables used though...

- benoît
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: sharing config via a dynamically compiled module?

ILYA Khlopotov-2
config as module approach is also used in CouchDB plugin system here
https://github.com/apache/couchdb/blob/master/src/couch_epi/src/couch_epi_data_gen.erl

On October 28, 2017 12:39:07 AM PDT, Benoit Chesneau <[hidden email]> wrote:


On 28 Oct 2017, at 08:38, Max Lapshin <[hidden email]> wrote:

Take a look at thread nearby.

All literals belong to a loaded module instance. When that (old) module instance is purged, all process heaps are scanned and heaps containing those literals will do a special garbage collection where the literals are copied.

Each config change may lead to rescanning heaps of all processes.


Frankly speaking global ets is much easier =)



Right. One another advantage to ETS i that you can store any terms even refs where it's not possible . At the cost of increasing the number of ETS tables used though...< br />
- benoît


erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions