How to get configuration data to a large number of threads?

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

How to get configuration data to a large number of threads?

Heinrich Venter-5
Hello all

I need a bit of advice.  I have a transaction based system that spawns a
thread to handle every incoming transaction.  Unfortunately there is
quite a large chunk of relatively static configuration information that
is needed by every thread.
With relatively static I mean that it is changeable through user
intervention at any time.

The question is, how do I get this information to every thread without
signifficantly slowing things down or using up all the available memory?

I am not sure, but I suspect passing it as a parameter will cause
thousands of copies to be made.  Is this correct?
I was thinking of making a globally registered ets table, but this will
cause copies to be made again once it is read from the table.
The last option might be to declare all the information in a hrl file,
but then changing the configuration would require recompilation.
As I understand, using a binary formated chunk of data will prevent
copying, but when converting back to a more usable format the data will
be copied again.

Any other ways of getting large chunks of configuration data quickly to
a large number of transient threads?

Help and advice appreciated.

-]-[einrich Venter



Look 4 Help - Click here for more info
http://asp1.rocketseed.com/RocketSeed/mail/433a32353a3733353731353a33303639373a2d323a313933

 
http://asp1.rocketseed.com/RocketSeed/mail/433a32353a3733353731353a33303639373a2d323a313836
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20041026/93871925/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: MTk2LjMwLjc5LjE1NQ==
Type: image/gif
Size: 19652 bytes
Desc: not available
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20041026/93871925/attachment.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: MTk2LjMwLjc5LjE1NQ==
Type: image/gif
Size: 556 bytes
Desc: not available
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20041026/93871925/attachment-0001.gif>

Reply | Threaded
Open this post in threaded view
|

How to get configuration data to a large number of threads?

Joe Williams-2

   The naive solution might be  to keep *all* the configuration data in
a single server.

   When one of  threads wants configuration data it  just does a normal
server RPC and the configuration data is returned to the client.

   If you are smart then only  the server would have a complete copy of
the configuration  data, the  individual processes would  only request
the  sub-set  of the  configuration  data  that  they actually  need
(assuming here that they did not need all the configuration data).

   Once the  client processes are done  with their data  the space needed
for the configuration data will be garbed away.

   It's probably  not a  good idea letting  the clients carry  around a
complete  copy of the  configuration data,  just in  case they  need a
small sub-set of this data for a small time.

   Try writing the code in a  lazy manner, so that the clients carry no
configuration data and  only request exactly what they  need when they
need it, and try  to reduce the amount of time that  they have to hold
copies of the sub-sets of the configuration data.

   If the configuration  data needs any complex processing  that can be
pre-computed then you can cache the necessary values in the server.

   This might help.  Without being  more explicit about your problem it
is difficult to give a better answer. Often the "pure Erlang" solution
(just use processes, no gets/puts ets/dets/mnesia) is the fastest way,
but in particular  cases ets/dets etc will be faster  - it all depends
upon your problem.

   Cheers

   /Joe

   BTW -  Erlang doesn't have threads  - it has  processes.  What's the
difference?  Formally, threads share  stuff - processes don't.

   Process provide much more protection than threads since they provide
you  with  error encapsulation  domains  -  Erlang  has "light  weight
processes" meaning that  creating an Erlang process is  a much quicker
operation that creating a thread in "Most other" (TM) languages.







On Tue, 26 Oct 2004, Heinrich Venter wrote:

> Hello all
>
> I need a bit of advice.  I have a transaction based system that spawns a
> thread to handle every incoming transaction.  Unfortunately there is
> quite a large chunk of relatively static configuration information that
> is needed by every thread.
> With relatively static I mean that it is changeable through user
> intervention at any time.
>
> The question is, how do I get this information to every thread without
> signifficantly slowing things down or using up all the available memory?
>
> I am not sure, but I suspect passing it as a parameter will cause
> thousands of copies to be made.  Is this correct?
> I was thinking of making a globally registered ets table, but this will
> cause copies to be made again once it is read from the table.
> The last option might be to declare all the information in a hrl file,
> but then changing the configuration would require recompilation.
> As I understand, using a binary formated chunk of data will prevent
> copying, but when converting back to a more usable format the data will
> be copied again.
>
> Any other ways of getting large chunks of configuration data quickly to
> a large number of transient threads?
>
> Help and advice appreciated.
>
> -]-[einrich Venter
>
>
>
> Look 4 Help - Click here for more info
> http://asp1.rocketseed.com/RocketSeed/mail/433a32353a3733353731353a33303639373a2d323a313933
>
>
> http://asp1.rocketseed.com/RocketSeed/mail/433a32353a3733353731353a33303639373a2d323a313836
>


Reply | Threaded
Open this post in threaded view
|

How to get configuration data to a large number of threads?

Hal Snyder-2
Joe Armstrong <joe> writes:

> The naive solution might be to keep *all* the configuration data in
> a single server.
>
> When one of threads wants configuration data it just does a normal
> server RPC and the configuration data is returned to the client.


Taking it to the next level, suppose things are distributed on several
nodes and you don't want a single point of failure.

Do you have any techniques or recommendations, besides the basic idea
of making duplicate config servers (which can get messy in
implementation)?


Reply | Threaded
Open this post in threaded view
|

How to get configuration data to a large number of threads?

Luke Gorrie-3
Hal Snyder <hal> writes:

> Do you have any techniques or recommendations, besides the basic idea
> of making duplicate config servers (which can get messy in
> implementation)?

Mnesia!




Reply | Threaded
Open this post in threaded view
|

How to get configuration data to a large number of threads?

Joe Williams-2
In reply to this post by Heinrich Venter-5

   How about some lateral thinking here:

> I have a transaction based system that spawns a
> thread to handle every incoming transaction.  Unfortunately there is
> quite a large chunk of relatively static configuration information that
> is needed by every thread....
> The question is, how do I get this information to every thread without
> significantly slowing things down or using up all the available memory?


   You don't - you move the functions to the data - not the data to the
functions :-)

   Assume the  configuration data is HUGE  (GBytes) - if  this case I'd
send the computations to  the configuration data. Exactly the opposite
of what you did :-)

   So you make a global server like this

   -module(config).

   start() ->
  AHugeDataStructure = read_initiaal_config_data(),
  loop(AHugeDataStructure).


   loop(AHugeDataStructure) ->
  receive
    {upgrade, AnotherHugeDataStructure} ->
  loop(AnotherHugeDataStruce);
    {compute, From, Fun} ->
  Val = (catch Fun(AHugeDataStructure)),
         From ! {self(), Val},
         loop(AHugeDataStructure)
  end.

... now about the clients ...

    Suppose the client wants a username and password

    Make a query

  Query = fun(Config) ->
    {get_from_config(username, Config),
     get_from_config(password, Config)
  end,

   Send it to the config process with a

  Config ! {compute, self(), Query}

  and wait for the reply


   Note that sending Funs in messages (locally) is a very lightweight
operation (as I have explained before :-)

   Now you possibly have a bottleneck in the server - how to solve this?

   Make several servers with identical copies of the configuration data.

   Remember - it might be cheaper to move the functions to the data
than moving the data to the functions.

   RPC style programming in "Other Languages" (TM) makes you believe
that there is only only way of doing RPCs (ie moving the data to the functions)

   Erlang offers freedom of choice.

   Cheers

/Joe








Reply | Threaded
Open this post in threaded view
|

How to get configuration data to a large number of threads?

Michał Ptaszek
Hi,

Why is not application:get_env/2 a good solution for huge configuration data
management?


thanks,
Eduardo Figoli
INSwitch Solutions


----- Original Message -----
From: "Joe Armstrong" <joe>
To: "Heinrich Venter" <heinrich>
Cc: <erlang-questions>
Sent: Wednesday, October 27, 2004 7:32 AM
Subject: Re: How to get configuration data to a large number of threads?


>
>    How about some lateral thinking here:
>
> > I have a transaction based system that spawns a
> > thread to handle every incoming transaction.  Unfortunately there is
> > quite a large chunk of relatively static configuration information that
> > is needed by every thread....
> > The question is, how do I get this information to every thread without
> > significantly slowing things down or using up all the available memory?
>
>
>    You don't - you move the functions to the data - not the data to the
> functions :-)
>
>    Assume the  configuration data is HUGE  (GBytes) - if  this case I'd
> send the computations to  the configuration data. Exactly the opposite
> of what you did :-)
>
>    So you make a global server like this
>
>    -module(config).
>
>    start() ->
>   AHugeDataStructure = read_initiaal_config_data(),
>   loop(AHugeDataStructure).
>
>
>    loop(AHugeDataStructure) ->
>   receive
>      {upgrade, AnotherHugeDataStructure} ->
>   loop(AnotherHugeDataStruce);
>      {compute, From, Fun} ->
>   Val = (catch Fun(AHugeDataStructure)),
>           From ! {self(), Val},
>           loop(AHugeDataStructure)
>   end.
>
> ... now about the clients ...
>
>     Suppose the client wants a username and password
>
>     Make a query
>
>   Query = fun(Config) ->
>      {get_from_config(username, Config),
>       get_from_config(password, Config)
>   end,
>
>    Send it to the config process with a
>
>   Config ! {compute, self(), Query}
>
>   and wait for the reply
>
>
>    Note that sending Funs in messages (locally) is a very lightweight
> operation (as I have explained before :-)
>
>    Now you possibly have a bottleneck in the server - how to solve this?
>
>    Make several servers with identical copies of the configuration data.
>
>    Remember - it might be cheaper to move the functions to the data
> than moving the data to the functions.
>
>    RPC style programming in "Other Languages" (TM) makes you believe
> that there is only only way of doing RPCs (ie moving the data to the
functions)

>
>    Erlang offers freedom of choice.
>
>    Cheers
>
> /Joe
>
>
>
>
>
>



Reply | Threaded
Open this post in threaded view
|

How to get configuration data to a large number of threads?

Michał Ptaszek
In reply to this post by Joe Williams-2
Hi,

Why is not application:get_env/2 a good solution for huge configuration data
management?


thanks,
Eduardo Figoli
INSwitch Solutions


----- Original Message -----
From: "Joe Armstrong" <joe>
To: "Heinrich Venter" <heinrich>
Cc: <erlang-questions>
Sent: Wednesday, October 27, 2004 7:32 AM
Subject: Re: How to get configuration data to a large number of threads?


>
>    How about some lateral thinking here:
>
> > I have a transaction based system that spawns a
> > thread to handle every incoming transaction.  Unfortunately there is
> > quite a large chunk of relatively static configuration information that
> > is needed by every thread....
> > The question is, how do I get this information to every thread without
> > significantly slowing things down or using up all the available memory?
>
>
>    You don't - you move the functions to the data - not the data to the
> functions :-)
>
>    Assume the  configuration data is HUGE  (GBytes) - if  this case I'd
> send the computations to  the configuration data. Exactly the opposite
> of what you did :-)
>
>    So you make a global server like this
>
>    -module(config).
>
>    start() ->
>   AHugeDataStructure = read_initiaal_config_data(),
>   loop(AHugeDataStructure).
>
>
>    loop(AHugeDataStructure) ->
>   receive
>      {upgrade, AnotherHugeDataStructure} ->
>   loop(AnotherHugeDataStruce);
>      {compute, From, Fun} ->
>   Val = (catch Fun(AHugeDataStructure)),
>           From ! {self(), Val},
>           loop(AHugeDataStructure)
>   end.
>
> ... now about the clients ...
>
>     Suppose the client wants a username and password
>
>     Make a query
>
>   Query = fun(Config) ->
>      {get_from_config(username, Config),
>       get_from_config(password, Config)
>   end,
>
>    Send it to the config process with a
>
>   Config ! {compute, self(), Query}
>
>   and wait for the reply
>
>
>    Note that sending Funs in messages (locally) is a very lightweight
> operation (as I have explained before :-)
>
>    Now you possibly have a bottleneck in the server - how to solve this?
>
>    Make several servers with identical copies of the configuration data.
>
>    Remember - it might be cheaper to move the functions to the data
> than moving the data to the functions.
>
>    RPC style programming in "Other Languages" (TM) makes you believe
> that there is only only way of doing RPCs (ie moving the data to the
functions)

>
>    Erlang offers freedom of choice.
>
>    Cheers
>
> /Joe
>
>
>
>
>
>