Private ets table vs. Process directory

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Private ets table vs. Process directory

Charles Hixson-2
When should a private ets table be preferred over the process directory?

To give some context, I'm expecting to has nearly as many processes as I
can run, and that each one will need internal mutable state.  Also, that
the mutable state will be complex (partially because of the limited
number of processes), so passing the state as function parameters would
entail huge amounts of copying.  (Essentially I'd be modifying nodes
deep within trees.)

Mutable state would allow me to avoid the copying, and the state is not
exported from the process.  I'm concerned that a huge number of private
ets tables would use excessive memory, decreasing the number of
processes I could use...but all the references keep saying not to use
the process directory.

I'm still designing things now, so this is the ideal time to decide.  An
alternative is that I could use a single public ets table, with each
process only accessing its own data, but I suspect that might involve a
lot of locking overhead, even though in principle nothing should need to
be locked.

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Private ets table vs. Process directory

zxq9-2
On 2018年2月7日水曜日 8時56分01秒 JST Charles Hixson wrote:
> ...so passing the state as function parameters would
> entail huge amounts of copying.  (Essentially I'd be modifying nodes
> deep within trees.)
>
> Mutable state would allow me to avoid the copying, and the state is not
> exported from the process...

You seem to be confused a bit about the nature of mutability. If I set a variable X and in my service loop alter X, the next time the service loop recurses (loops) X will be a different value -- it will have mutated, but within the context of a single call of the service loop function the thing labelled X at the time of the function call will be immutable.

-module(simple).
-export([start/1]).

start(X) ->
  spawn(fun() -> loop(X) end).

loop(X) ->
  ok = io:format("X is ~p~n", [X]),
  receive
    {add, Y} ->
      NewX = X + Y,
      loop(NewX);
    {sub, Y} ->
      NewX = X - Y,
      loop(NewX);
    stop ->
      ok = io:format("Bye!~n"),
      exit(normal);
    Unexpected ->
      ok = io:format("I don't understand ~tp~n", [Unexpected]),
      loop(X)
  end.


1> c(simple).
{ok,simple}
2> P = simple:start(10).
X is 10
<0.72.0>
3> P ! {add, 15}.
X is 25
{add,15}
4> P ! {sub, 100}.                                                                                                                                                                                                                                                          
X is -75                                                                                                                                                                                                                                                                    
{sub,100}


That is all there is to state maintenance, and this is how gen_servers work. This is also the form that has the least mysterious memory management model in the normal case, and the form that gives you all that nifty memory isolation and fault tolerance Erlang is famous for. Note that X is *not* copied every time we enter loop/1. If we send a message containing X to another process, though, *then* X is copied into the context of the process receiving that message.

It doesn't matter at all what sort of a structure X is. Here it is a number, but it could be anything. Gigantic tuples chock full of maps and gb_trees and other process references and lists of things and queues and whatnot are the norm -- and none of this causes trouble in the normal case.

As for mucking around in deep tree structures, altering nodes in trees does not necessarily entail making a copy of the whole tree. To you as a programmer there are two versions of the data which are effectively distinct, but that does not necessarily mean that they are two complete versions of the data in memory. The nature of copying (or whether copying happens at all under the hood) and how fast things can be garbage collected has to do with the nature of the task and what kind of data structures you are using. Because of immutability you *actually* get to share more data in the underlying implementation than otherwise.

Fred provided a great explanation a while back here:
http://erlang.org/pipermail/erlang-questions/2015-December/087040.html

The general approach to performance issues -- whether memory, I/O bottlenecks, messaging bottlenecks, or raw thunk time -- is to start out writing your processes in the vanilla way using state variables in a loop and only stepping away from that when some extreme deficiency is demonstrated. If you are going to be spawning a ton of processes at once to do things then you've really got no way of knowing what is going to break first until you actually have some working code and can see it break for yourself. People get themselves into trouble with the process dictionary, ETS, NIFs, etc. all the time because the use cases often do not warrant the use of these techniques.

So keep it simple. Write an example of what you want to do. Try it out. You might wind up just saturating your processor or memory bus way before you hit an actual space problem. If something breaks try to measure why -- but right now without telling anyone the kind of data you're dealing with or what kinds of operations you're doing or any example code that is known to break in a certain way at a certain scale we can't really give you much helpful advice.

-Craig
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Private ets table vs. Process directory

Charles Hixson-2
That works fine in the simple case, but I'm contemplating repeatedly adjusting weights deep within a nested data structure. Your approach would result in creating an altered copy of the entire structure for each recursion.  This is probably only about 1KB or so of information, so doing this a few times isn't a problem, but doing it millions of time quickly becomes a problem.

This can be addressed by either ets or the process directory, and those allow the internal structure to be safely modified.  In the process directory it's safe because the information is never exported from the process (except for i/o, which must be special cased).  Similarly a private ets can handle it without problems. And so can a global ets, as then a unique process specific id (NOT pid, as this needs to survive restarts) can be used as a part of the key.  So those three methods would work.  The question in my mind is how to predict the tradeoffs as it scales up.  I suspect that the process directory would use the least memory, though possibly it would be the global ets table.  A private ets table seems the most natural approach, but it looks, to my naive eyes, as if it would scale poorly WRT memory use.

What I'd really like is to use a Mnesia system which kept a cache of active entries, but didn't require everything to be rolled in from disk.  AFAIKT, however, my choices with a Mnesia table are to keep everything in memory or to keep everything rolled out to disk.

I also haven't been able to determine whether processes that are waiting to receive a message can be rolled out to inactive memory.  There are some indications ("use enough processes, but not too many") that they can't.  This means that I need to adapt my memory use to the systems that are being run on rather carefully.  If background processes keep activating every live process to check it's status I could easily end up with severe thrashing.  And *THAT* will affect the design.  If I need to hand manage the caching, then I loose a lot of the benefits that I'm hoping to get from Erlang.

The basic design calls for a huge number of "processes" to be doing n x m communication, and the simple design calls for each "process" to be able to send messages to each other process, though only a subset of the messages would be actually sent.  My first sketch of a design called for each "process" to be mapped to a separate Erlang process, but this doesn't work, because Erlang doesn't like to have that many processes.  Even this simple design, however, required to figure for allowing 1000 inputs and 1000 outputs to each "process", and probably well over 100,000 "processes".  Most of them would be idle most of the time, but all would need to be "activatable" when messaged, and all would need to become dormant when just waiting for a message.  The idea is not a neural net, but it has certain similarities.

Now if I could actually have one process per "process", then your proposal, which I recognize as the normal Erlang approach, would make sense, but that isn't going to work.  This could be done in that case by having lots of variables, so that there wouldn't be the need to have any modifications of deeply nested items, so not much would need to be copied.

As for KISS, that's a great approach, but it doesn't reveal scaling problems.  When one is adapting an approach one should always KISS, but when designing which approach to try it's important to pick one that will work when the system approaches its initial design goal.


On 02/07/2018 03:45 PM, [hidden email] wrote:
On 2018年2月7日水曜日 8時56分01秒 JST Charles Hixson wrote:
...so passing the state as function parameters would
entail huge amounts of copying.  (Essentially I'd be modifying nodes
deep within trees.)

Mutable state would allow me to avoid the copying, and the state is not
exported from the process...
You seem to be confused a bit about the nature of mutability. If I set a variable X and in my service loop alter X, the next time the service loop recurses (loops) X will be a different value -- it will have mutated, but within the context of a single call of the service loop function the thing labelled X at the time of the function call will be immutable.

-module(simple).
-export([start/1]).

start(X) ->
   spawn(fun() -> loop(X) end).

loop(X) ->
   ok = io:format("X is ~p~n", [X]),
   receive
     {add, Y} ->
       NewX = X + Y,
       loop(NewX);
     {sub, Y} ->
       NewX = X - Y,
       loop(NewX);
     stop ->
       ok = io:format("Bye!~n"),
       exit(normal);
     Unexpected ->
       ok = io:format("I don't understand ~tp~n", [Unexpected]),
       loop(X)
   end.


1> c(simple).
{ok,simple}
2> P = simple:start(10).
X is 10
<0.72.0>
3> P ! {add, 15}.
X is 25
{add,15}
4> P ! {sub, 100}.
X is -75
{sub,100}


That is all there is to state maintenance, and this is how gen_servers work. This is also the form that has the least mysterious memory management model in the normal case, and the form that gives you all that nifty memory isolation and fault tolerance Erlang is famous for. Note that X is *not* copied every time we enter loop/1. If we send a message containing X to another process, though, *then* X is copied into the context of the process receiving that message.

It doesn't matter at all what sort of a structure X is. Here it is a number, but it could be anything. Gigantic tuples chock full of maps and gb_trees and other process references and lists of things and queues and whatnot are the norm -- and none of this causes trouble in the normal case.

As for mucking around in deep tree structures, altering nodes in trees does not necessarily entail making a copy of the whole tree. To you as a programmer there are two versions of the data which are effectively distinct, but that does not necessarily mean that they are two complete versions of the data in memory. The nature of copying (or whether copying happens at all under the hood) and how fast things can be garbage collected has to do with the nature of the task and what kind of data structures you are using. Because of immutability you *actually* get to share more data in the underlying implementation than otherwise.

Fred provided a great explanation a while back here:
http://erlang.org/pipermail/erlang-questions/2015-December/087040.html

The general approach to performance issues -- whether memory, I/O bottlenecks, messaging bottlenecks, or raw thunk time -- is to start out writing your processes in the vanilla way using state variables in a loop and only stepping away from that when some extreme deficiency is demonstrated. If you are going to be spawning a ton of processes at once to do things then you've really got no way of knowing what is going to break first until you actually have some working code and can see it break for yourself. People get themselves into trouble with the process dictionary, ETS, NIFs, etc. all the time because the use cases often do not warrant the use of these techniques.

So keep it simple. Write an example of what you want to do. Try it out. You might wind up just saturating your processor or memory bus way before you hit an actual space problem. If something breaks try to measure why -- but right now without telling anyone the kind of data you're dealing with or what kinds of operations you're doing or any example code that is known to break in a certain way at a certain scale we can't really give you much helpful advice.

-Craig
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Private ets table vs. Process directory

Joe Armstrong-2
In reply to this post by Charles Hixson-2
In order to even think about your question I'd need certain data -
words like "huge" as in "huge amounts of copying" and "limited numbers
of processes"
etc. do not convey much meaning.

Huge means different things to different people - to some people Huge
means Gbytes
(I talked the other day to somebody who used the word Huge - and I
said "how big"
he said tens of PetaBytes)

To me huge means a data structure that is larger than the RAM on my machine
(which is 16GB) - so not only do you have to say what you meant by huge but also
how your numbers relate to your machine(s).

Also how long do you have to do what? - Handling huge amounts of data
is easy if you have a big enough disks and enough time - you also need
to say (roughly) how long you have to do what (are we talking seconds,
milliseconds,
hours, days???)

The more numbers you add to questions like this the better answers
you'll get :-)

Cheers

/Joe



On Wed, Feb 7, 2018 at 5:56 PM, Charles Hixson
<[hidden email]> wrote:

> When should a private ets table be preferred over the process directory?
>
> To give some context, I'm expecting to has nearly as many processes as I can
> run, and that each one will need internal mutable state.  Also, that the
> mutable state will be complex (partially because of the limited number of
> processes), so passing the state as function parameters would entail huge
> amounts of copying.  (Essentially I'd be modifying nodes deep within trees.)
>
> Mutable state would allow me to avoid the copying, and the state is not
> exported from the process.  I'm concerned that a huge number of private ets
> tables would use excessive memory, decreasing the number of processes I
> could use...but all the references keep saying not to use the process
> directory.
>
> I'm still designing things now, so this is the ideal time to decide.  An
> alternative is that I could use a single public ets table, with each process
> only accessing its own data, but I suspect that might involve a lot of
> locking overhead, even though in principle nothing should need to be locked.
>
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Private ets table vs. Process directory

Joe Armstrong-2
In reply to this post by Charles Hixson-2
On Thu, Feb 8, 2018 at 7:28 AM, Charles Hixson
<[hidden email]> wrote:
> That works fine in the simple case, but I'm contemplating repeatedly
> adjusting weights deep within a nested data structure. Your approach would
> result in creating an altered copy of the entire structure for each
> recursion.  This is probably only about 1KB or so of information, so doing
> this a few times isn't a problem, but doing it millions of time quickly
> becomes a problem.

1 KB is tiny in my mind - If you have 1 GB of RAM you can have a
million of these things

If it's a tree your modifying you don't copy all the nodes only the
nodes from the
top of the tree to the modified node - all nodes that don't change are shared.

Since each process has its own stack and heap data is well localized -
which improves
cache performance (so even though it costs to copy data between
processes, you might win in
cache performance) - actually this is in theory only - you'd have to
measure to see if this
is the case.


>
> This can be addressed by either ets or the process directory, and those
> allow the internal structure to be safely modified.  In the process
> directory it's safe because the information is never exported from the
> process (except for i/o, which must be special cased).  Similarly a private
> ets can handle it without problems. And so can a global ets, as then a
> unique process specific id (NOT pid, as this needs to survive restarts) can
> be used as a part of the key.  So those three methods would work.  The
> question in my mind is how to predict the tradeoffs as it scales up.  I
> suspect that the process directory would use the least memory, though
> possibly it would be the global ets table.  A private ets table seems the
> most natural approach, but it looks, to my naive eyes, as if it would scale
> poorly WRT memory use.

You have to write the code and measure. Actually the code might be
very different
you can do fancy things in ets tables that you can't do in the process
dictionary.

>
> What I'd really like is to use a Mnesia system which kept a cache of active
> entries, but didn't require everything to be rolled in from disk.  AFAIKT,
> however, my choices with a Mnesia table are to keep everything in memory or
> to keep everything rolled out to disk.

Or RAM replicate over two nodes :-)

>
> I also haven't been able to determine whether processes that are waiting to
> receive a message can be rolled out to inactive memory.  There are some
> indications ("use enough processes, but not too many") that they can't.
> This means that I need to adapt my memory use to the systems that are being
> run on rather carefully.  If background processes keep activating every live
> process to check it's status I could easily end up with severe thrashing.
> And *THAT* will affect the design.  If I need to hand manage the caching,
> then I loose a lot of the benefits that I'm hoping to get from Erlang.

Again "it all depends" - how many processes have you, how big are they, how
much main memory have you, what requirements have you for dynamic code change
latency, fault tolerance etc.

Even with Erlang you can get dramatic performance differences if you
want 'hot standbys' or 'no error recovery at all' - provisioning for
hot standby costs.

>
> The basic design calls for a huge number of "processes" to be doing n x m
> communication, and the simple design calls for each "process" to be able to
> send messages to each other process, though only a subset of the messages
> would be actually sent.  My first sketch of a design called for each
> "process" to be mapped to a separate Erlang process, but this doesn't work,
> because Erlang doesn't like to have that many processes.  Even this simple
> design, however, required to figure for allowing 1000 inputs and 1000
> outputs to each "process", and probably well over 100,000 "processes".  Most
> of them would be idle most of the time, but all would need to be
> "activatable" when messaged, and all would need to become dormant when just
> waiting for a message.  The idea is not a neural net, but it has certain
> similarities.

Some numbers at last :-)

Actually I think Erlang does like to have lots of processes. At say
10KB per process
you'd get 100 processes/MB or 100K processes per GB - if you have say
8-16GB then
there would be 80-160K per process and you mentioned stat data of 1K earlier

I know messaging servers are handling "a few million sessions" per node
(which is a few million processes) - Erlang was designed to handle millions of
simultaneous connections where most of them are idle most of the time.



>
> Now if I could actually have one process per "process", then your proposal,
> which I recognize as the normal Erlang approach, would make sense, but that
> isn't going to work.

How do you know? - It might it might not - you have to do the experiment first.

> This could be done in that case by having lots of
> variables, so that there wouldn't be the need to have any modifications of
> deeply nested items, so not much would need to be copied.
>
> As for KISS, that's a great approach, but it doesn't reveal scaling
> problems.  When one is adapting an approach one should always KISS, but when
> designing which approach to try it's important to pick one that will work
> when the system approaches its initial design goal.

If all your processes are isolated and share no memory and only interact
through message passing then if you *do* run out of steam then you can
also go distributed - the whole idea of the design is to scale up by adding
more nodes when you run out of power.

Making something scalable/fault-tolerant/secure will mean that the
individual nodes
are less efficient than a non-scalable/error-prone/insecure system

BUT you win it all back if you discover one day that you *need* to scale-up

In all my time programming I'd say the most difficult thing there
is, is to predict performance *before* you've written the code.

Why is this?

Basically there is no algebra for non-functional behaviour.

If X takes time T1 and Y takes time T2 (both on empty machines)
how long does it take to do X and Y (is it T1 + T2) ????

Answer: Nobody knows - and what does "and" mean here???
try it and see.

Cheers

/Joe



>
>
> On 02/07/2018 03:45 PM, [hidden email] wrote:
>
> On 2018年2月7日水曜日 8時56分01秒 JST Charles Hixson wrote:
>
> ...so passing the state as function parameters would
> entail huge amounts of copying.  (Essentially I'd be modifying nodes
> deep within trees.)
>
> Mutable state would allow me to avoid the copying, and the state is not
> exported from the process...
>
> You seem to be confused a bit about the nature of mutability. If I set a
> variable X and in my service loop alter X, the next time the service loop
> recurses (loops) X will be a different value -- it will have mutated, but
> within the context of a single call of the service loop function the thing
> labelled X at the time of the function call will be immutable.
>
> -module(simple).
> -export([start/1]).
>
> start(X) ->
>    spawn(fun() -> loop(X) end).
>
> loop(X) ->
>    ok = io:format("X is ~p~n", [X]),
>    receive
>      {add, Y} ->
>        NewX = X + Y,
>        loop(NewX);
>      {sub, Y} ->
>        NewX = X - Y,
>        loop(NewX);
>      stop ->
>        ok = io:format("Bye!~n"),
>        exit(normal);
>      Unexpected ->
>        ok = io:format("I don't understand ~tp~n", [Unexpected]),
>        loop(X)
>    end.
>
>
> 1> c(simple).
> {ok,simple}
> 2> P = simple:start(10).
> X is 10
> <0.72.0>
> 3> P ! {add, 15}.
> X is 25
> {add,15}
> 4> P ! {sub, 100}.
> X is -75
> {sub,100}
>
>
> That is all there is to state maintenance, and this is how gen_servers work.
> This is also the form that has the least mysterious memory management model
> in the normal case, and the form that gives you all that nifty memory
> isolation and fault tolerance Erlang is famous for. Note that X is *not*
> copied every time we enter loop/1. If we send a message containing X to
> another process, though, *then* X is copied into the context of the process
> receiving that message.
>
> It doesn't matter at all what sort of a structure X is. Here it is a number,
> but it could be anything. Gigantic tuples chock full of maps and gb_trees
> and other process references and lists of things and queues and whatnot are
> the norm -- and none of this causes trouble in the normal case.
>
> As for mucking around in deep tree structures, altering nodes in trees does
> not necessarily entail making a copy of the whole tree. To you as a
> programmer there are two versions of the data which are effectively
> distinct, but that does not necessarily mean that they are two complete
> versions of the data in memory. The nature of copying (or whether copying
> happens at all under the hood) and how fast things can be garbage collected
> has to do with the nature of the task and what kind of data structures you
> are using. Because of immutability you *actually* get to share more data in
> the underlying implementation than otherwise.
>
> Fred provided a great explanation a while back here:
> http://erlang.org/pipermail/erlang-questions/2015-December/087040.html
>
> The general approach to performance issues -- whether memory, I/O
> bottlenecks, messaging bottlenecks, or raw thunk time -- is to start out
> writing your processes in the vanilla way using state variables in a loop
> and only stepping away from that when some extreme deficiency is
> demonstrated. If you are going to be spawning a ton of processes at once to
> do things then you've really got no way of knowing what is going to break
> first until you actually have some working code and can see it break for
> yourself. People get themselves into trouble with the process dictionary,
> ETS, NIFs, etc. all the time because the use cases often do not warrant the
> use of these techniques.
>
> So keep it simple. Write an example of what you want to do. Try it out. You
> might wind up just saturating your processor or memory bus way before you
> hit an actual space problem. If something breaks try to measure why -- but
> right now without telling anyone the kind of data you're dealing with or
> what kinds of operations you're doing or any example code that is known to
> break in a certain way at a certain scale we can't really give you much
> helpful advice.
>
> -Craig
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
>
>
>
>
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
>
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Private ets table vs. Process directory

Charles Hixson-2
In reply to this post by Joe Armstrong-2
The problem with quantifying those numbers is I've got several different plausible designs for the system, and they have different values for those numbers.

E.g., one design called for one process/cpu processor.  In that design each process would need an ets table and a mnesia table.  The mnesia table would be disk-only.  The ets table would hold perhaps 100,000 entries, each of which would maintain a time stamp for time of last access.  When the table started getting full, stale entries would need to be rolled out to the database and purged. 
This design uses a lot less RAM, and an extremely smaller number of processes.

Now the tables would be keyed by a programatically generated key value to allow unique items to be referred to when they are rolled out, so that it's possible to roll them back in. 
In this design there would be perhaps (at a wild guess!!) one i/o operation for every sqrt (# of CPUs * # of entries/table) function calls.  But they would tend to come in bursts, so i/o would definitely slow things down considerably.

Well, that design wasn't optimized for Erlang.  I've been contemplating variations of it over many different languages. 

Now if I can have one process/entry, and if dormant processes (waiting for a receive) can sleep in virtual memory, then I will need a few million processes, but I will be able to let the system manage the activation/sleep cycle of the processes.  In that case each external signal is likely to induce around 1,000 to 100,000 activations in a chain before relaxing into a settled state.  Each activation will likely only cause about 500 bytes of data to be copied (another wild guess, with part of the uncertainty being how many internal pointers Erlang will need to adjust).  But in this case the data can be passed on the stack, and tail calls can be used.

My problem has been that when I searched for limits on the number of Erlang processes I got:
The maximum number of simultaneously alive Erlang processes is by default 32,768. This limit can be configured at startup. For more information, see the +P command-line flag in the erl(1) manual page in ERTS.
and:
The best thing to do is create a lagom number of processes. Not too many, not too few.
and:
The actual scalability achieved depends on your problem, on your design choices, and on the underlying execution framework.

Erlang has some things going for it, and while synthetic benchmarks have been produced, e.g. that show linear scalability within one node up to some 30-40 cores, and linear scalability in an Erlang cluster up to 100 nodes and a total of 1200 cores, the scalability story in Erlang is not so much about that, as it is about achieving real-world scalability in systems that actually do something useful.

Since I have a single system, this left me with the impression that I shouldn't use too many processes, and the best guess of the system at a reasonable maximum was a bit under 32,768.  It *was* clear that I could raise that limit, but raising it by more than an order of magnitude, while allowed, appeared probably unwise.

It appears now that this was a mistaken assumption, but I still don't see why I should have guessed differently.
 
On 02/08/2018 12:05 PM, Joe Armstrong wrote:
In order to even think about your question I'd need certain data -
words like "huge" as in "huge amounts of copying" and "limited numbers
of processes"
etc. do not convey much meaning.

Huge means different things to different people - to some people Huge
means Gbytes
(I talked the other day to somebody who used the word Huge - and I
said "how big"
he said tens of PetaBytes)

To me huge means a data structure that is larger than the RAM on my machine
(which is 16GB) - so not only do you have to say what you meant by huge but also
how your numbers relate to your machine(s).

Also how long do you have to do what? - Handling huge amounts of data
is easy if you have a big enough disks and enough time - you also need
to say (roughly) how long you have to do what (are we talking seconds,
milliseconds,
hours, days???)

The more numbers you add to questions like this the better answers
you'll get :-)

Cheers

/Joe



On Wed, Feb 7, 2018 at 5:56 PM, Charles Hixson
[hidden email] wrote:
When should a private ets table be preferred over the process directory?

To give some context, I'm expecting to has nearly as many processes as I can
run, and that each one will need internal mutable state.  Also, that the
mutable state will be complex (partially because of the limited number of
processes), so passing the state as function parameters would entail huge
amounts of copying.  (Essentially I'd be modifying nodes deep within trees.)

Mutable state would allow me to avoid the copying, and the state is not
exported from the process.  I'm concerned that a huge number of private ets
tables would use excessive memory, decreasing the number of processes I
could use...but all the references keep saying not to use the process
directory.

I'm still designing things now, so this is the ideal time to decide.  An
alternative is that I could use a single public ets table, with each process
only accessing its own data, but I suspect that might involve a lot of
locking overhead, even though in principle nothing should need to be locked.

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

    


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Private ets table vs. Process directory

Roger Lipscombe-2
On 9 February 2018 at 02:17, Charles Hixson <[hidden email]> wrote:
> My problem has been that when I searched for limits on the number of Erlang
> processes I got:
> The maximum number of simultaneously alive Erlang processes is by default
> 32,768. This limit can be configured at startup. For more information, see
> the +P command-line flag in the erl(1) manual page in ERTS.

The documentation at
http://erlang.org/doc/efficiency_guide/advanced.html is wrong. The
default (on x86_64, anyway) is 262,144 processes. See
http://erlang.org/doc/man/erl.html#max_processes.

Data point: I just took a look at our production servers. We've got
just under 40K processes per node. You could almost certainly go
higher. We're constrained by other factors, so we scale horizontally.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions