Erlang Process Limits

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Erlang Process Limits

Bruce Fitzsimons-2
Hello again,

Is the maximum number of processes on an Erlang node a hard limit, or is
there the potential for this to be increased (beyond 32767)?

I am aware that requiring such a huge number may be an indicator that I
should rethink my design, but it just maps cleanly to having 1 process for
every call, with a parent process to manage them(2+ calls per real call).
However this limits me to only 10000 simultaneous calls, which isn't that
much for my application.

Is there any other way I can get around this limit without going to multiple
nodes?

Your suggestions would be welcome.

Cheers,
Bruce




Reply | Threaded
Open this post in threaded view
|

Erlang Process Limits

Raimo Niskanen-3
Bruce Fitzsimons wrote:
>
> Hello again,
>
> Is the maximum number of processes on an Erlang node a hard limit, or is
> there the potential for this to be increased (beyond 32767)?
>

It is a fairly hard limit since in the internal representation of a pid,
15 bits are used for the process table index on the node, and all other
bits are also used for essential things. This applies at least for the
Beam virtual machine up to release R8. We may change the internal
representation in future releases due to other reasons as well, but no
guarantees..., and in that case this limit might be raised, as well as
when a port to a 64 bit host processor comes.

> I am aware that requiring such a huge number may be an indicator that I
> should rethink my design, but it just maps cleanly to having 1 process for
> every call, with a parent process to manage them(2+ calls per real call).
> However this limits me to only 10000 simultaneous calls, which isn't that
> much for my application.
>
> Is there any other way I can get around this limit without going to multiple
> nodes?
>

I heard a customer that found that the scheduler became too sluggish
when they had a similar solution, so they changed their design to
increase performance, not because they had hit the number of processes
limit.

They found that not all calls were always particulary active, so during
idle states in the call lifetime they save the process state into an ets
table and stops the process. Their manager process then detects if
something happens on such a call, spawns a new process (processes are
cheap to spawn) and gives it the saved state.

/ Raimo Niskanen, Erlang/OTP, Ericsson UAB


Reply | Threaded
Open this post in threaded view
|

Erlang Process Limits

Ulf Wiger-4
In reply to this post by Bruce Fitzsimons-2
On Mon, 13 Aug 2001, Bruce Fitzsimons wrote:

>Hello again,
>
>Is the maximum number of processes on an Erlang node a hard
>limit, or is there the potential for this to be increased
>(beyond 32767)?

I don't know that this has been tested.


>I am aware that requiring such a huge number may be an
>indicator that I should rethink my design, but it just maps
>cleanly to having 1 process for every call, with a parent
>process to manage them(2+ calls per real call). However this
>limits me to only 10000 simultaneous calls, which isn't that
>much for my application.

I suggest you look at the dispatcher user contrib. It is similar
to the solution used in AXD 301, where we have verified that we
can handle some 100,000 calls per call handling processor (I am
not up to date with the actual number.) We actually have two
processes per call (A- and B-side, what's called a half-call
model.)

I have a newer version, dispatcher-1.1, which is more efficient,
and also provides an improved framework. If you are interested, I
could clean it up[*] and publish it.

The idea of the dispatcher solution is that when handling calls
(establishing sessions of some sort), there is usually only
activity in the beginning and at the end of each call. In the
meantime, it's more efficient to store the call state in ETS,
than keeping one or two process(es) for each. The overhead is
slight, as long as each call activity takes at least a few
milliseconds.

There are actually a couple of other advantages to the model:

- The "call process" can be implemented as a bare-bones FSM,
  which is probably the most beautiful state machine programming
  model around. Since the process is short lived, it doesn't
  have to worry about supporting upgrade etc.

- The heap size of the process can be set very high, allowing
  you to short-circuit the GC.


/Uffe

[*] I had been playing with the type checker, and created type
specifications for the dispatcher modules. I don't remember what
state they were in. Also, I had apparently cheated on the upgrade
support for the dispatcher process -- unforgivable.

--
Ulf Wiger                                    tfn: +46  8 719 81 95
Senior System Architect                      mob: +46 70 519 81 95
Strategic Product & System Management    ATM Multiservice Networks
Data Backbone & Optical Services Division      Ericsson Telecom AB