shared memory in Erlang

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

shared memory in Erlang

Karel Van Oudheusden
Hello,


Are there Erlang compilers that support shared memory between processes?

I do not really understand how database applications or network
applications can be implemented in Erlang efficiently without some kind
of form of shared memory.  Only providing message passing does not seem
to be a good choice for these kind of applications.

Comparing Java with Erlang, I must conclude that Erlang is my first
choice except for this very important memory issue!

I would really appreciate a thorough answer or some links to this topic
if possible.


thank you in advance,
KVO.



Reply | Threaded
Open this post in threaded view
|

shared memory in Erlang

David Gould
On Tue, Jan 30, 2001 at 09:12:44PM +0100, Karel Van Oudheusden wrote:
> Hello,
>
>
> Are there Erlang compilers that support shared memory between processes?
>
> I do not really understand how database applications or network
> applications can be implemented in Erlang efficiently without some kind
> of form of shared memory.  Only providing message passing does not seem
> to be a good choice for these kind of applications.

Erlang has a very strong process and communication model. I wonder what
shared memory support between processes would look like in Erlang? I mean,
what would the language need in terms of syntax and semantics to support it?

I am having trouble picturing it fitting in without a great deal of
collateral damage...

Or do you mean, message passing implemented transparently by means of a
shared memory?

> Comparing Java with Erlang, I must conclude that Erlang is my first
> choice except for this very important memory issue!

But I certainly would not want to see Erlang become more like Java!

-dg

--
David Gould                                                 dg
SuSE, Inc.,  580 2cd St. #210,  Oakland, CA 94607          510.628.3380
You left them alone in a room with a penguin?!  Mr Gates, your men are
already dead.


Reply | Threaded
Open this post in threaded view
|

shared memory in Erlang

Scott Lystig Fritchie-2
>>>>> "dg" == David Gould <dg> writes:


dg> On Tue, Jan 30, 2001 at 09:12:44PM +0100, Karel Van Oudheusden wrote:

>> Are there Erlang compilers that support shared memory between
>> processes?

If you mean between Erlang processes, the Binary data type is as close
as you can currently get.

ETS tables can be used by multiple processes (if created as a 'public'
or 'protected' table), but its implementation makes copies of

dg> I am having trouble picturing it fitting in without a great deal
dg> of collateral damage...

Assuming a "process" in this context is a UNIX, not Erlang,
process....

I've dabbled with a hack that had all binary storage allocated from a
shared memory segment.  A new BIF would spit out the address and
length of a binary.  That address and size could be given to another
UNIX process which had also mapped that memory region in the same
manner.

Heh.  "Collateral damage" is a good phrase.  There are several big
problems with this hack.

1. Erlang's automatic memory management can reclaim a binary's memory
and re-use it for something else, *before* the other UNIX process
retrieved the data it wants from the shared memory buffer.  A hack was
to have an Erlang server process keep a reference to that binary, and
when the other UNIX process was done with the shared memory buffer, it
would signal back to the Erlang server to unreference the binary.

2. The Erlang VM was the sole owner of the shared memory region.  No
other UNIX process could allocate memory within it.  To do otherwise
was a good way to crash the VM.

3. An external UNIX process could violate Erlang's immutable data
property: it could modify the contents of a binary object.  This was
half the reason for this hack in the first place, but it void's the
Erlang warranty.  :-)  (Performance, by eliminating data copying, was
the other reason.)

I did that hack probably a year ago.  I haven't returned to it, due to
collater^Wsheer ugliness.  Every now and then Ithink of it.  But a few
random thoughts every 6 months isn't enough to cook up reasonable
semantics for this kind of thing, much less try to implement them.

-Scott


Reply | Threaded
Open this post in threaded view
|

shared memory in Erlang

Andrew Thompson-3
In reply to this post by David Gould
David Gould wrote:

>
> On Tue, Jan 30, 2001 at 09:12:44PM +0100, Karel Van Oudheusden wrote:
> > Hello,
> >
> >
> > Are there Erlang compilers that support shared memory between processes?
> >
> > I do not really understand how database applications or network
> > applications can be implemented in Erlang efficiently without some kind
> > of form of shared memory.  Only providing message passing does not seem
> > to be a good choice for these kind of applications.
>
> Erlang has a very strong process and communication model. I wonder what
> shared memory support between processes would look like in Erlang?

Karel might be talking about internal sharing memory between processes
which I find hard to picture in Erlang, ets tables maybe?

We've also thought about implementing shared memory for communication
between Erlang and the rest of the world.  If the external (non-Erlang)
stuff wrote to the shared memory buffer and Erlang could read from it
when it needed the data. (Or it could go the other way around.)


> I mean,
> what would the language need in terms of syntax and semantics to support it?
>

Nothing special, some understanding of normal data structures.  It
wouldn't have to  be too complicated.

Cheers, Andrew


> I am having trouble picturing it fitting in without a great deal of
> collateral damage...
>
> Or do you mean, message passing implemented transparently by means of a
> shared memory?
>
> > Comparing Java with Erlang, I must conclude that Erlang is my first
> > choice except for this very important memory issue!
>
> But I certainly would not want to see Erlang become more like Java!
>
> -dg
>
> --
> David Gould                                                 dg
> SuSE, Inc.,  580 2cd St. #210,  Oakland, CA 94607          510.628.3380
> You left them alone in a room with a penguin?!  Mr Gates, your men are
> already dead.


Reply | Threaded
Open this post in threaded view
|

shared memory in Erlang

Claes Wikström
In reply to this post by David Gould
On Tue, Jan 30, 2001 at 01:33:23PM -0800, David Gould wrote:

> On Tue, Jan 30, 2001 at 09:12:44PM +0100, Karel Van Oudheusden wrote:
> > Hello,
> >
> >
> > Are there Erlang compilers that support shared memory between processes?
> >
> > I do not really understand how database applications or network
> > applications can be implemented in Erlang efficiently without some kind
> > of form of shared memory.  Only providing message passing does not seem
> > to be a good choice for these kind of applications.
>
> Erlang has a very strong process and communication model. I wonder what
> shared memory support between processes would look like in Erlang? I mean,
> what would the language need in terms of syntax and semantics to support it?
>

I've fiddled with shared memory between different erlang nodes on the
same machine. A master node and a bunch of slave nodes,

1. Master node creates segment initiall with shmget() in
   a linked in driver.

2. Slave node attach to the segment.

3. Spinlocks in the segment itself to ensure exclusivity.

We (have used it (and dropped it)) to store SSL cached sesions
and share them between several erlang nodes on the smae machine.


/klacke


--
Claes Wikstrom                        -- Caps lock is nowhere and
Alteon WebSystems                     -- everything is under control          
http://www.bluetail.com/~klacke       --



Reply | Threaded
Open this post in threaded view
|

shared memory in Erlang

Jim Larson
In reply to this post by Karel Van Oudheusden
In message <3A77203C.DAE68BEB> Karel Van Oudheusden writes:
>Are there Erlang compilers that support shared memory between processes?

I can interpret your question in one of three ways:

1) "Are there Erlang implementations that allow you to send some data
between Erlang processes on the same Erlang node without copying?"

        Yes - in Open-Source Erlang, binary-type data is passed by
        reference in messages.

2) "Are there Erlang implementations that pass *all* data between
Erlang processes without copying?"

        Not to my knowledge.  Although the functional properties
        of Erlang should allow such an implementation, I don't
        think it's desirable for two reasons:

                - it would imply a single global heap, or massive
                  synchronization between per-process heaps, which
                  would prevent independent per-process GC, which
                  would hurt Erlang's soft real-time properties;

                - I recall hearing that the overhead of message
                  passing within an Erlang node is high enough that
                  the cost of copying message data isn't overwhelming,
                  for typically-sized messages.

3) "Are there Erlang implementations that allow shared memory IPC
between and Erlang node and another OS process?"

        Depending on your needs, you might be able to create a
        special-purpose linked-in driver that can transform messages
        into shared-memory updates, and vice-versa.  Using this
        mechanism, you incur an overhead in copying data, even from
        binaries, into the shared memory, and vice-versa.

        Scott Lystig Fritchie implemented hacks to R6B that put
        all of Erlang's binaries in a shared-memory space, so that
        bulk data received from the network could be processed by
        Erlang code and sent to another C application, with zero-copy
        overhead.  Unfortunately, the performance wasn't as great
        as we'd hoped, possibly due to the locking overheads of
        the shared-memory malloc() library we needed to use.

>I do not really understand how database applications or network
>applications can be implemented in Erlang efficiently without some kind
>of form of shared memory.  Only providing message passing does not seem
>to be a good choice for these kind of applications.

I don't understand your concern here.  Shared memory is only useful
for communicating with other OS processes on the same host, so
you're either having to integrate with applications written in
other languages, or perhaps take advantage of a multiprocessor
through running several Erlang nodes on the same host.  Other than
that, network and database applications seem to work fine in Erlang.

Jim


Reply | Threaded
Open this post in threaded view
|

shared memory in Erlang

David Gould
On Tue, Jan 30, 2001 at 04:50:32PM -0800, Jim Larson wrote:
 

> >I do not really understand how database applications or network
> >applications can be implemented in Erlang efficiently without some kind
> >of form of shared memory.  Only providing message passing does not seem
> >to be a good choice for these kind of applications.
>
> I don't understand your concern here.  Shared memory is only useful
> for communicating with other OS processes on the same host, so
> you're either having to integrate with applications written in
> other languages, or perhaps take advantage of a multiprocessor
> through running several Erlang nodes on the same host.  Other than
> that, network and database applications seem to work fine in Erlang.

As a database engine guy in a former life, I think I do understand. Basically
database engines run as a collection of processes (one per physical CPU)
sharing a common set of datastructures:

   the database page cache
   database metadata (table names, schema definitions, procedure defs etc)
   lock manager tables
   transaction / logging information
   scheduler information
   thread data including execution stacks

The database engines typically implement their own lightweight threading
(somewhat akin to Erlang processes), and are able to schedule threads
across multiple physical CPUs. That is, threads can migrate. So there is
not any local data, everything is in the shared memory, even if it is
private to a thread because that thread may next be scheduled onto a
different process/cpu.

It does not really look easy to build such a thing in Erlang. Possibly
a linked in driver could provide a suffcient interface to the critical
parts, eg lock manager, page cache, page qualification (get rows on this
page that match a predicate), page update etc.

Indeed, such a beast might be much cleaner and more stable than existing
db systems because of the enforced strict isolation of the shared memory
behind a defined interface (ie, no random pointer poking). Interesting
project for someone with a big need for a high performance Erlang dbms.

-dg

--
David Gould                                                 dg
SuSE, Inc.,  580 2cd St. #210,  Oakland, CA 94607          510.628.3380
You left them alone in a room with a penguin?!  Mr Gates, your men are
already dead.


Reply | Threaded
Open this post in threaded view
|

shared memory in Erlang

Jim Larson
In message <20010130171922.B6028> David Gould writes:
>It does not really look easy to build such a thing in Erlang. Possibly
>a linked in driver could provide a suffcient interface to the critical
>parts, eg lock manager, page cache, page qualification (get rows on this
>page that match a predicate), page update etc.
>
>Indeed, such a beast might be much cleaner and more stable than existing
>db systems because of the enforced strict isolation of the shared memory
>behind a defined interface (ie, no random pointer poking). Interesting
>project for someone with a big need for a high performance Erlang dbms.

It sounds like what's *really* desired here is a multiprocessor-friendly
implementation of Erlang, so that a single Erlang node could use
several CPUs on an SMP at once.  (In fact, Erlang is probably a
nice fit to NUMA architecutres too, not just SMP.)

Jim


Reply | Threaded
Open this post in threaded view
|

shared memory in Erlang

David Gould
On Tue, Jan 30, 2001 at 05:30:36PM -0800, Jim Larson wrote:
>
> It sounds like what's *really* desired here is a multiprocessor-friendly
> implementation of Erlang, so that a single Erlang node could use
> several CPUs on an SMP at once.


Hmmm. What would be the benefit as opposed to multiple nodes on one SMP?
Semantically Erlang makes it reasonably easy to ignore the node a process is
on. So, presumably it would be a performance issue?

But, since Erlang does not have large shared datastructures like a db engine,
and would not let processes directly share them anyway, where is the
performance gain of MP friendly Erlang going to come from?

Message passing?

More control of scheduling?

Something else?

The only one that comes to mind is message passing, in which case
it would be possible to do a shared memory transport (as I think you
mentioned doing).

But any kind of sharing on an MP is going to involve some amount of
overhead for spinlocks and other sychronization. This can eat most of the
gains of an MP pretty fast. And MP friendly programs are much harder to
debug for some kinds of bug.

 
> (In fact, Erlang is probably a
> nice fit to NUMA architecutres too, not just SMP.)

I like this thought. Traditional MP programs are too tightly coupled
to take good advantage of an MP. So they end up getting revamped with
separate zoned memory allocators, and local copies of stuff. And lock
contention becomes an even uglier headache than on an MP.

Erlang, not being coupled at all, avoids all this, but presumably can
take advantage of the low latency message that could be built for nodes
on a NUMA.

-dg

--
David Gould                                                 dg
SuSE, Inc.,  580 2cd St. #210,  Oakland, CA 94607          510.628.3380
You left them alone in a room with a penguin?!  Mr Gates, your men are
already dead.


Reply | Threaded
Open this post in threaded view
|

shared memory in Erlang

Thomas Lindgren-2
In reply to this post by Karel Van Oudheusden

> I do not really understand how database applications or network
> applications can be implemented in Erlang efficiently without some kind
> of form of shared memory.  Only providing message passing does not seem
> to be a good choice for these kind of applications.

Logically, the Erlang processes have private memories. In the
implementation, there is some room to wiggle.

In the Erlang/OTP implementation, all processes (in a single node)
reside in the same address space (e.g., the same OS process). Some
structures, such as binaries and ets-tables, are on the other hand
normally shared by all processes. The Erlang emulator, compiler and
runtime system ensure that everything is done properly (no wild
writing to shared data, etc).

But Erlang/OTP is just one data point. Another data point: it is
possible to implement Erlang so that _no_ data are private (all
processes share data arbitrarily); Robert Virding wrote such an
implementation. The great problem then is memory management, which
ended up costly in that implementation, but that might be solvable.

(Erlang/OTP does not support sharing memory between nodes, however.
At least as far as I know :-)

Finally, what is your measure of efficiency?  Our experience with a
highly scalable mail frontend is that Erlang is _very_ good at
handling large numbers of latency-insensitive concurrent sessions.

On the other hand, latency-critical stuff (like, say, packet switching
or digital signal processing) is probably better done in hardware, or
perhaps assembly/C. In that case, use Erlang's concept of linked-in
drivers to interface with this code.

                          Thomas
--
Thomas Lindgren thomas+junk
Alteon WebSystems


Reply | Threaded
Open this post in threaded view
|

shared memory in Erlang

Ulf Wiger-4
In reply to this post by David Gould
On Tue, 30 Jan 2001, David Gould wrote:

>On Tue, Jan 30, 2001 at 04:50:32PM -0800, Jim Larson wrote:
>
>> >I do not really understand how database applications or network
>> >applications can be implemented in Erlang efficiently without some kind
>> >of form of shared memory.  Only providing message passing does not seem
>> >to be a good choice for these kind of applications.
>>
>> I don't understand your concern here.  Shared memory is only useful
>> for communicating with other OS processes on the same host, so
>> you're either having to integrate with applications written in
>> other languages, or perhaps take advantage of a multiprocessor
>> through running several Erlang nodes on the same host.  Other than
>> that, network and database applications seem to work fine in Erlang.
>
>As a database engine guy in a former life, I think I do understand.
>Basically database engines run as a collection of processes (one per
>physical CPU) sharing a common set of datastructures:
>
>   the database page cache
>   database metadata (table names, schema definitions, procedure defs etc)
>   lock manager tables
>   transaction / logging information
>   scheduler information
>   thread data including execution stacks

A few comments from my end:

- There was an Erlang implementation with a global heap, called VEE,
  developed by Robert Virding. It used incremental GC, and did not
  copy messages. It didn't catch on though. Implementing message
  passing can of course be done transparently to erlang. One should
  also consider latency when dealing with message passing programs.

- Implementing ets:insert and ets:lookup without copying could
  probably be done, but it's not obvious how it should be done.
  Apparently, experiments have been made with ets tables on the
  process heap, but this didn't turn out as well as expected
  (mainly due to GC.) The obvious problem with non-copying ets
  shows up when multiple processes update the same object;
  you would then need to keep and garbage collect the old object,
  if it is referenced by some process(es). Private ets tables
  could be implemented without copying, but for various reasons
  (mainly debugging), private ets tables are not used much.

- The erlang processor, presented at EUC 2000, does not copy messages.
  The main thing wrong with the erlang processor at this time is that
  it is not ready yet.

- There was also a multipro erlang implementation
  http://www.erlang.se/publications/xjobb/0089-hedqvist.pdf
  Good reading for those who are interested. Demand for MP erlang
  was not high enough to warrant the extra effort needed to make
  it a product... at the time.

- We (AXD 301) are discussing the idea of having a shared memory
  database that has erlang and C applications as clients.

- We're also starting to look into a port of erlang to OSE/Delta
  (http://www.enea.com/PDF/rtk.pdf). We will at least study the
  possibility to make a native port, which maps erlang processes
  onto OSE threads. There are several good reasons to look into
  this.

- I've also played with the idea of using something like Encore's
  Reflective Memory for really fast replication of process state
  between processors - distributed shared memory if you will.
(http://www.encore.com/products/hardware/reflective/980012.shtml)
  It's not exactly clear to me if it could be reasonably well
  integrated into erlang, semantically, or if one would simply
  write a driver that interfaces to it. Our main problem with
  Encore's solution is that the memory size is too small for us.

/Uffe
--
Ulf Wiger                                    tfn: +46  8 719 81 95
Senior System Architect                      mob: +46 70 519 81 95
Strategic Product & System Management    ATM Multiservice Networks
Data Backbone & Optical Services Division      Ericsson Telecom AB



Reply | Threaded
Open this post in threaded view
|

shared memory in Erlang

Pekka Hedqvist
In reply to this post by David Gould
>-----Original Message-----
>From: owner-erlang-questions
>[mailto:owner-erlang-questions]On Behalf Of David Gould

>
>Message passing?

In a way yes, msg-passing + context switch between two Erlang processes
running on different nodes with shared mem (same physical machine) is way
slower than if they ran in the same process space with either native threads
or (preferably) worker threads scheduling Erlang processes.

>
>More control of scheduling?

Yes, with the node modell one has to ensure that not all heavy Erlang
processes ends on one node and the the lighter ones ends up on the other
etc. With all processes scheduled within one process, either on native
threads or with worker-model this is taken care automagically.

>
>Something else?

>But any kind of sharing on an MP is going to involve some amount of
>overhead for spinlocks and other sychronization. This can eat most of the
>gains of an MP pretty fast. And MP friendly programs are much harder to
>debug for some kinds of bug.

The cost of locking really depends on the design of the runtime. The one
Tony and I did many years back basically kept most of the original JAM/BEAM
Erlang (multiple heaps, per-heap-gc, copying msg-passing, copying ets tables
etc) characteristics, with a few tweaks though. There locking-bottlenecks
would be hard to get even with many cpu:s. With a unified heap (VEE, or like
some parallel prologs has and most Java runtimes has) things becomes more
difficult and unpredictable. But the one we did mainly kept most of the good
(and bad) things from the separate heaps (which todays Erlang still use)
implementation but also took advantage of MP machines. I belive that a
productified version of that one would given users few backlash-surprises,
just run better on multiple CPU machines than any multi-Erlang-node aproach
could ever do.

Maybe obvious, but with an MP-Erlang the MP part would not be visible. Just
plain ol'Erlang.

>Erlang, not being coupled at all, avoids all this, but presumably can
>take advantage of the low latency message that could be built for nodes
>on a NUMA.

BTW, improving Erlang-communication between nodes on the same machine and on
separate machines can be done with new aproaches like VIA (www.viaarch.org).
And new IO devices (NGIO etc). Erlang node communication on LAN/SAN:s with
VIA could be really fast with via (avoiding tcp) etc and still portable.
There is a free VIA around for Linux called M-VIA
(http://www.nersc.gov/research/FTG/via/faq.html), but development around VIA
does not happen as fast as I thought a few years back, but the basic ideas
are just right (superfast direct userspace network IO, smart IO cards,
portability etc)

/pekka



Reply | Threaded
Open this post in threaded view
|

shared memory in Erlang

Pekka Hedqvist
In reply to this post by David Gould
>-----Original Message-----
>From: owner-erlang-questions
>[mailto:owner-erlang-questions]On Behalf Of David Gould

>
>Message passing?

In a way yes, msg-passing + context switch between two Erlang processes
running on different nodes with shared mem (same physical machine) is way
slower than if they ran in the same process space with either native threads
or (preferably) worker threads scheduling Erlang processes.

>
>More control of scheduling?

Yes, with the node modell one has to ensure that not all heavy Erlang
processes ends on one node and the the lighter ones ends up on the other
etc. With all processes scheduled within one process, either on native
threads or with worker-model this is taken care automagically.

>
>Something else?

>But any kind of sharing on an MP is going to involve some amount of
>overhead for spinlocks and other sychronization. This can eat most of the
>gains of an MP pretty fast. And MP friendly programs are much harder to
>debug for some kinds of bug.

The cost of locking really depends on the design of the runtime. The one
Tony and I did many years back basically kept most of the original JAM/BEAM
Erlang (multiple heaps, per-heap-gc, copying msg-passing, copying ets tables
etc) characteristics, with a few tweaks though. There locking-bottlenecks
would be hard to get even with many cpu:s. With a unified heap (VEE, or like
some parallel prologs has and most Java runtimes has) things becomes more
difficult and unpredictable. But the one we did mainly kept most of the good
(and bad) things from the separate heaps (which todays Erlang still use)
implementation but also took advantage of MP machines. I belive that a
productified version of that one would given users few backlash-surprises,
just run better on multiple CPU machines than any multi-Erlang-node aproach
could ever do.

Maybe obvious, but with an MP-Erlang the MP part would not be visible. Just
plain ol'Erlang.

>Erlang, not being coupled at all, avoids all this, but presumably can
>take advantage of the low latency message that could be built for nodes
>on a NUMA.

BTW, improving Erlang-communication between nodes on the same machine and on
separate machines can be done with new aproaches like VIA (www.viaarch.org).
And new IO devices (NGIO etc). Erlang node communication on LAN/SAN:s with
VIA could be really fast with via (avoiding tcp) etc and still portable.
There is a free VIA around for Linux called M-VIA
(http://www.nersc.gov/research/FTG/via/faq.html), but development around VIA
does not happen as fast as I thought a few years back, but the basic ideas
are just right (superfast direct userspace network IO, smart IO cards,
portability etc)

/pekka