Controlled interaction of two erlang distributed networks

classic Classic list List threaded Threaded
19 messages Options
Reply | Threaded
Open this post in threaded view
|

Controlled interaction of two erlang distributed networks

Richard Andrews-5
I have a distributed system that needs to run on two geographically
isolated sites. Each site has a central manager node.

I want a way to have the manager nodes from each site use erlang
monitoring of each other and some specific processes only on the
manager nodes; but I want to avoid node lists and global registrations
expanding such that the nodes on different sites try and become
connected together.

How can I achieve this?

--
  Rich

________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

Reply | Threaded
Open this post in threaded view
|

Re: Controlled interaction of two erlang distributed networks

Jayson Vantuyl-2
You could use global_group to split each site into its own group and  
then make a node list of just the managers (which you already said you  
don't want to do).

I think what you want is to roll you're own gossiped mesh like this.  
Once you "introduce" a node to any other node in the mesh, it all just  
syncs up eventually.  This is a bit of work, but it really pays off.

Version 1:

Create an Mnesia table that holds tuples of the form  
{manager_node,Name,Clock,IpAddr}.  Name is an atom that identifies a  
node (used to detect when they switch IP), and it needs to be unique  
per node.  IpAddr is exactly what it looks like.  Clock is a value  
that only increases and is used to detect old entries.  This table  
should start out empty, although it should persist between restarts.

Create a process that periodically gets the current IpAddress of the  
node and updates the local table with it.  It should bump up the clock  
each time it changes the record.  One way to do this is to start it  
out and zero and increment.  Another is to just use the current  
machine timestamp.  I'd recommend incrementing, as time-sync problems  
are a pain to track down.

Create another process that periodically contacts all of the nodes and  
distributes the table over some well known port, probably using UDP.  
You don't have to reliably deliver it.  If you're pedantic about  
security, you might use TCP + SSL.  Formatting this request is not  
hard.  Just use term_to_binary and binary_to_term.

Create another process that listens on some well known port for the  
updates, and updates node entries.  It should overwrite an entry with  
the same Name, but only if the Clock is higher.

Now just create a utility that will "introduce" two nodes by putting  
an entry for the other Node in its table.  For this version, just  
start with 0.  After a little gossiping, they should know about each  
other.  The persistence of the table should allow changes of IP  
address to be easily tolerated.

Version 2:

The previous version has a few problems.  Most importantly, if you're  
using term_to_binary, it's possible for people to leak atoms.  Also,  
you probably didn't encrypt it or put any sort of secret-key  
encryption.  Add this to make it secure.

Also, if you are starting with 0 on an "introduction", now is the time  
to enhance that.  If you lose the node and want to deploy a new one,  
you can't just introduce it with 0, as the clock will already be  
higher.  Extend the "introduction" to ask the remote end for the clock  
value it last saw for Name.  This allows you to avoid that particular  
pitfall.

Similarly, if you need to remove a node, use a special atom 'dead' in  
the place of IpAddr.  You might add this to the utility as well.  This  
should only be needed when a Name is permanently gone, and it mostly  
exists to prevent gossips to an address that will never need them and  
to prevent the node from being gossiped back into existence after you  
delete its entry.

Version 3:

Now add functionality to gossip the processes you want to monitor.

NOTE:  Just because a process isn't communicating, doesn't mean its  
not there.  Be very careful with what you are trying to do (assuming  
you're doing automated failover, not just alerting).  It's been well  
established for decades that a system can only sport two out of the  
three big qualities: "highly available", "tolerant of network  
partitions", and "data is consistent".  If you think you've made  
something that does it, some very nasty mathematics will eventually  
bite you in the rear.

You might ask why I wouldn't just monkey with Distributed Erlang to  
separate the globals into groups (see the global_group module).  The  
problem is that Distributed Erlang still just isn't built for this  
sort of use-case.  In Distributed Erlang getting the SSL stuff working  
right is hard, firewalls suck, NAT sucks even more, and the proper DNS  
setup is more trouble than maintaining a node-list ever was (and  
impossible to quickly fix, if you use high enough TTLs).

On Aug 25, 2009, at 5:17 PM, Richard Andrews wrote:

> I have a distributed system that needs to run on two geographically
> isolated sites. Each site has a central manager node.
>
> I want a way to have the manager nodes from each site use erlang
> monitoring of each other and some specific processes only on the
> manager nodes; but I want to avoid node lists and global registrations
> expanding such that the nodes on different sites try and become
> connected together.
>
> How can I achieve this?
>
> --
>  Rich
>
> ________________________________________________________________
> erlang-questions mailing list. See http://www.erlang.org/faq.html
> erlang-questions (at) erlang.org
>



--
Jayson Vantuyl
[hidden email]




________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

Reply | Threaded
Open this post in threaded view
|

Re: Controlled interaction of two erlang distributed networks

Richard Andrews-5
On Wed, Aug 26, 2009 at 11:28 AM, Jayson Vantuyl<[hidden email]> wrote:
> You might ask why I wouldn't just monkey with Distributed Erlang to separate
> the globals into groups (see the global_group module).  The problem is that
> Distributed Erlang still just isn't built for this sort of use-case.  In
> Distributed Erlang getting the SSL stuff working right is hard, firewalls
> suck, NAT sucks even more, and the proper DNS setup is more trouble than
> maintaining a node-list ever was (and impossible to quickly fix, if you use
> high enough TTLs).

Security, NAT, etc. is a non-issue as any non-LAN traffic travels via
IPSec between sites. Do you think this modifies your suggestions?

________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

Reply | Threaded
Open this post in threaded view
|

Re: Controlled interaction of two erlang distributed networks

Jayson Vantuyl-2
Actually, IPsec was mostly what I was worried about.

That said, DNS is what will bite you.  If you're cool with that, then  
you can try using global_group.

On Aug 25, 2009, at 8:20 PM, Richard Andrews wrote:

> On Wed, Aug 26, 2009 at 11:28 AM, Jayson Vantuyl<[hidden email]>  
> wrote:
>> You might ask why I wouldn't just monkey with Distributed Erlang to  
>> separate
>> the globals into groups (see the global_group module).  The problem  
>> is that
>> Distributed Erlang still just isn't built for this sort of use-
>> case.  In
>> Distributed Erlang getting the SSL stuff working right is hard,  
>> firewalls
>> suck, NAT sucks even more, and the proper DNS setup is more trouble  
>> than
>> maintaining a node-list ever was (and impossible to quickly fix, if  
>> you use
>> high enough TTLs).
>
> Security, NAT, etc. is a non-issue as any non-LAN traffic travels via
> IPSec between sites. Do you think this modifies your suggestions?
>
> ________________________________________________________________
> erlang-questions mailing list. See http://www.erlang.org/faq.html
> erlang-questions (at) erlang.org
>



--
Jayson Vantuyl
[hidden email]




________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

Reply | Threaded
Open this post in threaded view
|

Re: Controlled interaction of two erlang distributed networks

Jayson Vantuyl-2
In reply to this post by Richard Andrews-5
One more note.  You may also need to use the option "-connect_all  
false".  It keeps from having the nodes automatically connect to all  
of the nodes that a partner is connected to.  I always forget about  
that one.

On Aug 25, 2009, at 8:20 PM, Richard Andrews wrote:

> On Wed, Aug 26, 2009 at 11:28 AM, Jayson Vantuyl<[hidden email]>  
> wrote:
>> You might ask why I wouldn't just monkey with Distributed Erlang to  
>> separate
>> the globals into groups (see the global_group module).  The problem  
>> is that
>> Distributed Erlang still just isn't built for this sort of use-
>> case.  In
>> Distributed Erlang getting the SSL stuff working right is hard,  
>> firewalls
>> suck, NAT sucks even more, and the proper DNS setup is more trouble  
>> than
>> maintaining a node-list ever was (and impossible to quickly fix, if  
>> you use
>> high enough TTLs).
>
> Security, NAT, etc. is a non-issue as any non-LAN traffic travels via
> IPSec between sites. Do you think this modifies your suggestions?

--
Jayson Vantuyl
[hidden email]




________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

Reply | Threaded
Open this post in threaded view
|

Re: Controlled interaction of two erlang distributed networks

Richard Andrews-5
On Wed, Aug 26, 2009 at 4:34 PM, Jayson Vantuyl<[hidden email]> wrote:
> One more note.  You may also need to use the option "-connect_all false".
>  It keeps from having the nodes automatically connect to all of the nodes
> that a partner is connected to.  I always forget about that one.

Using this is actually what spawned this thread. Adding this option
appears to lead to breakage of the global name registry. Nodes get
stuck doing what I suspect is a global:sync() with hosts that will not
cooperate. So I'm looking for a way to quarantine global to each LAN
and provide a proxy for communication across the WAN link.

________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

Reply | Threaded
Open this post in threaded view
|

Re: Controlled interaction of two erlang distributed networks

Ulf Wiger-3
In reply to this post by Jayson Vantuyl-2
Jayson Vantuyl wrote:
> One more note.  You may also need to use the option "-connect_all
> false".  It keeps from having the nodes automatically connect to all of
> the nodes that a partner is connected to.  I always forget about that one.

I don't think mixing global and connect_all false is a good idea.
The docs for global mention connect_all, but also say that only
locking, not name registration, will work then.


BTW, I found this fairly confusing wording in the global_group
man page:

"For the processes and nodes to run smoothly using the global group
functiontionality, the following criteria must be met:

- ...
- *All* nodes in the system should belong to exactly one global group."

Presumably, it should be "Every node in the system should belong
to exactly one global group."

As it reads now, it could easily be interpreted as requiring that
there must be exactly one global group, and all nodes must be
members of that group - apart from the fact that this would
clearly be absurd, as it would eliminate any reason for using
global_group in the first place...

BR,
Ulf W


> On Aug 25, 2009, at 8:20 PM, Richard Andrews wrote:
>
>> On Wed, Aug 26, 2009 at 11:28 AM, Jayson Vantuyl<[hidden email]> wrote:
>>> You might ask why I wouldn't just monkey with Distributed Erlang to
>>> separate
>>> the globals into groups (see the global_group module).  The problem
>>> is that
>>> Distributed Erlang still just isn't built for this sort of use-case.  In
>>> Distributed Erlang getting the SSL stuff working right is hard,
>>> firewalls
>>> suck, NAT sucks even more, and the proper DNS setup is more trouble than
>>> maintaining a node-list ever was (and impossible to quickly fix, if
>>> you use
>>> high enough TTLs).
>>
>> Security, NAT, etc. is a non-issue as any non-LAN traffic travels via
>> IPSec between sites. Do you think this modifies your suggestions?
>


--
Ulf Wiger
CTO, Erlang Training & Consulting Ltd
http://www.erlang-consulting.com

________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

Reply | Threaded
Open this post in threaded view
|

Re: Controlled interaction of two erlang distributed networks

Kenji Rikitake
In reply to this post by Jayson Vantuyl-2
I understand Erlang rpc module and the related ones including
global_group module are designed only for where arbitrary communication
is possible and allowed between the BEAM instances *and* epmd programs.
This assumption is unfortunately not feasible on a network across
unsecure links, such as those over global Internet.

I was once thinking about managing the two or more separated Erlang rpc
links as a single network, but I still have no idea with that.

In the message <[hidden email]>
dated Tue, Aug 25, 2009 at 08:49:50PM -0700,
Jayson Vantuyl <[hidden email]> writes:
> Actually, IPsec was mostly what I was worried about.

IPsec over a NAT conversion is complicated and difficult.

Configuring the IKE policy rules is a bit of headache unless making
*all* communication between two IP addresses under IPsec.

For running something depending on rpc module, Encryption of
communication between epmd daemons is mandatory, as well as that between
BEAM instances.  Mandating IPsec to *all* involving hosts is a simple
way, but enforcing the policy is a difficult task too.

(And using inet_ssl_dist is actually *incomplete* for encrypting all
necessary traffics, because it does not encrypt empd traffic at all.)

> That said, DNS is what will bite you.  If you're cool with that, then  
> you can try using global_group.

Maintaining unified DNS domain across multiple private (RFC1918)
networks is surely another headache.

My 2 JPY worth,

Kenji Rikitake

________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

Reply | Threaded
Open this post in threaded view
|

Re: Controlled interaction of two erlang distributed networks

Chandru-4
In reply to this post by Richard Andrews-5
2009/8/26 Richard Andrews <[hidden email]>

> I have a distributed system that needs to run on two geographically
> isolated sites. Each site has a central manager node.
>
> I want a way to have the manager nodes from each site use erlang
> monitoring of each other and some specific processes only on the
> manager nodes; but I want to avoid node lists and global registrations
> expanding such that the nodes on different sites try and become
> connected together.
>
> How can I achieve this?
>
>
Start up the manager nodes with the '-hidden' parameter. That way, a fully
meshed network is not formed when the two managers talk to each other.

cheers
Chandru
Reply | Threaded
Open this post in threaded view
|

Re: Controlled interaction of two erlang distributed networks

Kenneth Lundin
In reply to this post by Kenji Rikitake
>
> (And using inet_ssl_dist is actually *incomplete* for encrypting all
> necessary traffics, because it does not encrypt empd traffic at all.)
>
Why do you think it is important to encrypt the epmd traffic?
Is there really any sensitive information exchanged there?
It is really very little data with low frequency exchanged between epmd
and the nodes. It is actually in practice only used during
establishment of a new connection to an Erlang node.

I am not saying that the Erlang distribution is perfect for the use
over global internet but
is really epmd a problem?

/Kenneth Erlang/OTP Ericsson

________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

Reply | Threaded
Open this post in threaded view
|

Re: Controlled interaction of two erlang distributed networks

Witold Baryluk
Dnia 2009-08-26, śro o godzinie 18:28 +0200, Kenneth Lundin pisze:

> >
> > (And using inet_ssl_dist is actually *incomplete* for encrypting all
> > necessary traffics, because it does not encrypt empd traffic at all.)
> >
> Why do you think it is important to encrypt the epmd traffic?
> Is there really any sensitive information exchanged there?
> It is really very little data with low frequency exchanged between epmd
> and the nodes. It is actually in practice only used during
> establishment of a new connection to an Erlang node.
>
> I am not saying that the Erlang distribution is perfect for the use
> over global internet but
> is really epmd a problem?
>
> /Kenneth Erlang/OTP Ericsson
>
I think it allows spoofing registration of nodes. This can cause denial
of service.

--
Witold Baryluk

signature.asc (204 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Controlled interaction of two erlang distributed networks

Richard Andrews-5
In reply to this post by Chandru-4
On Wed, Aug 26, 2009 at 11:22 PM,
Chandru<[hidden email]> wrote:
> Start up the manager nodes with the '-hidden' parameter. That way, a fully
> meshed network is not formed when the two managers talk to each other.

This doesn't work because then the manager nodes are then also hidden
in the site where they are meant to be a normal member.

________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

Reply | Threaded
Open this post in threaded view
|

Re: Controlled interaction of two erlang distributed networks

Kenji Rikitake
In reply to this post by Witold Baryluk
Dear Kenneth, Witold and all:

It is not just about the importance, but the encryption should be
mandated on all protocols between BEAMs and epmds (or anything related
to distributed operation of Erlang systems), if Ericsson and current
Erlang users want to earn financial support of Erlang from the
security-aware (i.e. ordinary) users.

Port-mapping based RPCs in general, not only Erlang's but that of Sun
RPC (or ONC RPC), have been a long-time source of security problems.
You can learn this from the various security advisories regarding Sun
RPC in 1990s, also known as "portmap" problems.

Port-mapping based RPC is extremely unfriendly against firewalls, or
proxies and packet filters.  For example, allowing arbitrary ports for
BEAM communication is almost infeasible in the modern end-user
environment, due to entirely disabling incoming TCP connections, or at
least minimizing it to those absolutely necessary (e.g., ports 80 and
443.)  And under such circumstances IPsec is not a practical solution
either, since UDP exchange other than DNS and NTP is usually prohibited.

As Witold explains, information exchanged between epmds is an easy
target for killing BEAMs.  It includes P2P port mappings between the
BEAMs, so you can easily locate the targets to attack.  Communication
between epmds must be encrypted to prevent this kind of attack.

Of course epmd itself could be a target of DoS attack, but that's
another issue.

I am not denying the usefulness of current rpc module in Erlang.  It's
well-written, transparent, low programming overhead for parallelization,
and is OK so long as being used in a network where arbitrary use of
TCP ports are allowed.  This style of RPC, however, does not scale in
the hostile real-world Internet, unfortunately.

Erlang has SSL and SSH built-in (with the help of crypto linked-in
drivers), and I think the CPUs nowadays are fast enough to run something
equivalent to epmd purely under Erlang without using a dedicated C
program.  So Erlang has a lot of possibilities in implementing secure
protocols on top of it.  

I think making a new RPC protocol from scratch, such as:

* with restricting the usage of TCP connection between two BEAMs to only
one well-known destination port;

* preferably being able to forwarded through proxies (i.e. the
addressing mechanism of BEAMs does not depend on DNS, IP addresses, or
port numbers); and

* running everything within a BEAM (and linked-in drivers) without
anything like epmd
 
will open a new opportunity for Erlang to become a practical system for
monitoring/controlling distant systems over Internet.  This is a
challenging but an interesting project.

Regards,
Kenji Rikitake

In the message <1251304615.18875.39.camel@sredniczarny>
dated Wed, Aug 26, 2009 at 06:36:31PM +0200,
Witold Baryluk <[hidden email]> writes:

> Dnia 2009-08-26, śro o godzinie 18:28 +0200, Kenneth Lundin pisze:
> > >
> > > (And using inet_ssl_dist is actually *incomplete* for encrypting all
> > > necessary traffics, because it does not encrypt empd traffic at all.)
> > >
> > Why do you think it is important to encrypt the epmd traffic?
> > Is there really any sensitive information exchanged there?
> > It is really very little data with low frequency exchanged between epmd
> > and the nodes. It is actually in practice only used during
> > establishment of a new connection to an Erlang node.
> >
> > I am not saying that the Erlang distribution is perfect for the use
> > over global internet but
> > is really epmd a problem?
> >
> > /Kenneth Erlang/OTP Ericsson
> >
>
> I think it allows spoofing registration of nodes. This can cause denial
> of service.
>
> --
> Witold Baryluk

________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

Reply | Threaded
Open this post in threaded view
|

Re: Controlled interaction of two erlang distributed networks

Dave Smith-10
I'm not sure it's worth anything, but I designed and implemented
something along these lines for a class I took last year. You can read
about it here:

http://dizzyd.com/sdist.pdf

I could also make the code available if it's interesting. However, it
does have most of the security
properties Kenji was describing.

It would be very nice to have something which permits better port
control than the current EPMD approach. Security guidelines in trusted
environments have grown more stringent over the years and the
"everyone grab a port and register it!" approach doesn't fit well with
semi-trusted environments.

Anyways, I hope this paper will at least stimulate some further discussion. :)

D.

2009/8/26 Kenji Rikitake <[hidden email]>:

> Dear Kenneth, Witold and all:
>
> It is not just about the importance, but the encryption should be
> mandated on all protocols between BEAMs and epmds (or anything related
> to distributed operation of Erlang systems), if Ericsson and current
> Erlang users want to earn financial support of Erlang from the
> security-aware (i.e. ordinary) users.
>
> Port-mapping based RPCs in general, not only Erlang's but that of Sun
> RPC (or ONC RPC), have been a long-time source of security problems.
> You can learn this from the various security advisories regarding Sun
> RPC in 1990s, also known as "portmap" problems.
>
> Port-mapping based RPC is extremely unfriendly against firewalls, or
> proxies and packet filters.  For example, allowing arbitrary ports for
> BEAM communication is almost infeasible in the modern end-user
> environment, due to entirely disabling incoming TCP connections, or at
> least minimizing it to those absolutely necessary (e.g., ports 80 and
> 443.)  And under such circumstances IPsec is not a practical solution
> either, since UDP exchange other than DNS and NTP is usually prohibited.
>
> As Witold explains, information exchanged between epmds is an easy
> target for killing BEAMs.  It includes P2P port mappings between the
> BEAMs, so you can easily locate the targets to attack.  Communication
> between epmds must be encrypted to prevent this kind of attack.
>
> Of course epmd itself could be a target of DoS attack, but that's
> another issue.
>
> I am not denying the usefulness of current rpc module in Erlang.  It's
> well-written, transparent, low programming overhead for parallelization,
> and is OK so long as being used in a network where arbitrary use of
> TCP ports are allowed.  This style of RPC, however, does not scale in
> the hostile real-world Internet, unfortunately.
>
> Erlang has SSL and SSH built-in (with the help of crypto linked-in
> drivers), and I think the CPUs nowadays are fast enough to run something
> equivalent to epmd purely under Erlang without using a dedicated C
> program.  So Erlang has a lot of possibilities in implementing secure
> protocols on top of it.
>
> I think making a new RPC protocol from scratch, such as:
>
> * with restricting the usage of TCP connection between two BEAMs to only
> one well-known destination port;
>
> * preferably being able to forwarded through proxies (i.e. the
> addressing mechanism of BEAMs does not depend on DNS, IP addresses, or
> port numbers); and
>
> * running everything within a BEAM (and linked-in drivers) without
> anything like epmd
>
> will open a new opportunity for Erlang to become a practical system for
> monitoring/controlling distant systems over Internet.  This is a
> challenging but an interesting project.
>
> Regards,
> Kenji Rikitake
>
> In the message <1251304615.18875.39.camel@sredniczarny>
> dated Wed, Aug 26, 2009 at 06:36:31PM +0200,
> Witold Baryluk <[hidden email]> writes:
>> Dnia 2009-08-26, śro o godzinie 18:28 +0200, Kenneth Lundin pisze:
>> > >
>> > > (And using inet_ssl_dist is actually *incomplete* for encrypting all
>> > > necessary traffics, because it does not encrypt empd traffic at all.)
>> > >
>> > Why do you think it is important to encrypt the epmd traffic?
>> > Is there really any sensitive information exchanged there?
>> > It is really very little data with low frequency exchanged between epmd
>> > and the nodes. It is actually in practice only used during
>> > establishment of a new connection to an Erlang node.
>> >
>> > I am not saying that the Erlang distribution is perfect for the use
>> > over global internet but
>> > is really epmd a problem?
>> >
>> > /Kenneth Erlang/OTP Ericsson
>> >
>>
>> I think it allows spoofing registration of nodes. This can cause denial
>> of service.
>>
>> --
>> Witold Baryluk
>
> ________________________________________________________________
> erlang-questions mailing list. See http://www.erlang.org/faq.html
> erlang-questions (at) erlang.org
>
>

________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

Reply | Threaded
Open this post in threaded view
|

Re: Controlled interaction of two erlang distributed networks

Chandru-4
In reply to this post by Richard Andrews-5
2009/8/27 Richard Andrews <[hidden email]>

> On Wed, Aug 26, 2009 at 11:22 PM,
> Chandru<[hidden email]> wrote:
> > Start up the manager nodes with the '-hidden' parameter. That way, a
> fully
> > meshed network is not formed when the two managers talk to each other.
>
> This doesn't work because then the manager nodes are then also hidden
> in the site where they are meant to be a normal member.
>

True - but can you not configure the nodes you are supposed to connect to
and just ping them? That'll setup your internal mesh. And if an mnesia
schema is shared across these nodes, they'll connect up to each other
regardless of whether the node is set to be hidden.

Chandru
Reply | Threaded
Open this post in threaded view
|

Re: Controlled interaction of two erlang distributed networks

Richard Andrews-5
In reply to this post by Kenji Rikitake
> I think making a new RPC protocol from scratch, such as:
>
> * with restricting the usage of TCP connection between two BEAMs to only
> one well-known destination port;

inet_dist_listen_min + inet_dist_listen_max does this.

> * preferably being able to forwarded through proxies (i.e. the
> addressing mechanism of BEAMs does not depend on DNS, IP addresses, or
> port numbers); and

It would be nice if nodes did not keep opening more and more TCP
connections between the same two nodes in different directions.
Distribution over SCTP might solve this issue. Does this exist?

> * running everything within a BEAM (and linked-in drivers) without
> anything like epmd

The directory (epmd) needs to be in a predictable location and it
cannot be in a node as it must monitor those. I do dislike the inet
and ssl helper processes that hang off beam nodes; although having
them separate has helped me debug some problems.

> will open a new opportunity for Erlang to become a practical system for
> monitoring/controlling distant systems over Internet.  This is a
> challenging but an interesting project.

You may be trying to re-cast erlang distribution for something it was
not intended to do.

--
  Rich

________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

Reply | Threaded
Open this post in threaded view
|

Re: Controlled interaction of two erlang distributed networks

Kenji Rikitake
Disclaimer: I am not insisting to change Erlang rpc module as of now.
The rpc module model works fine for the intended usage.  (What I'm
writing may be on a different track from the beginning of this thread.)

I want to have something additional.

In the message <[hidden email]>
dated Thu, Aug 27, 2009 at 10:21:25PM +1000,
Richard Andrews <[hidden email]> writes:
> > * with restricting the usage of TCP connection between two BEAMs to only
> > one well-known destination port;
>
> inet_dist_listen_min + inet_dist_listen_max does this.

About rpc with inet_dist, true.  This should be configured for all BEAMs
individually though.  

Question: Why not only one destination port number is sufficient?

> > * preferably being able to forwarded through proxies (i.e. the
> > addressing mechanism of BEAMs does not depend on DNS, IP addresses, or
> > port numbers); and

> It would be nice if nodes did not keep opening more and more TCP
> connections between the same two nodes in different directions.

Agreed.

Why not overlaying multiple Erlang node connections between two IP hosts
into one TCP (or SCTP if needed) connection?

> Distribution over SCTP might solve this issue. Does this exist?

I've never heard of Erlang SCTP distribution.  Surely an interesting
project to try out.

> > * running everything within a BEAM (and linked-in drivers) without
> > anything like epmd

> The directory (epmd) needs to be in a predictable location and it
> cannot be in a node as it must monitor those. I do dislike the inet
> and ssl helper processes that hang off beam nodes; although having
> them separate has helped me debug some problems.

I am not claiming to remove epmd functionality for rpc.  I'd rather want
it in an Erlang BEAM or something directly controllable from BEAM.

One long-time argument for epmd: at least the source port address with
bind() system call should be freely specified other than
INADDR_ANY. I've seen this argument repeatedly on the list and I wonder
why this has not been implemented yet.  Binding to INADDR_ANY is the
least preferable choice, especially a host has multiple addresses bound
into a network interface.

> > will open a new opportunity for Erlang to become a practical system for
> > monitoring/controlling distant systems over Internet. ?This is a
> > challenging but an interesting project.
>
> You may be trying to re-cast erlang distribution for something it was
> not intended to do.

Disclaimer repeated: I am not trying to re-cast the current rpc
module-based distribution.

I need something in Erlang to safely monitor/control/manipulate/whatever
each other between BEAMs across the Internet.

Read Dave Smith's paper as well.

>   Rich

Kenji Rikitake

________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

Reply | Threaded
Open this post in threaded view
|

Re: Controlled interaction of two erlang distributed networks

Richard Andrews-5
>> It would be nice if nodes did not keep opening more and more TCP
>> connections between the same two nodes in different directions.
>
> Agreed.
>
> Why not overlaying multiple Erlang node connections between two IP hosts
> into one TCP (or SCTP if needed) connection?

TCP is unsuitable due to head of line blocking. I expect this is the
main reason for the many connections. SCTP would remove such an issue
as a stream could be established for each purpose.

> One long-time argument for epmd: at least the source port address with
> bind() system call should be freely specified other than
> INADDR_ANY. I've seen this argument repeatedly on the list and I wonder
> why this has not been implemented yet.  Binding to INADDR_ANY is the
> least preferable choice, especially a host has multiple addresses bound
> into a network interface.

This is undesirable to me.

> I need something in Erlang to safely monitor/control/manipulate/whatever
> each other between BEAMs across the Internet.

I just use sockets and term_to_bin.

________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

Reply | Threaded
Open this post in threaded view
|

Re: Controlled interaction of two erlang distributed networks

Kenji Rikitake
In the message <[hidden email]>
dated Fri, Aug 28, 2009 at 08:20:38AM +1000,
Richard Andrews <[hidden email]> writes:
> TCP is unsuitable due to head of line blocking. I expect this is the
> main reason for the many connections. SCTP would remove such an issue
> as a stream could be established for each purpose.

Streams within a connection will be useful indeed.

> > One long-time argument for epmd: at least the source port address with
> > bind() system call should be freely specified other than
> > INADDR_ANY. I've seen this argument repeatedly on the list and I wonder
> > why this has not been implemented yet. ?Binding to INADDR_ANY is the
> > least preferable choice, especially a host has multiple addresses bound
> > into a network interface.
>
> This is undesirable to me.

Running BEAM on a multiple interface machine often needs restricting
showing epmd to one trusted network, e.g., on a dual-homed firewall, or
in a FreeBSD Jail which requires processes in Jails to bind() on the
specific addresses other than INADDR_ANY.
Packet filters should be used anyway to prevent unwanted packets to
reach epmd, but narrowing the acceptable packets to specify a
non-INADDR_ANY address solves this issue in much simpler way.  All I
need is a command option to set the bind()ing address(es).
An environment variable ERL_EPMD_PORT is recently added;
ERL_EPMD_IPV4_ADDRESS (or the IPv6 one) looks easy to be added too.

I know I can patch, but I'd rather want it as an epmd official function.

> > I need something in Erlang to safely monitor/control/manipulate/whatever
> > each other between BEAMs across the Internet.
>
> I just use sockets and term_to_bin.

Yes this is a well-known first step; and I want it to do over an
encrypted channel.  inet_ssl_dist is not bad if epmd *were* encrypted.
Unfortunately this is not the case so I need to think about something
else.  Fortunately Erlang has its own ssh module and I'm now playing
around with it.  Note: the default ssh_cli.erl has so many functions
for command-line editing implemented, and those are not necessary at all
for just passing binaries in Erlang External Term Format (at
http://erlang.org/doc/apps/erts/erl_ext_dist.html on Web).

Regards,
Kenji Rikitake



________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org