Worth minimizing the number of inter-node messages?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Worth minimizing the number of inter-node messages?

Stefan Hellkvist-2
Hi all enlightened people, 

Suppose that you, in a distributed Erlang setup, have a process A, running on one of the nodes, which, in its state, holds pids B1, B2, ..., Bn, where each process B could be running on any of the nodes in the cluster. Suppose further that you want to typically broadcast messages to all processes (B1 through Bn) - the same message to all processes (you can think of it as a "group chat" if you like where "A" is the room and B1-Bn are the users joining the room - seeing everything posted there - or you could see it as a message bus for a topic "A"). The number n here could be in the order of "2-10 thousands" and messages are typically not huge but perhaps not tiny either (< 1kB). 

In such a setup, if you want to minimize the time to distribute the message to all processes B1 to Bn (the time between when the first B receives the message to the last), would you consider it be worth to try to minimize the amount of traffic between the nodes, by, let's say, grouping all processes belonging to the same node and instead adding such groups to the process A? Then there would be one message going between the nodes and then the local groups on each node take care of sending it to all its local processes (under the assumption that local message passing has a lower cost than inter-node message passing). Is there even such library out there that does something similar? I know of pg and pg2 but the goals of these modules is not about minimizing node-to-node traffic if I understand them correctly. 

Or perhaps others are approaching such broadcasting of messages in different ways that does not rely on Erlang message passing at all - such as relying on multicast or using message buses (such as Rabbit or NATS)? Perhaps that is a better approach? I suppose "measuring" is the answer here...but perhaps someone has done similar comparisons already?

/Stefan


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Worth minimizing the number of inter-node messages?

Sergej Jurečko
You neglected to mention a critical aspect. How often are you broadcasting?

Regards,
Sergej

> On 23 Nov 2017, at 08:49, Stefan Hellkvist <[hidden email]> wrote:
>
> Hi all enlightened people,
>
> Suppose that you, in a distributed Erlang setup, have a process A, running on one of the nodes, which, in its state, holds pids B1, B2, ..., Bn, where each process B could be running on any of the nodes in the cluster. Suppose further that you want to typically broadcast messages to all processes (B1 through Bn) - the same message to all processes (you can think of it as a "group chat" if you like where "A" is the room and B1-Bn are the users joining the room - seeing everything posted there - or you could see it as a message bus for a topic "A"). The number n here could be in the order of "2-10 thousands" and messages are typically not huge but perhaps not tiny either (< 1kB).
>
> In such a setup, if you want to minimize the time to distribute the message to all processes B1 to Bn (the time between when the first B receives the message to the last), would you consider it be worth to try to minimize the amount of traffic between the nodes, by, let's say, grouping all processes belonging to the same node and instead adding such groups to the process A? Then there would be one message going between the nodes and then the local groups on each node take care of sending it to all its local processes (under the assumption that local message passing has a lower cost than inter-node message passing). Is there even such library out there that does something similar? I know of pg and pg2 but the goals of these modules is not about minimizing node-to-node traffic if I understand them correctly.
>
> Or perhaps others are approaching such broadcasting of messages in different ways that does not rely on Erlang message passing at all - such as relying on multicast or using message buses (such as Rabbit or NATS)? Perhaps that is a better approach? I suppose "measuring" is the answer here...but perhaps someone has done similar comparisons already?
>
> /Stefan
>
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Worth minimizing the number of inter-node messages?

Stefan Hellkvist-2
Each process (or "user" if you prefer the group chat analogy) could send a message, say once every second. So if there are 1000 processes in one such "room" there are 1000 messages per second. Then there could be many such "rooms" also in the system, running on different nodes, but that is in the order of 10 at most. 

On Thu, Nov 23, 2017 at 8:55 AM, Sergej Jurečko <[hidden email]> wrote:
You neglected to mention a critical aspect. How often are you broadcasting?

Regards,
Sergej

> On 23 Nov 2017, at 08:49, Stefan Hellkvist <[hidden email]> wrote:
>
> Hi all enlightened people,
>
> Suppose that you, in a distributed Erlang setup, have a process A, running on one of the nodes, which, in its state, holds pids B1, B2, ..., Bn, where each process B could be running on any of the nodes in the cluster. Suppose further that you want to typically broadcast messages to all processes (B1 through Bn) - the same message to all processes (you can think of it as a "group chat" if you like where "A" is the room and B1-Bn are the users joining the room - seeing everything posted there - or you could see it as a message bus for a topic "A"). The number n here could be in the order of "2-10 thousands" and messages are typically not huge but perhaps not tiny either (< 1kB).
>
> In such a setup, if you want to minimize the time to distribute the message to all processes B1 to Bn (the time between when the first B receives the message to the last), would you consider it be worth to try to minimize the amount of traffic between the nodes, by, let's say, grouping all processes belonging to the same node and instead adding such groups to the process A? Then there would be one message going between the nodes and then the local groups on each node take care of sending it to all its local processes (under the assumption that local message passing has a lower cost than inter-node message passing). Is there even such library out there that does something similar? I know of pg and pg2 but the goals of these modules is not about minimizing node-to-node traffic if I understand them correctly.
>
> Or perhaps others are approaching such broadcasting of messages in different ways that does not rely on Erlang message passing at all - such as relying on multicast or using message buses (such as Rabbit or NATS)? Perhaps that is a better approach? I suppose "measuring" is the answer here...but perhaps someone has done similar comparisons already?
>
> /Stefan
>
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Worth minimizing the number of inter-node messages?

Tobias Schlager-2
In reply to this post by Stefan Hellkvist-2
Hi Stefan,

your question reminds me of a Discord blog about inter-node messaging I read, some time ago. I was lucky enough to be able to find it again. I know, it's about Elixir, but the concepts apply nonetheless. Here you go [1].

Regards
Tobias



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Worth minimizing the number of inter-node messages?

Stefan Hellkvist-2
Ah! Thanks! That ”Manifold” thingy sounds interesting. I’ll have a look

Stefan

23 nov. 2017 kl. 09:13 skrev Tobias Schlager <[hidden email]>:

Hi Stefan,

your question reminds me of a Discord blog about inter-node messaging I read, some time ago. I was lucky enough to be able to find it again. I know, it's about Elixir, but the concepts apply nonetheless. Here you go [1].

Regards
Tobias



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Worth minimizing the number of inter-node messages?

Dmitry Kolesnikov-2
In reply to this post by Stefan Hellkvist-2
Hello,

The Erlang distribution is good for signal plane and not optimal for data plane…

1000 messages per second is less then 1MB for interprocess communication. You’ve also missed the number of nodes in your cluster. Therefore, it is not clear what is the bandwidth requirements per link.

The broadcast of messages would not an optimal for highly-loaded production systems but pre-mature optimisation is an evil.

* Go with broadcast as first and rapid PoC implementation. 
* Optimise it with group proxy to save bandwidth of inter-node links.
* Go with external data plane channels (e.g. TCP) or external queue systems.

- Dmitry



On 23 Nov 2017, at 10.03, Stefan Hellkvist <[hidden email]> wrote:

Each process (or "user" if you prefer the group chat analogy) could send a message, say once every second. So if there are 1000 processes in one such "room" there are 1000 messages per second. Then there could be many such "rooms" also in the system, running on different nodes, but that is in the order of 10 at most. 

On Thu, Nov 23, 2017 at 8:55 AM, Sergej Jurečko <[hidden email]> wrote:
You neglected to mention a critical aspect. How often are you broadcasting?

Regards,
Sergej

> On 23 Nov 2017, at 08:49, Stefan Hellkvist <[hidden email]> wrote:
>
> Hi all enlightened people,
>
> Suppose that you, in a distributed Erlang setup, have a process A, running on one of the nodes, which, in its state, holds pids B1, B2, ..., Bn, where each process B could be running on any of the nodes in the cluster. Suppose further that you want to typically broadcast messages to all processes (B1 through Bn) - the same message to all processes (you can think of it as a "group chat" if you like where "A" is the room and B1-Bn are the users joining the room - seeing everything posted there - or you could see it as a message bus for a topic "A"). The number n here could be in the order of "2-10 thousands" and messages are typically not huge but perhaps not tiny either (< 1kB).
>
> In such a setup, if you want to minimize the time to distribute the message to all processes B1 to Bn (the time between when the first B receives the message to the last), would you consider it be worth to try to minimize the amount of traffic between the nodes, by, let's say, grouping all processes belonging to the same node and instead adding such groups to the process A? Then there would be one message going between the nodes and then the local groups on each node take care of sending it to all its local processes (under the assumption that local message passing has a lower cost than inter-node message passing). Is there even such library out there that does something similar? I know of pg and pg2 but the goals of these modules is not about minimizing node-to-node traffic if I understand them correctly.
>
> Or perhaps others are approaching such broadcasting of messages in different ways that does not rely on Erlang message passing at all - such as relying on multicast or using message buses (such as Rabbit or NATS)? Perhaps that is a better approach? I suppose "measuring" is the answer here...but perhaps someone has done similar comparisons already?
>
> /Stefan
>
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Worth minimizing the number of inter-node messages?

Stefan Hellkvist-2
Thank you for your valuable input. Yes, I think that is probably the way - to take it step wise in that order that you mention and avoid optimizing before it is really needed. The number of nodes needed, as you mention, is not known at the moment. I'm currently using ONE node. This work is just to try and understand how one would scale things out if there is a need to support more traffic. 

/Stefan



On Thu, Nov 23, 2017 at 9:21 AM, Dmitry Kolesnikov <[hidden email]> wrote:
Hello,

The Erlang distribution is good for signal plane and not optimal for data plane…

1000 messages per second is less then 1MB for interprocess communication. You’ve also missed the number of nodes in your cluster. Therefore, it is not clear what is the bandwidth requirements per link.

The broadcast of messages would not an optimal for highly-loaded production systems but pre-mature optimisation is an evil.

* Go with broadcast as first and rapid PoC implementation. 
* Optimise it with group proxy to save bandwidth of inter-node links.
* Go with external data plane channels (e.g. TCP) or external queue systems.

- Dmitry



On 23 Nov 2017, at 10.03, Stefan Hellkvist <[hidden email]> wrote:

Each process (or "user" if you prefer the group chat analogy) could send a message, say once every second. So if there are 1000 processes in one such "room" there are 1000 messages per second. Then there could be many such "rooms" also in the system, running on different nodes, but that is in the order of 10 at most. 

On Thu, Nov 23, 2017 at 8:55 AM, Sergej Jurečko <[hidden email]> wrote:
You neglected to mention a critical aspect. How often are you broadcasting?

Regards,
Sergej

> On 23 Nov 2017, at 08:49, Stefan Hellkvist <[hidden email]> wrote:
>
> Hi all enlightened people,
>
> Suppose that you, in a distributed Erlang setup, have a process A, running on one of the nodes, which, in its state, holds pids B1, B2, ..., Bn, where each process B could be running on any of the nodes in the cluster. Suppose further that you want to typically broadcast messages to all processes (B1 through Bn) - the same message to all processes (you can think of it as a "group chat" if you like where "A" is the room and B1-Bn are the users joining the room - seeing everything posted there - or you could see it as a message bus for a topic "A"). The number n here could be in the order of "2-10 thousands" and messages are typically not huge but perhaps not tiny either (< 1kB).
>
> In such a setup, if you want to minimize the time to distribute the message to all processes B1 to Bn (the time between when the first B receives the message to the last), would you consider it be worth to try to minimize the amount of traffic between the nodes, by, let's say, grouping all processes belonging to the same node and instead adding such groups to the process A? Then there would be one message going between the nodes and then the local groups on each node take care of sending it to all its local processes (under the assumption that local message passing has a lower cost than inter-node message passing). Is there even such library out there that does something similar? I know of pg and pg2 but the goals of these modules is not about minimizing node-to-node traffic if I understand them correctly.
>
> Or perhaps others are approaching such broadcasting of messages in different ways that does not rely on Erlang message passing at all - such as relying on multicast or using message buses (such as Rabbit or NATS)? Perhaps that is a better approach? I suppose "measuring" is the answer here...but perhaps someone has done similar comparisons already?
>
> /Stefan
>
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions